2024-04-06T02:55:58.852 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-06T02:55:58.852 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906 --description orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --owner scheduled_teuthology@teuthology --name teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/orig.config.yaml 2024-04-06T02:55:58.883 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T02:55:58.968 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906 branch: squid description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7641906' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 workunit: branch: squid sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 5609 sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 sleep_before_teardown: 0 subset: 32/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 584a67ccb9eb224c5312dccd96a82e67b0645518 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHS3NGNDwyc+9N/zANprQcxqgUJmXFIpW4mIZqLY+jc+WOAMTCLVpRk4pKDfPLyZwX4lfia2o4km4YptNsAHsm0= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmbjn0la71LFvZQWmzMqaxBgPwdL/4dFlwQaaVNmnqDOz98eHa9HNzEVBS9MV5Hl9pqHF17g7WyUfrDOdPXADI= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-06T02:55:58.968 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa; will attempt to use it 2024-04-06T02:55:58.969 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_584a67ccb9eb224c5312dccd96a82e67b0645518/qa/tasks 2024-04-06T02:55:58.969 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-06T02:55:58.970 INFO:teuthology.task.internal:Checking packages... 2024-04-06T02:55:58.992 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6' 2024-04-06T02:55:58.992 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-06T02:55:58.992 INFO:teuthology.packaging:ref: None 2024-04-06T02:55:58.993 INFO:teuthology.packaging:tag: None 2024-04-06T02:55:58.993 INFO:teuthology.packaging:branch: squid 2024-04-06T02:55:58.993 INFO:teuthology.packaging:sha1: 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T02:55:58.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-06T02:55:59.290 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2041-g584a67cc-1jammy 2024-04-06T02:55:59.291 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-06T02:55:59.298 INFO:teuthology.task.internal:no buildpackages task found 2024-04-06T02:55:59.298 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-06T02:55:59.306 INFO:teuthology.task.internal:Saving configuration 2024-04-06T02:55:59.318 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-06T02:55:59.324 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T02:55:59.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:50:04.853886', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMV4etqe7OfD9zTyRb5THLNfBopUWmzotGM0GTyOwIJ4'} 2024-04-06T02:55:59.367 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:50:04.855288', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHKkyW1k6IP91FleVWivc00wWZf45pt/OivXtEGqzsCk'} 2024-04-06T02:55:59.367 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-06T02:55:59.374 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-06T02:55:59.374 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-06T02:55:59.374 INFO:teuthology.run_tasks:Running task console_log... 2024-04-06T02:55:59.436 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7face0c763a0>, signals=[15]) 2024-04-06T02:55:59.437 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-06T02:55:59.443 INFO:teuthology.task.internal:Opening connections... 2024-04-06T02:55:59.443 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-04-06T02:55:59.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:55:59.525 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-06T02:55:59.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:55:59.604 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-06T02:55:59.611 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-04-06T02:55:59.616 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-04-06T02:55:59.617 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Ubuntu" 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="22.04" 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_CODENAME=jammy 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:ID=ubuntu 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE=debian 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-06T02:55:59.667 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-06T02:55:59.668 INFO:teuthology.orchestra.run.smithi050.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-06T02:55:59.668 INFO:teuthology.orchestra.run.smithi050.stdout:UBUNTU_CODENAME=jammy 2024-04-06T02:55:59.668 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-04-06T02:55:59.687 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-06T02:55:59.692 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-06T02:55:59.692 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="22.04" 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=jammy 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-06T02:55:59.739 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-06T02:55:59.740 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-06T02:55:59.740 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-06T02:55:59.740 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=jammy 2024-04-06T02:55:59.740 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-06T02:55:59.761 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-06T02:55:59.770 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-06T02:55:59.784 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-06T02:55:59.784 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-04-06T02:55:59.786 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-06T02:55:59.792 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-06T02:55:59.798 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-06T02:55:59.798 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-04-06T02:55:59.832 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-06T02:55:59.853 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-06T02:55:59.910 INFO:teuthology.run_tasks:Running task kernel... 2024-04-06T02:55:59.924 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-06T02:55:59.924 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-06T02:55:59.924 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T02:55:59.925 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-06T02:55:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:55:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T02:55:59.930 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-04-06T02:55:59.931 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-06T02:55:59.979 INFO:teuthology.orchestra.run.smithi050.stdout:5.15.0-56-generic 2024-04-06T02:55:59.979 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.15.0-56-generic 2024-04-06T02:55:59.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get clean 2024-04-06T02:55:59.980 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-56-generic 2024-04-06T02:55:59.980 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.15.0-56-generic 2024-04-06T02:55:59.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2024-04-06T02:56:00.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2024-04-06T02:56:00.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-04-06T02:56:00.262 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T02:56:00.264 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-06T02:56:00.274 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T02:56:00.274 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-06T02:56:00.274 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-06T02:56:00.279 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-06T02:56:00.414 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-06T02:56:00.417 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-06T02:56:00.486 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-06T02:56:00.506 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-04-06T02:56:00.560 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-06T02:56:00.583 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-06T02:56:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-06T02:56:00.643 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-04-06T02:56:00.645 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-04-06T02:56:00.650 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-06T02:56:00.653 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-04-06T02:56:00.656 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-06T02:56:00.656 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-06T02:56:00.657 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-06T02:56:00.660 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-06T02:56:00.660 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-04-06T02:56:00.661 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-04-06T02:56:00.699 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-06T02:56:00.705 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-04-06T02:56:00.705 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T02:56:00.705 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-06T02:56:00.711 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T02:56:00.712 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-06T02:56:00.717 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-06T02:56:00.729 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-04-06T02:56:00.730 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-06T02:56:00.730 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-04-06T02:56:00.742 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-06T02:56:00.745 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-06T02:56:00.746 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.2 kB] 2024-04-06T02:56:00.746 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-06T02:56:00.753 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-04-06T02:56:00.757 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-06T02:56:00.767 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,648 kB] 2024-04-06T02:56:00.768 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-06T02:56:00.770 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-06T02:56:00.771 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-06T02:56:00.771 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-04-06T02:56:00.772 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-06T02:56:00.786 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-06T02:56:00.787 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-06T02:56:00.787 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-06T02:56:00.787 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-04-06T02:56:00.788 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-06T02:56:00.788 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,648 kB] 2024-04-06T02:56:00.795 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [275 kB] 2024-04-06T02:56:00.800 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T02:56:00.801 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-06T02:56:00.816 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-06T02:56:00.821 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.2 kB] 2024-04-06T02:56:00.822 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [275 kB] 2024-04-06T02:56:00.824 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-06T02:56:00.825 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-06T02:56:00.825 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-04-06T02:56:00.826 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-06T02:56:00.827 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-06T02:56:00.828 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-06T02:56:00.828 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-06T02:56:00.836 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-04-06T02:56:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-06T02:56:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-06T02:56:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-06T02:56:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-06T02:56:00.852 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-04-06T02:56:00.856 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-06T02:56:00.857 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-04-06T02:56:00.858 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-04-06T02:56:00.858 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-04-06T02:56:00.859 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-06T02:56:00.859 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-06T02:56:00.860 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-06T02:56:00.865 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-06T02:56:00.865 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-06T02:56:00.865 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-06T02:56:00.866 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-06T02:56:00.866 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-06T02:56:00.878 INFO:teuthology.orchestra.run.smithi050.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-06T02:56:00.879 INFO:teuthology.orchestra.run.smithi050.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-06T02:56:00.899 INFO:teuthology.orchestra.run.smithi050.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-06T02:56:00.899 INFO:teuthology.orchestra.run.smithi050.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-06T02:56:00.900 INFO:teuthology.orchestra.run.smithi050.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-06T02:56:00.900 INFO:teuthology.orchestra.run.smithi050.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-06T02:56:06.261 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 12.6 MB in 2s (6,980 kB/s) 2024-04-06T02:56:06.725 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 12.6 MB in 2s (6,601 kB/s) 2024-04-06T02:56:07.603 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-04-06T02:56:07.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T02:56:07.681 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-04-06T02:56:07.878 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-04-06T02:56:07.879 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-04-06T02:56:08.050 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T02:56:08.051 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T02:56:08.051 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T02:56:08.051 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T02:56:08.052 INFO:teuthology.orchestra.run.smithi153.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-06T02:56:08.052 INFO:teuthology.orchestra.run.smithi153.stdout: python2.7-minimal 2024-04-06T02:56:08.052 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T02:56:08.054 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2024-04-06T02:56:08.055 INFO:teuthology.orchestra.run.smithi153.stdout: thermald 2024-04-06T02:56:08.103 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-06T02:56:08.105 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2024-04-06T02:56:08.105 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T02:56:08.105 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-06T02:56:08.105 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-06T02:56:08.107 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages will be upgraded: 2024-04-06T02:56:08.107 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-06T02:56:08.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T02:56:08.175 INFO:teuthology.orchestra.run.smithi153.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T02:56:08.175 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 113 MB of archives. 2024-04-06T02:56:08.175 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-06T02:56:08.175 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-06T02:56:08.188 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-06T02:56:08.392 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2024-04-06T02:56:08.393 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2024-04-06T02:56:08.573 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T02:56:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T02:56:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T02:56:08.573 INFO:teuthology.orchestra.run.smithi050.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T02:56:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-06T02:56:08.574 INFO:teuthology.orchestra.run.smithi050.stdout: python2.7-minimal 2024-04-06T02:56:08.574 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T02:56:08.575 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2024-04-06T02:56:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T02:56:08.575 INFO:teuthology.orchestra.run.smithi050.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-04-06T02:56:08.576 INFO:teuthology.orchestra.run.smithi050.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-04-06T02:56:08.577 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2024-04-06T02:56:08.577 INFO:teuthology.orchestra.run.smithi050.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-06T02:56:08.577 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2024-04-06T02:56:08.577 INFO:teuthology.orchestra.run.smithi050.stdout: thermald 2024-04-06T02:56:08.631 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2024-04-06T02:56:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-04-06T02:56:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-04-06T02:56:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: linux-modules-extra-5.15.0-101-generic 2024-04-06T02:56:08.633 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages will be upgraded: 2024-04-06T02:56:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-06T02:56:08.812 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-06T02:56:08.813 INFO:teuthology.orchestra.run.smithi050.stdout:3 upgraded, 5 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T02:56:08.813 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 113 MB of archives. 2024-04-06T02:56:08.813 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 582 MB of additional disk space will be used. 2024-04-06T02:56:08.813 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-04-06T02:56:09.052 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-06T02:56:10.241 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-04-06T02:56:10.473 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-06T02:56:10.473 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-06T02:56:10.474 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-06T02:56:10.579 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-04-06T02:56:10.664 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-06T02:56:10.712 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-06T02:56:11.097 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 113 MB in 3s (43.5 MB/s) 2024-04-06T02:56:11.291 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-06T02:56:12.755 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-04-06T02:56:12.756 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-04-06T02:56:12.756 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-04-06T02:56:13.264 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-06T02:56:13.268 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:13.334 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-04-06T02:56:13.421 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:13.462 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-04-06T02:56:13.894 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 113 MB in 5s (23.5 MB/s) 2024-04-06T02:56:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-04-06T02:56:15.656 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-06T02:56:15.678 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:15.772 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:16.127 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-06T02:56:16.131 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:16.201 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:16.326 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-06T02:56:16.346 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:16.569 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:18.402 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-04-06T02:56:18.423 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:18.520 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:18.988 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-04-06T02:56:19.009 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:19.046 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:23.467 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T02:56:23.611 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T02:56:23.972 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T02:56:24.100 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T02:56:24.436 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-06T02:56:24.459 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-06T02:56:24.528 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T02:56:25.855 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T02:56:25.981 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T02:56:26.298 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T02:56:26.418 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T02:56:26.729 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-04-06T02:56:26.751 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-04-06T02:56:26.795 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T02:56:30.445 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-06T02:56:30.482 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:30.553 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:32.845 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-04-06T02:56:32.883 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-04-06T02:56:32.937 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:33.862 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T02:56:33.963 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T02:56:34.320 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T02:56:34.476 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:34.631 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-06T02:56:34.772 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:35.614 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-04-06T02:56:35.914 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-04-06T02:56:36.199 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-06T02:56:36.199 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-06T02:56:36.199 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-06T02:56:36.199 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:36.560 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:36.807 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-04-06T02:56:36.928 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:37.054 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-04-06T02:56:37.183 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:38.002 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:38.541 INFO:teuthology.orchestra.run.smithi050.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-06T02:56:38.541 INFO:teuthology.orchestra.run.smithi050.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-06T02:56:38.541 INFO:teuthology.orchestra.run.smithi050.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-04-06T02:56:38.541 INFO:teuthology.orchestra.run.smithi050.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:38.685 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:39.378 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-06T02:56:39.537 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-06T02:56:39.663 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:39.730 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-06T02:56:39.730 INFO:teuthology.orchestra.run.smithi153.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:40.119 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:41.529 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-04-06T02:56:42.213 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-04-06T02:56:42.523 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-04-06T02:56:42.590 INFO:teuthology.orchestra.run.smithi050.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-06T02:56:42.590 INFO:teuthology.orchestra.run.smithi050.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:53.946 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-06T02:56:53.946 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub' 2024-04-06T02:56:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T02:56:53.999 INFO:teuthology.orchestra.run.smithi153.stdout:Generating grub configuration file ... 2024-04-06T02:56:54.308 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T02:56:54.336 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:54.662 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T02:56:54.664 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T02:56:54.735 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T02:56:54.740 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T02:56:54.933 INFO:teuthology.orchestra.run.smithi153.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T02:56:54.933 INFO:teuthology.orchestra.run.smithi153.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T02:56:54.933 INFO:teuthology.orchestra.run.smithi153.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T02:56:54.976 INFO:teuthology.orchestra.run.smithi153.stdout:done 2024-04-06T02:56:55.470 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 21 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2024-04-06T02:56:55.490 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.101.98 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2024-04-06T02:56:55.491 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2024-04-06T02:56:55.491 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-06T02:56:55.491 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T02:56:55.491 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T02:56:55.492 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-06T02:56:55.492 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T02:56:55.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2024-04-06T02:56:55.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-04-06T02:56:55.854 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T02:56:55.939 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T02:56:56.020 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T02:56:56.102 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T02:56:57.097 INFO:teuthology.orchestra.run.smithi050.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-06T02:56:57.098 INFO:teuthology.orchestra.run.smithi050.stdout:Sourcing file `/etc/default/grub' 2024-04-06T02:56:57.110 INFO:teuthology.orchestra.run.smithi050.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T02:56:57.126 INFO:teuthology.orchestra.run.smithi050.stdout:Generating grub configuration file ... 2024-04-06T02:56:57.239 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-04-06T02:56:57.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T02:56:57.318 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-04-06T02:56:57.434 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T02:56:57.448 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:57.530 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-04-06T02:56:57.531 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-04-06T02:56:57.702 INFO:teuthology.orchestra.run.smithi153.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-06T02:56:57.703 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T02:56:57.703 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T02:56:57.703 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T02:56:57.703 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T02:56:57.703 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T02:56:57.704 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T02:56:57.704 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T02:56:57.743 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T02:56:57.745 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2024-04-06T02:56:57.757 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2024-04-06T02:56:57.757 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2024-04-06T02:56:57.757 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2024-04-06T02:56:57.757 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2024-04-06T02:56:57.757 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 21 2024-04-06T02:56:57.757 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T02:56:57.758 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2024-04-06T02:56:57.758 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2024-04-06T02:56:57.758 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.101.98 2024-04-06T02:56:57.758 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T02:56:57.758 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T02:56:57.758 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2024-04-06T02:56:57.759 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2024-04-06T02:56:57.759 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2024-04-06T02:56:57.759 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2024-04-06T02:56:57.759 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-04-06T02:56:57.807 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.OBkZPAUG77 2024-04-06T02:56:57.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub/grub.cfg /tmp/tmp.OBkZPAUG77 2024-04-06T02:56:57.814 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T02:56:57.820 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T02:56:57.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.OBkZPAUG77 2024-04-06T02:56:57.893 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T02:56:57.898 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T02:56:57.989 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.OBkZPAUG77 is 10KB 2024-04-06T02:56:58.002 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.OBkZPAUG77 2024-04-06T02:56:58.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-06T02:56:58.061 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T02:56:58.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-06T02:56:58.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-06T02:56:58.070 INFO:teuthology.orchestra.run.smithi050.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T02:56:58.070 INFO:teuthology.orchestra.run.smithi050.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T02:56:58.070 INFO:teuthology.orchestra.run.smithi050.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T02:56:58.110 INFO:teuthology.orchestra.run.smithi050.stdout:done 2024-04-06T02:56:58.135 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-06T02:56:58.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo update-grub 2024-04-06T02:56:58.541 DEBUG:teuthology.orchestra.run.smithi050:> dpkg -s linux-image-generic 2024-04-06T02:56:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:Package: linux-image-generic 2024-04-06T02:56:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:Status: install ok installed 2024-04-06T02:56:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:Priority: optional 2024-04-06T02:56:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:Section: kernel 2024-04-06T02:56:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:Installed-Size: 21 2024-04-06T02:56:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: amd64 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Source: linux-meta 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Version: 5.15.0.101.98 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Recommends: thermald 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout:Description: Generic Linux kernel image 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: This package will always depend on the latest generic kernel image 2024-04-06T02:56:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: available. 2024-04-06T02:56:58.562 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-04-06T02:56:58.562 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-06T02:56:58.563 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-06T02:56:58.563 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-06T02:56:58.563 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-04-06T02:56:58.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get clean 2024-04-06T02:56:58.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2024-04-06T02:56:58.989 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T02:56:59.003 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T02:56:59.085 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T02:56:59.167 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T02:56:59.463 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub' 2024-04-06T02:56:59.464 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T02:56:59.468 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-04-06T02:56:59.666 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T02:56:59.671 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T02:56:59.917 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T02:56:59.922 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T02:56:59.964 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T02:56:59.968 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T02:57:00.088 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T02:57:00.088 INFO:teuthology.orchestra.run.smithi153.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T02:57:00.088 INFO:teuthology.orchestra.run.smithi153.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T02:57:00.111 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-04-06T02:57:00.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-04-06T02:57:00.356 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-06T02:57:00.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-06T02:57:00.438 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-06T02:57:00.644 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2024-04-06T02:57:00.644 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2024-04-06T02:57:00.824 INFO:teuthology.orchestra.run.smithi050.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-04-06T02:57:00.824 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T02:57:00.824 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T02:57:00.824 INFO:teuthology.orchestra.run.smithi050.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T02:57:00.824 INFO:teuthology.orchestra.run.smithi050.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T02:57:00.825 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T02:57:00.825 INFO:teuthology.orchestra.run.smithi050.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T02:57:00.825 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T02:57:00.869 INFO:teuthology.orchestra.run.smithi050.stdout:0 upgraded, 0 newly installed, 0 to remove and 312 not upgraded. 2024-04-06T02:57:00.871 DEBUG:teuthology.orchestra.run.smithi050:> dpkg -s linux-image-generic 2024-04-06T02:57:00.887 INFO:teuthology.orchestra.run.smithi050.stdout:Package: linux-image-generic 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Status: install ok installed 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Priority: optional 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Section: kernel 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Installed-Size: 21 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Maintainer: Ubuntu Kernel Team 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: amd64 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Source: linux-meta 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Version: 5.15.0.101.98 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Recommends: thermald 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:Description: Generic Linux kernel image 2024-04-06T02:57:00.888 INFO:teuthology.orchestra.run.smithi050.stdout: This package will always depend on the latest generic kernel image 2024-04-06T02:57:00.889 INFO:teuthology.orchestra.run.smithi050.stdout: available. 2024-04-06T02:57:00.889 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-04-06T02:57:00.937 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.pdzQN4dF2X 2024-04-06T02:57:00.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub/grub.cfg /tmp/tmp.pdzQN4dF2X 2024-04-06T02:57:00.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.pdzQN4dF2X 2024-04-06T02:57:01.110 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.pdzQN4dF2X is 10KB 2024-04-06T02:57:01.124 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.pdzQN4dF2X 2024-04-06T02:57:01.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-06T02:57:01.179 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T02:57:01.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-06T02:57:01.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-06T02:57:01.243 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-04-06T02:57:01.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo update-grub 2024-04-06T02:57:02.428 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub' 2024-04-06T02:57:02.429 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-06T02:57:02.432 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-04-06T02:57:02.661 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-04-06T02:57:02.667 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-04-06T02:57:02.954 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-06T02:57:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-06T02:57:03.002 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-06T02:57:03.007 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-06T02:57:03.140 INFO:teuthology.orchestra.run.smithi050.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-06T02:57:03.140 INFO:teuthology.orchestra.run.smithi050.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-06T02:57:03.140 INFO:teuthology.orchestra.run.smithi050.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-06T02:57:03.161 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-04-06T02:57:03.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-04-06T02:57:30.149 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-06T02:57:30.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T02:57:30.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:57:33.186 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-06T02:57:33.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-06T02:57:33.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:57:48.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-04-06T02:57:51.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-04-06T02:57:57.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T02:57:57.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:58:00.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-04-06T02:58:00.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-06T02:58:00.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:58:12.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T02:58:12.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:59:00.697 DEBUG:teuthology.orchestra.remote:timed out 2024-04-06T02:59:12.654 DEBUG:teuthology.orchestra.remote:timed out 2024-04-06T02:59:12.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-06T02:59:12.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:59:12.901 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-06T02:59:13.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-06T02:59:13.542 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-04-06T02:59:13.542 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-06T02:59:13.547 INFO:teuthology.orchestra.run.smithi050.stdout:5.15.0-101-generic 2024-04-06T02:59:13.548 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-06T02:59:13.548 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T02:59:13.548 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-06T02:59:14.548 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-06T02:59:14.549 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T02:59:14.618 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-04-06T02:59:14.665 DEBUG:teuthology.parallel:result is None 2024-04-06T02:59:27.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T02:59:27.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T02:59:27.975 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-06T02:59:28.568 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T02:59:28.568 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-04-06T02:59:28.569 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-04-06T02:59:28.618 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-101-generic 2024-04-06T02:59:28.618 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-04-06T02:59:28.618 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-06T02:59:28.618 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-06T02:59:29.619 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-06T02:59:29.620 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-06T02:59:29.950 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-04-06T02:59:29.983 DEBUG:teuthology.parallel:result is None 2024-04-06T02:59:29.984 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-06T02:59:29.991 INFO:teuthology.task.internal:Creating test directory... 2024-04-06T02:59:29.991 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T02:59:29.994 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-06T02:59:30.000 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-06T02:59:30.006 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-06T02:59:30.012 INFO:teuthology.task.internal:Creating archive directory... 2024-04-06T02:59:30.013 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T02:59:30.040 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-06T02:59:30.069 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-06T02:59:30.076 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-06T02:59:30.076 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T02:59:30.092 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-06T02:59:30.106 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:59:30.116 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:59:30.123 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:59:30.131 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-06T02:59:30.133 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-06T02:59:30.170 INFO:teuthology.task.internal:Configuring sudo... 2024-04-06T02:59:30.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T02:59:30.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-06T02:59:30.212 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-06T02:59:30.221 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-06T02:59:30.221 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T02:59:30.228 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-06T02:59:30.258 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:59:30.279 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:59:30.327 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T02:59:30.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T02:59:30.393 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-06T02:59:30.399 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-06T02:59:30.450 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T02:59:30.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-06T02:59:30.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-04-06T02:59:30.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-06T02:59:30.594 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-06T02:59:30.602 INFO:teuthology.task.internal:Starting timer... 2024-04-06T02:59:30.602 INFO:teuthology.run_tasks:Running task pcp... 2024-04-06T02:59:30.667 INFO:teuthology.run_tasks:Running task selinux... 2024-04-06T02:59:30.677 DEBUG:teuthology.task.selinux:Excluding smithi050: OS 'ubuntu' does not support SELinux 2024-04-06T02:59:30.677 DEBUG:teuthology.task.selinux:Excluding smithi153: OS 'ubuntu' does not support SELinux 2024-04-06T02:59:30.677 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-06T02:59:30.677 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-06T02:59:30.678 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-06T02:59:30.678 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-06T02:59:30.688 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-06T02:59:30.704 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-06T02:59:30.706 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-06T03:06:37.043 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-04-06T03:06:37.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-06T03:06:37.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:06:37.131 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-06T03:06:37.215 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-06T03:06:37.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T03:06:37.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-06T03:06:37.294 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-06T03:06:37.370 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-06T03:06:37.370 INFO:teuthology.run_tasks:Running task clock... 2024-04-06T03:06:37.381 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-06T03:06:37.382 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T03:06:37.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:06:37.385 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-06T03:06:37.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T03:06:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-06T03:06:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Command line: ntpd -gq 2024-04-06T03:06:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: ---------------------------------------------------- 2024-04-06T03:06:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: ntp-4 is maintained by Network Time Foundation, 2024-04-06T03:06:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-06T03:06:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: corporation. Support and training for ntp-4 are 2024-04-06T03:06:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: available at https://www.nwtime.org/support 2024-04-06T03:06:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: ---------------------------------------------------- 2024-04-06T03:06:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: proto: precision = 0.046 usec (-24) 2024-04-06T03:06:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: basedate set to 2022-02-04 2024-04-06T03:06:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: gps base set to 2022-02-06 (week 2196) 2024-04-06T03:06:37.414 INFO:teuthology.orchestra.run.smithi050.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T03:06:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T03:06:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: restrict ::: KOD does nothing without LIMITED. 2024-04-06T03:06:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-06T03:06:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listen and drop on 0 v6wildcard [::]:123 2024-04-06T03:06:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-06T03:06:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listen normally on 2 lo 127.0.0.1:123 2024-04-06T03:06:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listen normally on 3 ens1f0 172.21.15.50:123 2024-04-06T03:06:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listen normally on 4 lo [::1]:123 2024-04-06T03:06:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1584%4]:123 2024-04-06T03:06:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:37 ntpd[17093]: Listening on routing socket on fd #22 for interface updates 2024-04-06T03:06:37.442 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-06T03:06:37.442 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Command line: ntpd -gq 2024-04-06T03:06:37.442 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: ---------------------------------------------------- 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: ntp-4 is maintained by Network Time Foundation, 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: corporation. Support and training for ntp-4 are 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: available at https://www.nwtime.org/support 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: ---------------------------------------------------- 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: proto: precision = 0.043 usec (-24) 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: basedate set to 2022-02-04 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: gps base set to 2022-02-06 (week 2196) 2024-04-06T03:06:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T03:06:37.444 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-06T03:06:37.444 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: restrict ::: KOD does nothing without LIMITED. 2024-04-06T03:06:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listen and drop on 0 v6wildcard [::]:123 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listen normally on 2 lo 127.0.0.1:123 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listen normally on 3 enp3s0f1 172.21.15.153:123 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listen normally on 4 lo [::1]:123 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6deb%5]:123 2024-04-06T03:06:37.445 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:37 ntpd[17173]: Listening on routing socket on fd #22 for interface updates 2024-04-06T03:06:38.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:38 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:38.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:38 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:38.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:38 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:38.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:38 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:38.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:38 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:38.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:38 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:38.442 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:38.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:38 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:39.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:39 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:39.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:39 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:39.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:39 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:39.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:39 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:39.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:39 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:39.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:39 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:40.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:40.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:40.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:40.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:40.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:40.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:40 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:40 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:40 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:40 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:40 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:41.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:41 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:41.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:41 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:41.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:41 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:41.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:41 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:41.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:41 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:41.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:41 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.416 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:42.416 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:42 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:42.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:42.447 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:42 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:43.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:43 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:43.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:43 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:43.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:43 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:43.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:43 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:43.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:43 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:43.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:43 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.413 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:44.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: 6 Apr 03:06:44 ntpd[17093]: ntpd: time slew -0.001528 s 2024-04-06T03:06:44.414 INFO:teuthology.orchestra.run.smithi050.stdout:ntpd: time slew -0.001528s 2024-04-06T03:06:44.414 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-06T03:06:44.415 INFO:teuthology.orchestra.run.smithi050.stderr: 6 Apr 03:06:44 ntpd[17093]: can't open /var/log/ntpstats/loopstats.20240406: Permission denied 2024-04-06T03:06:44.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.443 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-06T03:06:44.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: can't open /var/log/ntpstats/rawstats.20240406: Permission denied 2024-04-06T03:06:44.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-06T03:06:44.444 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: can't open /var/log/ntpstats/peerstats.20240406: Permission denied 2024-04-06T03:06:44.444 INFO:teuthology.orchestra.run.smithi153.stdout: 6 Apr 03:06:44 ntpd[17173]: ntpd: time slew -0.001725 s 2024-04-06T03:06:44.444 INFO:teuthology.orchestra.run.smithi153.stdout:ntpd: time slew -0.001725s 2024-04-06T03:06:44.445 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-06T03:06:44.445 INFO:teuthology.orchestra.run.smithi153.stderr: 6 Apr 03:06:44 ntpd[17173]: can't open /var/log/ntpstats/loopstats.20240406: Permission denied 2024-04-06T03:06:44.465 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T03:06:44.465 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2024-04-06T03:06:44.465 INFO:teuthology.orchestra.run.smithi050.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.465 INFO:teuthology.orchestra.run.smithi050.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.465 INFO:teuthology.orchestra.run.smithi050.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.465 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.572 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T03:06:44.572 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2024-04-06T03:06:44.572 INFO:teuthology.orchestra.run.smithi153.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.572 INFO:teuthology.orchestra.run.smithi153.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.572 INFO:teuthology.orchestra.run.smithi153.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.572 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-06T03:06:44.573 INFO:teuthology.run_tasks:Running task install... 2024-04-06T03:06:44.582 DEBUG:teuthology.task.install:project ceph 2024-04-06T03:06:44.582 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'}} 2024-04-06T03:06:44.582 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'} 2024-04-06T03:06:44.582 INFO:teuthology.task.install:Using flavor: default 2024-04-06T03:06:44.588 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-06T03:06:44.588 INFO:teuthology.task.install:extra packages: [] 2024-04-06T03:06:44.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-key list | grep Ceph 2024-04-06T03:06:44.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-key list | grep Ceph 2024-04-06T03:06:44.650 INFO:teuthology.orchestra.run.smithi050.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-06T03:06:44.680 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-06T03:06:44.680 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph.com (release key) 2024-04-06T03:06:44.681 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-06T03:06:44.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:06:44.684 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-06T03:06:44.717 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-06T03:06:44.717 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph.com (release key) 2024-04-06T03:06:44.718 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-06T03:06:44.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:06:44.828 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default/ 2024-04-06T03:06:44.828 INFO:teuthology.task.install.deb:Package version is 19.0.0-2027-g9f192350-1jammy 2024-04-06T03:06:45.007 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default/ 2024-04-06T03:06:45.007 INFO:teuthology.task.install.deb:Package version is 19.0.0-2027-g9f192350-1jammy 2024-04-06T03:06:45.061 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:06:45.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-06T03:06:45.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2024-04-06T03:06:45.229 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:06:45.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-06T03:06:45.244 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-04-06T03:06:45.348 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy InRelease 2024-04-06T03:06:45.370 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-06T03:06:45.393 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-06T03:06:45.414 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T03:06:45.415 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,402 B] 2024-04-06T03:06:45.438 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T03:06:45.447 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-06T03:06:45.486 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T03:06:45.542 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-06T03:06:45.542 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-06T03:06:45.552 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy InRelease 2024-04-06T03:06:45.568 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T03:06:45.600 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-06T03:06:45.614 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-06T03:06:45.647 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-06T03:06:45.696 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-06T03:06:45.697 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,402 B] 2024-04-06T03:06:45.758 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-06T03:06:45.794 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 28.9 kB in 0s (58.4 kB/s) 2024-04-06T03:06:45.918 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 28.9 kB in 0s (59.0 kB/s) 2024-04-06T03:06:46.770 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-06T03:06:46.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2027-g9f192350-1jammy cephadm=19.0.0-2027-g9f192350-1jammy ceph-mds=19.0.0-2027-g9f192350-1jammy ceph-mgr=19.0.0-2027-g9f192350-1jammy ceph-common=19.0.0-2027-g9f192350-1jammy ceph-fuse=19.0.0-2027-g9f192350-1jammy ceph-test=19.0.0-2027-g9f192350-1jammy ceph-volume=19.0.0-2027-g9f192350-1jammy radosgw=19.0.0-2027-g9f192350-1jammy python3-rados=19.0.0-2027-g9f192350-1jammy python3-rgw=19.0.0-2027-g9f192350-1jammy python3-cephfs=19.0.0-2027-g9f192350-1jammy python3-rbd=19.0.0-2027-g9f192350-1jammy libcephfs2=19.0.0-2027-g9f192350-1jammy libcephfs-dev=19.0.0-2027-g9f192350-1jammy librados2=19.0.0-2027-g9f192350-1jammy librbd1=19.0.0-2027-g9f192350-1jammy rbd-fuse=19.0.0-2027-g9f192350-1jammy 2024-04-06T03:06:46.838 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-06T03:06:46.905 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-04-06T03:06:46.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2027-g9f192350-1jammy cephadm=19.0.0-2027-g9f192350-1jammy ceph-mds=19.0.0-2027-g9f192350-1jammy ceph-mgr=19.0.0-2027-g9f192350-1jammy ceph-common=19.0.0-2027-g9f192350-1jammy ceph-fuse=19.0.0-2027-g9f192350-1jammy ceph-test=19.0.0-2027-g9f192350-1jammy ceph-volume=19.0.0-2027-g9f192350-1jammy radosgw=19.0.0-2027-g9f192350-1jammy python3-rados=19.0.0-2027-g9f192350-1jammy python3-rgw=19.0.0-2027-g9f192350-1jammy python3-cephfs=19.0.0-2027-g9f192350-1jammy python3-rbd=19.0.0-2027-g9f192350-1jammy libcephfs2=19.0.0-2027-g9f192350-1jammy libcephfs-dev=19.0.0-2027-g9f192350-1jammy librados2=19.0.0-2027-g9f192350-1jammy librbd1=19.0.0-2027-g9f192350-1jammy rbd-fuse=19.0.0-2027-g9f192350-1jammy 2024-04-06T03:06:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-04-06T03:06:47.017 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2024-04-06T03:06:47.018 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2024-04-06T03:06:47.180 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-04-06T03:06:47.180 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-04-06T03:06:47.195 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T03:06:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T03:06:47.195 INFO:teuthology.orchestra.run.smithi050.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T03:06:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T03:06:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T03:06:47.196 INFO:teuthology.orchestra.run.smithi050.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T03:06:47.196 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-06T03:06:47.198 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T03:06:47.199 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet zip 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-06T03:06:47.200 INFO:teuthology.orchestra.run.smithi050.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-06T03:06:47.201 INFO:teuthology.orchestra.run.smithi050.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-06T03:06:47.201 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2024-04-06T03:06:47.201 INFO:teuthology.orchestra.run.smithi050.stdout: btrfs-tools 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-06T03:06:47.304 INFO:teuthology.orchestra.run.smithi050.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-06T03:06:47.305 INFO:teuthology.orchestra.run.smithi050.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-06T03:06:47.305 INFO:teuthology.orchestra.run.smithi050.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-06T03:06:47.305 INFO:teuthology.orchestra.run.smithi050.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-06T03:06:47.305 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-06T03:06:47.305 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-06T03:06:47.306 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-06T03:06:47.307 INFO:teuthology.orchestra.run.smithi050.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-06T03:06:47.361 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-04-06T03:06:47.361 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-06T03:06:47.361 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-06T03:06:47.361 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-06T03:06:47.361 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-06T03:06:47.362 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-06T03:06:47.362 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-06T03:06:47.362 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2024-04-06T03:06:47.363 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-06T03:06:47.363 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-06T03:06:47.363 INFO:teuthology.orchestra.run.smithi153.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-06T03:06:47.363 INFO:teuthology.orchestra.run.smithi153.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-06T03:06:47.364 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-06T03:06:47.364 INFO:teuthology.orchestra.run.smithi153.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-06T03:06:47.364 INFO:teuthology.orchestra.run.smithi153.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-06T03:06:47.365 INFO:teuthology.orchestra.run.smithi153.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet zip 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-06T03:06:47.366 INFO:teuthology.orchestra.run.smithi153.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-06T03:06:47.367 INFO:teuthology.orchestra.run.smithi153.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-06T03:06:47.367 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2024-04-06T03:06:47.367 INFO:teuthology.orchestra.run.smithi153.stdout: btrfs-tools 2024-04-06T03:06:47.409 INFO:teuthology.orchestra.run.smithi050.stdout:0 upgraded, 118 newly installed, 0 to remove and 299 not upgraded. 2024-04-06T03:06:47.410 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 178 MB of archives. 2024-04-06T03:06:47.410 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 815 MB of additional disk space will be used. 2024-04-06T03:06:47.410 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-06T03:06:47.463 INFO:teuthology.orchestra.run.smithi153.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-06T03:06:47.464 INFO:teuthology.orchestra.run.smithi153.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-06T03:06:47.464 INFO:teuthology.orchestra.run.smithi153.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-06T03:06:47.464 INFO:teuthology.orchestra.run.smithi153.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-06T03:06:47.464 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-06T03:06:47.464 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-06T03:06:47.464 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-06T03:06:47.465 INFO:teuthology.orchestra.run.smithi153.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-06T03:06:47.629 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-06T03:06:47.629 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2027-g9f192350-1jammy [3,545 kB] 2024-04-06T03:06:47.635 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-06T03:06:47.651 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 118 newly installed, 0 to remove and 299 not upgraded. 2024-04-06T03:06:47.651 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 178 MB of archives. 2024-04-06T03:06:47.651 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 815 MB of additional disk space will be used. 2024-04-06T03:06:47.651 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-06T03:06:47.693 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-06T03:06:47.833 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2027-g9f192350-1jammy [3,545 kB] 2024-04-06T03:06:47.903 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-06T03:06:47.915 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-06T03:06:47.948 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-06T03:06:47.955 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-06T03:06:47.955 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-06T03:06:47.957 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-06T03:06:48.026 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-06T03:06:48.031 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-06T03:06:48.035 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-06T03:06:48.037 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-06T03:06:48.038 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-06T03:06:48.042 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-06T03:06:48.047 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-06T03:06:48.050 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-06T03:06:48.052 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-06T03:06:48.060 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-06T03:06:48.077 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-06T03:06:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-06T03:06:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-06T03:06:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-06T03:06:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-06T03:06:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-06T03:06:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-06T03:06:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-06T03:06:48.088 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-06T03:06:48.089 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-06T03:06:48.089 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-06T03:06:48.116 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-06T03:06:48.117 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-06T03:06:48.118 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-06T03:06:48.119 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-06T03:06:48.120 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-06T03:06:48.123 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-06T03:06:48.124 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-06T03:06:48.145 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-06T03:06:48.155 INFO:teuthology.orchestra.run.smithi050.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-06T03:06:48.156 INFO:teuthology.orchestra.run.smithi050.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-06T03:06:48.172 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-06T03:06:48.175 INFO:teuthology.orchestra.run.smithi050.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-06T03:06:48.176 INFO:teuthology.orchestra.run.smithi050.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-06T03:06:48.179 INFO:teuthology.orchestra.run.smithi050.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-06T03:06:48.183 INFO:teuthology.orchestra.run.smithi050.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-06T03:06:48.184 INFO:teuthology.orchestra.run.smithi050.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-06T03:06:48.187 INFO:teuthology.orchestra.run.smithi050.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-06T03:06:48.188 INFO:teuthology.orchestra.run.smithi050.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-06T03:06:48.212 INFO:teuthology.orchestra.run.smithi050.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-06T03:06:48.213 INFO:teuthology.orchestra.run.smithi050.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-06T03:06:48.214 INFO:teuthology.orchestra.run.smithi050.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-06T03:06:48.247 INFO:teuthology.orchestra.run.smithi050.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-06T03:06:48.401 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-06T03:06:48.413 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-06T03:06:48.430 INFO:teuthology.orchestra.run.smithi050.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-06T03:06:48.432 INFO:teuthology.orchestra.run.smithi050.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-06T03:06:48.435 INFO:teuthology.orchestra.run.smithi050.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-06T03:06:48.436 INFO:teuthology.orchestra.run.smithi050.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-06T03:06:48.438 INFO:teuthology.orchestra.run.smithi050.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-06T03:06:48.442 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-06T03:06:48.452 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-06T03:06:48.453 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-06T03:06:48.454 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-06T03:06:48.471 INFO:teuthology.orchestra.run.smithi050.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-06T03:06:48.472 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-06T03:06:48.474 INFO:teuthology.orchestra.run.smithi050.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-06T03:06:48.474 INFO:teuthology.orchestra.run.smithi050.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-06T03:06:48.495 INFO:teuthology.orchestra.run.smithi050.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-06T03:06:48.495 INFO:teuthology.orchestra.run.smithi050.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-06T03:06:48.517 INFO:teuthology.orchestra.run.smithi050.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-06T03:06:48.518 INFO:teuthology.orchestra.run.smithi050.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-06T03:06:48.518 INFO:teuthology.orchestra.run.smithi050.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-06T03:06:48.519 INFO:teuthology.orchestra.run.smithi050.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-06T03:06:48.519 INFO:teuthology.orchestra.run.smithi050.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-06T03:06:48.520 INFO:teuthology.orchestra.run.smithi050.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-06T03:06:48.525 INFO:teuthology.orchestra.run.smithi050.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-06T03:06:48.559 INFO:teuthology.orchestra.run.smithi050.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-06T03:06:48.562 INFO:teuthology.orchestra.run.smithi050.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-06T03:06:48.563 INFO:teuthology.orchestra.run.smithi050.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-06T03:06:48.599 INFO:teuthology.orchestra.run.smithi050.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-06T03:06:48.605 INFO:teuthology.orchestra.run.smithi050.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-06T03:06:48.606 INFO:teuthology.orchestra.run.smithi050.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-06T03:06:48.608 INFO:teuthology.orchestra.run.smithi050.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-06T03:06:48.618 INFO:teuthology.orchestra.run.smithi050.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-06T03:06:48.619 INFO:teuthology.orchestra.run.smithi050.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-06T03:06:48.626 INFO:teuthology.orchestra.run.smithi050.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-06T03:06:48.641 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-06T03:06:48.646 INFO:teuthology.orchestra.run.smithi050.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-06T03:06:48.647 INFO:teuthology.orchestra.run.smithi050.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-06T03:06:48.648 INFO:teuthology.orchestra.run.smithi050.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-06T03:06:48.675 INFO:teuthology.orchestra.run.smithi050.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-06T03:06:48.677 INFO:teuthology.orchestra.run.smithi050.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-06T03:06:48.692 INFO:teuthology.orchestra.run.smithi050.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-06T03:06:48.693 INFO:teuthology.orchestra.run.smithi050.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-06T03:06:48.694 INFO:teuthology.orchestra.run.smithi050.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-06T03:06:48.698 INFO:teuthology.orchestra.run.smithi050.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-06T03:06:48.700 INFO:teuthology.orchestra.run.smithi050.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-06T03:06:49.052 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-06T03:06:49.086 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-06T03:06:49.088 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-06T03:06:49.169 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-06T03:06:49.220 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-06T03:06:49.242 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-06T03:06:49.249 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-06T03:06:49.250 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-06T03:06:49.261 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-06T03:06:49.262 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-06T03:06:49.342 INFO:teuthology.orchestra.run.smithi050.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2027-g9f192350-1jammy [3,218 kB] 2024-04-06T03:06:49.428 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-06T03:06:49.511 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-06T03:06:49.540 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-06T03:06:49.557 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-06T03:06:49.601 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-06T03:06:49.817 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-06T03:06:49.832 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-06T03:06:49.839 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-06T03:06:49.844 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-06T03:06:49.866 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-06T03:06:49.869 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-06T03:06:49.910 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2027-g9f192350-1jammy [3,218 kB] 2024-04-06T03:06:50.035 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-06T03:06:50.353 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-06T03:06:50.391 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-06T03:06:50.402 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-06T03:06:50.604 INFO:teuthology.orchestra.run.smithi050.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2027-g9f192350-1jammy [712 kB] 2024-04-06T03:06:50.607 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-06T03:06:50.607 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-06T03:06:50.613 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-06T03:06:50.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-06T03:06:50.635 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-06T03:06:50.656 INFO:teuthology.orchestra.run.smithi153.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-06T03:06:50.662 INFO:teuthology.orchestra.run.smithi153.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-06T03:06:50.821 INFO:teuthology.orchestra.run.smithi050.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2027-g9f192350-1jammy [356 kB] 2024-04-06T03:06:50.837 INFO:teuthology.orchestra.run.smithi153.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-06T03:06:50.964 INFO:teuthology.orchestra.run.smithi050.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2027-g9f192350-1jammy [32.8 kB] 2024-04-06T03:06:50.967 INFO:teuthology.orchestra.run.smithi050.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2027-g9f192350-1jammy [180 kB] 2024-04-06T03:06:51.018 INFO:teuthology.orchestra.run.smithi050.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2027-g9f192350-1jammy [64.6 kB] 2024-04-06T03:06:51.063 INFO:teuthology.orchestra.run.smithi050.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2027-g9f192350-1jammy [330 kB] 2024-04-06T03:06:51.100 INFO:teuthology.orchestra.run.smithi153.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-06T03:06:51.172 INFO:teuthology.orchestra.run.smithi050.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2027-g9f192350-1jammy [6,581 kB] 2024-04-06T03:06:51.278 INFO:teuthology.orchestra.run.smithi153.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-06T03:06:51.291 INFO:teuthology.orchestra.run.smithi153.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-06T03:06:51.311 INFO:teuthology.orchestra.run.smithi153.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-06T03:06:51.375 INFO:teuthology.orchestra.run.smithi153.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-06T03:06:51.399 INFO:teuthology.orchestra.run.smithi153.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2027-g9f192350-1jammy [712 kB] 2024-04-06T03:06:51.663 INFO:teuthology.orchestra.run.smithi153.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-06T03:06:51.665 INFO:teuthology.orchestra.run.smithi153.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-06T03:06:51.668 INFO:teuthology.orchestra.run.smithi153.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-06T03:06:51.668 INFO:teuthology.orchestra.run.smithi153.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-06T03:06:51.671 INFO:teuthology.orchestra.run.smithi153.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-06T03:06:51.696 INFO:teuthology.orchestra.run.smithi153.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2027-g9f192350-1jammy [356 kB] 2024-04-06T03:06:51.802 INFO:teuthology.orchestra.run.smithi153.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2027-g9f192350-1jammy [32.8 kB] 2024-04-06T03:06:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2027-g9f192350-1jammy [180 kB] 2024-04-06T03:06:51.873 INFO:teuthology.orchestra.run.smithi153.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2027-g9f192350-1jammy [64.6 kB] 2024-04-06T03:06:51.875 INFO:teuthology.orchestra.run.smithi153.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2027-g9f192350-1jammy [330 kB] 2024-04-06T03:06:51.895 INFO:teuthology.orchestra.run.smithi153.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-06T03:06:51.981 INFO:teuthology.orchestra.run.smithi153.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2027-g9f192350-1jammy [6,581 kB] 2024-04-06T03:06:52.327 INFO:teuthology.orchestra.run.smithi153.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-06T03:06:52.333 INFO:teuthology.orchestra.run.smithi153.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-06T03:06:52.626 INFO:teuthology.orchestra.run.smithi153.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-06T03:06:52.627 INFO:teuthology.orchestra.run.smithi153.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-06T03:06:52.629 INFO:teuthology.orchestra.run.smithi153.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-06T03:06:52.634 INFO:teuthology.orchestra.run.smithi153.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-06T03:06:52.634 INFO:teuthology.orchestra.run.smithi153.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-06T03:06:52.636 INFO:teuthology.orchestra.run.smithi153.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-06T03:06:52.636 INFO:teuthology.orchestra.run.smithi153.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-06T03:06:52.636 INFO:teuthology.orchestra.run.smithi153.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-06T03:06:52.816 INFO:teuthology.orchestra.run.smithi153.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-06T03:06:52.926 INFO:teuthology.orchestra.run.smithi050.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2027-g9f192350-1jammy [112 kB] 2024-04-06T03:06:52.929 INFO:teuthology.orchestra.run.smithi050.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2027-g9f192350-1jammy [466 kB] 2024-04-06T03:06:53.035 INFO:teuthology.orchestra.run.smithi050.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2027-g9f192350-1jammy [25.2 MB] 2024-04-06T03:06:53.220 INFO:teuthology.orchestra.run.smithi153.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-06T03:06:53.267 INFO:teuthology.orchestra.run.smithi153.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-06T03:06:53.301 INFO:teuthology.orchestra.run.smithi153.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-06T03:06:53.399 INFO:teuthology.orchestra.run.smithi153.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-06T03:06:53.438 INFO:teuthology.orchestra.run.smithi153.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-06T03:06:53.451 INFO:teuthology.orchestra.run.smithi153.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-06T03:06:53.455 INFO:teuthology.orchestra.run.smithi153.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-06T03:06:53.503 INFO:teuthology.orchestra.run.smithi153.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-06T03:06:53.506 INFO:teuthology.orchestra.run.smithi153.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-06T03:06:53.530 INFO:teuthology.orchestra.run.smithi153.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-06T03:06:53.692 INFO:teuthology.orchestra.run.smithi153.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-06T03:06:53.747 INFO:teuthology.orchestra.run.smithi153.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2027-g9f192350-1jammy [112 kB] 2024-04-06T03:06:53.780 INFO:teuthology.orchestra.run.smithi153.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2027-g9f192350-1jammy [466 kB] 2024-04-06T03:06:53.886 INFO:teuthology.orchestra.run.smithi153.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2027-g9f192350-1jammy [25.2 MB] 2024-04-06T03:06:53.958 INFO:teuthology.orchestra.run.smithi153.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-06T03:06:53.977 INFO:teuthology.orchestra.run.smithi153.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-06T03:06:54.130 INFO:teuthology.orchestra.run.smithi153.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-06T03:06:54.159 INFO:teuthology.orchestra.run.smithi153.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-06T03:06:54.317 INFO:teuthology.orchestra.run.smithi153.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-06T03:06:54.320 INFO:teuthology.orchestra.run.smithi153.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-06T03:06:54.322 INFO:teuthology.orchestra.run.smithi153.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-06T03:06:54.343 INFO:teuthology.orchestra.run.smithi153.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-06T03:06:54.349 INFO:teuthology.orchestra.run.smithi153.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-06T03:06:57.520 INFO:teuthology.orchestra.run.smithi050.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2027-g9f192350-1jammy [5,077 kB] 2024-04-06T03:06:58.546 INFO:teuthology.orchestra.run.smithi050.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2027-g9f192350-1jammy [244 kB] 2024-04-06T03:06:58.621 INFO:teuthology.orchestra.run.smithi050.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2027-g9f192350-1jammy [124 kB] 2024-04-06T03:06:58.633 INFO:teuthology.orchestra.run.smithi050.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2027-g9f192350-1jammy [1,525 kB] 2024-04-06T03:06:58.913 INFO:teuthology.orchestra.run.smithi153.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2027-g9f192350-1jammy [5,077 kB] 2024-04-06T03:06:59.189 INFO:teuthology.orchestra.run.smithi050.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2027-g9f192350-1jammy [6,247 kB] 2024-04-06T03:06:59.880 INFO:teuthology.orchestra.run.smithi153.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2027-g9f192350-1jammy [244 kB] 2024-04-06T03:06:59.927 INFO:teuthology.orchestra.run.smithi153.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2027-g9f192350-1jammy [124 kB] 2024-04-06T03:06:59.965 INFO:teuthology.orchestra.run.smithi153.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2027-g9f192350-1jammy [1,525 kB] 2024-04-06T03:07:00.303 INFO:teuthology.orchestra.run.smithi153.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2027-g9f192350-1jammy [6,247 kB] 2024-04-06T03:07:00.865 INFO:teuthology.orchestra.run.smithi050.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2027-g9f192350-1jammy [22.7 MB] 2024-04-06T03:07:01.490 INFO:teuthology.orchestra.run.smithi153.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2027-g9f192350-1jammy [22.7 MB] 2024-04-06T03:07:05.207 INFO:teuthology.orchestra.run.smithi153.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2027-g9f192350-1jammy [14.1 kB] 2024-04-06T03:07:05.208 INFO:teuthology.orchestra.run.smithi153.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2027-g9f192350-1jammy [869 kB] 2024-04-06T03:07:05.225 INFO:teuthology.orchestra.run.smithi153.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2027-g9f192350-1jammy [2,433 kB] 2024-04-06T03:07:05.824 INFO:teuthology.orchestra.run.smithi153.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2027-g9f192350-1jammy [777 kB] 2024-04-06T03:07:05.977 INFO:teuthology.orchestra.run.smithi153.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2027-g9f192350-1jammy [145 kB] 2024-04-06T03:07:06.005 INFO:teuthology.orchestra.run.smithi153.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2027-g9f192350-1jammy [2,304 kB] 2024-04-06T03:07:06.133 INFO:teuthology.orchestra.run.smithi050.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2027-g9f192350-1jammy [14.1 kB] 2024-04-06T03:07:06.135 INFO:teuthology.orchestra.run.smithi050.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2027-g9f192350-1jammy [869 kB] 2024-04-06T03:07:06.365 INFO:teuthology.orchestra.run.smithi050.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2027-g9f192350-1jammy [2,433 kB] 2024-04-06T03:07:06.506 INFO:teuthology.orchestra.run.smithi153.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2027-g9f192350-1jammy [8,625 kB] 2024-04-06T03:07:06.833 INFO:teuthology.orchestra.run.smithi050.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2027-g9f192350-1jammy [777 kB] 2024-04-06T03:07:06.968 INFO:teuthology.orchestra.run.smithi050.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2027-g9f192350-1jammy [145 kB] 2024-04-06T03:07:07.001 INFO:teuthology.orchestra.run.smithi050.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2027-g9f192350-1jammy [2,304 kB] 2024-04-06T03:07:07.359 INFO:teuthology.orchestra.run.smithi050.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2027-g9f192350-1jammy [8,625 kB] 2024-04-06T03:07:07.704 INFO:teuthology.orchestra.run.smithi153.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2027-g9f192350-1jammy [14.2 kB] 2024-04-06T03:07:07.705 INFO:teuthology.orchestra.run.smithi153.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2027-g9f192350-1jammy [50.0 MB] 2024-04-06T03:07:08.648 INFO:teuthology.orchestra.run.smithi050.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2027-g9f192350-1jammy [14.2 kB] 2024-04-06T03:07:08.648 INFO:teuthology.orchestra.run.smithi050.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2027-g9f192350-1jammy [50.0 MB] 2024-04-06T03:07:16.807 INFO:teuthology.orchestra.run.smithi153.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2027-g9f192350-1jammy [121 kB] 2024-04-06T03:07:16.809 INFO:teuthology.orchestra.run.smithi153.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2027-g9f192350-1jammy [38.9 kB] 2024-04-06T03:07:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2027-g9f192350-1jammy [13.0 MB] 2024-04-06T03:07:17.749 INFO:teuthology.orchestra.run.smithi050.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2027-g9f192350-1jammy [121 kB] 2024-04-06T03:07:17.751 INFO:teuthology.orchestra.run.smithi050.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2027-g9f192350-1jammy [38.9 kB] 2024-04-06T03:07:17.752 INFO:teuthology.orchestra.run.smithi050.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2027-g9f192350-1jammy [13.0 MB] 2024-04-06T03:07:19.248 INFO:teuthology.orchestra.run.smithi153.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2027-g9f192350-1jammy [91.4 kB] 2024-04-06T03:07:19.776 INFO:teuthology.orchestra.run.smithi050.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2027-g9f192350-1jammy [91.4 kB] 2024-04-06T03:07:19.802 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 178 MB in 32s (5,594 kB/s) 2024-04-06T03:07:19.972 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-06T03:07:20.073 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-06T03:07:20.079 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-06T03:07:20.155 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T03:07:20.366 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 178 MB in 32s (5,476 kB/s) 2024-04-06T03:07:20.475 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-06T03:07:20.501 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-06T03:07:20.891 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T03:07:21.486 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-06T03:07:21.591 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-04-06T03:07:21.597 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-06T03:07:21.670 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-06T03:07:21.692 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T03:07:21.696 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-06T03:07:21.736 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T03:07:22.013 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-06T03:07:22.039 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-06T03:07:22.079 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T03:07:22.165 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-06T03:07:22.192 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T03:07:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:07:22.424 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-06T03:07:22.451 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-06T03:07:22.490 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T03:07:22.618 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-06T03:07:22.645 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T03:07:22.684 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:07:22.903 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-06T03:07:22.929 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T03:07:22.977 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:07:23.046 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-06T03:07:23.072 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T03:07:23.113 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:07:23.414 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-06T03:07:23.418 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-06T03:07:23.431 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-06T03:07:23.441 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T03:07:23.465 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T03:07:23.480 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:07:23.777 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librados2. 2024-04-06T03:07:23.804 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../007-librados2_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:23.834 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-06T03:07:23.843 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librados2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:23.860 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-06T03:07:23.900 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:07:24.237 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-06T03:07:24.264 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-06T03:07:24.303 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T03:07:24.321 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librbd1. 2024-04-06T03:07:24.348 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../008-librbd1_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:24.388 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librbd1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:24.614 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librados2. 2024-04-06T03:07:24.641 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../007-librados2_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:24.681 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librados2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:24.832 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs2. 2024-04-06T03:07:24.848 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:24.907 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:25.134 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librbd1. 2024-04-06T03:07:25.161 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../008-librbd1_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:25.200 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librbd1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:25.326 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rados. 2024-04-06T03:07:25.353 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:25.393 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rados (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:25.645 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs2. 2024-04-06T03:07:25.672 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:25.711 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:25.721 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-06T03:07:25.747 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:25.787 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-argparse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:25.989 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rados. 2024-04-06T03:07:26.013 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:26.047 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rados (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:26.048 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cephfs. 2024-04-06T03:07:26.064 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:26.107 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cephfs (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:26.409 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-06T03:07:26.436 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:26.476 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:26.500 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-06T03:07:26.527 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:26.567 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-argparse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:26.819 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cephfs. 2024-04-06T03:07:26.820 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-06T03:07:26.835 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:26.847 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-06T03:07:26.877 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cephfs (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:26.887 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T03:07:27.148 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-prettytable. 2024-04-06T03:07:27.164 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-06T03:07:27.197 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-06T03:07:27.198 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-06T03:07:27.223 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:27.264 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:27.459 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rbd. 2024-04-06T03:07:27.486 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:27.526 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rbd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:27.566 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-06T03:07:27.582 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-06T03:07:27.616 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T03:07:27.879 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-06T03:07:27.894 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-prettytable. 2024-04-06T03:07:27.906 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-06T03:07:27.921 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-06T03:07:27.945 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T03:07:27.961 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-06T03:07:28.223 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-06T03:07:28.230 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rbd. 2024-04-06T03:07:28.246 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:28.250 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-06T03:07:28.290 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-06T03:07:28.297 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rbd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:28.567 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-06T03:07:28.583 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-06T03:07:28.626 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-06T03:07:28.650 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-06T03:07:28.677 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-06T03:07:28.716 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T03:07:28.963 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua5.1. 2024-04-06T03:07:28.989 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-06T03:07:29.019 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-06T03:07:29.029 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-06T03:07:29.046 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-06T03:07:29.077 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-06T03:07:29.349 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-any. 2024-04-06T03:07:29.376 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-06T03:07:29.380 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-06T03:07:29.407 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-06T03:07:29.415 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-06T03:07:29.446 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-06T03:07:29.634 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package zip. 2024-04-06T03:07:29.650 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-06T03:07:29.684 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking zip (3.0-12build2) ... 2024-04-06T03:07:29.775 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua5.1. 2024-04-06T03:07:29.802 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-06T03:07:29.842 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-06T03:07:30.004 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package unzip. 2024-04-06T03:07:30.031 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-06T03:07:30.071 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-06T03:07:30.153 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua-any. 2024-04-06T03:07:30.180 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-06T03:07:30.220 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-06T03:07:30.447 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package zip. 2024-04-06T03:07:30.466 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package luarocks. 2024-04-06T03:07:30.474 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-06T03:07:30.493 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-06T03:07:30.522 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking zip (3.0-12build2) ... 2024-04-06T03:07:30.649 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-06T03:07:30.917 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package unzip. 2024-04-06T03:07:30.944 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-06T03:07:30.992 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-06T03:07:31.007 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librgw2. 2024-04-06T03:07:31.034 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../025-librgw2_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:31.074 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librgw2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:31.576 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rgw. 2024-04-06T03:07:31.592 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:31.621 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package luarocks. 2024-04-06T03:07:31.634 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:31.648 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-06T03:07:31.695 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-06T03:07:31.971 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-06T03:07:31.998 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-06T03:07:32.046 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T03:07:32.078 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librgw2. 2024-04-06T03:07:32.105 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../025-librgw2_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:32.145 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librgw2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:32.366 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libradosstriper1. 2024-04-06T03:07:32.392 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:32.432 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libradosstriper1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:32.714 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rgw. 2024-04-06T03:07:32.741 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:32.777 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-common. 2024-04-06T03:07:32.797 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:32.804 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:32.835 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:33.209 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-06T03:07:33.236 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-06T03:07:33.283 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T03:07:33.628 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libradosstriper1. 2024-04-06T03:07:33.655 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:33.694 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libradosstriper1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:34.123 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-common. 2024-04-06T03:07:34.149 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-base. 2024-04-06T03:07:34.150 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:34.176 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:34.189 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:34.225 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-base (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:34.795 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-06T03:07:34.822 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-06T03:07:34.861 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-06T03:07:35.139 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cheroot. 2024-04-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-06T03:07:35.206 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T03:07:35.475 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-06T03:07:35.491 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-06T03:07:35.534 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-06T03:07:35.620 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-base. 2024-04-06T03:07:35.647 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:35.706 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-base (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:35.836 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-06T03:07:35.864 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-06T03:07:36.053 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-06T03:07:36.333 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-06T03:07:36.341 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-06T03:07:36.359 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-06T03:07:36.368 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-06T03:07:36.399 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-06T03:07:36.400 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-06T03:07:36.652 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempora. 2024-04-06T03:07:36.679 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-06T03:07:36.702 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cheroot. 2024-04-06T03:07:36.719 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-06T03:07:36.729 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T03:07:37.005 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-portend. 2024-04-06T03:07:37.031 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-06T03:07:37.071 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-06T03:07:37.072 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-06T03:07:37.098 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-06T03:07:37.138 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-06T03:07:37.383 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-06T03:07:37.410 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-06T03:07:37.449 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-06T03:07:37.466 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-06T03:07:37.493 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-06T03:07:37.532 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-06T03:07:37.754 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-06T03:07:37.779 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-06T03:07:37.819 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-06T03:07:37.835 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-06T03:07:37.862 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-06T03:07:37.902 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-06T03:07:38.166 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-tempora. 2024-04-06T03:07:38.172 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-natsort. 2024-04-06T03:07:38.190 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-06T03:07:38.199 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-06T03:07:38.221 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-06T03:07:38.238 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-06T03:07:38.524 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-portend. 2024-04-06T03:07:38.525 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-logutils. 2024-04-06T03:07:38.551 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-06T03:07:38.552 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-06T03:07:38.591 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-06T03:07:38.591 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-06T03:07:38.836 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-06T03:07:38.862 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-06T03:07:38.893 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-06T03:07:38.902 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-06T03:07:38.921 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-06T03:07:38.960 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-06T03:07:39.205 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-mako. 2024-04-06T03:07:39.232 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-06T03:07:39.254 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-06T03:07:39.272 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T03:07:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-06T03:07:39.321 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-06T03:07:39.609 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-06T03:07:39.635 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-06T03:07:39.675 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-natsort. 2024-04-06T03:07:39.675 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-06T03:07:39.702 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-06T03:07:39.741 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-06T03:07:39.978 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-06T03:07:40.004 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-06T03:07:40.044 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-06T03:07:40.069 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-logutils. 2024-04-06T03:07:40.096 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-06T03:07:40.136 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-06T03:07:40.330 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webob. 2024-04-06T03:07:40.357 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-06T03:07:40.388 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-06T03:07:40.397 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-06T03:07:40.421 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-06T03:07:40.455 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-06T03:07:40.717 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-waitress. 2024-04-06T03:07:40.743 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-06T03:07:40.758 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-mako. 2024-04-06T03:07:40.785 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-06T03:07:40.786 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T03:07:40.826 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T03:07:41.086 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempita. 2024-04-06T03:07:41.113 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-06T03:07:41.152 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T03:07:41.169 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-06T03:07:41.197 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-06T03:07:41.238 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-06T03:07:41.413 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-paste. 2024-04-06T03:07:41.440 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-06T03:07:41.472 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T03:07:41.539 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-06T03:07:41.566 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-06T03:07:41.764 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-06T03:07:41.892 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-06T03:07:41.918 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-06T03:07:41.950 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T03:07:42.025 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webob. 2024-04-06T03:07:42.042 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-06T03:07:42.083 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-06T03:07:42.236 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-06T03:07:42.263 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-06T03:07:42.336 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-06T03:07:42.411 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-waitress. 2024-04-06T03:07:42.438 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-06T03:07:42.473 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T03:07:42.638 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webtest. 2024-04-06T03:07:42.665 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-06T03:07:42.706 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-06T03:07:42.748 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-tempita. 2024-04-06T03:07:42.775 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-06T03:07:42.814 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T03:07:43.016 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pecan. 2024-04-06T03:07:43.043 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-06T03:07:43.083 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T03:07:43.142 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-paste. 2024-04-06T03:07:43.169 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-06T03:07:43.208 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T03:07:43.437 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-certifi. 2024-04-06T03:07:43.464 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-06T03:07:43.504 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-06T03:07:43.545 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-06T03:07:43.572 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-06T03:07:43.612 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T03:07:43.832 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-idna. 2024-04-06T03:07:43.858 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-06T03:07:43.898 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-06T03:07:43.914 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-06T03:07:43.941 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-06T03:07:43.981 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-06T03:07:44.176 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-urllib3. 2024-04-06T03:07:44.191 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-06T03:07:44.226 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T03:07:44.234 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webtest. 2024-04-06T03:07:44.250 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-06T03:07:44.284 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-06T03:07:44.554 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests. 2024-04-06T03:07:44.579 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pecan. 2024-04-06T03:07:44.580 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-06T03:07:44.606 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-06T03:07:44.621 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T03:07:44.645 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T03:07:44.898 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-06T03:07:44.925 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-06T03:07:44.956 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T03:07:45.008 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-certifi. 2024-04-06T03:07:45.035 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-06T03:07:45.075 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-06T03:07:45.259 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-06T03:07:45.275 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:45.318 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:45.386 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-idna. 2024-04-06T03:07:45.413 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-06T03:07:45.453 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-06T03:07:45.714 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-06T03:07:45.742 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:45.780 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:45.797 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-urllib3. 2024-04-06T03:07:45.824 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-06T03:07:45.863 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T03:07:46.100 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-06T03:07:46.127 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-06T03:07:46.166 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-06T03:07:46.192 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-requests. 2024-04-06T03:07:46.219 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-06T03:07:46.258 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T03:07:46.444 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-openssl. 2024-04-06T03:07:46.471 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-06T03:07:46.511 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-06T03:07:46.561 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-06T03:07:46.578 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-06T03:07:46.611 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T03:07:46.855 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr. 2024-04-06T03:07:46.882 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:46.889 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-06T03:07:46.916 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:46.922 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:46.955 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:47.317 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mon. 2024-04-06T03:07:47.343 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:47.352 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-06T03:07:47.379 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:47.383 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mon (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:47.419 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:47.738 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-06T03:07:47.765 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-06T03:07:47.805 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-06T03:07:47.987 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-osd. 2024-04-06T03:07:48.013 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:48.053 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-osd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:48.074 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-openssl. 2024-04-06T03:07:48.101 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-06T03:07:48.132 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-06T03:07:48.502 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr. 2024-04-06T03:07:48.529 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:48.569 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:48.955 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mon. 2024-04-06T03:07:48.972 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:49.013 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mon (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:49.064 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph. 2024-04-06T03:07:49.080 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../066-ceph_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:49.122 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:49.442 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-fuse. 2024-04-06T03:07:49.469 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:49.509 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:49.567 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-osd. 2024-04-06T03:07:49.595 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:49.625 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-osd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:49.870 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mds. 2024-04-06T03:07:49.897 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:49.936 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mds (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:50.323 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package cephadm. 2024-04-06T03:07:50.349 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../069-cephadm_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:50.381 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:50.701 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-06T03:07:50.712 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph. 2024-04-06T03:07:50.727 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-06T03:07:50.739 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../066-ceph_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:50.767 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-06T03:07:50.778 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:51.098 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-fuse. 2024-04-06T03:07:51.111 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-06T03:07:51.125 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:51.138 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T03:07:51.164 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:51.177 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T03:07:51.526 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mds. 2024-04-06T03:07:51.553 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:51.592 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mds (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:51.962 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephadm. 2024-04-06T03:07:51.989 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../069-cephadm_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:07:52.028 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:52.357 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-babel. 2024-04-06T03:07:52.384 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T03:07:52.424 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T03:07:52.514 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-06T03:07:52.530 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-06T03:07:52.564 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-06T03:07:52.768 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jinja2. 2024-04-06T03:07:52.795 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-06T03:07:52.835 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T03:07:52.875 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-06T03:07:52.892 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T03:07:52.925 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T03:07:53.179 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-06T03:07:53.206 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:53.246 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:53.607 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-06T03:07:53.634 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-06T03:07:53.682 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-06T03:07:53.986 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-routes. 2024-04-06T03:07:53.988 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-babel. 2024-04-06T03:07:54.012 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-06T03:07:54.016 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-06T03:07:54.052 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T03:07:54.055 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T03:07:54.389 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-06T03:07:54.408 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jinja2. 2024-04-06T03:07:54.414 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:54.435 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-06T03:07:54.454 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:54.475 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T03:07:54.844 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-06T03:07:54.872 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:54.911 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:55.273 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-06T03:07:55.300 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-06T03:07:55.339 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-06T03:07:55.650 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-routes. 2024-04-06T03:07:55.678 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-06T03:07:55.717 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T03:07:55.726 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-06T03:07:55.753 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-06T03:07:55.793 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T03:07:56.062 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-06T03:07:56.089 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:56.128 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:56.296 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-joblib. 2024-04-06T03:07:56.323 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-06T03:07:56.362 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T03:07:56.732 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-06T03:07:56.759 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-06T03:07:56.799 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-06T03:07:57.068 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn. 2024-04-06T03:07:57.095 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-06T03:07:57.134 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T03:07:57.375 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-06T03:07:57.403 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-06T03:07:57.467 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T03:07:57.860 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-06T03:07:57.888 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:57.987 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-joblib. 2024-04-06T03:07:58.014 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-06T03:07:58.020 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:58.053 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T03:07:58.415 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-06T03:07:58.443 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-06T03:07:58.482 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-06T03:07:58.751 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn. 2024-04-06T03:07:58.779 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-06T03:07:58.809 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T03:07:58.915 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cachetools. 2024-04-06T03:07:58.942 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-06T03:07:58.981 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-06T03:07:59.293 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-06T03:07:59.320 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-06T03:07:59.351 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-06T03:07:59.543 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-06T03:07:59.571 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:07:59.662 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-06T03:07:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-06T03:07:59.695 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:07:59.738 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T03:08:00.049 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rsa. 2024-04-06T03:08:00.076 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-06T03:08:00.115 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-06T03:08:00.410 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-google-auth. 2024-04-06T03:08:00.438 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-06T03:08:00.477 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-06T03:08:00.707 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cachetools. 2024-04-06T03:08:00.734 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-06T03:08:00.773 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-06T03:08:00.796 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-06T03:08:00.824 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-06T03:08:00.863 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T03:08:01.036 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-06T03:08:01.064 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-06T03:08:01.103 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-06T03:08:01.174 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-websocket. 2024-04-06T03:08:01.202 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-06T03:08:01.241 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-06T03:08:01.423 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-06T03:08:01.451 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-06T03:08:01.489 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T03:08:01.577 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-06T03:08:01.605 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-06T03:08:01.644 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T03:08:01.817 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rsa. 2024-04-06T03:08:01.845 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-06T03:08:01.884 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-06T03:08:02.205 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-google-auth. 2024-04-06T03:08:02.232 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-06T03:08:02.271 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-06T03:08:02.483 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-06T03:08:02.511 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:08:02.549 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:02.608 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-06T03:08:02.635 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-06T03:08:02.674 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T03:08:02.853 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-06T03:08:02.879 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-06T03:08:02.918 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T03:08:02.944 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-websocket. 2024-04-06T03:08:02.972 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-06T03:08:03.010 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-06T03:08:03.238 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-06T03:08:03.265 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T03:08:03.304 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T03:08:03.330 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-06T03:08:03.358 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-06T03:08:03.397 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T03:08:03.582 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package jq. 2024-04-06T03:08:03.609 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T03:08:03.648 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-06T03:08:03.909 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package socat. 2024-04-06T03:08:03.925 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-06T03:08:03.968 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-06T03:08:04.027 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-06T03:08:04.044 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:08:04.085 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:04.287 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package xmlstarlet. 2024-04-06T03:08:04.303 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-06T03:08:04.337 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-06T03:08:04.372 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-06T03:08:04.401 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-06T03:08:04.438 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T03:08:04.691 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-test. 2024-04-06T03:08:04.707 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:04.741 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-06T03:08:04.749 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-test (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:04.769 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T03:08:04.809 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T03:08:05.077 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package jq. 2024-04-06T03:08:05.102 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-06T03:08:05.152 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-06T03:08:05.430 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package socat. 2024-04-06T03:08:05.458 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-06T03:08:05.496 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-06T03:08:05.824 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package xmlstarlet. 2024-04-06T03:08:05.853 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-06T03:08:05.899 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-06T03:08:06.278 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-test. 2024-04-06T03:08:06.307 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:06.345 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-test (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:07.004 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-volume. 2024-04-06T03:08:07.031 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:08:07.070 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-volume (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:07.407 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-06T03:08:07.435 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:07.474 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs-dev (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:07.760 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-06T03:08:07.776 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-06T03:08:07.810 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-06T03:08:08.164 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-06T03:08:08.192 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-06T03:08:08.231 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-06T03:08:08.627 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package nvme-cli. 2024-04-06T03:08:08.654 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-06T03:08:08.693 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T03:08:08.716 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-volume. 2024-04-06T03:08:08.744 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2027-g9f192350-1jammy_all.deb ... 2024-04-06T03:08:08.783 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-volume (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:09.087 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-06T03:08:09.115 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package pkg-config. 2024-04-06T03:08:09.115 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:09.143 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-06T03:08:09.153 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs-dev (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:09.181 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-06T03:08:09.450 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-06T03:08:09.464 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-06T03:08:09.467 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-06T03:08:09.489 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-06T03:08:09.509 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-06T03:08:09.531 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-06T03:08:09.860 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-06T03:08:09.888 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-06T03:08:09.921 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-06T03:08:09.975 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-06T03:08:09.975 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-06T03:08:09.997 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-06T03:08:10.249 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package nvme-cli. 2024-04-06T03:08:10.273 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastescript. 2024-04-06T03:08:10.276 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-06T03:08:10.301 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-06T03:08:10.305 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T03:08:10.339 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-06T03:08:10.669 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package pkg-config. 2024-04-06T03:08:10.684 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pluggy. 2024-04-06T03:08:10.685 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-06T03:08:10.712 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-06T03:08:10.735 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-06T03:08:10.750 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-06T03:08:11.011 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-psutil. 2024-04-06T03:08:11.018 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-06T03:08:11.036 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-06T03:08:11.046 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-06T03:08:11.078 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-06T03:08:11.088 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-06T03:08:11.439 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-py. 2024-04-06T03:08:11.467 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-06T03:08:11.514 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-06T03:08:11.533 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-06T03:08:11.561 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-06T03:08:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-06T03:08:11.809 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pygments. 2024-04-06T03:08:11.837 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-06T03:08:11.875 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T03:08:11.885 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastescript. 2024-04-06T03:08:11.913 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-06T03:08:11.960 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-06T03:08:12.280 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pluggy. 2024-04-06T03:08:12.309 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-06T03:08:12.347 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-06T03:08:12.363 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-06T03:08:12.391 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-06T03:08:12.429 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-06T03:08:12.641 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-psutil. 2024-04-06T03:08:12.670 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-06T03:08:12.699 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-06T03:08:12.773 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-toml. 2024-04-06T03:08:12.801 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-06T03:08:12.840 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-06T03:08:13.061 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-py. 2024-04-06T03:08:13.090 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-06T03:08:13.127 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-06T03:08:13.136 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pytest. 2024-04-06T03:08:13.164 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-06T03:08:13.202 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T03:08:13.456 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pygments. 2024-04-06T03:08:13.484 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-06T03:08:13.522 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T03:08:13.523 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplejson. 2024-04-06T03:08:13.539 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-06T03:08:13.572 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-06T03:08:13.833 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-06T03:08:13.861 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-06T03:08:13.900 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-06T03:08:14.035 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-06T03:08:14.063 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-06T03:08:14.101 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-06T03:08:14.404 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-toml. 2024-04-06T03:08:14.429 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-06T03:08:14.470 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-06T03:08:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package radosgw. 2024-04-06T03:08:14.656 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../116-radosgw_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:14.695 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking radosgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:14.715 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pytest. 2024-04-06T03:08:14.743 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-06T03:08:14.773 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T03:08:15.110 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplejson. 2024-04-06T03:08:15.127 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-06T03:08:15.167 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-06T03:08:15.429 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-06T03:08:15.457 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-06T03:08:15.487 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-06T03:08:15.489 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package rbd-fuse. 2024-04-06T03:08:15.505 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:15.547 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking rbd-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:15.852 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-06T03:08:16.111 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T03:08:16.237 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T03:08:16.258 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package radosgw. 2024-04-06T03:08:16.286 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../116-radosgw_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:16.324 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking radosgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:16.501 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-06T03:08:16.845 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-06T03:08:17.144 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package rbd-fuse. 2024-04-06T03:08:17.172 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2027-g9f192350-1jammy_amd64.deb ... 2024-04-06T03:08:17.210 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking rbd-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:17.272 INFO:teuthology.orchestra.run.smithi153.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-06T03:08:17.338 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:17.515 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-06T03:08:17.782 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-06T03:08:17.925 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-06T03:08:18.079 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user cephadm....done 2024-04-06T03:08:18.193 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-06T03:08:18.228 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T03:08:18.504 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-06T03:08:18.554 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-06T03:08:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-06T03:08:18.840 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-06T03:08:19.015 INFO:teuthology.orchestra.run.smithi050.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-06T03:08:19.082 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-06T03:08:19.101 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:19.359 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T03:08:19.484 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-06T03:08:19.795 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T03:08:20.094 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user cephadm....done 2024-04-06T03:08:20.156 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-06T03:08:20.253 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-06T03:08:20.389 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-06T03:08:20.535 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-06T03:08:20.648 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-06T03:08:20.803 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-06T03:08:20.875 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-06T03:08:20.946 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-06T03:08:21.126 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-06T03:08:21.230 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-06T03:08:21.385 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-06T03:08:21.490 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-06T03:08:21.616 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-06T03:08:21.670 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-argparse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:21.926 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-06T03:08:22.029 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T03:08:22.130 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-06T03:08:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-06T03:08:22.303 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-06T03:08:22.332 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-06T03:08:22.571 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-06T03:08:22.608 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-06T03:08:22.855 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-06T03:08:23.003 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-06T03:08:23.123 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-06T03:08:23.172 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-06T03:08:23.236 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T03:08:23.362 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-06T03:08:23.400 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-06T03:08:23.667 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-06T03:08:23.722 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T03:08:23.822 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-06T03:08:23.927 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-argparse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:24.074 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T03:08:24.194 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-06T03:08:24.337 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-06T03:08:24.349 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-06T03:08:24.463 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-06T03:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-06T03:08:24.667 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up zip (3.0-12build2) ... 2024-04-06T03:08:24.802 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T03:08:24.815 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-06T03:08:25.184 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-06T03:08:25.237 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-06T03:08:25.302 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-06T03:08:25.436 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-06T03:08:25.487 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T03:08:25.763 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T03:08:25.837 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-06T03:08:25.881 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-06T03:08:25.982 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T03:08:25.988 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-06T03:08:26.273 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-06T03:08:26.282 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-06T03:08:26.557 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-06T03:08:26.684 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-06T03:08:26.875 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up zip (3.0-12build2) ... 2024-04-06T03:08:26.935 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T03:08:27.001 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-06T03:08:27.328 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T03:08:27.621 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T03:08:27.703 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-06T03:08:27.947 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-06T03:08:27.996 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-06T03:08:28.122 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-06T03:08:28.173 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-06T03:08:28.223 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-06T03:08:28.299 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:28.507 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-06T03:08:28.592 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-idna (3.3-1) ... 2024-04-06T03:08:28.884 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-06T03:08:28.910 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T03:08:29.135 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-06T03:08:29.511 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-06T03:08:29.829 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-06T03:08:30.073 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-06T03:08:30.189 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-06T03:08:30.333 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:08:30.457 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-06T03:08:30.468 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-06T03:08:30.558 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:30.727 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T03:08:30.853 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-06T03:08:30.901 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-idna (3.3-1) ... 2024-04-06T03:08:30.979 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T03:08:31.235 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-06T03:08:31.246 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-06T03:08:31.489 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-06T03:08:31.732 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:08:31.858 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-06T03:08:32.160 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-06T03:08:32.410 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-06T03:08:32.474 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-06T03:08:32.662 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-06T03:08:32.767 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:08:32.913 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-06T03:08:32.919 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-06T03:08:33.139 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-06T03:08:33.195 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-06T03:08:33.321 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-06T03:08:33.381 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T03:08:33.447 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-06T03:08:33.717 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-06T03:08:33.832 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-06T03:08:34.000 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T03:08:34.091 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-06T03:08:34.143 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T03:08:34.359 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:08:34.436 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-06T03:08:34.485 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-06T03:08:34.745 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-06T03:08:34.795 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-06T03:08:34.996 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-06T03:08:35.054 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-06T03:08:35.122 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-06T03:08:35.248 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T03:08:35.322 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-06T03:08:35.336 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-06T03:08:35.573 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-06T03:08:35.583 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T03:08:35.709 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T03:08:35.824 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-06T03:08:36.084 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-06T03:08:36.110 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-06T03:08:36.209 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-06T03:08:36.370 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-06T03:08:36.495 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-06T03:08:36.502 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-06T03:08:36.628 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-06T03:08:36.738 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:08:36.864 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T03:08:36.954 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-06T03:08:37.174 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T03:08:37.264 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-06T03:08:37.475 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T03:08:37.532 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-06T03:08:37.641 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-06T03:08:37.758 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-06T03:08:37.784 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-06T03:08:37.848 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-06T03:08:37.918 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-06T03:08:38.085 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-06T03:08:38.194 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-06T03:08:38.211 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-06T03:08:38.470 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T03:08:38.613 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-06T03:08:38.832 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-06T03:08:38.897 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-06T03:08:38.998 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-06T03:08:39.275 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-06T03:08:39.577 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T03:08:39.600 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-06T03:08:39.751 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-06T03:08:39.896 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T03:08:40.022 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librados2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:40.093 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-06T03:08:40.148 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:40.274 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-06T03:08:40.411 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-06T03:08:40.619 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:40.704 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-06T03:08:40.719 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libradosstriper1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:40.821 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-06T03:08:40.855 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-06T03:08:41.089 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librbd1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:41.148 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-06T03:08:41.189 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-modules-core (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:41.316 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:41.407 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-06T03:08:41.536 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T03:08:41.536 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T03:08:41.794 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-06T03:08:42.033 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs-dev (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:42.168 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rados (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:42.196 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-06T03:08:42.294 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librgw2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:42.420 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T03:08:42.498 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-06T03:08:42.850 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-06T03:08:43.001 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librados2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.135 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.261 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-06T03:08:43.614 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.632 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rbd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.757 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up rbd-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.765 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libradosstriper1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.883 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:43.916 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-06T03:08:44.034 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cephfs (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:44.144 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:44.234 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librbd1 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:44.361 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-modules-core (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:44.487 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:44.636 INFO:teuthology.orchestra.run.smithi153.stdout:Adding group ceph....done 2024-04-06T03:08:44.706 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T03:08:44.706 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-06T03:08:45.038 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user ceph....done 2024-04-06T03:08:45.181 INFO:teuthology.orchestra.run.smithi153.stdout:Setting system user ceph properties....done 2024-04-06T03:08:45.194 INFO:teuthology.orchestra.run.smithi153.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-06T03:08:45.230 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs-dev (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:45.336 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-06T03:08:45.364 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rados (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:45.515 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librgw2 (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:45.666 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-06T03:08:45.666 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-06T03:08:46.186 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-test (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:46.295 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up radosgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:46.770 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T03:08:46.770 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T03:08:46.962 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rbd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:47.087 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up rbd-fuse (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:47.213 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:47.272 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-base (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:47.339 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cephfs (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:47.466 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-common (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:47.670 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-06T03:08:48.042 INFO:teuthology.orchestra.run.smithi050.stdout:Adding group ceph....done 2024-04-06T03:08:48.141 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mds (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:48.375 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T03:08:48.375 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T03:08:48.418 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user ceph....done 2024-04-06T03:08:48.570 INFO:teuthology.orchestra.run.smithi050.stdout:Setting system user ceph properties....done 2024-04-06T03:08:48.586 INFO:teuthology.orchestra.run.smithi050.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-06T03:08:48.733 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-06T03:08:48.859 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:49.062 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-06T03:08:49.078 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T03:08:49.078 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T03:08:49.567 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-test (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:49.611 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-osd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:49.710 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up radosgw (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:49.894 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T03:08:49.894 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T03:08:50.193 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T03:08:50.193 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-06T03:08:50.421 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:50.564 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:50.672 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-base (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:50.731 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mon (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:50.941 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T03:08:50.942 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T03:08:51.035 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-06T03:08:51.424 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:51.549 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mds (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:51.575 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:51.701 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-dashboard (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:51.797 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T03:08:51.797 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-06T03:08:51.852 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-volume (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:52.222 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-06T03:08:52.301 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:52.322 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-06T03:08:52.492 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-06T03:08:52.517 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T03:08:52.517 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-06T03:08:53.002 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-osd (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:53.298 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T03:08:53.298 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-06T03:08:53.534 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-06T03:08:53.837 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:53.989 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:54.165 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mon (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:54.201 INFO:teuthology.orchestra.run.smithi153.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-06T03:08:54.208 DEBUG:teuthology.parallel:result is None 2024-04-06T03:08:54.405 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T03:08:54.405 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-06T03:08:55.298 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-cephadm (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:55.447 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:55.573 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-dashboard (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:55.724 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-volume (19.0.0-2027-g9f192350-1jammy) ... 2024-04-06T03:08:56.119 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-06T03:08:56.229 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-06T03:08:56.356 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-06T03:08:57.507 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-06T03:08:58.182 INFO:teuthology.orchestra.run.smithi050.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-06T03:08:58.186 DEBUG:teuthology.parallel:result is None 2024-04-06T03:08:58.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:08:58.337 DEBUG:teuthology.orchestra.run.smithi050:> dpkg-query -W -f '${Version}' ceph 2024-04-06T03:08:58.354 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-2027-g9f192350-1jammy 2024-04-06T03:08:58.355 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2027-g9f192350-1jammy 2024-04-06T03:08:58.355 INFO:teuthology.task.install:The correct ceph version 19.0.0-2027-g9f192350-1jammy is installed. 2024-04-06T03:08:58.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:08:58.503 DEBUG:teuthology.orchestra.run.smithi153:> dpkg-query -W -f '${Version}' ceph 2024-04-06T03:08:58.523 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2027-g9f192350-1jammy 2024-04-06T03:08:58.523 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2027-g9f192350-1jammy 2024-04-06T03:08:58.523 INFO:teuthology.task.install:The correct ceph version 19.0.0-2027-g9f192350-1jammy is installed. 2024-04-06T03:08:58.525 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-06T03:08:58.525 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:08:58.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T03:08:58.540 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:08:58.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-06T03:08:58.579 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-06T03:08:58.579 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:08:58.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T03:08:58.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T03:08:58.651 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:08:58.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-06T03:08:58.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-06T03:08:58.717 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-06T03:08:58.717 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:08:58.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T03:08:58.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T03:08:58.786 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:08:58.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-06T03:08:58.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-06T03:08:58.854 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-06T03:08:58.854 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:08:58.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T03:08:58.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T03:08:58.923 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:08:58.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-06T03:08:58.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-06T03:08:58.991 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-06T03:08:59.086 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6', 'cephadm_mode': 'root'} 2024-04-06T03:08:59.086 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:08:59.087 INFO:tasks.cephadm:Cluster fsid is 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:08:59.087 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-06T03:08:59.087 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.50', 'mon.c': '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]', 'mon.b': '172.21.15.153'} 2024-04-06T03:08:59.087 INFO:tasks.cephadm:First mon is mon.a on smithi050 2024-04-06T03:08:59.087 INFO:tasks.cephadm:First mgr is y 2024-04-06T03:08:59.087 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-06T03:08:59.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2024-04-06T03:08:59.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-06T03:08:59.117 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-06T03:08:59.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:08:59.349 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/54816/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2027-g9f192350', 'node_name': '172.21.2.9+braggi09', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2027-g9f192350-1jammy'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-04-05 15:07:26.188552', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-04-06T03:08:59.564 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:08:59.566 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-04-06T03:08:59.566 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:00.339 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-rw-r-- 1 ubuntu ubuntu 775038 Apr 6 03:09 /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:00.340 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/9f192350ba5fe6d0a365b251a8c2af61f56a5fa6/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:01.106 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-rw-r-- 1 ubuntu ubuntu 775038 Apr 6 03:09 /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:01.107 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:01.115 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-06T03:09:01.135 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 on all hosts... 2024-04-06T03:09:01.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 pull 2024-04-06T03:09:01.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 pull 2024-04-06T03:09:01.329 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6... 2024-04-06T03:09:01.335 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6... 2024-04-06T03:10:14.237 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:10:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)", 2024-04-06T03:10:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "image_id": "a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880", 2024-04-06T03:10:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "repo_digests": [ 2024-04-06T03:10:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c" 2024-04-06T03:10:14.238 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:10:14.238 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:10:14.484 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:10:14.485 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev)", 2024-04-06T03:10:14.485 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "a5511c7deba913038976ce236d2528051568e0aa5cd4df4989ad160a0a4c3880", 2024-04-06T03:10:14.485 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-06T03:10:14.485 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c" 2024-04-06T03:10:14.485 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:10:14.485 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:10:14.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2024-04-06T03:10:14.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-06T03:10:14.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2024-04-06T03:10:14.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-06T03:10:14.608 INFO:tasks.cephadm:Writing seed config... 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-06T03:10:14.609 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-06T03:10:14.610 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-04-06T03:10:14.611 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:10:14.611 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-06T03:10:14.643 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0313e520-f3c3-11ee-b649-cb9ed24678a4 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-06T03:10:14.643 DEBUG:teuthology.orchestra.run.smithi050:mon.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a.service 2024-04-06T03:10:14.692 DEBUG:teuthology.orchestra.run.smithi050:mgr.y> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y.service 2024-04-06T03:10:14.735 INFO:tasks.cephadm:Bootstrapping... 2024-04-06T03:10:14.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 -v bootstrap --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.50 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:10:14.952 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-06T03:10:14.952 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6', '-v', 'bootstrap', '--fsid', '0313e520-f3c3-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.50', '--skip-admin-label'] 2024-04-06T03:10:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-06T03:10:14.953 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying podman|docker is present... 2024-04-06T03:10:14.953 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying lvm2 is present... 2024-04-06T03:10:14.953 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying time synchronization is in place... 2024-04-06T03:10:14.958 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-06T03:10:14.958 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T03:10:14.962 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-06T03:10:14.962 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:14.966 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-06T03:10:14.966 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-06T03:10:14.970 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-06T03:10:14.970 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:14.974 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-06T03:10:14.974 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout masked 2024-04-06T03:10:14.978 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-06T03:10:14.978 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:14.982 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-06T03:10:14.982 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-06T03:10:14.987 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-06T03:10:14.987 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:14.991 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout enabled 2024-04-06T03:10:14.995 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout active 2024-04-06T03:10:14.995 INFO:teuthology.orchestra.run.smithi050.stdout:Unit ntp.service is enabled and running 2024-04-06T03:10:14.996 INFO:teuthology.orchestra.run.smithi050.stdout:Repeating the final host check... 2024-04-06T03:10:14.996 INFO:teuthology.orchestra.run.smithi050.stdout:docker (/usr/bin/docker) is present 2024-04-06T03:10:14.996 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl is present 2024-04-06T03:10:14.996 INFO:teuthology.orchestra.run.smithi050.stdout:lvcreate is present 2024-04-06T03:10:14.999 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-06T03:10:14.999 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-06T03:10:15.003 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-06T03:10:15.003 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:15.006 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-06T03:10:15.006 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-06T03:10:15.010 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-06T03:10:15.010 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:15.013 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-06T03:10:15.013 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout masked 2024-04-06T03:10:15.017 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-06T03:10:15.017 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:15.021 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-06T03:10:15.021 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-06T03:10:15.024 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-06T03:10:15.024 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-06T03:10:15.028 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout enabled 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout active 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Unit ntp.service is enabled and running 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Host looks OK 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Cluster fsid: 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Acquiring lock 140544218131728 on /run/cephadm/0313e520-f3c3-11ee-b649-cb9ed24678a4.lock 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Lock 140544218131728 acquired on /run/cephadm/0313e520-f3c3-11ee-b649-cb9ed24678a4.lock 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying IP 172.21.15.50 port 3300 ... 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying IP 172.21.15.50 port 6789 ... 2024-04-06T03:10:15.032 INFO:teuthology.orchestra.run.smithi050.stdout:Base mon IP(s) is [172.21.15.50:3300, 172.21.15.50:6789], mon addrv is [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2024-04-06T03:10:15.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 2024-04-06T03:10:15.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-04-06T03:10:15.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.50 2024-04-06T03:10:15.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-06T03:10:15.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-06T03:10:15.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1584/64 scope link 2024-04-06T03:10:15.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-06T03:10:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Mon IP `172.21.15.50` is in CIDR network `172.21.0.0/20` 2024-04-06T03:10:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Mon IP `172.21.15.50` is in CIDR network `172.21.0.0/20` 2024-04-06T03:10:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-06T03:10:15.040 INFO:teuthology.orchestra.run.smithi050.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-06T03:10:15.041 INFO:teuthology.orchestra.run.smithi050.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6... 2024-04-06T03:10:15.301 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout 9f192350ba5fe6d0a365b251a8c2af61f56a5fa6: Pulling from ceph-ci/ceph 2024-04-06T03:10:15.301 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout Digest: sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c 2024-04-06T03:10:15.301 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:10:15.301 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 2024-04-06T03:10:16.782 INFO:teuthology.orchestra.run.smithi050.stdout:ceph: stdout ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev) 2024-04-06T03:10:16.782 INFO:teuthology.orchestra.run.smithi050.stdout:Ceph version: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev) 2024-04-06T03:10:16.782 INFO:teuthology.orchestra.run.smithi050.stdout:Extracting ceph user uid/gid from container image... 2024-04-06T03:10:17.970 INFO:teuthology.orchestra.run.smithi050.stdout:stat: stdout 167 167 2024-04-06T03:10:17.970 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial keys... 2024-04-06T03:10:19.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAavRBmXQXlKhAAcQGihhonHNDSxL7GOTK27A== 2024-04-06T03:10:20.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAcvRBmKnpjAhAA73smpiTPn2c4U0gKcoB1/Q== 2024-04-06T03:10:21.787 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAdvRBmY0zZDRAAwcQPpl/FmzKf5hUOAtZ+1w== 2024-04-06T03:10:21.788 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial monmap... 2024-04-06T03:10:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T03:10:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-06T03:10:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T03:10:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool for a [v2:172.21.15.50:3300,v1:172.21.15.50:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-06T03:10:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = quincy 2024-04-06T03:10:23.102 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: set fsid to 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:23.102 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-06T03:10:23.102 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:10:23.102 INFO:teuthology.orchestra.run.smithi050.stdout:Creating mon... 2024-04-06T03:10:24.358 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.841+0000 7f997d074c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:10:24.358 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.841+0000 7f997d074c80 1 imported monmap: 2024-04-06T03:10:24.358 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-04-06T03:10:24.358 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:24.358 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-04-06T03:10:22.523083+0000 2024-04-06T03:10:24.358 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:10:24.359 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-04-06T03:10:24.359 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-04-06T03:10:24.359 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:10:24.359 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.359 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.841+0000 7f997d074c80 0 /usr/bin/ceph-mon: set fsid to 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Git sha 0 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: DB SUMMARY 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.374 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: DB Session ID: 3684Y2G9M29T1RGOQZBQ 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.error_if_exists: 0 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.create_if_missing: 1 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.env: 0x55eb33bf1c60 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-06T03:10:24.375 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.info_log: 0x55eb34e9ed60 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.statistics: (nil) 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.use_fsync: 0 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:10:24.376 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.db_log_dir: 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.wal_dir: 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:10:24.377 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.write_buffer_manager: 0x55eb34e73360 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.unordered_write: 0 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:10:24.378 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.row_cache: None 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.wal_filter: None 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.two_write_queues: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.wal_compression: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.atomic_flush: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:10:24.379 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:10:24.380 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_open_files: -1 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Compression algorithms supported: 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kZSTD supported: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kXpressCompression supported: 0 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:10:24.381 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kZlibCompression supported: 1 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.merge_operator: 2024-04-06T03:10:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_filter: None 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eb34e9ee60) 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-04-06T03:10:24.383 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x55eb34ea3090 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-04-06T03:10:24.384 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-04-06T03:10:24.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression: NoCompression 2024-04-06T03:10:24.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.num_levels: 7 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:10:24.402 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:10:24.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:10:24.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:10:24.405 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.table_properties_collectors: 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.bloom_locality: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.ttl: 2592000 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:10:24.406 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.enable_blob_files: false 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.min_blob_size: 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-06T03:10:24.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f3119995-4ffc-4e45-8ba1-4ea2450a54cc 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.845+0000 7f997d074c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f997d074c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eb34f6e000 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f997d074c80 4 rocksdb: DB pointer 0x55eb34f5e000 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f996d1df700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f996d1df700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:10:24.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x55eb34ea3090#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:10:24.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-06T03:10:24.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f997d074c80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-04-06T03:10:24.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f997d074c80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-04-06T03:10:24.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-06T03:10:23.849+0000 7f997d074c80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-04-06T03:10:24.411 INFO:teuthology.orchestra.run.smithi050.stdout:create mon.a on 2024-04-06T03:10:24.722 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-06T03:10:24.929 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-04-06T03:10:25.177 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4.target -> /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4.target. 2024-04-06T03:10:25.178 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4.target -> /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4.target. 2024-04-06T03:10:25.478 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a 2024-04-06T03:10:25.478 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a.service: Unit ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a.service not loaded. 2024-04-06T03:10:25.701 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4.target.wants/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a.service -> /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service. 2024-04-06T03:10:25.710 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-06T03:10:25.711 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to enable service . firewalld.service is not available 2024-04-06T03:10:25.711 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mon to start... 2024-04-06T03:10:25.711 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mon... 2024-04-06T03:10:27.183 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:26 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.964850+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout cluster: 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout id: 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout services: 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.218874s) 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-06T03:10:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:27.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout data: 2024-04-06T03:10:27.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-06T03:10:27.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-06T03:10:27.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-06T03:10:27.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout pgs: 2024-04-06T03:10:27.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:27.719 INFO:teuthology.orchestra.run.smithi050.stdout:mon is available 2024-04-06T03:10:27.719 INFO:teuthology.orchestra.run.smithi050.stdout:Assimilating anything we can from ceph.conf... 2024-04-06T03:10:28.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968310+0000 mon.a (mon.0) 2 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-04-06T03:10:28.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968477+0000 mon.a (mon.0) 3 : cluster 0 monmap epoch 1 2024-04-06T03:10:28.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968495+0000 mon.a (mon.0) 4 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:28.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968506+0000 mon.a (mon.0) 5 : cluster 0 last_changed 2024-04-06T03:10:22.523083+0000 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968514+0000 mon.a (mon.0) 6 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968523+0000 mon.a (mon.0) 7 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968532+0000 mon.a (mon.0) 8 : cluster 0 election_strategy: 1 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.968546+0000 mon.a (mon.0) 9 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.970643+0000 mon.a (mon.0) 10 : cluster 0 fsmap 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.972836+0000 mon.a (mon.0) 11 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: cluster 2024-04-06T03:10:26.973322+0000 mon.a (mon.0) 12 : cluster 0 mgrmap e1: no daemons active 2024-04-06T03:10:28.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:27 smithi050 bash[20949]: audit 2024-04-06T03:10:27.187142+0000 mon.a (mon.0) 13 : audit 0 from='client.? 172.21.15.50:0/345660859' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-06T03:10:29.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:29.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [global] 2024-04-06T03:10:29.204 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout fsid = 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:29.205 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2024-04-06T03:10:29.205 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [osd] 2024-04-06T03:10:29.206 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-06T03:10:29.207 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-06T03:10:29.207 INFO:teuthology.orchestra.run.smithi050.stdout:Generating new minimal ceph.conf... 2024-04-06T03:10:29.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:28 smithi050 bash[20949]: audit 2024-04-06T03:10:28.695227+0000 mon.a (mon.0) 14 : audit 1 from='client.? 172.21.15.50:0/2091506034' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T03:10:29.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:28 smithi050 bash[20949]: audit 2024-04-06T03:10:28.698371+0000 mon.a (mon.0) 15 : audit 1 from='client.? 172.21.15.50:0/2091506034' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-06T03:10:30.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:30 smithi050 bash[20949]: audit 2024-04-06T03:10:30.264516+0000 mon.a (mon.0) 16 : audit 0 from='client.? 172.21.15.50:0/3128168547' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:10:30.801 INFO:teuthology.orchestra.run.smithi050.stdout:Restarting the monitor... 2024-04-06T03:10:31.104 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:30 smithi050 systemd[1]: Stopping Ceph mon.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:10:31.105 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:30 smithi050 bash[20949]: debug 2024-04-06T03:10:30.944+0000 7f21f84fe700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-06T03:10:31.105 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:30 smithi050 bash[20949]: debug 2024-04-06T03:10:30.944+0000 7f21f84fe700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-06T03:10:31.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:31 smithi050 bash[21369]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-mon-a 2024-04-06T03:10:31.493 INFO:teuthology.orchestra.run.smithi050.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-06T03:10:31.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:31 smithi050 bash[21428]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-mon-a 2024-04-06T03:10:31.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:31 smithi050 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-06T03:10:31.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:31 smithi050 systemd[1]: Stopped Ceph mon.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:10:31.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:31 smithi050 systemd[1]: Started Ceph mon.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:10:32.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.508+0000 7f7039eb8c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.508+0000 7f7039eb8c80 0 ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 7 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.508+0000 7f7039eb8c80 0 pidfile_write: ignore empty --pid-file 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 0 load: jerasure load: lrc 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Git sha 0 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: DB SUMMARY 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: DB Session ID: S5T4PQZGNVE528HKF8PQ 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: CURRENT file: CURRENT 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-06T03:10:32.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94949 ; 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.error_if_exists: 0 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.create_if_missing: 0 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.env: 0x555d7ec6ec60 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.info_log: 0x555d81341b20 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:10:32.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.statistics: (nil) 2024-04-06T03:10:32.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.use_fsync: 0 2024-04-06T03:10:32.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-06T03:10:32.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:10:32.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:10:32.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:10:32.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:10:32.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-06T03:10:32.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.db_log_dir: 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.wal_dir: 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:10:32.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.write_buffer_manager: 0x555d813141e0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.unordered_write: 0 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:10:32.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.row_cache: None 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.wal_filter: None 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.two_write_queues: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.wal_compression: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.atomic_flush: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:10:32.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:10:32.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_open_files: -1 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Compression algorithms supported: 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kZSTD supported: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kXpressCompression supported: 0 2024-04-06T03:10:32.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kZlibCompression supported: 1 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.merge_operator: 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_filter: None 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:10:32.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555d81341c40) 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cache_index_and_filter_blocks: 1 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: pin_top_level_index_and_filter: 1 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: index_type: 0 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: data_block_index_type: 0 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: index_shortening: 1 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: checksum: 4 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: no_block_cache: 0 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_cache: 0x555d81343090 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_cache_name: BinnedLRUCache 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_cache_options: 2024-04-06T03:10:32.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: capacity : 536870912 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: num_shard_bits : 4 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: strict_capacity_limit : 0 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: high_pri_pool_ratio: 0.000 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_cache_compressed: (nil) 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: persistent_cache: (nil) 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_size: 4096 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_size_deviation: 10 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_restart_interval: 16 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: index_block_restart_interval: 1 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: metadata_block_size: 4096 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: partition_filters: 0 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: use_delta_encoding: 1 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: filter_policy: bloomfilter 2024-04-06T03:10:32.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: whole_key_filtering: 1 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: verify_compression: 0 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: read_amp_bytes_per_bit: 0 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: format_version: 5 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: enable_index_compression: 1 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: block_align: 0 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: max_auto_readahead_size: 262144 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: prepopulate_block_cache: 0 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: initial_auto_readahead_size: 8192 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: num_file_reads_for_auto_readahead: 2 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression: NoCompression 2024-04-06T03:10:32.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.num_levels: 7 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:10:32.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:10:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T03:10:32.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.table_properties_collectors: 2024-04-06T03:10:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.bloom_locality: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.ttl: 2592000 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.enable_blob_files: false 2024-04-06T03:10:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.min_blob_size: 0 2024-04-06T03:10:32.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:10:32.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:10:32.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.512+0000 7f7039eb8c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f3119995-4ffc-4e45-8ba1-4ea2450a54cc 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373032521122, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373032522184, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90910, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 284, "table_properties": {"data_size": 88978, "index_size": 278, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11754, "raw_average_key_size": 48, "raw_value_size": 82497, "raw_average_value_size": 339, "num_data_blocks": 12, "num_entries": 243, "num_filter_entries": 243, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712373032, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f3119995-4ffc-4e45-8ba1-4ea2450a54cc", "db_session_id": "S5T4PQZGNVE528HKF8PQ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373032522271, "job": 1, "event": "recovery_finished"} 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-06T03:10:32.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555d81438000 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f7039eb8c80 4 rocksdb: DB pointer 0x555d81422000 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f702861b700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.516+0000 7f702861b700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: ** DB Stats ** 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: ** Compaction Stats [default] ** 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T03:10:32.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: L0 2/0 90.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Sum 2/0 90.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: ** Compaction Stats [default] ** 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Cumulative compaction: 0.00 GB write, 23.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Interval compaction: 0.00 GB write, 23.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Block cache BinnedLRUCache@0x555d81343090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] at bind addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 1 mon.a@-1(???) e1 preinit fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 mon.a@-1(???).mds e1 new map 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 mon.a@-1(???).mds e1 print_map 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: e1 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: legacy client fscid: -1 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: 2024-04-06T03:10:32.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: No filesystems configured 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: debug 2024-04-06T03:10:32.520+0000 7f7039eb8c80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.548887+0000 mon.a (mon.0) 1 : cluster 1 mon.a is new leader, mons a in quorum (ranks 0) 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.548990+0000 mon.a (mon.0) 2 : cluster 0 monmap epoch 1 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549001+0000 mon.a (mon.0) 3 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549015+0000 mon.a (mon.0) 4 : cluster 0 last_changed 2024-04-06T03:10:22.523083+0000 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549028+0000 mon.a (mon.0) 5 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549043+0000 mon.a (mon.0) 6 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549055+0000 mon.a (mon.0) 7 : cluster 0 election_strategy: 1 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549069+0000 mon.a (mon.0) 8 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:10:32.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549481+0000 mon.a (mon.0) 9 : cluster 0 fsmap 2024-04-06T03:10:32.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549517+0000 mon.a (mon.0) 10 : cluster 0 osdmap e1: 0 total, 0 up, 0 in 2024-04-06T03:10:32.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:32 smithi050 bash[21467]: cluster 2024-04-06T03:10:32.549880+0000 mon.a (mon.0) 11 : cluster 0 mgrmap e1: no daemons active 2024-04-06T03:10:33.195 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-06T03:10:33.213 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:10:33.213 INFO:teuthology.orchestra.run.smithi050.stdout:Creating mgr... 2024-04-06T03:10:33.213 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-06T03:10:33.213 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-06T03:10:33.495 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y 2024-04-06T03:10:33.495 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y.service: Unit ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y.service not loaded. 2024-04-06T03:10:33.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:33 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:10:33.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:33 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:10:33.704 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:33 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:10:33.705 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4.target.wants/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y.service -> /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service. 2024-04-06T03:10:33.713 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-06T03:10:33.713 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to enable service . firewalld.service is not available 2024-04-06T03:10:33.713 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-06T03:10:33.713 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-06T03:10:33.713 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr to start... 2024-04-06T03:10:33.714 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr... 2024-04-06T03:10:33.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:33 smithi050 bash[21467]: audit 2024-04-06T03:10:32.720445+0000 mon.a (mon.0) 12 : audit 1 from='client.? 172.21.15.50:0/3096340462' entity='client.admin' 2024-04-06T03:10:33.975 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:33 smithi050 systemd[1]: Started Ceph mgr.y for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:10:35.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:35 smithi050 bash[21467]: audit 2024-04-06T03:10:35.081615+0000 mon.a (mon.0) 13 : audit 0 from='client.? 172.21.15.50:0/4068777812' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:35.475 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:35 smithi050 bash[21725]: debug 2024-04-06T03:10:35.096+0000 7f3a6b69e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T03:10:35.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T03:10:35.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.691 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T03:10:35.692 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T03:10:35.695 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.695 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T03:10:35.695 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T03:10:35.695 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T03:10:35.696 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T03:10:26.969670+0000", 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:10:35.697 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (1/15)... 2024-04-06T03:10:36.975 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:36 smithi050 bash[21725]: debug 2024-04-06T03:10:36.728+0000 7f3a6b69e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T03:10:36.975 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:36 smithi050 bash[21725]: debug 2024-04-06T03:10:36.828+0000 7f3a6b69e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T03:10:37.475 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:37 smithi050 bash[21725]: debug 2024-04-06T03:10:37.084+0000 7f3a6b69e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T03:10:37.975 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:37 smithi050 bash[21725]: debug 2024-04-06T03:10:37.916+0000 7f3a6b69e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T03:10:38.396 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:38 smithi050 bash[21725]: debug 2024-04-06T03:10:38.028+0000 7f3a6b69e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T03:10:38.396 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:38 smithi050 bash[21725]: debug 2024-04-06T03:10:38.136+0000 7f3a6b69e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T03:10:38.725 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:38 smithi050 bash[21725]: debug 2024-04-06T03:10:38.500+0000 7f3a6b69e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T03:10:38.725 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:38 smithi050 bash[21725]: debug 2024-04-06T03:10:38.632+0000 7f3a6b69e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T03:10:39.097 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:38 smithi050 bash[21467]: audit 2024-04-06T03:10:38.733377+0000 mon.a (mon.0) 14 : audit 0 from='client.? 172.21.15.50:0/3174499007' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:39.097 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:38 smithi050 bash[21725]: debug 2024-04-06T03:10:38.740+0000 7f3a6b69e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T03:10:39.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:39.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:10:39.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:39.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T03:10:39.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:39.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T03:10:39.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T03:10:39.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:39.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T03:10:26.969670+0000", 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:10:39.290 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (2/15)... 2024-04-06T03:10:39.475 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:39 smithi050 bash[21725]: debug 2024-04-06T03:10:39.112+0000 7f3a6b69e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T03:10:39.476 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:39 smithi050 bash[21725]: debug 2024-04-06T03:10:39.356+0000 7f3a6b69e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T03:10:39.975 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:39 smithi050 bash[21725]: debug 2024-04-06T03:10:39.624+0000 7f3a6b69e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T03:10:39.976 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:39 smithi050 bash[21725]: debug 2024-04-06T03:10:39.740+0000 7f3a6b69e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T03:10:40.431 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:40 smithi050 bash[21725]: debug 2024-04-06T03:10:40.024+0000 7f3a6b69e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T03:10:40.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:40 smithi050 bash[21725]: debug 2024-04-06T03:10:40.444+0000 7f3a6b69e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:10:40.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:40 smithi050 bash[21725]: debug 2024-04-06T03:10:40.688+0000 7f3a6b69e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T03:10:41.051 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:40 smithi050 bash[21725]: debug 2024-04-06T03:10:40.812+0000 7f3a6b69e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T03:10:41.302 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:41 smithi050 bash[21725]: debug 2024-04-06T03:10:41.064+0000 7f3a6b69e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:10:42.217 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:42 smithi050 bash[21725]: debug 2024-04-06T03:10:42.112+0000 7f3a6b69e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T03:10:42.476 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:42 smithi050 bash[21725]: debug 2024-04-06T03:10:42.232+0000 7f3a6b69e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T03:10:42.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:42 smithi050 bash[21467]: audit 2024-04-06T03:10:42.310113+0000 mon.a (mon.0) 15 : audit 0 from='client.? 172.21.15.50:0/3692663168' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T03:10:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:42.853 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.854 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T03:10:42.858 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T03:10:26.969670+0000", 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:42.859 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:42.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T03:10:42.860 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:10:42.860 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (3/15)... 2024-04-06T03:10:42.978 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:42 smithi050 bash[21725]: debug 2024-04-06T03:10:42.540+0000 7f3a6b69e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T03:10:42.978 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:42 smithi050 bash[21725]: debug 2024-04-06T03:10:42.656+0000 7f3a6b69e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T03:10:43.476 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21725]: debug 2024-04-06T03:10:43.124+0000 7f3a6b69e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T03:10:43.476 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21725]: debug 2024-04-06T03:10:43.248+0000 7f3a6b69e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T03:10:43.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: cluster 2024-04-06T03:10:43.274893+0000 mon.a (mon.0) 16 : cluster 1 Activating manager daemon y 2024-04-06T03:10:43.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: cluster 2024-04-06T03:10:43.278600+0000 mon.a (mon.0) 17 : cluster 0 mgrmap e2: y(active, starting, since 0.00404009s) 2024-04-06T03:10:43.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.283380+0000 mon.a (mon.0) 18 : audit 0 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:10:43.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.284062+0000 mon.a (mon.0) 19 : audit 0 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:10:43.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.284693+0000 mon.a (mon.0) 20 : audit 0 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:10:43.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.285276+0000 mon.a (mon.0) 21 : audit 0 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:10:43.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.285879+0000 mon.a (mon.0) 22 : audit 0 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T03:10:43.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: cluster 2024-04-06T03:10:43.310650+0000 mon.a (mon.0) 23 : cluster 1 Manager daemon y is now available 2024-04-06T03:10:43.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.322040+0000 mon.a (mon.0) 24 : audit 1 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:10:43.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.324763+0000 mon.a (mon.0) 25 : audit 1 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T03:10:43.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.328349+0000 mon.a (mon.0) 26 : audit 1 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' 2024-04-06T03:10:43.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.330952+0000 mon.a (mon.0) 27 : audit 1 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' 2024-04-06T03:10:43.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:43 smithi050 bash[21467]: audit 2024-04-06T03:10:43.333194+0000 mon.a (mon.0) 28 : audit 1 from='mgr.14102 172.21.15.50:0/2937635316' entity='mgr.y' 2024-04-06T03:10:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:45 smithi050 bash[21467]: cluster 2024-04-06T03:10:44.290419+0000 mon.a (mon.0) 29 : cluster 0 mgrmap e3: y(active, since 1.01584s) 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.978 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-06T03:10:46.979 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-06T03:10:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-06T03:10:46.981 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-06T03:10:26.969670+0000", 2024-04-06T03:10:46.982 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-06T03:10:46.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-06T03:10:46.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-06T03:10:46.983 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:10:46.983 INFO:teuthology.orchestra.run.smithi050.stdout:mgr is available 2024-04-06T03:10:47.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:47 smithi050 bash[21467]: cluster 2024-04-06T03:10:46.286477+0000 mon.a (mon.0) 30 : cluster 0 mgrmap e4: y(active, since 3s) 2024-04-06T03:10:47.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:47 smithi050 bash[21467]: audit 2024-04-06T03:10:46.322547+0000 mon.a (mon.0) 31 : audit 0 from='client.? 172.21.15.50:0/113673562' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-06T03:10:48.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:48 smithi050 bash[21467]: audit 2024-04-06T03:10:48.191669+0000 mon.a (mon.0) 32 : audit 1 from='client.? 172.21.15.50:0/426292391' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [global] 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout fsid = 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-06T03:10:48.770 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-06T03:10:48.771 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [osd] 2024-04-06T03:10:48.771 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-06T03:10:48.771 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-06T03:10:48.771 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling cephadm module... 2024-04-06T03:10:50.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:50 smithi050 bash[21467]: audit 2024-04-06T03:10:50.218512+0000 mon.a (mon.0) 33 : audit 1 from='client.? 172.21.15.50:0/2015579976' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-06T03:10:50.519 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:50 smithi050 bash[21725]: ignoring --setuser ceph since I am not root 2024-04-06T03:10:50.519 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:50 smithi050 bash[21725]: ignoring --setgroup ceph since I am not root 2024-04-06T03:10:50.519 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:50 smithi050 bash[21725]: debug 2024-04-06T03:10:50.459+0000 7f71f426c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T03:10:51.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:51 smithi050 bash[21467]: audit 2024-04-06T03:10:50.270442+0000 mon.a (mon.0) 34 : audit 1 from='client.? 172.21.15.50:0/2015579976' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-06T03:10:51.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:51 smithi050 bash[21467]: cluster 2024-04-06T03:10:50.275797+0000 mon.a (mon.0) 35 : cluster 0 mgrmap e5: y(active, since 7s) 2024-04-06T03:10:52.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:52 smithi050 bash[21467]: audit 2024-04-06T03:10:52.131985+0000 mon.a (mon.0) 36 : audit 0 from='client.? 172.21.15.50:0/4176805314' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T03:10:52.368 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:52 smithi050 bash[21725]: debug 2024-04-06T03:10:52.091+0000 7f71f426c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T03:10:52.369 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:52 smithi050 bash[21725]: debug 2024-04-06T03:10:52.195+0000 7f71f426c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T03:10:52.679 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:10:52.679 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-06T03:10:52.679 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T03:10:52.679 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-06T03:10:52.679 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-06T03:10:52.680 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:10:52.680 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for the mgr to restart... 2024-04-06T03:10:52.680 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr epoch 5... 2024-04-06T03:10:52.698 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:52 smithi050 bash[21725]: debug 2024-04-06T03:10:52.451+0000 7f71f426c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T03:10:53.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:53 smithi050 bash[21725]: debug 2024-04-06T03:10:53.275+0000 7f71f426c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T03:10:53.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:53 smithi050 bash[21725]: debug 2024-04-06T03:10:53.391+0000 7f71f426c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T03:10:53.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:53 smithi050 bash[21725]: debug 2024-04-06T03:10:53.499+0000 7f71f426c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T03:10:54.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:53 smithi050 bash[21725]: debug 2024-04-06T03:10:53.863+0000 7f71f426c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T03:10:54.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:53 smithi050 bash[21725]: debug 2024-04-06T03:10:53.979+0000 7f71f426c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T03:10:54.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:54 smithi050 bash[21725]: debug 2024-04-06T03:10:54.087+0000 7f71f426c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T03:10:54.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:54 smithi050 bash[21725]: debug 2024-04-06T03:10:54.459+0000 7f71f426c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T03:10:54.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:54 smithi050 bash[21725]: debug 2024-04-06T03:10:54.703+0000 7f71f426c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T03:10:55.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:54 smithi050 bash[21725]: debug 2024-04-06T03:10:54.971+0000 7f71f426c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T03:10:55.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:55 smithi050 bash[21725]: debug 2024-04-06T03:10:55.087+0000 7f71f426c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T03:10:55.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:55 smithi050 bash[21725]: debug 2024-04-06T03:10:55.371+0000 7f71f426c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T03:10:56.148 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:55 smithi050 bash[21725]: debug 2024-04-06T03:10:55.791+0000 7f71f426c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:10:56.149 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:56 smithi050 bash[21725]: debug 2024-04-06T03:10:56.039+0000 7f71f426c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T03:10:56.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:56 smithi050 bash[21725]: debug 2024-04-06T03:10:56.159+0000 7f71f426c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T03:10:56.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:56 smithi050 bash[21725]: debug 2024-04-06T03:10:56.415+0000 7f71f426c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:10:57.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:57 smithi050 bash[21725]: debug 2024-04-06T03:10:57.451+0000 7f71f426c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T03:10:57.727 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:57 smithi050 bash[21725]: debug 2024-04-06T03:10:57.567+0000 7f71f426c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T03:10:58.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:57 smithi050 bash[21725]: debug 2024-04-06T03:10:57.871+0000 7f71f426c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T03:10:58.226 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:57 smithi050 bash[21725]: debug 2024-04-06T03:10:57.991+0000 7f71f426c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T03:10:58.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21725]: debug 2024-04-06T03:10:58.463+0000 7f71f426c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T03:10:58.726 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21725]: debug 2024-04-06T03:10:58.587+0000 7f71f426c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T03:10:58.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: cluster 2024-04-06T03:10:58.613999+0000 mon.a (mon.0) 37 : cluster 1 Active manager daemon y restarted 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: cluster 2024-04-06T03:10:58.614290+0000 mon.a (mon.0) 38 : cluster 1 Activating manager daemon y 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: cluster 2024-04-06T03:10:58.620933+0000 mon.a (mon.0) 39 : cluster 0 osdmap e2: 0 total, 0 up, 0 in 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: cluster 2024-04-06T03:10:58.621161+0000 mon.a (mon.0) 40 : cluster 0 mgrmap e6: y(active, starting, since 0.00696891s) 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: audit 2024-04-06T03:10:58.623055+0000 mon.a (mon.0) 41 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: audit 2024-04-06T03:10:58.625491+0000 mon.a (mon.0) 42 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: audit 2024-04-06T03:10:58.626630+0000 mon.a (mon.0) 43 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: audit 2024-04-06T03:10:58.627159+0000 mon.a (mon.0) 44 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: audit 2024-04-06T03:10:58.627477+0000 mon.a (mon.0) 45 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:10:58.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:58 smithi050 bash[21467]: cluster 2024-04-06T03:10:58.650593+0000 mon.a (mon.0) 46 : cluster 1 Manager daemon y is now available 2024-04-06T03:10:59.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: cephadm 2024-04-06T03:10:58.669114+0000 mgr.y (mgr.14120) 1 : cephadm 1 Found migration_current of "None". Setting to last migration. 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:58.671416+0000 mon.a (mon.0) 47 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:58.674765+0000 mon.a (mon.0) 48 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:58.678637+0000 mon.a (mon.0) 49 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:58.683554+0000 mon.a (mon.0) 50 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:58.693702+0000 mon.a (mon.0) 51 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:58.697340+0000 mon.a (mon.0) 52 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:59.135385+0000 mon.a (mon.0) 53 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: audit 2024-04-06T03:10:59.141314+0000 mon.a (mon.0) 54 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:10:59.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:10:59 smithi050 bash[21467]: cluster 2024-04-06T03:10:59.621044+0000 mon.a (mon.0) 55 : cluster 0 mgrmap e7: y(active, since 1.00686s) 2024-04-06T03:11:00.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:11:00.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-06T03:11:00.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-06T03:11:00.244 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:11:00.244 INFO:teuthology.orchestra.run.smithi050.stdout:mgr epoch 5 is available 2024-04-06T03:11:00.244 INFO:teuthology.orchestra.run.smithi050.stdout:Setting orchestrator backend to cephadm... 2024-04-06T03:11:00.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: cephadm 2024-04-06T03:10:59.486652+0000 mgr.y (mgr.14120) 2 : cephadm 1 [06/Apr/2024:03:10:59] ENGINE Bus STARTING 2024-04-06T03:11:00.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: cephadm 2024-04-06T03:10:59.588526+0000 mgr.y (mgr.14120) 3 : cephadm 1 [06/Apr/2024:03:10:59] ENGINE Serving on http://172.21.15.50:8765 2024-04-06T03:11:00.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: audit 2024-04-06T03:10:59.620199+0000 mgr.y (mgr.14120) 4 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T03:11:00.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: audit 2024-04-06T03:10:59.632255+0000 mgr.y (mgr.14120) 5 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T03:11:00.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: cephadm 2024-04-06T03:10:59.710205+0000 mgr.y (mgr.14120) 6 : cephadm 1 [06/Apr/2024:03:10:59] ENGINE Serving on https://172.21.15.50:7150 2024-04-06T03:11:00.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: cephadm 2024-04-06T03:10:59.710332+0000 mgr.y (mgr.14120) 7 : cephadm 1 [06/Apr/2024:03:10:59] ENGINE Bus STARTED 2024-04-06T03:11:00.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:00 smithi050 bash[21467]: audit 2024-04-06T03:10:59.753702+0000 mon.a (mon.0) 56 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:01.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:01 smithi050 bash[21467]: cluster 2024-04-06T03:11:00.684139+0000 mon.a (mon.0) 57 : cluster 0 mgrmap e8: y(active, since 2s) 2024-04-06T03:11:01.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:01 smithi050 bash[21467]: audit 2024-04-06T03:11:01.550322+0000 mon.a (mon.0) 58 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:01.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:01 smithi050 bash[21467]: audit 2024-04-06T03:11:01.574175+0000 mon.a (mon.0) 59 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:02.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:02 smithi050 bash[21467]: audit 2024-04-06T03:11:01.544119+0000 mgr.y (mgr.14120) 8 : audit 0 from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:03.830 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-06T03:11:03.830 INFO:teuthology.orchestra.run.smithi050.stdout:Generating ssh key... 2024-04-06T03:11:03.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:03 smithi050 bash[21467]: audit 2024-04-06T03:11:03.311936+0000 mgr.y (mgr.14120) 9 : audit 0 from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:05.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: Generating public/private rsa key pair. 2024-04-06T03:11:05.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: Your identification has been saved in /tmp/tmpeei9j7t9/key. 2024-04-06T03:11:05.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: Your public key has been saved in /tmp/tmpeei9j7t9/key.pub. 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: The key fingerprint is: 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: SHA256:x7Ghw9cGjRx08QUIwZrSLT/dZ32nTTmwvXfZzaOGPQQ ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: The key's randomart image is: 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: +---[RSA 3072]----+ 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | o=ooo...| 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | ..=.. . | 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | . +* . . | 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | ..=o.E . | 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | .So=.+.+ o| 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | +o..o.+B| 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | .+ BO| 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | . +.+B| 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: | ..o +| 2024-04-06T03:11:05.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:05 smithi050 bash[21725]: +----[SHA256]-----+ 2024-04-06T03:11:06.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:06 smithi050 bash[21467]: audit 2024-04-06T03:11:05.136301+0000 mgr.y (mgr.14120) 10 : audit 0 from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:06.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:06 smithi050 bash[21467]: cephadm 2024-04-06T03:11:05.136549+0000 mgr.y (mgr.14120) 11 : cephadm 1 Generating ssh key... 2024-04-06T03:11:06.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:06 smithi050 bash[21467]: audit 2024-04-06T03:11:05.228316+0000 mon.a (mon.0) 60 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:06.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:06 smithi050 bash[21467]: audit 2024-04-06T03:11:05.233024+0000 mon.a (mon.0) 61 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:07.615 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDafv2twjgzr3B2xPjm0sc97lYrCoS8ruhJq1UzD2aL4ReqLqCgSYcQWl7J2XuUgncU8GGlFsBBhBvtlowJf69N0ms1/ocCzxfLa4m//cxiTK1QctfwZk/PLu7yRj4IxWlkHexMS/RyG6jD9Zpx+0Nb9Rpx4UvUvhlHjW5p1g5Vb68WBFw1DwbT+THleUWuuA03ylx0uLQTADnFneDSFsryqPdNLpU5S0VDxn6ILALyIa/7Ij41ZLKcjIq/Uma4s8eU/qxiBsLa0jTABaq11CwsGultjpivsl39B73jOiSgP54CsX7PG0zRaFa4jTJUrF3eLe/QktUkP0/bzTS1LJYNy8Wn3WxhNuL2qnCLI6Yk11kiDQOO7S2cthKXieK3/zNMJ2MiHYXxqzBTsSQpHiJH24WAKkcFN7BV6gW3oiHwpqo7GPo6re3exqVYDvEAWhkezUCX3ykPlU9W571l7ULwgtI2Mg+kdbcr5pkyA9KuslQRUSr3ORTH/ig2XWCM+rk= ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:11:07.615 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:11:07.616 INFO:teuthology.orchestra.run.smithi050.stdout:Adding key to root@localhost authorized_keys... 2024-04-06T03:11:07.616 INFO:teuthology.orchestra.run.smithi050.stdout:Adding host smithi050... 2024-04-06T03:11:07.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:07 smithi050 bash[21467]: audit 2024-04-06T03:11:07.040995+0000 mgr.y (mgr.14120) 12 : audit 0 from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:09.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:09 smithi050 bash[21467]: audit 2024-04-06T03:11:08.889146+0000 mgr.y (mgr.14120) 13 : audit 0 from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:10.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:10 smithi050 bash[21467]: cephadm 2024-04-06T03:11:09.612592+0000 mgr.y (mgr.14120) 14 : cephadm 1 Deploying cephadm binary to smithi050 2024-04-06T03:11:15.422 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Added host 'smithi050' with addr '172.21.15.50' 2024-04-06T03:11:15.422 INFO:teuthology.orchestra.run.smithi050.stdout:Deploying unmanaged mon service... 2024-04-06T03:11:15.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:15 smithi050 bash[21467]: audit 2024-04-06T03:11:14.605512+0000 mon.a (mon.0) 62 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:15.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:15 smithi050 bash[21467]: cephadm 2024-04-06T03:11:14.606035+0000 mgr.y (mgr.14120) 15 : cephadm 1 Added host smithi050 2024-04-06T03:11:15.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:15 smithi050 bash[21467]: audit 2024-04-06T03:11:14.653022+0000 mon.a (mon.0) 63 : audit 0 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:17.975 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-06T03:11:17.975 INFO:teuthology.orchestra.run.smithi050.stdout:Deploying unmanaged mgr service... 2024-04-06T03:11:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:18 smithi050 bash[21467]: audit 2024-04-06T03:11:17.285269+0000 mgr.y (mgr.14120) 16 : audit 0 from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:18 smithi050 bash[21467]: cephadm 2024-04-06T03:11:17.288516+0000 mgr.y (mgr.14120) 17 : cephadm 1 Saving service mon spec with placement count:5 2024-04-06T03:11:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:18 smithi050 bash[21467]: audit 2024-04-06T03:11:17.293366+0000 mon.a (mon.0) 64 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:18 smithi050 bash[21467]: audit 2024-04-06T03:11:17.628011+0000 mon.a (mon.0) 65 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:18 smithi050 bash[21467]: audit 2024-04-06T03:11:18.163721+0000 mon.a (mon.0) 66 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:19.782 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-06T03:11:20.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:20 smithi050 bash[21467]: cluster 2024-04-06T03:11:18.628049+0000 mgr.y (mgr.14120) 18 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:20.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:20 smithi050 bash[21467]: audit 2024-04-06T03:11:19.233177+0000 mgr.y (mgr.14120) 19 : audit 0 from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:20.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:20 smithi050 bash[21467]: cephadm 2024-04-06T03:11:19.235722+0000 mgr.y (mgr.14120) 20 : cephadm 1 Saving service mgr spec with placement count:2 2024-04-06T03:11:20.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:20 smithi050 bash[21467]: audit 2024-04-06T03:11:19.238909+0000 mon.a (mon.0) 67 : audit 1 from='mgr.14120 172.21.15.50:0/622446554' entity='mgr.y' 2024-04-06T03:11:22.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:22 smithi050 bash[21467]: cluster 2024-04-06T03:11:20.628334+0000 mgr.y (mgr.14120) 21 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:22.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:22 smithi050 bash[21467]: audit 2024-04-06T03:11:21.103603+0000 mon.a (mon.0) 68 : audit 1 from='client.? 172.21.15.50:0/992285771' entity='client.admin' 2024-04-06T03:11:23.492 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling the dashboard module... 2024-04-06T03:11:24.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:23 smithi050 bash[21467]: cluster 2024-04-06T03:11:22.628742+0000 mgr.y (mgr.14120) 22 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:24.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:23 smithi050 bash[21467]: audit 2024-04-06T03:11:22.952594+0000 mon.a (mon.0) 69 : audit 1 from='client.? 172.21.15.50:0/577912285' entity='client.admin' 2024-04-06T03:11:25.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:24 smithi050 bash[21467]: audit 2024-04-06T03:11:24.833217+0000 mon.a (mon.0) 70 : audit 1 from='client.? 172.21.15.50:0/2143499670' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-06T03:11:25.198 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:25 smithi050 bash[21725]: ignoring --setuser ceph since I am not root 2024-04-06T03:11:25.198 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:25 smithi050 bash[21725]: ignoring --setgroup ceph since I am not root 2024-04-06T03:11:25.198 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:25 smithi050 bash[21725]: debug 2024-04-06T03:11:25.174+0000 7ff0acb05200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T03:11:26.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:25 smithi050 bash[21467]: audit 2024-04-06T03:11:24.958564+0000 mon.a (mon.0) 71 : audit 1 from='client.? 172.21.15.50:0/2143499670' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-06T03:11:26.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:25 smithi050 bash[21467]: cluster 2024-04-06T03:11:24.965047+0000 mon.a (mon.0) 72 : cluster 0 mgrmap e9: y(active, since 26s) 2024-04-06T03:11:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:27 smithi050 bash[21467]: audit 2024-04-06T03:11:26.959833+0000 mon.a (mon.0) 73 : audit 0 from='client.? 172.21.15.50:0/2454344247' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-06T03:11:27.133 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:26 smithi050 bash[21725]: debug 2024-04-06T03:11:26.826+0000 7ff0acb05200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T03:11:27.133 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:26 smithi050 bash[21725]: debug 2024-04-06T03:11:26.926+0000 7ff0acb05200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T03:11:27.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:27 smithi050 bash[21725]: debug 2024-04-06T03:11:27.190+0000 7ff0acb05200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for the mgr to restart... 2024-04-06T03:11:27.536 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr epoch 9... 2024-04-06T03:11:28.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:28 smithi050 bash[21725]: debug 2024-04-06T03:11:28.018+0000 7ff0acb05200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T03:11:28.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:28 smithi050 bash[21725]: debug 2024-04-06T03:11:28.134+0000 7ff0acb05200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T03:11:28.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:28 smithi050 bash[21725]: debug 2024-04-06T03:11:28.242+0000 7ff0acb05200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T03:11:28.977 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:28 smithi050 bash[21725]: debug 2024-04-06T03:11:28.614+0000 7ff0acb05200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T03:11:28.978 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:28 smithi050 bash[21725]: debug 2024-04-06T03:11:28.730+0000 7ff0acb05200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T03:11:28.978 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:28 smithi050 bash[21725]: debug 2024-04-06T03:11:28.838+0000 7ff0acb05200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T03:11:29.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:29 smithi050 bash[21725]: debug 2024-04-06T03:11:29.214+0000 7ff0acb05200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T03:11:29.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:29 smithi050 bash[21725]: debug 2024-04-06T03:11:29.458+0000 7ff0acb05200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T03:11:29.977 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:29 smithi050 bash[21725]: debug 2024-04-06T03:11:29.730+0000 7ff0acb05200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T03:11:29.977 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:29 smithi050 bash[21725]: debug 2024-04-06T03:11:29.850+0000 7ff0acb05200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T03:11:30.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:30 smithi050 bash[21725]: debug 2024-04-06T03:11:30.138+0000 7ff0acb05200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T03:11:30.830 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:30 smithi050 bash[21725]: debug 2024-04-06T03:11:30.586+0000 7ff0acb05200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:11:31.215 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:30 smithi050 bash[21725]: debug 2024-04-06T03:11:30.838+0000 7ff0acb05200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T03:11:31.215 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:30 smithi050 bash[21725]: debug 2024-04-06T03:11:30.966+0000 7ff0acb05200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T03:11:31.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:31 smithi050 bash[21725]: debug 2024-04-06T03:11:31.226+0000 7ff0acb05200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:11:32.690 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:32 smithi050 bash[21725]: debug 2024-04-06T03:11:32.274+0000 7ff0acb05200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T03:11:32.690 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:32 smithi050 bash[21725]: debug 2024-04-06T03:11:32.394+0000 7ff0acb05200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T03:11:32.977 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:32 smithi050 bash[21725]: debug 2024-04-06T03:11:32.698+0000 7ff0acb05200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T03:11:32.977 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:32 smithi050 bash[21725]: debug 2024-04-06T03:11:32.818+0000 7ff0acb05200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: cluster 2024-04-06T03:11:33.442318+0000 mon.a (mon.0) 74 : cluster 1 Active manager daemon y restarted 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: cluster 2024-04-06T03:11:33.442676+0000 mon.a (mon.0) 75 : cluster 1 Activating manager daemon y 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: cluster 2024-04-06T03:11:33.449330+0000 mon.a (mon.0) 76 : cluster 0 osdmap e3: 0 total, 0 up, 0 in 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: cluster 2024-04-06T03:11:33.449557+0000 mon.a (mon.0) 77 : cluster 0 mgrmap e10: y(active, starting, since 0.00704964s) 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: audit 2024-04-06T03:11:33.451572+0000 mon.a (mon.0) 78 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: audit 2024-04-06T03:11:33.453219+0000 mon.a (mon.0) 79 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: audit 2024-04-06T03:11:33.454120+0000 mon.a (mon.0) 80 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:11:33.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: audit 2024-04-06T03:11:33.454579+0000 mon.a (mon.0) 81 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:11:33.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: audit 2024-04-06T03:11:33.455008+0000 mon.a (mon.0) 82 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:11:33.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21467]: cluster 2024-04-06T03:11:33.478482+0000 mon.a (mon.0) 83 : cluster 1 Manager daemon y is now available 2024-04-06T03:11:33.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21725]: debug 2024-04-06T03:11:33.290+0000 7ff0acb05200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T03:11:33.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:11:33 smithi050 bash[21725]: debug 2024-04-06T03:11:33.418+0000 7ff0acb05200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T03:11:34.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: audit 2024-04-06T03:11:33.503238+0000 mon.a (mon.0) 84 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: audit 2024-04-06T03:11:33.523641+0000 mon.a (mon.0) 85 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: audit 2024-04-06T03:11:33.527086+0000 mon.a (mon.0) 86 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: cephadm 2024-04-06T03:11:33.726405+0000 mgr.y (mgr.14152) 1 : cephadm 1 [06/Apr/2024:03:11:33] ENGINE Bus STARTING 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: cephadm 2024-04-06T03:11:33.849682+0000 mgr.y (mgr.14152) 2 : cephadm 1 [06/Apr/2024:03:11:33] ENGINE Serving on https://172.21.15.50:7150 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: cephadm 2024-04-06T03:11:33.951143+0000 mgr.y (mgr.14152) 3 : cephadm 1 [06/Apr/2024:03:11:33] ENGINE Serving on http://172.21.15.50:8765 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: cephadm 2024-04-06T03:11:33.951318+0000 mgr.y (mgr.14152) 4 : cephadm 1 [06/Apr/2024:03:11:33] ENGINE Bus STARTED 2024-04-06T03:11:34.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:34 smithi050 bash[21467]: cluster 2024-04-06T03:11:34.456158+0000 mon.a (mon.0) 87 : cluster 0 mgrmap e11: y(active, since 1.01364s) 2024-04-06T03:11:35.151 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-06T03:11:35.151 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-06T03:11:35.151 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-06T03:11:35.151 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-06T03:11:35.151 INFO:teuthology.orchestra.run.smithi050.stdout:mgr epoch 9 is available 2024-04-06T03:11:35.151 INFO:teuthology.orchestra.run.smithi050.stdout:Generating a dashboard self-signed certificate... 2024-04-06T03:11:35.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:35 smithi050 bash[21467]: audit 2024-04-06T03:11:34.454239+0000 mgr.y (mgr.14152) 5 : audit 0 from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-06T03:11:35.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:35 smithi050 bash[21467]: audit 2024-04-06T03:11:34.468483+0000 mgr.y (mgr.14152) 6 : audit 0 from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-06T03:11:36.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:36 smithi050 bash[21467]: cluster 2024-04-06T03:11:35.502897+0000 mon.a (mon.0) 88 : cluster 0 mgrmap e12: y(active, since 2s) 2024-04-06T03:11:37.138 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-06T03:11:37.138 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial admin user... 2024-04-06T03:11:37.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:37 smithi050 bash[21467]: audit 2024-04-06T03:11:36.420594+0000 mgr.y (mgr.14152) 7 : audit 0 from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:37.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:37 smithi050 bash[21467]: audit 2024-04-06T03:11:36.566014+0000 mon.a (mon.0) 89 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:37.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:37 smithi050 bash[21467]: audit 2024-04-06T03:11:36.570864+0000 mon.a (mon.0) 90 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:39.320 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nc.2QUEpEBThCmVBggrHvOILxrYvCnQYetUhMxFf21fy30NrMCf8C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712373098, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-06T03:11:39.320 INFO:teuthology.orchestra.run.smithi050.stdout:Fetching dashboard port number... 2024-04-06T03:11:39.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:39 smithi050 bash[21467]: audit 2024-04-06T03:11:38.468796+0000 mgr.y (mgr.14152) 8 : audit 0 from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:11:39.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:39 smithi050 bash[21467]: audit 2024-04-06T03:11:38.734846+0000 mon.a (mon.0) 91 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:40 smithi050 bash[21467]: cluster 2024-04-06T03:11:39.744127+0000 mon.a (mon.0) 92 : cluster 0 mgrmap e13: y(active, since 6s) 2024-04-06T03:11:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:40 smithi050 bash[21467]: audit 2024-04-06T03:11:40.411186+0000 mon.a (mon.0) 93 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:40 smithi050 bash[21467]: audit 2024-04-06T03:11:40.646414+0000 mon.a (mon.0) 94 : audit 0 from='client.? 172.21.15.50:0/2331480599' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-06T03:11:41.241 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 8443 2024-04-06T03:11:41.242 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-06T03:11:41.242 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-06T03:11:41.244 INFO:teuthology.orchestra.run.smithi050.stdout:Ceph Dashboard is now available at: 2024-04-06T03:11:41.245 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:41.245 INFO:teuthology.orchestra.run.smithi050.stdout: URL: https://smithi050.front.sepia.ceph.com:8443/ 2024-04-06T03:11:41.245 INFO:teuthology.orchestra.run.smithi050.stdout: User: admin 2024-04-06T03:11:41.245 INFO:teuthology.orchestra.run.smithi050.stdout: Password: hfc4krqn2q 2024-04-06T03:11:41.245 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:41.245 INFO:teuthology.orchestra.run.smithi050.stdout:Saving cluster configuration to /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config directory 2024-04-06T03:11:41.246 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling autotune for osd_memory_target 2024-04-06T03:11:42.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:42 smithi050 bash[21467]: audit 2024-04-06T03:11:41.452138+0000 mon.a (mon.0) 95 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout:Or, if you are only running a single cluster on this host: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.045 INFO:teuthology.orchestra.run.smithi050.stdout: ceph telemetry on 2024-04-06T03:11:45.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.046 INFO:teuthology.orchestra.run.smithi050.stdout:For more information see: 2024-04-06T03:11:45.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.046 INFO:teuthology.orchestra.run.smithi050.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-06T03:11:45.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:11:45.046 INFO:teuthology.orchestra.run.smithi050.stdout:Bootstrap complete. 2024-04-06T03:11:45.096 INFO:tasks.cephadm:Fetching config... 2024-04-06T03:11:45.096 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:11:45.096 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-06T03:11:45.102 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-06T03:11:45.102 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:11:45.102 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-06T03:11:45.150 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-06T03:11:45.151 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:11:45.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-06T03:11:45.211 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-06T03:11:45.211 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:11:45.211 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-06T03:11:45.259 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-06T03:11:45.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDafv2twjgzr3B2xPjm0sc97lYrCoS8ruhJq1UzD2aL4ReqLqCgSYcQWl7J2XuUgncU8GGlFsBBhBvtlowJf69N0ms1/ocCzxfLa4m//cxiTK1QctfwZk/PLu7yRj4IxWlkHexMS/RyG6jD9Zpx+0Nb9Rpx4UvUvhlHjW5p1g5Vb68WBFw1DwbT+THleUWuuA03ylx0uLQTADnFneDSFsryqPdNLpU5S0VDxn6ILALyIa/7Ij41ZLKcjIq/Uma4s8eU/qxiBsLa0jTABaq11CwsGultjpivsl39B73jOiSgP54CsX7PG0zRaFa4jTJUrF3eLe/QktUkP0/bzTS1LJYNy8Wn3WxhNuL2qnCLI6Yk11kiDQOO7S2cthKXieK3/zNMJ2MiHYXxqzBTsSQpHiJH24WAKkcFN7BV6gW3oiHwpqo7GPo6re3exqVYDvEAWhkezUCX3ykPlU9W571l7ULwgtI2Mg+kdbcr5pkyA9KuslQRUSr3ORTH/ig2XWCM+rk= ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T03:11:45.328 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDafv2twjgzr3B2xPjm0sc97lYrCoS8ruhJq1UzD2aL4ReqLqCgSYcQWl7J2XuUgncU8GGlFsBBhBvtlowJf69N0ms1/ocCzxfLa4m//cxiTK1QctfwZk/PLu7yRj4IxWlkHexMS/RyG6jD9Zpx+0Nb9Rpx4UvUvhlHjW5p1g5Vb68WBFw1DwbT+THleUWuuA03ylx0uLQTADnFneDSFsryqPdNLpU5S0VDxn6ILALyIa/7Ij41ZLKcjIq/Uma4s8eU/qxiBsLa0jTABaq11CwsGultjpivsl39B73jOiSgP54CsX7PG0zRaFa4jTJUrF3eLe/QktUkP0/bzTS1LJYNy8Wn3WxhNuL2qnCLI6Yk11kiDQOO7S2cthKXieK3/zNMJ2MiHYXxqzBTsSQpHiJH24WAKkcFN7BV6gW3oiHwpqo7GPo6re3exqVYDvEAWhkezUCX3ykPlU9W571l7ULwgtI2Mg+kdbcr5pkyA9KuslQRUSr3ORTH/ig2XWCM+rk= ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:11:45.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDafv2twjgzr3B2xPjm0sc97lYrCoS8ruhJq1UzD2aL4ReqLqCgSYcQWl7J2XuUgncU8GGlFsBBhBvtlowJf69N0ms1/ocCzxfLa4m//cxiTK1QctfwZk/PLu7yRj4IxWlkHexMS/RyG6jD9Zpx+0Nb9Rpx4UvUvhlHjW5p1g5Vb68WBFw1DwbT+THleUWuuA03ylx0uLQTADnFneDSFsryqPdNLpU5S0VDxn6ILALyIa/7Ij41ZLKcjIq/Uma4s8eU/qxiBsLa0jTABaq11CwsGultjpivsl39B73jOiSgP54CsX7PG0zRaFa4jTJUrF3eLe/QktUkP0/bzTS1LJYNy8Wn3WxhNuL2qnCLI6Yk11kiDQOO7S2cthKXieK3/zNMJ2MiHYXxqzBTsSQpHiJH24WAKkcFN7BV6gW3oiHwpqo7GPo6re3exqVYDvEAWhkezUCX3ykPlU9W571l7ULwgtI2Mg+kdbcr5pkyA9KuslQRUSr3ORTH/ig2XWCM+rk= ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-06T03:11:45.361 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDafv2twjgzr3B2xPjm0sc97lYrCoS8ruhJq1UzD2aL4ReqLqCgSYcQWl7J2XuUgncU8GGlFsBBhBvtlowJf69N0ms1/ocCzxfLa4m//cxiTK1QctfwZk/PLu7yRj4IxWlkHexMS/RyG6jD9Zpx+0Nb9Rpx4UvUvhlHjW5p1g5Vb68WBFw1DwbT+THleUWuuA03ylx0uLQTADnFneDSFsryqPdNLpU5S0VDxn6ILALyIa/7Ij41ZLKcjIq/Uma4s8eU/qxiBsLa0jTABaq11CwsGultjpivsl39B73jOiSgP54CsX7PG0zRaFa4jTJUrF3eLe/QktUkP0/bzTS1LJYNy8Wn3WxhNuL2qnCLI6Yk11kiDQOO7S2cthKXieK3/zNMJ2MiHYXxqzBTsSQpHiJH24WAKkcFN7BV6gW3oiHwpqo7GPo6re3exqVYDvEAWhkezUCX3ykPlU9W571l7ULwgtI2Mg+kdbcr5pkyA9KuslQRUSr3ORTH/ig2XWCM+rk= ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:11:45.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-06T03:11:45.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:45 smithi050 bash[21467]: audit 2024-04-06T03:11:44.522186+0000 mon.a (mon.0) 96 : audit 1 from='client.? 172.21.15.50:0/2309302374' entity='client.admin' 2024-04-06T03:11:50.036 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:11:53.869 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-06T03:11:53.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-06T03:11:54.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:52.848735+0000 mon.a (mon.0) 97 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:54.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:52.851386+0000 mon.a (mon.0) 98 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:54.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:52.852205+0000 mon.a (mon.0) 99 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:11:54.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:52.856169+0000 mon.a (mon.0) 100 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:54.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:52.873155+0000 mon.a (mon.0) 101 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:54.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:52.877589+0000 mon.a (mon.0) 102 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:54.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:53.180916+0000 mon.a (mon.0) 103 : audit 1 from='client.? 172.21.15.50:0/3006321225' entity='client.admin' 2024-04-06T03:11:54.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:53.208548+0000 mon.a (mon.0) 104 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:11:54.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:53 smithi050 bash[21467]: audit 2024-04-06T03:11:53.213697+0000 mon.a (mon.0) 105 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:11:55.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:54 smithi050 bash[21467]: cluster 2024-04-06T03:11:53.455758+0000 mgr.y (mgr.14152) 9 : cluster 0 pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:57.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:56 smithi050 bash[21467]: cluster 2024-04-06T03:11:55.456266+0000 mgr.y (mgr.14152) 10 : cluster 0 pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:11:58.583 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:11:58.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:11:58 smithi050 bash[21467]: cluster 2024-04-06T03:11:57.456796+0000 mgr.y (mgr.14152) 11 : cluster 0 pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:00.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:00 smithi050 bash[21467]: cluster 2024-04-06T03:11:59.457180+0000 mgr.y (mgr.14152) 12 : cluster 0 pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:01.793 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-04-06T03:12:01.793 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:12:01.793 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-04-06T03:12:01.799 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:12:01.799 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:01.846 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-04-06T03:12:01.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi153 2024-04-06T03:12:02.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.120352+0000 mgr.y (mgr.14152) 13 : audit 0 from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:02.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.125585+0000 mon.a (mon.0) 106 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:02.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.165521+0000 mon.a (mon.0) 107 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:02.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.167419+0000 mon.a (mon.0) 108 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:02.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.168339+0000 mon.a (mon.0) 109 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:02.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: cephadm 2024-04-06T03:12:01.169763+0000 mgr.y (mgr.14152) 14 : cephadm 1 Updating smithi050:/etc/ceph/ceph.conf 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: cephadm 2024-04-06T03:12:01.283926+0000 mgr.y (mgr.14152) 15 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: cephadm 2024-04-06T03:12:01.398351+0000 mgr.y (mgr.14152) 16 : cephadm 1 Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: cluster 2024-04-06T03:12:01.457642+0000 mgr.y (mgr.14152) 17 : cluster 0 pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: cephadm 2024-04-06T03:12:01.513312+0000 mgr.y (mgr.14152) 18 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.683525+0000 mon.a (mon.0) 110 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.688834+0000 mon.a (mon.0) 111 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:02.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:02 smithi050 bash[21467]: audit 2024-04-06T03:12:01.706281+0000 mon.a (mon.0) 112 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:04.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:04 smithi050 bash[21467]: cluster 2024-04-06T03:12:03.458116+0000 mgr.y (mgr.14152) 19 : cluster 0 pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:06.556 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:12:06.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:06 smithi050 bash[21467]: cluster 2024-04-06T03:12:05.458561+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:08.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:08 smithi050 bash[21467]: cluster 2024-04-06T03:12:07.458901+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:10.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:10 smithi050 bash[21467]: audit 2024-04-06T03:12:09.097518+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:10.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:10 smithi050 bash[21467]: cluster 2024-04-06T03:12:09.459271+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:10.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:10 smithi050 bash[21467]: cephadm 2024-04-06T03:12:09.912014+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi153 2024-04-06T03:12:12.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:12 smithi050 bash[21467]: cluster 2024-04-06T03:12:11.459672+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:14.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:14 smithi050 bash[21467]: cluster 2024-04-06T03:12:13.460024+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:15.472 INFO:teuthology.orchestra.run.smithi050.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-04-06T03:12:16.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-06T03:12:16.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:16 smithi050 bash[21467]: cluster 2024-04-06T03:12:15.460454+0000 mgr.y (mgr.14152) 27 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:16.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:16 smithi050 bash[21467]: audit 2024-04-06T03:12:15.483738+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:16.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:16 smithi050 bash[21467]: cephadm 2024-04-06T03:12:15.484674+0000 mgr.y (mgr.14152) 28 : cephadm 1 Added host smithi153 2024-04-06T03:12:16.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:16 smithi050 bash[21467]: audit 2024-04-06T03:12:15.529257+0000 mon.a (mon.0) 114 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:16.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:16 smithi050 bash[21467]: audit 2024-04-06T03:12:15.951814+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:18.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:18 smithi050 bash[21467]: audit 2024-04-06T03:12:17.427485+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:18.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:18 smithi050 bash[21467]: cluster 2024-04-06T03:12:17.460883+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:19.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:19 smithi050 bash[21467]: audit 2024-04-06T03:12:18.626095+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:12:20.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:20 smithi050 bash[21467]: cluster 2024-04-06T03:12:19.461286+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:22.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:22 smithi050 bash[21467]: cluster 2024-04-06T03:12:21.461722+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:23.764 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:12:23.764 INFO:teuthology.orchestra.run.smithi050.stdout:[{"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-04-06T03:12:24.393 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-06T03:12:24.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: cluster 2024-04-06T03:12:23.462028+0000 mgr.y (mgr.14152) 32 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.776145+0000 mgr.y (mgr.14152) 33 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.857372+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.859595+0000 mon.a (mon.0) 119 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.862068+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.864345+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.865090+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:12:25.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.866284+0000 mon.a (mon.0) 123 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:25.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:23.867300+0000 mon.a (mon.0) 124 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:25.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: cephadm 2024-04-06T03:12:23.868625+0000 mgr.y (mgr.14152) 34 : cephadm 1 Updating smithi153:/etc/ceph/ceph.conf 2024-04-06T03:12:25.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: cephadm 2024-04-06T03:12:24.032961+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:12:25.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:24.532582+0000 mon.a (mon.0) 125 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:25.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:24.536598+0000 mon.a (mon.0) 126 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:25.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:24 smithi050 bash[21467]: audit 2024-04-06T03:12:24.541017+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:26.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:25 smithi050 bash[21467]: cephadm 2024-04-06T03:12:24.198084+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:26.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:25 smithi050 bash[21467]: cephadm 2024-04-06T03:12:24.363190+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:12:27.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:26 smithi050 bash[21467]: cluster 2024-04-06T03:12:25.462396+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:29.120 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:12:29.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:28 smithi050 bash[21467]: cluster 2024-04-06T03:12:27.462839+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:31.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:30 smithi050 bash[21467]: cluster 2024-04-06T03:12:29.463270+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:31.863 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-04-06T03:12:32.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:31 smithi050 bash[21467]: audit 2024-04-06T03:12:31.673577+0000 mon.a (mon.0) 128 : audit 1 from='client.? 172.21.15.50:0/816141983' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T03:12:32.484 INFO:tasks.cephadm:Adding mon.a on smithi050 2024-04-06T03:12:32.484 INFO:tasks.cephadm:Adding mon.c on smithi050 2024-04-06T03:12:32.484 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-04-06T03:12:32.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b' 2024-04-06T03:12:33.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:32 smithi050 bash[21467]: cluster 2024-04-06T03:12:31.463556+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:33.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:32 smithi050 bash[21467]: audit 2024-04-06T03:12:31.871165+0000 mon.a (mon.0) 129 : audit 1 from='client.? 172.21.15.50:0/816141983' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T03:12:33.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:32 smithi050 bash[21467]: cluster 2024-04-06T03:12:31.876983+0000 mon.a (mon.0) 130 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:33.694 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:12:35.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:34 smithi050 bash[21467]: cluster 2024-04-06T03:12:33.463934+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:36.203 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-04-06T03:12:36.707 DEBUG:teuthology.orchestra.run.smithi050:mon.c> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.c.service 2024-04-06T03:12:36.711 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.b.service 2024-04-06T03:12:36.714 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-06T03:12:36.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: cluster 2024-04-06T03:12:35.464435+0000 mgr.y (mgr.14152) 43 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.214907+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.249248+0000 mon.a (mon.0) 132 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.253296+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.255182+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.263336+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:36.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.267422+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:36.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:36 smithi050 bash[21467]: audit 2024-04-06T03:12:36.269278+0000 mon.a (mon.0) 137 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:37.942 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:12:38.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:37 smithi050 bash[21467]: audit 2024-04-06T03:12:36.203756+0000 mgr.y (mgr.14152) 44 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:38.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:37 smithi050 bash[21467]: cephadm 2024-04-06T03:12:36.209458+0000 mgr.y (mgr.14152) 45 : cephadm 1 Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b;count:3 2024-04-06T03:12:38.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:37 smithi050 bash[21467]: cephadm 2024-04-06T03:12:36.271385+0000 mgr.y (mgr.14152) 46 : cephadm 1 Deploying daemon mon.b on smithi153 2024-04-06T03:12:39.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:38 smithi050 bash[21467]: cluster 2024-04-06T03:12:37.464802+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:41.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:40 smithi050 bash[21467]: cluster 2024-04-06T03:12:39.465189+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:41.403 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-06T03:12:41.403 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:10:22.523083Z","created":"2024-04-06T03:10:22.523083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-06T03:12:41.406 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-06T03:12:41.786 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:41 smithi153 systemd[1]: Started Ceph mon.b for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:12:42.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:41 smithi050 bash[21467]: audit 2024-04-06T03:12:41.415182+0000 mon.a (mon.0) 138 : audit 0 from='client.? 172.21.15.153:0/532629589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:42.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:41 smithi050 bash[21467]: audit 2024-04-06T03:12:41.827065+0000 mon.a (mon.0) 139 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:42.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:41 smithi050 bash[21467]: audit 2024-04-06T03:12:41.833138+0000 mon.a (mon.0) 140 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:42.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:41 smithi050 bash[21467]: audit 2024-04-06T03:12:41.855325+0000 mon.a (mon.0) 141 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:42.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:41 smithi050 bash[21467]: audit 2024-04-06T03:12:41.856825+0000 mon.a (mon.0) 142 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:42.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:41 smithi050 bash[21467]: audit 2024-04-06T03:12:41.858799+0000 mon.a (mon.0) 143 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.812+0000 7f6c5d385c80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.812+0000 7f6c5d385c80 0 ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 7 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.812+0000 7f6c5d385c80 0 pidfile_write: ignore empty --pid-file 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 0 load: jerasure load: lrc 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 4 rocksdb: RocksDB version: 7.9.2 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 4 rocksdb: Git sha 0 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 4 rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T03:12:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 4 rocksdb: DB SUMMARY 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 4 rocksdb: DB Session ID: 8R4SDHAYEBGWIF84IZ6R 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.816+0000 7f6c5d385c80 4 rocksdb: CURRENT file: CURRENT 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.error_if_exists: 0 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.create_if_missing: 0 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.paranoid_checks: 1 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:12:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.env: 0x55fc4a9b2c60 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.info_log: 0x55fc4c4ddb00 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.statistics: (nil) 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.use_fsync: 0 2024-04-06T03:12:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_log_file_size: 0 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.allow_fallocate: 1 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.use_direct_reads: 0 2024-04-06T03:12:43.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.db_log_dir: 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.wal_dir: 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:12:43.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.write_buffer_manager: 0x55fc4c4b01e0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.unordered_write: 0 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:12:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.row_cache: None 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.wal_filter: None 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.two_write_queues: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.wal_compression: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.atomic_flush: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.log_readahead_size: 0 2024-04-06T03:12:43.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_background_jobs: 2 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_background_compactions: -1 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_subcompactions: 1 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:12:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_open_files: -1 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_background_flushes: -1 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Compression algorithms supported: 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kZSTD supported: 0 2024-04-06T03:12:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kXpressCompression supported: 0 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kZlibCompression supported: 1 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kSnappyCompression supported: 1 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kLZ4Compression supported: 1 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: kBZip2Compression supported: 0 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-06T03:12:43.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.merge_operator: 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_filter: None 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:12:43.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fc4c4ddc40) 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cache_index_and_filter_blocks: 1 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: pin_top_level_index_and_filter: 1 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: index_type: 0 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: data_block_index_type: 0 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: index_shortening: 1 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: checksum: 4 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: no_block_cache: 0 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_cache: 0x55fc4c4df090 2024-04-06T03:12:43.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_cache_name: BinnedLRUCache 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_cache_options: 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: capacity : 536870912 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: num_shard_bits : 4 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: strict_capacity_limit : 0 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: high_pri_pool_ratio: 0.000 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_cache_compressed: (nil) 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: persistent_cache: (nil) 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_size: 4096 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_size_deviation: 10 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_restart_interval: 16 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: index_block_restart_interval: 1 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: metadata_block_size: 4096 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: partition_filters: 0 2024-04-06T03:12:43.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: use_delta_encoding: 1 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: filter_policy: bloomfilter 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: whole_key_filtering: 1 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: verify_compression: 0 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: read_amp_bytes_per_bit: 0 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: format_version: 5 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: enable_index_compression: 1 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: block_align: 0 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: max_auto_readahead_size: 262144 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: prepopulate_block_cache: 0 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: initial_auto_readahead_size: 8192 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: num_file_reads_for_auto_readahead: 2 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:12:43.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression: NoCompression 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.num_levels: 7 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:12:43.052 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:12:43.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:12:43.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:12:43.055 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.table_properties_collectors: 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.inplace_update_support: 0 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.bloom_locality: 0 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.max_successive_merges: 0 2024-04-06T03:12:43.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.ttl: 2592000 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.enable_blob_files: false 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.min_blob_size: 0 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:12:43.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fbcf521c-d1aa-4f84-8b05-e8223972f69a 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373162824283, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373162824736, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712373162, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fbcf521c-d1aa-4f84-8b05-e8223972f69a", "db_session_id": "8R4SDHAYEBGWIF84IZ6R", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373162824831, "job": 1, "event": "recovery_finished"} 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-06T03:12:43.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fc4c5d4000 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 4 rocksdb: DB pointer 0x55fc4c5be000 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c4bae8700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c4bae8700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: ** DB Stats ** 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:12:43.059 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: ** Compaction Stats [default] ** 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: ** Compaction Stats [default] ** 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:12:43.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Block cache BinnedLRUCache@0x55fc4c4df090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-06T03:12:43.061 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.820+0000 7f6c5d385c80 0 using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.824+0000 7f6c5d385c80 0 starting mon.b rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.824+0000 7f6c5d385c80 1 mon.b@-1(???) e0 preinit fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 0 mon.b@-1(synchronizing).mds e1 new map 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: e1 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: legacy client fscid: -1 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: No filesystems configured 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4a2e5700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-06T03:12:43.062 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c532f7700 4 rocksdb: (Original Log Time 2024/04/06-03:12:42.852372) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c532f7700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c532f7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373162852430, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1118, "num_deletes": 24, "total_data_size": 2250358, "memory_usage": 2278928, "flush_reason": "Manual Compaction"} 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c532f7700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.848+0000 7f6c4eaee700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:05.458561+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:07.458901+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:09.097518+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:09.459271+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:09.912014+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi153 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:11.459672+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:13.460024+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:15.460454+0000 mgr.y (mgr.14152) 27 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:15.483738+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:15.484674+0000 mgr.y (mgr.14152) 28 : cephadm 1 Added host smithi153 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:15.529257+0000 mon.a (mon.0) 114 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:15.951814+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:17.427485+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:17.460883+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:18.626095+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:19.461286+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:21.461722+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:23.462028+0000 mgr.y (mgr.14152) 32 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.776145+0000 mgr.y (mgr.14152) 33 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.857372+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.859595+0000 mon.a (mon.0) 119 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.862068+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.864345+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.865090+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.866284+0000 mon.a (mon.0) 123 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:23.867300+0000 mon.a (mon.0) 124 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:23.868625+0000 mgr.y (mgr.14152) 34 : cephadm 1 Updating smithi153:/etc/ceph/ceph.conf 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:24.032961+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:24.532582+0000 mon.a (mon.0) 125 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.065 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:24.536598+0000 mon.a (mon.0) 126 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:24.541017+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:24.198084+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:24.363190+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:25.462396+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:27.462839+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:29.463270+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:31.673577+0000 mon.a (mon.0) 128 : audit 1 from='client.? 172.21.15.50:0/816141983' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:31.463556+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:31.871165+0000 mon.a (mon.0) 129 : audit 1 from='client.? 172.21.15.50:0/816141983' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:31.876983+0000 mon.a (mon.0) 130 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:33.463934+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:35.464435+0000 mgr.y (mgr.14152) 43 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.066 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.214907+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.249248+0000 mon.a (mon.0) 132 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.253296+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.255182+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.263336+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.267422+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.269278+0000 mon.a (mon.0) 137 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:36.203756+0000 mgr.y (mgr.14152) 44 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:36.209458+0000 mgr.y (mgr.14152) 45 : cephadm 1 Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b;count:3 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cephadm 2024-04-06T03:12:36.271385+0000 mgr.y (mgr.14152) 46 : cephadm 1 Deploying daemon mon.b on smithi153 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:37.464802+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: cluster 2024-04-06T03:12:39.465189+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:41.415182+0000 mon.a (mon.0) 138 : audit 0 from='client.? 172.21.15.153:0/532629589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:41.827065+0000 mon.a (mon.0) 139 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:41.833138+0000 mon.a (mon.0) 140 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:41.855325+0000 mon.a (mon.0) 141 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:43.068 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:41.856825+0000 mon.a (mon.0) 142 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:43.068 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: audit 2024-04-06T03:12:41.858799+0000 mon.a (mon.0) 143 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:43.068 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.852+0000 7f6c4eaee700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-06T03:12:43.068 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.852+0000 7f6c532f7700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373162857077, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1987747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1123, "table_properties": {"data_size": 1982140, "index_size": 2453, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2117, "raw_key_size": 24980, "raw_average_key_size": 30, "raw_value_size": 1965066, "raw_average_value_size": 2381, "num_data_blocks": 115, "num_entries": 825, "num_filter_entries": 825, "num_deletions": 24, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712373162, "oldest_key_time": 1712373162, "file_creation_time": 1712373162, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fbcf521c-d1aa-4f84-8b05-e8223972f69a", "db_session_id": "8R4SDHAYEBGWIF84IZ6R", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-06T03:12:43.068 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:42 smithi153 bash[23852]: debug 2024-04-06T03:12:42.852+0000 7f6c532f7700 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 4735 microseconds, and 4100 cpu microseconds. 2024-04-06T03:12:43.460 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-06T03:12:43.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-06T03:12:45.536 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:12:45 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:12:45.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:45 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:12:45.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:45 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:12:45.818 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:12:45 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:12:45.818 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:45 smithi050 systemd[1]: Started Ceph mon.c for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:12:46.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.712+0000 7fcb8d36dc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-06T03:12:46.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.712+0000 7fcb8d36dc80 0 ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev), process ceph-mon, pid 7 2024-04-06T03:12:46.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.712+0000 7fcb8d36dc80 0 pidfile_write: ignore empty --pid-file 2024-04-06T03:12:46.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 0 load: jerasure load: lrc 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: RocksDB version: 7.9.2 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Git sha 0 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Compile date 2024-04-05 08:39:05 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: DB SUMMARY 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: DB Session ID: 1GKX3FTWR9QMUNXARUJC 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: CURRENT file: CURRENT 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.error_if_exists: 0 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.create_if_missing: 0 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.paranoid_checks: 1 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-06T03:12:46.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.env: 0x55f2f27e0c60 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.info_log: 0x55f2f4295b20 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.statistics: (nil) 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.use_fsync: 0 2024-04-06T03:12:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_log_file_size: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.allow_fallocate: 1 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.use_direct_reads: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-06T03:12:46.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.db_log_dir: 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.wal_dir: 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.write_buffer_manager: 0x55f2f42681e0 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-06T03:12:46.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.unordered_write: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.row_cache: None 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.wal_filter: None 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-06T03:12:46.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.two_write_queues: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.wal_compression: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.atomic_flush: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.log_readahead_size: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-06T03:12:46.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_background_jobs: 2 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_background_compactions: -1 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_subcompactions: 1 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_open_files: -1 2024-04-06T03:12:46.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_background_flushes: -1 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Compression algorithms supported: 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kZSTD supported: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kXpressCompression supported: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kZlibCompression supported: 1 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kSnappyCompression supported: 1 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kLZ4Compression supported: 1 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: kBZip2Compression supported: 0 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-06T03:12:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-06T03:12:46.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-06T03:12:46.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-06T03:12:46.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.merge_operator: 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_filter: None 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f2f4295c40) 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cache_index_and_filter_blocks: 1 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: pin_top_level_index_and_filter: 1 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: index_type: 0 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: data_block_index_type: 0 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: index_shortening: 1 2024-04-06T03:12:46.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: data_block_hash_table_util_ratio: 0.750000 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: checksum: 4 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: no_block_cache: 0 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_cache: 0x55f2f4297090 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_cache_name: BinnedLRUCache 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_cache_options: 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: capacity : 536870912 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: num_shard_bits : 4 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: strict_capacity_limit : 0 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: high_pri_pool_ratio: 0.000 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_cache_compressed: (nil) 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: persistent_cache: (nil) 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_size: 4096 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_size_deviation: 10 2024-04-06T03:12:46.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_restart_interval: 16 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: index_block_restart_interval: 1 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: metadata_block_size: 4096 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: partition_filters: 0 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: use_delta_encoding: 1 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: filter_policy: bloomfilter 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: whole_key_filtering: 1 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: verify_compression: 0 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: read_amp_bytes_per_bit: 0 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: format_version: 5 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: enable_index_compression: 1 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: block_align: 0 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: max_auto_readahead_size: 262144 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: prepopulate_block_cache: 0 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: initial_auto_readahead_size: 8192 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: num_file_reads_for_auto_readahead: 2 2024-04-06T03:12:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression: NoCompression 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.num_levels: 7 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:12:46.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-06T03:12:46.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-06T03:12:46.996 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-06T03:12:46.997 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.table_properties_collectors: 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.inplace_update_support: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.bloom_locality: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.max_successive_merges: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.ttl: 2592000 2024-04-06T03:12:46.998 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.enable_blob_files: false 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.min_blob_size: 0 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.716+0000 7fcb8d36dc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-06T03:12:46.999 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 61e2a7e9-161a-4770-865e-b11f40492693 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373166724574, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373166725220, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712373166, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "61e2a7e9-161a-4770-865e-b11f40492693", "db_session_id": "1GKX3FTWR9QMUNXARUJC", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373166725353, "job": 1, "event": "recovery_finished"} 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f2f438c000 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 4 rocksdb: DB pointer 0x55f2f4376000 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb7bad0700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb7bad0700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: ** DB Stats ** 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:12:47.000 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: ** Compaction Stats [default] ** 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: ** Compaction Stats [default] ** 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-06T03:12:47.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: AddFile(Total Files): cumulative 0, interval 0 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: AddFile(Keys): cumulative 0, interval 0 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Cumulative compaction: 0.00 GB write, 0.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Interval compaction: 0.00 GB write, 0.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-06T03:12:47.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Block cache BinnedLRUCache@0x55f2f4297090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: ** File Read Latency Histogram By Level [default] ** 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.720+0000 7fcb8d36dc80 0 using public_addrv [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.724+0000 7fcb8d36dc80 0 starting mon.c rank -1 at public addrs [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] at bind addrs [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.724+0000 7fcb8d36dc80 1 mon.c@-1(???) e0 preinit fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 0 mon.c@-1(synchronizing).mds e1 new map 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: e1 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: legacy client fscid: -1 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: No filesystems configured 2024-04-06T03:12:47.003 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7a2cd700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb832df700 4 rocksdb: (Original Log Time 2024/04/06-03:12:46.748877) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb832df700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb832df700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1712373166748943, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1122, "num_deletes": 24, "total_data_size": 2251231, "memory_usage": 2279808, "flush_reason": "Manual Compaction"} 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb832df700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-06T03:12:47.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:05.458561+0000 mgr.y (mgr.14152) 20 : cluster 0 pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:07.458901+0000 mgr.y (mgr.14152) 21 : cluster 0 pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:09.097518+0000 mgr.y (mgr.14152) 22 : audit 0 from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:09.459271+0000 mgr.y (mgr.14152) 23 : cluster 0 pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:09.912014+0000 mgr.y (mgr.14152) 24 : cephadm 1 Deploying cephadm binary to smithi153 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:11.459672+0000 mgr.y (mgr.14152) 25 : cluster 0 pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:13.460024+0000 mgr.y (mgr.14152) 26 : cluster 0 pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:15.460454+0000 mgr.y (mgr.14152) 27 : cluster 0 pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:15.483738+0000 mon.a (mon.0) 113 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:15.484674+0000 mgr.y (mgr.14152) 28 : cephadm 1 Added host smithi153 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:15.529257+0000 mon.a (mon.0) 114 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:15.951814+0000 mon.a (mon.0) 115 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:17.427485+0000 mon.a (mon.0) 116 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.005 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:17.460883+0000 mgr.y (mgr.14152) 29 : cluster 0 pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:18.626095+0000 mon.a (mon.0) 117 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:19.461286+0000 mgr.y (mgr.14152) 30 : cluster 0 pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:21.461722+0000 mgr.y (mgr.14152) 31 : cluster 0 pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:23.462028+0000 mgr.y (mgr.14152) 32 : cluster 0 pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.776145+0000 mgr.y (mgr.14152) 33 : audit 0 from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.857372+0000 mon.a (mon.0) 118 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.859595+0000 mon.a (mon.0) 119 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.862068+0000 mon.a (mon.0) 120 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.864345+0000 mon.a (mon.0) 121 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.865090+0000 mon.a (mon.0) 122 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.866284+0000 mon.a (mon.0) 123 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:23.867300+0000 mon.a (mon.0) 124 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:23.868625+0000 mgr.y (mgr.14152) 34 : cephadm 1 Updating smithi153:/etc/ceph/ceph.conf 2024-04-06T03:12:47.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:24.032961+0000 mgr.y (mgr.14152) 35 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:24.532582+0000 mon.a (mon.0) 125 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:24.536598+0000 mon.a (mon.0) 126 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:24.541017+0000 mon.a (mon.0) 127 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:24.198084+0000 mgr.y (mgr.14152) 36 : cephadm 1 Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:24.363190+0000 mgr.y (mgr.14152) 37 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:25.462396+0000 mgr.y (mgr.14152) 38 : cluster 0 pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:27.462839+0000 mgr.y (mgr.14152) 39 : cluster 0 pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:29.463270+0000 mgr.y (mgr.14152) 40 : cluster 0 pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:31.673577+0000 mon.a (mon.0) 128 : audit 1 from='client.? 172.21.15.50:0/816141983' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:31.463556+0000 mgr.y (mgr.14152) 41 : cluster 0 pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:31.871165+0000 mon.a (mon.0) 129 : audit 1 from='client.? 172.21.15.50:0/816141983' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:31.876983+0000 mon.a (mon.0) 130 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:33.463934+0000 mgr.y (mgr.14152) 42 : cluster 0 pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:35.464435+0000 mgr.y (mgr.14152) 43 : cluster 0 pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.214907+0000 mon.a (mon.0) 131 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.249248+0000 mon.a (mon.0) 132 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.253296+0000 mon.a (mon.0) 133 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.255182+0000 mon.a (mon.0) 134 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.263336+0000 mon.a (mon.0) 135 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.267422+0000 mon.a (mon.0) 136 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.269278+0000 mon.a (mon.0) 137 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:36.203756+0000 mgr.y (mgr.14152) 44 : audit 0 from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:36.209458+0000 mgr.y (mgr.14152) 45 : cephadm 1 Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi153:172.21.15.153=b;count:3 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cephadm 2024-04-06T03:12:36.271385+0000 mgr.y (mgr.14152) 46 : cephadm 1 Deploying daemon mon.b on smithi153 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:37.464802+0000 mgr.y (mgr.14152) 47 : cluster 0 pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: cluster 2024-04-06T03:12:39.465189+0000 mgr.y (mgr.14152) 48 : cluster 0 pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:41.415182+0000 mon.a (mon.0) 138 : audit 0 from='client.? 172.21.15.153:0/532629589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:41.827065+0000 mon.a (mon.0) 139 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:41.833138+0000 mon.a (mon.0) 140 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.008 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:41.855325+0000 mon.a (mon.0) 141 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:47.009 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:41.856825+0000 mon.a (mon.0) 142 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:47.009 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: audit 2024-04-06T03:12:41.858799+0000 mon.a (mon.0) 143 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:47.009 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:46 smithi050 bash[29221]: debug 2024-04-06T03:12:46.744+0000 7fcb7ead6700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-06T03:12:47.161 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:12:48.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:42.867798+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:12:48.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:42.868184+0000 mon.a (mon.0) 146 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:48.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:42.889646+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:43.465993+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:48.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:43.862901+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:44.862778+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:44.864661+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:45.466301+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:45.862317+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:46.754141+0000 mon.a (mon.0) 151 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:46.862944+0000 mon.a (mon.0) 152 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:47.755020+0000 mon.a (mon.0) 153 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:47.863194+0000 mon.a (mon.0) 154 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.909996+0000 mon.a (mon.0) 155 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916104+0000 mon.a (mon.0) 156 : cluster 0 monmap epoch 2 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916142+0000 mon.a (mon.0) 157 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916162+0000 mon.a (mon.0) 158 : cluster 0 last_changed 2024-04-06T03:12:42.862507+0000 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916187+0000 mon.a (mon.0) 159 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:48.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916237+0000 mon.a (mon.0) 160 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916260+0000 mon.a (mon.0) 161 : cluster 0 election_strategy: 1 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916284+0000 mon.a (mon.0) 162 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.916311+0000 mon.a (mon.0) 163 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.937300+0000 mon.a (mon.0) 164 : cluster 0 fsmap 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.937360+0000 mon.a (mon.0) 165 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.937638+0000 mon.a (mon.0) 166 : cluster 0 mgrmap e13: y(active, since 74s) 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: cluster 2024-04-06T03:12:47.937876+0000 mon.a (mon.0) 167 : cluster 1 overall HEALTH_OK 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:47.944764+0000 mon.a (mon.0) 168 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:48.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:47 smithi050 bash[21467]: audit 2024-04-06T03:12:47.952712+0000 mon.a (mon.0) 169 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:48.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:42.867798+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:12:48.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:42.868184+0000 mon.a (mon.0) 146 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:42.889646+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:43.465993+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:43.862901+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:44.862778+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:44.864661+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:45.466301+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:45.862317+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:46.754141+0000 mon.a (mon.0) 151 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:46.862944+0000 mon.a (mon.0) 152 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:47.755020+0000 mon.a (mon.0) 153 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:47.863194+0000 mon.a (mon.0) 154 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:48.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.909996+0000 mon.a (mon.0) 155 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916104+0000 mon.a (mon.0) 156 : cluster 0 monmap epoch 2 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916142+0000 mon.a (mon.0) 157 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916162+0000 mon.a (mon.0) 158 : cluster 0 last_changed 2024-04-06T03:12:42.862507+0000 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916187+0000 mon.a (mon.0) 159 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916237+0000 mon.a (mon.0) 160 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916260+0000 mon.a (mon.0) 161 : cluster 0 election_strategy: 1 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916284+0000 mon.a (mon.0) 162 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.916311+0000 mon.a (mon.0) 163 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.937300+0000 mon.a (mon.0) 164 : cluster 0 fsmap 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.937360+0000 mon.a (mon.0) 165 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.937638+0000 mon.a (mon.0) 166 : cluster 0 mgrmap e13: y(active, since 74s) 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: cluster 2024-04-06T03:12:47.937876+0000 mon.a (mon.0) 167 : cluster 1 overall HEALTH_OK 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:47.944764+0000 mon.a (mon.0) 168 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:48.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:47 smithi153 bash[23852]: audit 2024-04-06T03:12:47.952712+0000 mon.a (mon.0) 169 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:49.229 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:12:48 smithi050 bash[21725]: debug 2024-04-06T03:12:48.856+0000 7ff06b26d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-06T03:12:54.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:48.761145+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:54.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:48.762200+0000 mon.a (mon.0) 175 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:12:54.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:48.762791+0000 mon.a (mon.0) 176 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:54.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:48.786115+0000 mon.a (mon.0) 177 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:54.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:48.786458+0000 mon.a (mon.0) 178 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:49.467048+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:49.755428+0000 mon.a (mon.0) 179 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:50.755415+0000 mon.a (mon.0) 180 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:51.467504+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:51.755777+0000 mon.a (mon.0) 181 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:52.755769+0000 mon.a (mon.0) 182 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:53.755913+0000 mon.a (mon.0) 183 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.785912+0000 mon.a (mon.0) 184 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792207+0000 mon.a (mon.0) 185 : cluster 0 monmap epoch 3 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792245+0000 mon.a (mon.0) 186 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792270+0000 mon.a (mon.0) 187 : cluster 0 last_changed 2024-04-06T03:12:48.755878+0000 2024-04-06T03:12:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792297+0000 mon.a (mon.0) 188 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792320+0000 mon.a (mon.0) 189 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792343+0000 mon.a (mon.0) 190 : cluster 0 election_strategy: 1 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792371+0000 mon.a (mon.0) 191 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792395+0000 mon.a (mon.0) 192 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.792430+0000 mon.a (mon.0) 193 : cluster 0 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.813335+0000 mon.a (mon.0) 194 : cluster 0 fsmap 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.813392+0000 mon.a (mon.0) 195 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.813692+0000 mon.a (mon.0) 196 : cluster 0 mgrmap e13: y(active, since 80s) 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.813998+0000 mon.a (mon.0) 197 : cluster 3 Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-06T03:12:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.821218+0000 mon.a (mon.0) 198 : cluster 3 Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-06T03:12:54.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.821259+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-06T03:12:54.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.821281+0000 mon.a (mon.0) 200 : cluster 3 mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-06T03:12:54.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:53.826396+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:53.832340+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:53.838223+0000 mon.a (mon.0) 203 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:53 smithi050 bash[21467]: audit 2024-04-06T03:12:53.844076+0000 mon.a (mon.0) 204 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:48.761145+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:48.762200+0000 mon.a (mon.0) 175 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:48.762791+0000 mon.a (mon.0) 176 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:48.786115+0000 mon.a (mon.0) 177 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:48.786458+0000 mon.a (mon.0) 178 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:49.467048+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:54.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:49.755428+0000 mon.a (mon.0) 179 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:50.755415+0000 mon.a (mon.0) 180 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:51.467504+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:51.755777+0000 mon.a (mon.0) 181 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:52.755769+0000 mon.a (mon.0) 182 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:53.755913+0000 mon.a (mon.0) 183 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.785912+0000 mon.a (mon.0) 184 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792207+0000 mon.a (mon.0) 185 : cluster 0 monmap epoch 3 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792245+0000 mon.a (mon.0) 186 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792270+0000 mon.a (mon.0) 187 : cluster 0 last_changed 2024-04-06T03:12:48.755878+0000 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792297+0000 mon.a (mon.0) 188 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792320+0000 mon.a (mon.0) 189 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792343+0000 mon.a (mon.0) 190 : cluster 0 election_strategy: 1 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792371+0000 mon.a (mon.0) 191 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792395+0000 mon.a (mon.0) 192 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:54.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.792430+0000 mon.a (mon.0) 193 : cluster 0 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.813335+0000 mon.a (mon.0) 194 : cluster 0 fsmap 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.813392+0000 mon.a (mon.0) 195 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.813692+0000 mon.a (mon.0) 196 : cluster 0 mgrmap e13: y(active, since 80s) 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.813998+0000 mon.a (mon.0) 197 : cluster 3 Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.821218+0000 mon.a (mon.0) 198 : cluster 3 Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.821259+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.821281+0000 mon.a (mon.0) 200 : cluster 3 mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:53.826396+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:53.832340+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:53.838223+0000 mon.a (mon.0) 203 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:53 smithi153 bash[23852]: audit 2024-04-06T03:12:53.844076+0000 mon.a (mon.0) 204 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:54.292 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-06T03:12:54.292 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","modified":"2024-04-06T03:12:48.755878Z","created":"2024-04-06T03:10:22.523083Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-06T03:12:54.294 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-04-06T03:12:54.935 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-06T03:12:54.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-06T03:12:55.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:54 smithi050 bash[21467]: cluster 2024-04-06T03:12:53.467909+0000 mgr.y (mgr.14152) 56 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:55.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:54 smithi050 bash[21467]: audit 2024-04-06T03:12:54.303917+0000 mon.a (mon.0) 205 : audit 0 from='client.? 172.21.15.153:0/546369609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:55.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:54 smithi050 bash[21467]: audit 2024-04-06T03:12:54.522555+0000 mon.a (mon.0) 206 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:55.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:54 smithi050 bash[21467]: audit 2024-04-06T03:12:54.524640+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:55.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:54 smithi050 bash[21467]: audit 2024-04-06T03:12:54.755614+0000 mon.a (mon.0) 208 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:55.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:54 smithi153 bash[23852]: cluster 2024-04-06T03:12:53.467909+0000 mgr.y (mgr.14152) 56 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:55.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:54 smithi153 bash[23852]: audit 2024-04-06T03:12:54.303917+0000 mon.a (mon.0) 205 : audit 0 from='client.? 172.21.15.153:0/546369609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:55.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:54 smithi153 bash[23852]: audit 2024-04-06T03:12:54.522555+0000 mon.a (mon.0) 206 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:55.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:54 smithi153 bash[23852]: audit 2024-04-06T03:12:54.524640+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:55.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:54 smithi153 bash[23852]: audit 2024-04-06T03:12:54.755614+0000 mon.a (mon.0) 208 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:50.757910+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.789079+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.789439+0000 mon.b (mon.1) 3 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.790094+0000 mon.a (mon.0) 222 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.829770+0000 mon.a (mon.0) 223 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835675+0000 mon.a (mon.0) 224 : cluster 0 monmap epoch 3 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835715+0000 mon.a (mon.0) 225 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835738+0000 mon.a (mon.0) 226 : cluster 0 last_changed 2024-04-06T03:12:48.755878+0000 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835760+0000 mon.a (mon.0) 227 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835785+0000 mon.a (mon.0) 228 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835806+0000 mon.a (mon.0) 229 : cluster 0 election_strategy: 1 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835829+0000 mon.a (mon.0) 230 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835852+0000 mon.a (mon.0) 231 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.835877+0000 mon.a (mon.0) 232 : cluster 0 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.836892+0000 mon.a (mon.0) 233 : cluster 0 fsmap 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.836955+0000 mon.a (mon.0) 234 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.837239+0000 mon.a (mon.0) 235 : cluster 0 mgrmap e13: y(active, since 82s) 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.837521+0000 mon.a (mon.0) 236 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.837561+0000 mon.a (mon.0) 237 : cluster 1 Cluster is now healthy 2024-04-06T03:12:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.857144+0000 mon.a (mon.0) 238 : cluster 1 overall HEALTH_OK 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:42.867798+0000 mon.a (mon.0) 145 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:42.868184+0000 mon.a (mon.0) 146 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:42.889646+0000 mon.a (mon.0) 147 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:43.465993+0000 mgr.y (mgr.14152) 51 : cluster 0 pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:43.862901+0000 mon.a (mon.0) 148 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:44.862778+0000 mon.a (mon.0) 149 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:44.864661+0000 mon.b (mon.1) 1 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:45.466301+0000 mgr.y (mgr.14152) 52 : cluster 0 pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:45.862317+0000 mon.a (mon.0) 150 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:46.754141+0000 mon.a (mon.0) 151 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:46.862944+0000 mon.a (mon.0) 152 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:47.755020+0000 mon.a (mon.0) 153 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:47.863194+0000 mon.a (mon.0) 154 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.909996+0000 mon.a (mon.0) 155 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916104+0000 mon.a (mon.0) 156 : cluster 0 monmap epoch 2 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916142+0000 mon.a (mon.0) 157 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916162+0000 mon.a (mon.0) 158 : cluster 0 last_changed 2024-04-06T03:12:42.862507+0000 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916187+0000 mon.a (mon.0) 159 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916237+0000 mon.a (mon.0) 160 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916260+0000 mon.a (mon.0) 161 : cluster 0 election_strategy: 1 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916284+0000 mon.a (mon.0) 162 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.916311+0000 mon.a (mon.0) 163 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.937300+0000 mon.a (mon.0) 164 : cluster 0 fsmap 2024-04-06T03:12:56.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.937360+0000 mon.a (mon.0) 165 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.937638+0000 mon.a (mon.0) 166 : cluster 0 mgrmap e13: y(active, since 74s) 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:47.937876+0000 mon.a (mon.0) 167 : cluster 1 overall HEALTH_OK 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:47.944764+0000 mon.a (mon.0) 168 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:47.952712+0000 mon.a (mon.0) 169 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:48.761145+0000 mon.b (mon.1) 2 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:48.762200+0000 mon.a (mon.0) 175 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:48.762791+0000 mon.a (mon.0) 176 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:48.786115+0000 mon.a (mon.0) 177 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:48.786458+0000 mon.a (mon.0) 178 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:49.467048+0000 mgr.y (mgr.14152) 54 : cluster 0 pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:49.755428+0000 mon.a (mon.0) 179 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:50.755415+0000 mon.a (mon.0) 180 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:51.467504+0000 mgr.y (mgr.14152) 55 : cluster 0 pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:56.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:51.755777+0000 mon.a (mon.0) 181 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:52.755769+0000 mon.a (mon.0) 182 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:53.755913+0000 mon.a (mon.0) 183 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.785912+0000 mon.a (mon.0) 184 : cluster 1 mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792207+0000 mon.a (mon.0) 185 : cluster 0 monmap epoch 3 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792245+0000 mon.a (mon.0) 186 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792270+0000 mon.a (mon.0) 187 : cluster 0 last_changed 2024-04-06T03:12:48.755878+0000 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792297+0000 mon.a (mon.0) 188 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792320+0000 mon.a (mon.0) 189 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792343+0000 mon.a (mon.0) 190 : cluster 0 election_strategy: 1 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792371+0000 mon.a (mon.0) 191 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792395+0000 mon.a (mon.0) 192 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.792430+0000 mon.a (mon.0) 193 : cluster 0 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.813335+0000 mon.a (mon.0) 194 : cluster 0 fsmap 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.813392+0000 mon.a (mon.0) 195 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:56.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.813692+0000 mon.a (mon.0) 196 : cluster 0 mgrmap e13: y(active, since 80s) 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.813998+0000 mon.a (mon.0) 197 : cluster 3 Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.821218+0000 mon.a (mon.0) 198 : cluster 3 Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.821259+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.821281+0000 mon.a (mon.0) 200 : cluster 3 mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:53.826396+0000 mon.a (mon.0) 201 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:53.832340+0000 mon.a (mon.0) 202 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:53.838223+0000 mon.a (mon.0) 203 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:53.844076+0000 mon.a (mon.0) 204 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:53.467909+0000 mgr.y (mgr.14152) 56 : cluster 0 pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:54.303917+0000 mon.a (mon.0) 205 : audit 0 from='client.? 172.21.15.153:0/546369609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:54.522555+0000 mon.a (mon.0) 206 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:54.524640+0000 mon.a (mon.0) 207 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: audit 2024-04-06T03:12:54.755614+0000 mon.a (mon.0) 208 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:50.757910+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-04-06T03:12:56.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.789079+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.789439+0000 mon.b (mon.1) 3 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.790094+0000 mon.a (mon.0) 222 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.829770+0000 mon.a (mon.0) 223 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835675+0000 mon.a (mon.0) 224 : cluster 0 monmap epoch 3 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835715+0000 mon.a (mon.0) 225 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835738+0000 mon.a (mon.0) 226 : cluster 0 last_changed 2024-04-06T03:12:48.755878+0000 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835760+0000 mon.a (mon.0) 227 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835785+0000 mon.a (mon.0) 228 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835806+0000 mon.a (mon.0) 229 : cluster 0 election_strategy: 1 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835829+0000 mon.a (mon.0) 230 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835852+0000 mon.a (mon.0) 231 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.835877+0000 mon.a (mon.0) 232 : cluster 0 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.836892+0000 mon.a (mon.0) 233 : cluster 0 fsmap 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.836955+0000 mon.a (mon.0) 234 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:56.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.837239+0000 mon.a (mon.0) 235 : cluster 0 mgrmap e13: y(active, since 82s) 2024-04-06T03:12:56.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.837521+0000 mon.a (mon.0) 236 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-06T03:12:56.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.837561+0000 mon.a (mon.0) 237 : cluster 1 Cluster is now healthy 2024-04-06T03:12:56.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:55 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.857144+0000 mon.a (mon.0) 238 : cluster 1 overall HEALTH_OK 2024-04-06T03:12:56.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:50.757910+0000 mon.c (mon.2) 1 : cluster 1 mon.c calling monitor election 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.789079+0000 mon.c (mon.2) 2 : cluster 1 mon.c calling monitor election 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.789439+0000 mon.b (mon.1) 3 : cluster 1 mon.b calling monitor election 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.790094+0000 mon.a (mon.0) 222 : cluster 1 mon.a calling monitor election 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.829770+0000 mon.a (mon.0) 223 : cluster 1 mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835675+0000 mon.a (mon.0) 224 : cluster 0 monmap epoch 3 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835715+0000 mon.a (mon.0) 225 : cluster 0 fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835738+0000 mon.a (mon.0) 226 : cluster 0 last_changed 2024-04-06T03:12:48.755878+0000 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835760+0000 mon.a (mon.0) 227 : cluster 0 created 2024-04-06T03:10:22.523083+0000 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835785+0000 mon.a (mon.0) 228 : cluster 0 min_mon_release 19 (squid) 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835806+0000 mon.a (mon.0) 229 : cluster 0 election_strategy: 1 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835829+0000 mon.a (mon.0) 230 : cluster 0 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835852+0000 mon.a (mon.0) 231 : cluster 0 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.835877+0000 mon.a (mon.0) 232 : cluster 0 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-06T03:12:56.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.836892+0000 mon.a (mon.0) 233 : cluster 0 fsmap 2024-04-06T03:12:56.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.836955+0000 mon.a (mon.0) 234 : cluster 0 osdmap e4: 0 total, 0 up, 0 in 2024-04-06T03:12:56.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.837239+0000 mon.a (mon.0) 235 : cluster 0 mgrmap e13: y(active, since 82s) 2024-04-06T03:12:56.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.837521+0000 mon.a (mon.0) 236 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-06T03:12:56.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.837561+0000 mon.a (mon.0) 237 : cluster 1 Cluster is now healthy 2024-04-06T03:12:56.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:55 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.857144+0000 mon.a (mon.0) 238 : cluster 1 overall HEALTH_OK 2024-04-06T03:12:57.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:56 smithi153 bash[23852]: cluster 2024-04-06T03:12:55.468258+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:57.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:56 smithi153 bash[23852]: audit 2024-04-06T03:12:56.755997+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:57.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:56 smithi153 bash[23852]: audit 2024-04-06T03:12:56.967040+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:57.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:56 smithi153 bash[23852]: audit 2024-04-06T03:12:56.973142+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:57.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[21467]: cluster 2024-04-06T03:12:55.468258+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:57.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[21467]: audit 2024-04-06T03:12:56.755997+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:57.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[21467]: audit 2024-04-06T03:12:56.967040+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:57.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[21467]: audit 2024-04-06T03:12:56.973142+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:57.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[29221]: cluster 2024-04-06T03:12:55.468258+0000 mgr.y (mgr.14152) 63 : cluster 0 pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:57.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[29221]: audit 2024-04-06T03:12:56.755997+0000 mon.a (mon.0) 239 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:12:57.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[29221]: audit 2024-04-06T03:12:56.967040+0000 mon.a (mon.0) 240 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:57.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:56 smithi050 bash[29221]: audit 2024-04-06T03:12:56.973142+0000 mon.a (mon.0) 241 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:58.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:58 smithi153 bash[23852]: cephadm 2024-04-06T03:12:57.012700+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-04-06T03:12:58.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:58 smithi153 bash[23852]: audit 2024-04-06T03:12:57.013250+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:58.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:58 smithi153 bash[23852]: audit 2024-04-06T03:12:57.014636+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:58.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:58 smithi153 bash[23852]: audit 2024-04-06T03:12:57.015939+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:58.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:58 smithi153 bash[23852]: cephadm 2024-04-06T03:12:57.017604+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi050 2024-04-06T03:12:58.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[21467]: cephadm 2024-04-06T03:12:57.012700+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-04-06T03:12:58.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[21467]: audit 2024-04-06T03:12:57.013250+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:58.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[21467]: audit 2024-04-06T03:12:57.014636+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[21467]: audit 2024-04-06T03:12:57.015939+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[21467]: cephadm 2024-04-06T03:12:57.017604+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi050 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[29221]: cephadm 2024-04-06T03:12:57.012700+0000 mgr.y (mgr.14152) 64 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[29221]: audit 2024-04-06T03:12:57.013250+0000 mon.a (mon.0) 242 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[29221]: audit 2024-04-06T03:12:57.014636+0000 mon.a (mon.0) 243 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[29221]: audit 2024-04-06T03:12:57.015939+0000 mon.a (mon.0) 244 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:58.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:58 smithi050 bash[29221]: cephadm 2024-04-06T03:12:57.017604+0000 mgr.y (mgr.14152) 65 : cephadm 1 Reconfiguring daemon mon.c on smithi050 2024-04-06T03:12:59.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:59 smithi153 bash[23852]: cluster 2024-04-06T03:12:57.468651+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:59.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:59 smithi153 bash[23852]: audit 2024-04-06T03:12:58.871422+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:59.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:59 smithi153 bash[23852]: audit 2024-04-06T03:12:58.877350+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:59.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:59 smithi153 bash[23852]: audit 2024-04-06T03:12:58.887079+0000 mon.a (mon.0) 247 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:59.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:59 smithi153 bash[23852]: audit 2024-04-06T03:12:58.888760+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:59.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:12:59 smithi153 bash[23852]: audit 2024-04-06T03:12:58.890456+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:59.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[21467]: cluster 2024-04-06T03:12:57.468651+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:59.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[21467]: audit 2024-04-06T03:12:58.871422+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[21467]: audit 2024-04-06T03:12:58.877350+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[21467]: audit 2024-04-06T03:12:58.887079+0000 mon.a (mon.0) 247 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[21467]: audit 2024-04-06T03:12:58.888760+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[21467]: audit 2024-04-06T03:12:58.890456+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[29221]: cluster 2024-04-06T03:12:57.468651+0000 mgr.y (mgr.14152) 66 : cluster 0 pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[29221]: audit 2024-04-06T03:12:58.871422+0000 mon.a (mon.0) 245 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[29221]: audit 2024-04-06T03:12:58.877350+0000 mon.a (mon.0) 246 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[29221]: audit 2024-04-06T03:12:58.887079+0000 mon.a (mon.0) 247 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[29221]: audit 2024-04-06T03:12:58.888760+0000 mon.a (mon.0) 248 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-06T03:12:59.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:12:59 smithi050 bash[29221]: audit 2024-04-06T03:12:58.890456+0000 mon.a (mon.0) 249 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:12:59.665 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:13:00.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:00 smithi153 bash[23852]: cephadm 2024-04-06T03:12:58.886547+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-04-06T03:13:00.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:00 smithi153 bash[23852]: cephadm 2024-04-06T03:12:58.892616+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi153 2024-04-06T03:13:00.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:00 smithi050 bash[29221]: cephadm 2024-04-06T03:12:58.886547+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-04-06T03:13:00.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:00 smithi050 bash[29221]: cephadm 2024-04-06T03:12:58.892616+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi153 2024-04-06T03:13:00.415 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:00 smithi050 bash[21467]: cephadm 2024-04-06T03:12:58.886547+0000 mgr.y (mgr.14152) 67 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2024-04-06T03:13:00.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:00 smithi050 bash[21467]: cephadm 2024-04-06T03:12:58.892616+0000 mgr.y (mgr.14152) 68 : cephadm 1 Reconfiguring daemon mon.b on smithi153 2024-04-06T03:13:01.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: cluster 2024-04-06T03:12:59.469018+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: audit 2024-04-06T03:13:00.609856+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: audit 2024-04-06T03:13:00.614272+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: audit 2024-04-06T03:13:00.662142+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: audit 2024-04-06T03:13:00.665614+0000 mon.a (mon.0) 253 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: audit 2024-04-06T03:13:00.667319+0000 mon.a (mon.0) 254 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:01 smithi153 bash[23852]: audit 2024-04-06T03:13:00.675946+0000 mon.a (mon.0) 255 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: cluster 2024-04-06T03:12:59.469018+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: audit 2024-04-06T03:13:00.609856+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: audit 2024-04-06T03:13:00.614272+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: audit 2024-04-06T03:13:00.662142+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: audit 2024-04-06T03:13:00.665614+0000 mon.a (mon.0) 253 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: audit 2024-04-06T03:13:00.667319+0000 mon.a (mon.0) 254 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[21467]: audit 2024-04-06T03:13:00.675946+0000 mon.a (mon.0) 255 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: cluster 2024-04-06T03:12:59.469018+0000 mgr.y (mgr.14152) 69 : cluster 0 pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: audit 2024-04-06T03:13:00.609856+0000 mon.a (mon.0) 250 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: audit 2024-04-06T03:13:00.614272+0000 mon.a (mon.0) 251 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: audit 2024-04-06T03:13:00.662142+0000 mon.a (mon.0) 252 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:01.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: audit 2024-04-06T03:13:00.665614+0000 mon.a (mon.0) 253 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:01.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: audit 2024-04-06T03:13:00.667319+0000 mon.a (mon.0) 254 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:01.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:01 smithi050 bash[29221]: audit 2024-04-06T03:13:00.675946+0000 mon.a (mon.0) 255 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:02.305 INFO:teuthology.orchestra.run.smithi050.stdout:# minimal ceph.conf for 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:13:02.305 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-04-06T03:13:02.305 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:13:02.305 INFO:teuthology.orchestra.run.smithi050.stdout: mon_host = [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2024-04-06T03:13:02.916 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-06T03:13:02.917 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:13:02.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T03:13:02.931 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:13:02.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:13:02.984 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:13:02.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-06T03:13:02.999 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:13:02.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:13:03.056 INFO:tasks.cephadm:Adding mgr.y on smithi050 2024-04-06T03:13:03.056 INFO:tasks.cephadm:Adding mgr.x on smithi153 2024-04-06T03:13:03.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply mgr '2;smithi050=y;smithi153=x' 2024-04-06T03:13:03.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:03 smithi050 bash[21467]: cluster 2024-04-06T03:13:01.469464+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:03.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:03 smithi050 bash[21467]: audit 2024-04-06T03:13:02.317309+0000 mon.c (mon.2) 3 : audit 0 from='client.? 172.21.15.50:0/178697992' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:03.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:03 smithi050 bash[29221]: cluster 2024-04-06T03:13:01.469464+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:03.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:03 smithi050 bash[29221]: audit 2024-04-06T03:13:02.317309+0000 mon.c (mon.2) 3 : audit 0 from='client.? 172.21.15.50:0/178697992' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:03.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:03 smithi153 bash[23852]: cluster 2024-04-06T03:13:01.469464+0000 mgr.y (mgr.14152) 70 : cluster 0 pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:03.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:03 smithi153 bash[23852]: audit 2024-04-06T03:13:02.317309+0000 mon.c (mon.2) 3 : audit 0 from='client.? 172.21.15.50:0/178697992' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:05.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:05 smithi153 bash[23852]: cluster 2024-04-06T03:13:03.469924+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:05.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:05 smithi050 bash[21467]: cluster 2024-04-06T03:13:03.469924+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:05.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:05 smithi050 bash[29221]: cluster 2024-04-06T03:13:03.469924+0000 mgr.y (mgr.14152) 71 : cluster 0 pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:06.793 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:13:07.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:07 smithi153 bash[23852]: cluster 2024-04-06T03:13:05.470348+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:07.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:07 smithi050 bash[21467]: cluster 2024-04-06T03:13:05.470348+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:07.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:07 smithi050 bash[29221]: cluster 2024-04-06T03:13:05.470348+0000 mgr.y (mgr.14152) 72 : cluster 0 pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:09.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:09 smithi153 bash[23852]: cluster 2024-04-06T03:13:07.470782+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:09.295 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-04-06T03:13:09.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:09 smithi050 bash[21467]: cluster 2024-04-06T03:13:07.470782+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:09.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:09 smithi050 bash[29221]: cluster 2024-04-06T03:13:07.470782+0000 mgr.y (mgr.14152) 73 : cluster 0 pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:09.809 DEBUG:teuthology.orchestra.run.smithi153:mgr.x> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.x.service 2024-04-06T03:13:09.811 INFO:tasks.cephadm:Deploying OSDs... 2024-04-06T03:13:09.811 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:13:09.811 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T03:13:09.817 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T03:13:09.818 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-06 03:03:15.852082996 +0000 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-06 03:03:15.560085604 +0000 2024-04-06T03:13:09.865 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-06 03:03:15.560085604 +0000 2024-04-06T03:13:09.866 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-06T03:13:09.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T03:13:09.923 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-06T03:13:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-06T03:13:09.924 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000310475 s, 1.6 MB/s 2024-04-06T03:13:09.925 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T03:13:09.974 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-06 03:03:16.376078315 +0000 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-06 03:03:16.100080781 +0000 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-06 03:03:16.100080781 +0000 2024-04-06T03:13:10.025 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-06T03:13:10.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T03:13:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-06T03:13:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-06T03:13:10.084 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000318806 s, 1.6 MB/s 2024-04-06T03:13:10.085 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T03:13:10.134 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-04-06T03:13:10.180 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T03:13:10.180 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.180 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-04-06T03:13:10.181 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.181 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-06 03:03:16.900073635 +0000 2024-04-06T03:13:10.181 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-06 03:03:16.608076242 +0000 2024-04-06T03:13:10.181 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-06 03:03:16.608076242 +0000 2024-04-06T03:13:10.181 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-06T03:13:10.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T03:13:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-06T03:13:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-06T03:13:10.235 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000294698 s, 1.7 MB/s 2024-04-06T03:13:10.237 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T03:13:10.290 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-06 03:03:17.132071562 +0000 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-06 03:03:17.128071598 +0000 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-06 03:03:17.128071598 +0000 2024-04-06T03:13:10.337 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-06T03:13:10.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T03:13:10.392 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-06T03:13:10.392 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-06T03:13:10.392 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000313302 s, 1.6 MB/s 2024-04-06T03:13:10.393 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T03:13:10.442 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:13:10.442 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-06T03:13:10.448 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-06T03:13:10.448 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-06 03:03:10.194308721 +0000 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-06 03:03:09.910311254 +0000 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-06 03:03:09.910311254 +0000 2024-04-06T03:13:10.496 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-04-06T03:13:10.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.295780+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: cephadm 2024-04-06T03:13:09.299452+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi050=y;smithi153=x;count:2 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.306273+0000 mon.a (mon.0) 256 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.330764+0000 mon.a (mon.0) 257 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.334819+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.336711+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.346065+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.349387+0000 mon.a (mon.0) 261 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:10.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.351836+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T03:13:10.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.356689+0000 mon.a (mon.0) 263 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:10.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: audit 2024-04-06T03:13:09.358673+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: cephadm 2024-04-06T03:13:09.360772+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi153 2024-04-06T03:13:10.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:10 smithi153 bash[23852]: cluster 2024-04-06T03:13:09.471181+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-06T03:13:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-06T03:13:10.550 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000333644 s, 1.5 MB/s 2024-04-06T03:13:10.552 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-06T03:13:10.601 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-06T03:13:10.651 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-06T03:13:10.651 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.651 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-06T03:13:10.651 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.651 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-06 03:03:10.682304368 +0000 2024-04-06T03:13:10.651 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-06 03:03:10.434306580 +0000 2024-04-06T03:13:10.652 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-06 03:03:10.434306580 +0000 2024-04-06T03:13:10.652 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-04-06T03:13:10.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-06T03:13:10.707 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-06T03:13:10.707 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-06T03:13:10.707 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000243489 s, 2.1 MB/s 2024-04-06T03:13:10.708 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.295780+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: cephadm 2024-04-06T03:13:09.299452+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi050=y;smithi153=x;count:2 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.306273+0000 mon.a (mon.0) 256 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.330764+0000 mon.a (mon.0) 257 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.334819+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.336711+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.346065+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.349387+0000 mon.a (mon.0) 261 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.351836+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.356689+0000 mon.a (mon.0) 263 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: audit 2024-04-06T03:13:09.358673+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: cephadm 2024-04-06T03:13:09.360772+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi153 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[21467]: cluster 2024-04-06T03:13:09.471181+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.295780+0000 mgr.y (mgr.14152) 74 : audit 0 from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: cephadm 2024-04-06T03:13:09.299452+0000 mgr.y (mgr.14152) 75 : cephadm 1 Saving service mgr spec with placement smithi050=y;smithi153=x;count:2 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.306273+0000 mon.a (mon.0) 256 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.330764+0000 mon.a (mon.0) 257 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.334819+0000 mon.a (mon.0) 258 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.336711+0000 mon.a (mon.0) 259 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.346065+0000 mon.a (mon.0) 260 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.349387+0000 mon.a (mon.0) 261 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.351836+0000 mon.a (mon.0) 262 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.356689+0000 mon.a (mon.0) 263 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:10.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: audit 2024-04-06T03:13:09.358673+0000 mon.a (mon.0) 264 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: cephadm 2024-04-06T03:13:09.360772+0000 mgr.y (mgr.14152) 76 : cephadm 1 Deploying daemon mgr.x on smithi153 2024-04-06T03:13:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:10 smithi050 bash[29221]: cluster 2024-04-06T03:13:09.471181+0000 mgr.y (mgr.14152) 77 : cluster 0 pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:10.760 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-06T03:13:10.806 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-06 03:03:11.178299943 +0000 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-06 03:03:10.930302155 +0000 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-06 03:03:10.930302155 +0000 2024-04-06T03:13:10.807 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-04-06T03:13:10.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-06T03:13:10.859 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-06T03:13:10.859 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-06T03:13:10.859 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000244411 s, 2.1 MB/s 2024-04-06T03:13:10.860 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-06T03:13:10.908 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-06T03:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-06T03:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-06T03:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-04-06T03:13:10.959 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-06T03:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-06 03:03:11.394298016 +0000 2024-04-06T03:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-06 03:03:11.390298051 +0000 2024-04-06T03:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-06 03:03:11.390298051 +0000 2024-04-06T03:13:10.960 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-04-06T03:13:10.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-06T03:13:11.014 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-06T03:13:11.014 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-06T03:13:11.014 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00032975 s, 1.6 MB/s 2024-04-06T03:13:11.015 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-06T03:13:11.065 INFO:tasks.cephadm:Deploying osd.0 on smithi050 with /dev/vg_nvme/lv_4... 2024-04-06T03:13:11.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T03:13:11.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:11.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:11.873 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:11.873 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:11.873 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:11.873 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:12.286 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:11 smithi153 systemd[1]: Started Ceph mgr.x for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:13:12.921 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[24761]: debug 2024-04-06T03:13:12.764+0000 7ff500188700 1 -- 172.21.15.153:0/1745071505 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5585be52e5a0 con 0x5585bf2aec00 2024-04-06T03:13:12.921 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[24761]: debug 2024-04-06T03:13:12.904+0000 7ff50a38f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T03:13:13.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[21467]: cluster 2024-04-06T03:13:11.471611+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[21467]: audit 2024-04-06T03:13:11.934067+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[21467]: audit 2024-04-06T03:13:11.941939+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[21467]: audit 2024-04-06T03:13:11.987554+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[21467]: audit 2024-04-06T03:13:11.994550+0000 mon.a (mon.0) 268 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[21467]: audit 2024-04-06T03:13:12.065614+0000 mon.a (mon.0) 269 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[29221]: cluster 2024-04-06T03:13:11.471611+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[29221]: audit 2024-04-06T03:13:11.934067+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[29221]: audit 2024-04-06T03:13:11.941939+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[29221]: audit 2024-04-06T03:13:11.987554+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[29221]: audit 2024-04-06T03:13:11.994550+0000 mon.a (mon.0) 268 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:12 smithi050 bash[29221]: audit 2024-04-06T03:13:12.065614+0000 mon.a (mon.0) 269 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:13.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[23852]: cluster 2024-04-06T03:13:11.471611+0000 mgr.y (mgr.14152) 78 : cluster 0 pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:13.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[23852]: audit 2024-04-06T03:13:11.934067+0000 mon.a (mon.0) 265 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[23852]: audit 2024-04-06T03:13:11.941939+0000 mon.a (mon.0) 266 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[23852]: audit 2024-04-06T03:13:11.987554+0000 mon.a (mon.0) 267 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[23852]: audit 2024-04-06T03:13:11.994550+0000 mon.a (mon.0) 268 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:13.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:12 smithi153 bash[23852]: audit 2024-04-06T03:13:12.065614+0000 mon.a (mon.0) 269 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:14.786 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:14 smithi153 bash[24761]: debug 2024-04-06T03:13:14.452+0000 7ff50a38f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T03:13:14.787 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:14 smithi153 bash[24761]: debug 2024-04-06T03:13:14.544+0000 7ff50a38f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T03:13:14.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:14 smithi153 bash[23852]: cluster 2024-04-06T03:13:13.472036+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:14.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:14 smithi153 bash[23852]: audit 2024-04-06T03:13:13.539831+0000 mon.a (mon.0) 270 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:14.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:14 smithi050 bash[21467]: cluster 2024-04-06T03:13:13.472036+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:14.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:14 smithi050 bash[21467]: audit 2024-04-06T03:13:13.539831+0000 mon.a (mon.0) 270 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:14.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:14 smithi050 bash[29221]: cluster 2024-04-06T03:13:13.472036+0000 mgr.y (mgr.14152) 79 : cluster 0 pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:14.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:14 smithi050 bash[29221]: audit 2024-04-06T03:13:13.539831+0000 mon.a (mon.0) 270 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:15.036 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:14 smithi153 bash[24761]: debug 2024-04-06T03:13:14.792+0000 7ff50a38f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T03:13:15.792 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:13:16.036 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:15 smithi153 bash[24761]: debug 2024-04-06T03:13:15.580+0000 7ff50a38f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T03:13:16.037 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:15 smithi153 bash[24761]: debug 2024-04-06T03:13:15.688+0000 7ff50a38f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T03:13:16.037 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:15 smithi153 bash[24761]: debug 2024-04-06T03:13:15.788+0000 7ff50a38f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T03:13:16.530 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:16 smithi153 bash[24761]: debug 2024-04-06T03:13:16.140+0000 7ff50a38f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T03:13:16.530 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:16 smithi153 bash[24761]: debug 2024-04-06T03:13:16.252+0000 7ff50a38f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T03:13:16.530 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:16 smithi153 bash[24761]: debug 2024-04-06T03:13:16.352+0000 7ff50a38f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T03:13:16.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:16 smithi153 bash[23852]: cluster 2024-04-06T03:13:15.472593+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:16.787 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:16 smithi153 bash[24761]: debug 2024-04-06T03:13:16.704+0000 7ff50a38f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T03:13:16.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:16 smithi050 bash[21467]: cluster 2024-04-06T03:13:15.472593+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:16.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:16 smithi050 bash[29221]: cluster 2024-04-06T03:13:15.472593+0000 mgr.y (mgr.14152) 80 : cluster 0 pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:17.199 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:16 smithi153 bash[24761]: debug 2024-04-06T03:13:16.944+0000 7ff50a38f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T03:13:17.537 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:17 smithi153 bash[24761]: debug 2024-04-06T03:13:17.204+0000 7ff50a38f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T03:13:17.537 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:17 smithi153 bash[24761]: debug 2024-04-06T03:13:17.320+0000 7ff50a38f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T03:13:18.004 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:17 smithi153 bash[24761]: debug 2024-04-06T03:13:17.588+0000 7ff50a38f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[24761]: debug 2024-04-06T03:13:18.012+0000 7ff50a38f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[24761]: debug 2024-04-06T03:13:18.248+0000 7ff50a38f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.255598+0000 mon.a (mon.0) 271 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.262417+0000 mon.a (mon.0) 272 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.263976+0000 mon.a (mon.0) 273 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.265458+0000 mon.a (mon.0) 274 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.272589+0000 mon.a (mon.0) 275 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: cephadm 2024-04-06T03:13:17.339975+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-04-06T03:13:18.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.340487+0000 mon.a (mon.0) 276 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:18.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.341485+0000 mon.a (mon.0) 277 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:18.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: audit 2024-04-06T03:13:17.342284+0000 mon.a (mon.0) 278 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:18.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: cephadm 2024-04-06T03:13:17.343277+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi050 2024-04-06T03:13:18.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[23852]: cluster 2024-04-06T03:13:17.473023+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.255598+0000 mon.a (mon.0) 271 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.262417+0000 mon.a (mon.0) 272 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.263976+0000 mon.a (mon.0) 273 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.265458+0000 mon.a (mon.0) 274 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.272589+0000 mon.a (mon.0) 275 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: cephadm 2024-04-06T03:13:17.339975+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.340487+0000 mon.a (mon.0) 276 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.341485+0000 mon.a (mon.0) 277 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: audit 2024-04-06T03:13:17.342284+0000 mon.a (mon.0) 278 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: cephadm 2024-04-06T03:13:17.343277+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi050 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[29221]: cluster 2024-04-06T03:13:17.473023+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.255598+0000 mon.a (mon.0) 271 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.262417+0000 mon.a (mon.0) 272 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.263976+0000 mon.a (mon.0) 273 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.265458+0000 mon.a (mon.0) 274 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.272589+0000 mon.a (mon.0) 275 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: cephadm 2024-04-06T03:13:17.339975+0000 mgr.y (mgr.14152) 81 : cephadm 1 Reconfiguring mgr.y (unknown last config time)... 2024-04-06T03:13:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.340487+0000 mon.a (mon.0) 276 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-06T03:13:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.341485+0000 mon.a (mon.0) 277 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-06T03:13:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: audit 2024-04-06T03:13:17.342284+0000 mon.a (mon.0) 278 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: cephadm 2024-04-06T03:13:17.343277+0000 mgr.y (mgr.14152) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi050 2024-04-06T03:13:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:18 smithi050 bash[21467]: cluster 2024-04-06T03:13:17.473023+0000 mgr.y (mgr.14152) 83 : cluster 0 pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:18.786 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[24761]: debug 2024-04-06T03:13:18.364+0000 7ff50a38f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T03:13:18.787 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:18 smithi153 bash[24761]: debug 2024-04-06T03:13:18.608+0000 7ff50a38f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:13:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T03:13:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T03:13:19.030 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-06T03:13:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-06T03:13:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109283 s, 960 MB/s 2024-04-06T03:13:19.030 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-06T03:13:19.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi050:vg_nvme/lv_4 2024-04-06T03:13:20.025 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:19 smithi153 bash[24761]: debug 2024-04-06T03:13:19.612+0000 7ff50a38f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T03:13:20.025 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:19 smithi153 bash[24761]: debug 2024-04-06T03:13:19.728+0000 7ff50a38f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T03:13:20.286 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[24761]: debug 2024-04-06T03:13:20.032+0000 7ff50a38f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T03:13:20.287 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[24761]: debug 2024-04-06T03:13:20.152+0000 7ff50a38f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T03:13:20.707 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[24761]: debug 2024-04-06T03:13:20.592+0000 7ff50a38f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T03:13:20.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: audit 2024-04-06T03:13:19.425076+0000 mon.a (mon.0) 279 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: audit 2024-04-06T03:13:19.430592+0000 mon.a (mon.0) 280 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: audit 2024-04-06T03:13:19.463106+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:20.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: cluster 2024-04-06T03:13:19.473395+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:20.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: audit 2024-04-06T03:13:20.038316+0000 mon.a (mon.0) 282 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:20.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: audit 2024-04-06T03:13:20.039635+0000 mon.a (mon.0) 283 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:20.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[23852]: audit 2024-04-06T03:13:20.046199+0000 mon.a (mon.0) 284 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: audit 2024-04-06T03:13:19.425076+0000 mon.a (mon.0) 279 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: audit 2024-04-06T03:13:19.430592+0000 mon.a (mon.0) 280 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: audit 2024-04-06T03:13:19.463106+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: cluster 2024-04-06T03:13:19.473395+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: audit 2024-04-06T03:13:20.038316+0000 mon.a (mon.0) 282 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: audit 2024-04-06T03:13:20.039635+0000 mon.a (mon.0) 283 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[29221]: audit 2024-04-06T03:13:20.046199+0000 mon.a (mon.0) 284 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: audit 2024-04-06T03:13:19.425076+0000 mon.a (mon.0) 279 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: audit 2024-04-06T03:13:19.430592+0000 mon.a (mon.0) 280 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: audit 2024-04-06T03:13:19.463106+0000 mon.a (mon.0) 281 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: cluster 2024-04-06T03:13:19.473395+0000 mgr.y (mgr.14152) 84 : cluster 0 pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: audit 2024-04-06T03:13:20.038316+0000 mon.a (mon.0) 282 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: audit 2024-04-06T03:13:20.039635+0000 mon.a (mon.0) 283 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:13:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:20 smithi050 bash[21467]: audit 2024-04-06T03:13:20.046199+0000 mon.a (mon.0) 284 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:21.036 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:13:20 smithi153 bash[24761]: debug 2024-04-06T03:13:20.712+0000 7ff50a38f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T03:13:21.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[29221]: audit 2024-04-06T03:13:20.743411+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T03:13:21.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[29221]: cluster 2024-04-06T03:13:20.743502+0000 mon.a (mon.0) 285 : cluster 0 Standby manager daemon x started 2024-04-06T03:13:21.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[29221]: audit 2024-04-06T03:13:20.744005+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:13:21.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[29221]: audit 2024-04-06T03:13:20.745256+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T03:13:21.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[29221]: audit 2024-04-06T03:13:20.745685+0000 mon.b (mon.1) 7 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:13:21.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[21467]: audit 2024-04-06T03:13:20.743411+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T03:13:21.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[21467]: cluster 2024-04-06T03:13:20.743502+0000 mon.a (mon.0) 285 : cluster 0 Standby manager daemon x started 2024-04-06T03:13:21.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[21467]: audit 2024-04-06T03:13:20.744005+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:13:21.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[21467]: audit 2024-04-06T03:13:20.745256+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T03:13:21.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:21 smithi050 bash[21467]: audit 2024-04-06T03:13:20.745685+0000 mon.b (mon.1) 7 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:13:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:21 smithi153 bash[23852]: audit 2024-04-06T03:13:20.743411+0000 mon.b (mon.1) 4 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T03:13:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:21 smithi153 bash[23852]: cluster 2024-04-06T03:13:20.743502+0000 mon.a (mon.0) 285 : cluster 0 Standby manager daemon x started 2024-04-06T03:13:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:21 smithi153 bash[23852]: audit 2024-04-06T03:13:20.744005+0000 mon.b (mon.1) 5 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:13:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:21 smithi153 bash[23852]: audit 2024-04-06T03:13:20.745256+0000 mon.b (mon.1) 6 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T03:13:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:21 smithi153 bash[23852]: audit 2024-04-06T03:13:20.745685+0000 mon.b (mon.1) 7 : audit 0 from='mgr.? 172.21.15.153:0/596325919' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:13:22.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:22 smithi050 bash[21467]: cluster 2024-04-06T03:13:21.454854+0000 mon.a (mon.0) 286 : cluster 0 mgrmap e14: y(active, since 108s), standbys: x 2024-04-06T03:13:22.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:22 smithi050 bash[21467]: audit 2024-04-06T03:13:21.455173+0000 mon.a (mon.0) 287 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T03:13:22.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:22 smithi050 bash[21467]: cluster 2024-04-06T03:13:21.473805+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:22.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:22 smithi050 bash[29221]: cluster 2024-04-06T03:13:21.454854+0000 mon.a (mon.0) 286 : cluster 0 mgrmap e14: y(active, since 108s), standbys: x 2024-04-06T03:13:22.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:22 smithi050 bash[29221]: audit 2024-04-06T03:13:21.455173+0000 mon.a (mon.0) 287 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T03:13:22.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:22 smithi050 bash[29221]: cluster 2024-04-06T03:13:21.473805+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:22.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:22 smithi153 bash[23852]: cluster 2024-04-06T03:13:21.454854+0000 mon.a (mon.0) 286 : cluster 0 mgrmap e14: y(active, since 108s), standbys: x 2024-04-06T03:13:22.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:22 smithi153 bash[23852]: audit 2024-04-06T03:13:21.455173+0000 mon.a (mon.0) 287 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T03:13:22.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:22 smithi153 bash[23852]: cluster 2024-04-06T03:13:21.473805+0000 mgr.y (mgr.14152) 85 : cluster 0 pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:24.495 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:13:24.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:24 smithi153 bash[23852]: cluster 2024-04-06T03:13:23.474228+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:24.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:24 smithi050 bash[21467]: cluster 2024-04-06T03:13:23.474228+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:24.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:24 smithi050 bash[29221]: cluster 2024-04-06T03:13:23.474228+0000 mgr.y (mgr.14152) 86 : cluster 0 pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:26.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:26 smithi050 bash[21467]: cluster 2024-04-06T03:13:25.474762+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:26 smithi050 bash[29221]: cluster 2024-04-06T03:13:25.474762+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:26.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:26 smithi153 bash[23852]: cluster 2024-04-06T03:13:25.474762+0000 mgr.y (mgr.14152) 87 : cluster 0 pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:27.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:27 smithi153 bash[23852]: audit 2024-04-06T03:13:27.179936+0000 mon.a (mon.0) 288 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:27.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:27 smithi153 bash[23852]: audit 2024-04-06T03:13:27.185874+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:27.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:27 smithi153 bash[23852]: audit 2024-04-06T03:13:27.186861+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:27.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:27 smithi050 bash[21467]: audit 2024-04-06T03:13:27.179936+0000 mon.a (mon.0) 288 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:27 smithi050 bash[21467]: audit 2024-04-06T03:13:27.185874+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:27 smithi050 bash[21467]: audit 2024-04-06T03:13:27.186861+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:27.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:27 smithi050 bash[29221]: audit 2024-04-06T03:13:27.179936+0000 mon.a (mon.0) 288 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:13:27.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:27 smithi050 bash[29221]: audit 2024-04-06T03:13:27.185874+0000 mon.a (mon.0) 289 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:13:27.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:27 smithi050 bash[29221]: audit 2024-04-06T03:13:27.186861+0000 mon.a (mon.0) 290 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:28.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:28 smithi050 bash[21467]: audit 2024-04-06T03:13:27.172925+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:28.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:28 smithi050 bash[21467]: cluster 2024-04-06T03:13:27.475188+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:28.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:28 smithi050 bash[29221]: audit 2024-04-06T03:13:27.172925+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:28.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:28 smithi050 bash[29221]: cluster 2024-04-06T03:13:27.475188+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:29.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:28 smithi153 bash[23852]: audit 2024-04-06T03:13:27.172925+0000 mgr.y (mgr.14152) 88 : audit 0 from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:13:29.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:28 smithi153 bash[23852]: cluster 2024-04-06T03:13:27.475188+0000 mgr.y (mgr.14152) 89 : cluster 0 pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:30.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:30 smithi050 bash[29221]: cluster 2024-04-06T03:13:29.475653+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:30.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:30 smithi050 bash[21467]: cluster 2024-04-06T03:13:29.475653+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:31.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:30 smithi153 bash[23852]: cluster 2024-04-06T03:13:29.475653+0000 mgr.y (mgr.14152) 90 : cluster 0 pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:32.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:32 smithi050 bash[21467]: cluster 2024-04-06T03:13:31.476062+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:32.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:32 smithi050 bash[29221]: cluster 2024-04-06T03:13:31.476062+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:33.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:32 smithi153 bash[23852]: cluster 2024-04-06T03:13:31.476062+0000 mgr.y (mgr.14152) 91 : cluster 0 pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:34.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:34 smithi050 bash[29221]: cluster 2024-04-06T03:13:33.476542+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:34.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:34 smithi050 bash[21467]: cluster 2024-04-06T03:13:33.476542+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:35.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:34 smithi153 bash[23852]: cluster 2024-04-06T03:13:33.476542+0000 mgr.y (mgr.14152) 92 : cluster 0 pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:35.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[29221]: audit 2024-04-06T03:13:35.246145+0000 mon.c (mon.2) 4 : audit 1 from='client.? 172.21.15.50:0/1995826161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]: dispatch 2024-04-06T03:13:35.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[29221]: audit 2024-04-06T03:13:35.247016+0000 mon.a (mon.0) 291 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]: dispatch 2024-04-06T03:13:35.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[29221]: audit 2024-04-06T03:13:35.249859+0000 mon.a (mon.0) 292 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]': finished 2024-04-06T03:13:35.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[29221]: cluster 2024-04-06T03:13:35.252224+0000 mon.a (mon.0) 293 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-04-06T03:13:35.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[29221]: audit 2024-04-06T03:13:35.252804+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:35.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[21467]: audit 2024-04-06T03:13:35.246145+0000 mon.c (mon.2) 4 : audit 1 from='client.? 172.21.15.50:0/1995826161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]: dispatch 2024-04-06T03:13:35.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[21467]: audit 2024-04-06T03:13:35.247016+0000 mon.a (mon.0) 291 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]: dispatch 2024-04-06T03:13:35.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[21467]: audit 2024-04-06T03:13:35.249859+0000 mon.a (mon.0) 292 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]': finished 2024-04-06T03:13:35.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[21467]: cluster 2024-04-06T03:13:35.252224+0000 mon.a (mon.0) 293 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-04-06T03:13:35.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:35 smithi050 bash[21467]: audit 2024-04-06T03:13:35.252804+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:36.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:35 smithi153 bash[23852]: audit 2024-04-06T03:13:35.246145+0000 mon.c (mon.2) 4 : audit 1 from='client.? 172.21.15.50:0/1995826161' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]: dispatch 2024-04-06T03:13:36.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:35 smithi153 bash[23852]: audit 2024-04-06T03:13:35.247016+0000 mon.a (mon.0) 291 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]: dispatch 2024-04-06T03:13:36.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:35 smithi153 bash[23852]: audit 2024-04-06T03:13:35.249859+0000 mon.a (mon.0) 292 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e607f527-4592-4acd-a47f-76575615ae1b"}]': finished 2024-04-06T03:13:36.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:35 smithi153 bash[23852]: cluster 2024-04-06T03:13:35.252224+0000 mon.a (mon.0) 293 : cluster 0 osdmap e5: 1 total, 0 up, 1 in 2024-04-06T03:13:36.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:35 smithi153 bash[23852]: audit 2024-04-06T03:13:35.252804+0000 mon.a (mon.0) 294 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:13:36.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:36 smithi050 bash[21467]: cluster 2024-04-06T03:13:35.476872+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:36.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:36 smithi050 bash[21467]: audit 2024-04-06T03:13:35.976873+0000 mon.c (mon.2) 5 : audit 0 from='client.? 172.21.15.50:0/2302163241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:36.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:36 smithi050 bash[29221]: cluster 2024-04-06T03:13:35.476872+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:36.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:36 smithi050 bash[29221]: audit 2024-04-06T03:13:35.976873+0000 mon.c (mon.2) 5 : audit 0 from='client.? 172.21.15.50:0/2302163241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:37.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:36 smithi153 bash[23852]: cluster 2024-04-06T03:13:35.476872+0000 mgr.y (mgr.14152) 93 : cluster 0 pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:37.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:36 smithi153 bash[23852]: audit 2024-04-06T03:13:35.976873+0000 mon.c (mon.2) 5 : audit 0 from='client.? 172.21.15.50:0/2302163241' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:13:38.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:38 smithi050 bash[21467]: cluster 2024-04-06T03:13:37.477265+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:38.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:38 smithi050 bash[29221]: cluster 2024-04-06T03:13:37.477265+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:39.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:38 smithi153 bash[23852]: cluster 2024-04-06T03:13:37.477265+0000 mgr.y (mgr.14152) 94 : cluster 0 pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:40.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:40 smithi050 bash[21467]: cluster 2024-04-06T03:13:39.477560+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:40.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:40 smithi050 bash[29221]: cluster 2024-04-06T03:13:39.477560+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:41.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:40 smithi153 bash[23852]: cluster 2024-04-06T03:13:39.477560+0000 mgr.y (mgr.14152) 95 : cluster 0 pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:42.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:42 smithi050 bash[21467]: cluster 2024-04-06T03:13:41.477949+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:42.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:42 smithi050 bash[29221]: cluster 2024-04-06T03:13:41.477949+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:43.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:42 smithi153 bash[23852]: cluster 2024-04-06T03:13:41.477949+0000 mgr.y (mgr.14152) 96 : cluster 0 pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:44.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:44 smithi050 bash[21467]: cluster 2024-04-06T03:13:43.478364+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:44.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:44 smithi050 bash[29221]: cluster 2024-04-06T03:13:43.478364+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:45.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:44 smithi153 bash[23852]: cluster 2024-04-06T03:13:43.478364+0000 mgr.y (mgr.14152) 97 : cluster 0 pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:46.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:46 smithi050 bash[21467]: cluster 2024-04-06T03:13:45.478801+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:46.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:46 smithi050 bash[29221]: cluster 2024-04-06T03:13:45.478801+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:47.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:46 smithi153 bash[23852]: cluster 2024-04-06T03:13:45.478801+0000 mgr.y (mgr.14152) 98 : cluster 0 pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:47.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:47 smithi050 bash[29221]: audit 2024-04-06T03:13:47.466980+0000 mon.a (mon.0) 295 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:13:47.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:47 smithi050 bash[29221]: audit 2024-04-06T03:13:47.469176+0000 mon.a (mon.0) 296 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:47.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:47 smithi050 bash[21467]: audit 2024-04-06T03:13:47.466980+0000 mon.a (mon.0) 295 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:13:47.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:47 smithi050 bash[21467]: audit 2024-04-06T03:13:47.469176+0000 mon.a (mon.0) 296 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:48.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:47 smithi153 bash[23852]: audit 2024-04-06T03:13:47.466980+0000 mon.a (mon.0) 295 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-06T03:13:48.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:47 smithi153 bash[23852]: audit 2024-04-06T03:13:47.469176+0000 mon.a (mon.0) 296 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:13:48.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:48 smithi050 bash[21467]: cephadm 2024-04-06T03:13:47.470888+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi050 2024-04-06T03:13:48.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:48 smithi050 bash[21467]: cluster 2024-04-06T03:13:47.479205+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:48.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:48 smithi050 bash[29221]: cephadm 2024-04-06T03:13:47.470888+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi050 2024-04-06T03:13:48.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:48 smithi050 bash[29221]: cluster 2024-04-06T03:13:47.479205+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:49.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:48 smithi153 bash[23852]: cephadm 2024-04-06T03:13:47.470888+0000 mgr.y (mgr.14152) 99 : cephadm 1 Deploying daemon osd.0 on smithi050 2024-04-06T03:13:49.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:48 smithi153 bash[23852]: cluster 2024-04-06T03:13:47.479205+0000 mgr.y (mgr.14152) 100 : cluster 0 pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:50.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:50 smithi050 bash[21467]: cluster 2024-04-06T03:13:49.479586+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:50.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:50 smithi050 bash[29221]: cluster 2024-04-06T03:13:49.479586+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:51.037 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:50 smithi153 bash[23852]: cluster 2024-04-06T03:13:49.479586+0000 mgr.y (mgr.14152) 101 : cluster 0 pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:51.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:51.709 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:13:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:51.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:51.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:51.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:13:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:51.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:13:53.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[21467]: cluster 2024-04-06T03:13:51.479868+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[21467]: audit 2024-04-06T03:13:51.866306+0000 mon.a (mon.0) 297 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[21467]: audit 2024-04-06T03:13:51.872600+0000 mon.a (mon.0) 298 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[21467]: audit 2024-04-06T03:13:51.875062+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[29221]: cluster 2024-04-06T03:13:51.479868+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[29221]: audit 2024-04-06T03:13:51.866306+0000 mon.a (mon.0) 297 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[29221]: audit 2024-04-06T03:13:51.872600+0000 mon.a (mon.0) 298 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:53.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:52 smithi050 bash[29221]: audit 2024-04-06T03:13:51.875062+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:53.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:52 smithi153 bash[23852]: cluster 2024-04-06T03:13:51.479868+0000 mgr.y (mgr.14152) 102 : cluster 0 pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:53.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:52 smithi153 bash[23852]: audit 2024-04-06T03:13:51.866306+0000 mon.a (mon.0) 297 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:53.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:52 smithi153 bash[23852]: audit 2024-04-06T03:13:51.872600+0000 mon.a (mon.0) 298 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:53.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:52 smithi153 bash[23852]: audit 2024-04-06T03:13:51.875062+0000 mon.a (mon.0) 299 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:13:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:54 smithi050 bash[21467]: cluster 2024-04-06T03:13:53.480303+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:55.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:54 smithi050 bash[29221]: cluster 2024-04-06T03:13:53.480303+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:55.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:54 smithi153 bash[23852]: cluster 2024-04-06T03:13:53.480303+0000 mgr.y (mgr.14152) 103 : cluster 0 pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:57.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:56 smithi050 bash[21467]: cluster 2024-04-06T03:13:55.480764+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:57.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:56 smithi050 bash[29221]: cluster 2024-04-06T03:13:55.480764+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:57.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:56 smithi153 bash[23852]: cluster 2024-04-06T03:13:55.480764+0000 mgr.y (mgr.14152) 104 : cluster 0 pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:59.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[21467]: cluster 2024-04-06T03:13:57.481177+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:59.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[21467]: audit 2024-04-06T03:13:58.047201+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:13:59.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[21467]: audit 2024-04-06T03:13:58.047575+0000 mon.a (mon.0) 300 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[21467]: audit 2024-04-06T03:13:58.666406+0000 mon.a (mon.0) 301 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[21467]: audit 2024-04-06T03:13:58.670809+0000 mon.a (mon.0) 302 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[29221]: cluster 2024-04-06T03:13:57.481177+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[29221]: audit 2024-04-06T03:13:58.047201+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[29221]: audit 2024-04-06T03:13:58.047575+0000 mon.a (mon.0) 300 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[29221]: audit 2024-04-06T03:13:58.666406+0000 mon.a (mon.0) 301 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:59.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:58 smithi050 bash[29221]: audit 2024-04-06T03:13:58.670809+0000 mon.a (mon.0) 302 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:59.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:58 smithi153 bash[23852]: cluster 2024-04-06T03:13:57.481177+0000 mgr.y (mgr.14152) 105 : cluster 0 pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:13:59.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:58 smithi153 bash[23852]: audit 2024-04-06T03:13:58.047201+0000 mon.c (mon.2) 6 : audit 1 from='osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:13:59.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:58 smithi153 bash[23852]: audit 2024-04-06T03:13:58.047575+0000 mon.a (mon.0) 300 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-06T03:13:59.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:58 smithi153 bash[23852]: audit 2024-04-06T03:13:58.666406+0000 mon.a (mon.0) 301 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:13:59.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:58 smithi153 bash[23852]: audit 2024-04-06T03:13:58.670809+0000 mon.a (mon.0) 302 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:00.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:58.886260+0000 mon.a (mon.0) 303 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:14:00.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: cluster 2024-04-06T03:13:58.891234+0000 mon.a (mon.0) 304 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-04-06T03:14:00.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:58.891617+0000 mon.a (mon.0) 305 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:00.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:58.891770+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:58.892355+0000 mon.a (mon.0) 306 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:59.242079+0000 mon.a (mon.0) 307 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:59.242971+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[21467]: audit 2024-04-06T03:13:59.248085+0000 mon.a (mon.0) 309 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:58.886260+0000 mon.a (mon.0) 303 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: cluster 2024-04-06T03:13:58.891234+0000 mon.a (mon.0) 304 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:58.891617+0000 mon.a (mon.0) 305 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:58.891770+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:58.892355+0000 mon.a (mon.0) 306 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:59.242079+0000 mon.a (mon.0) 307 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:00.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:59.242971+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:00.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:13:59 smithi050 bash[29221]: audit 2024-04-06T03:13:59.248085+0000 mon.a (mon.0) 309 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:58.886260+0000 mon.a (mon.0) 303 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: cluster 2024-04-06T03:13:58.891234+0000 mon.a (mon.0) 304 : cluster 0 osdmap e6: 1 total, 0 up, 1 in 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:58.891617+0000 mon.a (mon.0) 305 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:58.891770+0000 mon.c (mon.2) 7 : audit 1 from='osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:58.892355+0000 mon.a (mon.0) 306 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:59.242079+0000 mon.a (mon.0) 307 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:59.242971+0000 mon.a (mon.0) 308 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:00.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:13:59 smithi153 bash[23852]: audit 2024-04-06T03:13:59.248085+0000 mon.a (mon.0) 309 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:01.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[21467]: cluster 2024-04-06T03:13:59.481693+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[21467]: audit 2024-04-06T03:13:59.888429+0000 mon.a (mon.0) 310 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[21467]: cluster 2024-04-06T03:13:59.894200+0000 mon.a (mon.0) 311 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[21467]: audit 2024-04-06T03:13:59.895677+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[21467]: audit 2024-04-06T03:13:59.904076+0000 mon.a (mon.0) 313 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[21467]: audit 2024-04-06T03:14:00.227613+0000 mon.a (mon.0) 314 : audit 1 from='osd.0 ' entity='osd.0' 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[29221]: cluster 2024-04-06T03:13:59.481693+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[29221]: audit 2024-04-06T03:13:59.888429+0000 mon.a (mon.0) 310 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[29221]: cluster 2024-04-06T03:13:59.894200+0000 mon.a (mon.0) 311 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[29221]: audit 2024-04-06T03:13:59.895677+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[29221]: audit 2024-04-06T03:13:59.904076+0000 mon.a (mon.0) 313 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:00 smithi050 bash[29221]: audit 2024-04-06T03:14:00.227613+0000 mon.a (mon.0) 314 : audit 1 from='osd.0 ' entity='osd.0' 2024-04-06T03:14:01.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:00 smithi153 bash[23852]: cluster 2024-04-06T03:13:59.481693+0000 mgr.y (mgr.14152) 106 : cluster 0 pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:14:01.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:00 smithi153 bash[23852]: audit 2024-04-06T03:13:59.888429+0000 mon.a (mon.0) 310 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:14:01.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:00 smithi153 bash[23852]: cluster 2024-04-06T03:13:59.894200+0000 mon.a (mon.0) 311 : cluster 0 osdmap e7: 1 total, 0 up, 1 in 2024-04-06T03:14:01.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:00 smithi153 bash[23852]: audit 2024-04-06T03:13:59.895677+0000 mon.a (mon.0) 312 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:01.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:00 smithi153 bash[23852]: audit 2024-04-06T03:13:59.904076+0000 mon.a (mon.0) 313 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:01.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:00 smithi153 bash[23852]: audit 2024-04-06T03:14:00.227613+0000 mon.a (mon.0) 314 : audit 1 from='osd.0 ' entity='osd.0' 2024-04-06T03:14:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 0 on host 'smithi050' 2024-04-06T03:14:02.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[21467]: cluster 2024-04-06T03:13:58.998670+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[21467]: cluster 2024-04-06T03:13:58.998786+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[21467]: audit 2024-04-06T03:14:00.905403+0000 mon.a (mon.0) 315 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[21467]: cluster 2024-04-06T03:14:01.251993+0000 mon.a (mon.0) 316 : cluster 1 osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646] boot 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[21467]: cluster 2024-04-06T03:14:01.252039+0000 mon.a (mon.0) 317 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[21467]: audit 2024-04-06T03:14:01.252214+0000 mon.a (mon.0) 318 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[29221]: cluster 2024-04-06T03:13:58.998670+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[29221]: cluster 2024-04-06T03:13:58.998786+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[29221]: audit 2024-04-06T03:14:00.905403+0000 mon.a (mon.0) 315 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[29221]: cluster 2024-04-06T03:14:01.251993+0000 mon.a (mon.0) 316 : cluster 1 osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646] boot 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[29221]: cluster 2024-04-06T03:14:01.252039+0000 mon.a (mon.0) 317 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-04-06T03:14:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:01 smithi050 bash[29221]: audit 2024-04-06T03:14:01.252214+0000 mon.a (mon.0) 318 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:02.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:01 smithi153 bash[23852]: cluster 2024-04-06T03:13:58.998670+0000 osd.0 (osd.0) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:14:02.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:01 smithi153 bash[23852]: cluster 2024-04-06T03:13:58.998786+0000 osd.0 (osd.0) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:14:02.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:01 smithi153 bash[23852]: audit 2024-04-06T03:14:00.905403+0000 mon.a (mon.0) 315 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:02.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:01 smithi153 bash[23852]: cluster 2024-04-06T03:14:01.251993+0000 mon.a (mon.0) 316 : cluster 1 osd.0 [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646] boot 2024-04-06T03:14:02.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:01 smithi153 bash[23852]: cluster 2024-04-06T03:14:01.252039+0000 mon.a (mon.0) 317 : cluster 0 osdmap e8: 1 total, 1 up, 1 in 2024-04-06T03:14:02.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:01 smithi153 bash[23852]: audit 2024-04-06T03:14:01.252214+0000 mon.a (mon.0) 318 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:14:02.690 DEBUG:teuthology.orchestra.run.smithi050:osd.0> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-06T03:14:02.692 INFO:tasks.cephadm:Deploying osd.1 on smithi050 with /dev/vg_nvme/lv_3... 2024-04-06T03:14:02.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[21467]: cluster 2024-04-06T03:14:01.482155+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[21467]: audit 2024-04-06T03:14:01.984155+0000 mon.a (mon.0) 319 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[21467]: audit 2024-04-06T03:14:01.989428+0000 mon.a (mon.0) 320 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[21467]: audit 2024-04-06T03:14:01.998729+0000 mon.a (mon.0) 321 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[29221]: cluster 2024-04-06T03:14:01.482155+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[29221]: audit 2024-04-06T03:14:01.984155+0000 mon.a (mon.0) 319 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[29221]: audit 2024-04-06T03:14:01.989428+0000 mon.a (mon.0) 320 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:02.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:02 smithi050 bash[29221]: audit 2024-04-06T03:14:01.998729+0000 mon.a (mon.0) 321 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:03.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:02 smithi153 bash[23852]: cluster 2024-04-06T03:14:01.482155+0000 mgr.y (mgr.14152) 107 : cluster 0 pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-06T03:14:03.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:02 smithi153 bash[23852]: audit 2024-04-06T03:14:01.984155+0000 mon.a (mon.0) 319 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:03.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:02 smithi153 bash[23852]: audit 2024-04-06T03:14:01.989428+0000 mon.a (mon.0) 320 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:03.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:02 smithi153 bash[23852]: audit 2024-04-06T03:14:01.998729+0000 mon.a (mon.0) 321 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:04.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:03 smithi153 bash[23852]: cluster 2024-04-06T03:14:02.999529+0000 mon.a (mon.0) 322 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-04-06T03:14:04.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:03 smithi050 bash[21467]: cluster 2024-04-06T03:14:02.999529+0000 mon.a (mon.0) 322 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-04-06T03:14:04.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:03 smithi050 bash[29221]: cluster 2024-04-06T03:14:02.999529+0000 mon.a (mon.0) 322 : cluster 0 osdmap e9: 1 total, 1 up, 1 in 2024-04-06T03:14:05.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:05 smithi153 bash[23852]: cluster 2024-04-06T03:14:03.482707+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:05.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:05 smithi050 bash[21467]: cluster 2024-04-06T03:14:03.482707+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:05.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:05 smithi050 bash[29221]: cluster 2024-04-06T03:14:03.482707+0000 mgr.y (mgr.14152) 108 : cluster 0 pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:06.970 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:14:07.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:07 smithi153 bash[23852]: cluster 2024-04-06T03:14:05.483130+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:07.480 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:07 smithi050 bash[21467]: cluster 2024-04-06T03:14:05.483130+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:07.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:07 smithi050 bash[29221]: cluster 2024-04-06T03:14:05.483130+0000 mgr.y (mgr.14152) 109 : cluster 0 pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:09.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:09 smithi153 bash[23852]: cluster 2024-04-06T03:14:07.483736+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:09 smithi050 bash[21467]: cluster 2024-04-06T03:14:07.483736+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:09.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:09 smithi050 bash[29221]: cluster 2024-04-06T03:14:07.483736+0000 mgr.y (mgr.14152) 110 : cluster 0 pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T03:14:11.097 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T03:14:11.097 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-06T03:14:11.097 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-06T03:14:11.097 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0106291 s, 987 MB/s 2024-04-06T03:14:11.097 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-06T03:14:11.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:11 smithi153 bash[23852]: cluster 2024-04-06T03:14:09.484181+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:11.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:11 smithi050 bash[21467]: cluster 2024-04-06T03:14:09.484181+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:11.321 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:11 smithi050 bash[29221]: cluster 2024-04-06T03:14:09.484181+0000 mgr.y (mgr.14152) 111 : cluster 0 pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:11.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi050:vg_nvme/lv_3 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: cluster 2024-04-06T03:14:11.484596+0000 mgr.y (mgr.14152) 112 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.392641+0000 mon.a (mon.0) 323 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.402019+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.404609+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.412610+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.417160+0000 mon.a (mon.0) 327 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.418901+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:13.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:13 smithi153 bash[23852]: audit 2024-04-06T03:14:12.428685+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: cluster 2024-04-06T03:14:11.484596+0000 mgr.y (mgr.14152) 112 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.392641+0000 mon.a (mon.0) 323 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.402019+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.404609+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.412610+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.417160+0000 mon.a (mon.0) 327 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.418901+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[21467]: audit 2024-04-06T03:14:12.428685+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: cluster 2024-04-06T03:14:11.484596+0000 mgr.y (mgr.14152) 112 : cluster 0 pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.392641+0000 mon.a (mon.0) 323 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.402019+0000 mon.a (mon.0) 324 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.404609+0000 mon.a (mon.0) 325 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.412610+0000 mon.a (mon.0) 326 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.417160+0000 mon.a (mon.0) 327 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.418901+0000 mon.a (mon.0) 328 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:13 smithi050 bash[29221]: audit 2024-04-06T03:14:12.428685+0000 mon.a (mon.0) 329 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:14.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:14 smithi153 bash[23852]: cephadm 2024-04-06T03:14:12.384459+0000 mgr.y (mgr.14152) 113 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:14:14.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:14 smithi153 bash[23852]: cephadm 2024-04-06T03:14:12.406011+0000 mgr.y (mgr.14152) 114 : cephadm 1 Adjusting osd_memory_target on smithi050 to 16245M 2024-04-06T03:14:14.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:14 smithi050 bash[21467]: cephadm 2024-04-06T03:14:12.384459+0000 mgr.y (mgr.14152) 113 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:14:14.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:14 smithi050 bash[21467]: cephadm 2024-04-06T03:14:12.406011+0000 mgr.y (mgr.14152) 114 : cephadm 1 Adjusting osd_memory_target on smithi050 to 16245M 2024-04-06T03:14:14.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:14 smithi050 bash[29221]: cephadm 2024-04-06T03:14:12.384459+0000 mgr.y (mgr.14152) 113 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:14:14.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:14 smithi050 bash[29221]: cephadm 2024-04-06T03:14:12.406011+0000 mgr.y (mgr.14152) 114 : cephadm 1 Adjusting osd_memory_target on smithi050 to 16245M 2024-04-06T03:14:15.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:15 smithi153 bash[23852]: cluster 2024-04-06T03:14:13.485040+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:15.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:15 smithi050 bash[21467]: cluster 2024-04-06T03:14:13.485040+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:15.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:15 smithi050 bash[29221]: cluster 2024-04-06T03:14:13.485040+0000 mgr.y (mgr.14152) 115 : cluster 0 pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:16.684 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:14:17.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:17 smithi153 bash[23852]: cluster 2024-04-06T03:14:15.485507+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:17 smithi050 bash[21467]: cluster 2024-04-06T03:14:15.485507+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:17.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:17 smithi050 bash[29221]: cluster 2024-04-06T03:14:15.485507+0000 mgr.y (mgr.14152) 116 : cluster 0 pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:19.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:19 smithi153 bash[23852]: cluster 2024-04-06T03:14:17.485939+0000 mgr.y (mgr.14152) 117 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:19.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:19 smithi050 bash[29221]: cluster 2024-04-06T03:14:17.485939+0000 mgr.y (mgr.14152) 117 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:19.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:19 smithi050 bash[21467]: cluster 2024-04-06T03:14:17.485939+0000 mgr.y (mgr.14152) 117 : cluster 0 pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:20.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:20 smithi153 bash[23852]: audit 2024-04-06T03:14:19.502756+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:20.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:20 smithi153 bash[23852]: audit 2024-04-06T03:14:19.505672+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:20.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:20 smithi153 bash[23852]: audit 2024-04-06T03:14:19.506836+0000 mon.a (mon.0) 332 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:20.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:20 smithi050 bash[21467]: audit 2024-04-06T03:14:19.502756+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:20.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:20 smithi050 bash[21467]: audit 2024-04-06T03:14:19.505672+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:20.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:20 smithi050 bash[21467]: audit 2024-04-06T03:14:19.506836+0000 mon.a (mon.0) 332 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:20 smithi050 bash[29221]: audit 2024-04-06T03:14:19.502756+0000 mon.a (mon.0) 330 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:14:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:20 smithi050 bash[29221]: audit 2024-04-06T03:14:19.505672+0000 mon.a (mon.0) 331 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:14:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:20 smithi050 bash[29221]: audit 2024-04-06T03:14:19.506836+0000 mon.a (mon.0) 332 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:21.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:21 smithi050 bash[21467]: cluster 2024-04-06T03:14:19.486253+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:21.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:21 smithi050 bash[21467]: audit 2024-04-06T03:14:19.500044+0000 mgr.y (mgr.14152) 119 : audit 0 from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:21.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:21 smithi050 bash[29221]: cluster 2024-04-06T03:14:19.486253+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:21.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:21 smithi050 bash[29221]: audit 2024-04-06T03:14:19.500044+0000 mgr.y (mgr.14152) 119 : audit 0 from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:21.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:21 smithi153 bash[23852]: cluster 2024-04-06T03:14:19.486253+0000 mgr.y (mgr.14152) 118 : cluster 0 pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:21.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:21 smithi153 bash[23852]: audit 2024-04-06T03:14:19.500044+0000 mgr.y (mgr.14152) 119 : audit 0 from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:14:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:23 smithi050 bash[21467]: cluster 2024-04-06T03:14:21.486776+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:23.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:23 smithi050 bash[29221]: cluster 2024-04-06T03:14:21.486776+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:23.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:23 smithi153 bash[23852]: cluster 2024-04-06T03:14:21.486776+0000 mgr.y (mgr.14152) 120 : cluster 0 pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:25.347 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:25 smithi050 bash[21467]: cluster 2024-04-06T03:14:23.487297+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:25.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:25 smithi050 bash[29221]: cluster 2024-04-06T03:14:23.487297+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:25.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:25 smithi153 bash[23852]: cluster 2024-04-06T03:14:23.487297+0000 mgr.y (mgr.14152) 121 : cluster 0 pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:27.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:27 smithi050 bash[21467]: cluster 2024-04-06T03:14:25.487837+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:27.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:27 smithi050 bash[29221]: cluster 2024-04-06T03:14:25.487837+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:27.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:27 smithi153 bash[23852]: cluster 2024-04-06T03:14:25.487837+0000 mgr.y (mgr.14152) 122 : cluster 0 pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:28.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[29221]: audit 2024-04-06T03:14:27.829657+0000 mon.a (mon.0) 333 : audit 1 from='client.? 172.21.15.50:0/1789058114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10"}]: dispatch 2024-04-06T03:14:28.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[29221]: audit 2024-04-06T03:14:27.832430+0000 mon.a (mon.0) 334 : audit 1 from='client.? 172.21.15.50:0/1789058114' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10"}]': finished 2024-04-06T03:14:28.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[29221]: cluster 2024-04-06T03:14:27.837114+0000 mon.a (mon.0) 335 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-04-06T03:14:28.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[29221]: audit 2024-04-06T03:14:27.837525+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:28.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[21467]: audit 2024-04-06T03:14:27.829657+0000 mon.a (mon.0) 333 : audit 1 from='client.? 172.21.15.50:0/1789058114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10"}]: dispatch 2024-04-06T03:14:28.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[21467]: audit 2024-04-06T03:14:27.832430+0000 mon.a (mon.0) 334 : audit 1 from='client.? 172.21.15.50:0/1789058114' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10"}]': finished 2024-04-06T03:14:28.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[21467]: cluster 2024-04-06T03:14:27.837114+0000 mon.a (mon.0) 335 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-04-06T03:14:28.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:28 smithi050 bash[21467]: audit 2024-04-06T03:14:27.837525+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:28.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:28 smithi153 bash[23852]: audit 2024-04-06T03:14:27.829657+0000 mon.a (mon.0) 333 : audit 1 from='client.? 172.21.15.50:0/1789058114' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10"}]: dispatch 2024-04-06T03:14:28.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:28 smithi153 bash[23852]: audit 2024-04-06T03:14:27.832430+0000 mon.a (mon.0) 334 : audit 1 from='client.? 172.21.15.50:0/1789058114' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10"}]': finished 2024-04-06T03:14:28.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:28 smithi153 bash[23852]: cluster 2024-04-06T03:14:27.837114+0000 mon.a (mon.0) 335 : cluster 0 osdmap e10: 2 total, 1 up, 2 in 2024-04-06T03:14:28.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:28 smithi153 bash[23852]: audit 2024-04-06T03:14:27.837525+0000 mon.a (mon.0) 336 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:29 smithi050 bash[21467]: cluster 2024-04-06T03:14:27.488191+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:29 smithi050 bash[21467]: audit 2024-04-06T03:14:28.561634+0000 mon.c (mon.2) 8 : audit 0 from='client.? 172.21.15.50:0/314887851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:29.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:29 smithi050 bash[29221]: cluster 2024-04-06T03:14:27.488191+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:29.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:29 smithi050 bash[29221]: audit 2024-04-06T03:14:28.561634+0000 mon.c (mon.2) 8 : audit 0 from='client.? 172.21.15.50:0/314887851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:29.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:29 smithi153 bash[23852]: cluster 2024-04-06T03:14:27.488191+0000 mgr.y (mgr.14152) 123 : cluster 0 pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:29.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:29 smithi153 bash[23852]: audit 2024-04-06T03:14:28.561634+0000 mon.c (mon.2) 8 : audit 0 from='client.? 172.21.15.50:0/314887851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:14:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:31 smithi050 bash[21467]: cluster 2024-04-06T03:14:29.488554+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:31.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:31 smithi050 bash[29221]: cluster 2024-04-06T03:14:29.488554+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:31.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:31 smithi153 bash[23852]: cluster 2024-04-06T03:14:29.488554+0000 mgr.y (mgr.14152) 124 : cluster 0 pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:33 smithi050 bash[21467]: cluster 2024-04-06T03:14:31.488858+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:33.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:33 smithi050 bash[29221]: cluster 2024-04-06T03:14:31.488858+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:33.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:33 smithi153 bash[23852]: cluster 2024-04-06T03:14:31.488858+0000 mgr.y (mgr.14152) 125 : cluster 0 pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:35 smithi050 bash[21467]: cluster 2024-04-06T03:14:33.489354+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:35.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:35 smithi050 bash[29221]: cluster 2024-04-06T03:14:33.489354+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:35.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:35 smithi153 bash[23852]: cluster 2024-04-06T03:14:33.489354+0000 mgr.y (mgr.14152) 126 : cluster 0 pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:36.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:36 smithi050 bash[21467]: cluster 2024-04-06T03:14:35.489828+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:36.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:36 smithi050 bash[29221]: cluster 2024-04-06T03:14:35.489828+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:36.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:36 smithi153 bash[23852]: cluster 2024-04-06T03:14:35.489828+0000 mgr.y (mgr.14152) 127 : cluster 0 pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:38.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:38 smithi050 bash[21467]: cluster 2024-04-06T03:14:37.490281+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:38.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:38 smithi050 bash[29221]: cluster 2024-04-06T03:14:37.490281+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:38.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:38 smithi153 bash[23852]: cluster 2024-04-06T03:14:37.490281+0000 mgr.y (mgr.14152) 128 : cluster 0 pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:40.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:40 smithi050 bash[21467]: cluster 2024-04-06T03:14:39.490701+0000 mgr.y (mgr.14152) 129 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:40.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:40 smithi050 bash[21467]: audit 2024-04-06T03:14:40.251037+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:14:40.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:40 smithi050 bash[21467]: audit 2024-04-06T03:14:40.253020+0000 mon.a (mon.0) 338 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:40 smithi050 bash[29221]: cluster 2024-04-06T03:14:39.490701+0000 mgr.y (mgr.14152) 129 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:40 smithi050 bash[29221]: audit 2024-04-06T03:14:40.251037+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:14:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:40 smithi050 bash[29221]: audit 2024-04-06T03:14:40.253020+0000 mon.a (mon.0) 338 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:41.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:40 smithi153 bash[23852]: cluster 2024-04-06T03:14:39.490701+0000 mgr.y (mgr.14152) 129 : cluster 0 pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:41.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:40 smithi153 bash[23852]: audit 2024-04-06T03:14:40.251037+0000 mon.a (mon.0) 337 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-06T03:14:41.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:40 smithi153 bash[23852]: audit 2024-04-06T03:14:40.253020+0000 mon.a (mon.0) 338 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:41.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:41 smithi050 bash[21467]: cephadm 2024-04-06T03:14:40.254793+0000 mgr.y (mgr.14152) 130 : cephadm 1 Deploying daemon osd.1 on smithi050 2024-04-06T03:14:41.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:41 smithi050 bash[29221]: cephadm 2024-04-06T03:14:40.254793+0000 mgr.y (mgr.14152) 130 : cephadm 1 Deploying daemon osd.1 on smithi050 2024-04-06T03:14:42.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:41 smithi153 bash[23852]: cephadm 2024-04-06T03:14:40.254793+0000 mgr.y (mgr.14152) 130 : cephadm 1 Deploying daemon osd.1 on smithi050 2024-04-06T03:14:42.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:42 smithi050 bash[21467]: cluster 2024-04-06T03:14:41.491072+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:42.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:42 smithi050 bash[29221]: cluster 2024-04-06T03:14:41.491072+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:43.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:42 smithi153 bash[23852]: cluster 2024-04-06T03:14:41.491072+0000 mgr.y (mgr.14152) 131 : cluster 0 pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:44.397 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.397 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.398 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.398 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.731 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.732 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:44 smithi050 bash[21467]: cluster 2024-04-06T03:14:43.491550+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:44.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:44 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:14:44.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:44 smithi050 bash[29221]: cluster 2024-04-06T03:14:43.491550+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:45.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:44 smithi153 bash[23852]: cluster 2024-04-06T03:14:43.491550+0000 mgr.y (mgr.14152) 132 : cluster 0 pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:45.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:45 smithi050 bash[21467]: audit 2024-04-06T03:14:44.688433+0000 mon.a (mon.0) 339 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:45.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:45 smithi050 bash[21467]: audit 2024-04-06T03:14:44.693757+0000 mon.a (mon.0) 340 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:45.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:45 smithi050 bash[21467]: audit 2024-04-06T03:14:44.697041+0000 mon.a (mon.0) 341 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:45.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:45 smithi050 bash[29221]: audit 2024-04-06T03:14:44.688433+0000 mon.a (mon.0) 339 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:45.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:45 smithi050 bash[29221]: audit 2024-04-06T03:14:44.693757+0000 mon.a (mon.0) 340 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:45.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:45 smithi050 bash[29221]: audit 2024-04-06T03:14:44.697041+0000 mon.a (mon.0) 341 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:46.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:45 smithi153 bash[23852]: audit 2024-04-06T03:14:44.688433+0000 mon.a (mon.0) 339 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:46.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:45 smithi153 bash[23852]: audit 2024-04-06T03:14:44.693757+0000 mon.a (mon.0) 340 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:46.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:45 smithi153 bash[23852]: audit 2024-04-06T03:14:44.697041+0000 mon.a (mon.0) 341 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:46.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:46 smithi050 bash[21467]: cluster 2024-04-06T03:14:45.492025+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:46.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:46 smithi050 bash[29221]: cluster 2024-04-06T03:14:45.492025+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:47.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:46 smithi153 bash[23852]: cluster 2024-04-06T03:14:45.492025+0000 mgr.y (mgr.14152) 133 : cluster 0 pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:48.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:48 smithi050 bash[21467]: cluster 2024-04-06T03:14:47.492541+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:48.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:48 smithi050 bash[29221]: cluster 2024-04-06T03:14:47.492541+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:49.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:48 smithi153 bash[23852]: cluster 2024-04-06T03:14:47.492541+0000 mgr.y (mgr.14152) 134 : cluster 0 pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[29221]: cluster 2024-04-06T03:14:49.493085+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[29221]: audit 2024-04-06T03:14:50.037836+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[29221]: audit 2024-04-06T03:14:50.044762+0000 mon.a (mon.0) 343 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[29221]: audit 2024-04-06T03:14:50.046201+0000 mon.a (mon.0) 344 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[29221]: audit 2024-04-06T03:14:50.047328+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[29221]: audit 2024-04-06T03:14:50.054069+0000 mon.a (mon.0) 346 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[21467]: cluster 2024-04-06T03:14:49.493085+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:51.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[21467]: audit 2024-04-06T03:14:50.037836+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[21467]: audit 2024-04-06T03:14:50.044762+0000 mon.a (mon.0) 343 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[21467]: audit 2024-04-06T03:14:50.046201+0000 mon.a (mon.0) 344 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:51.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[21467]: audit 2024-04-06T03:14:50.047328+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:51.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:51 smithi050 bash[21467]: audit 2024-04-06T03:14:50.054069+0000 mon.a (mon.0) 346 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:51 smithi153 bash[23852]: cluster 2024-04-06T03:14:49.493085+0000 mgr.y (mgr.14152) 135 : cluster 0 pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:51.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:51 smithi153 bash[23852]: audit 2024-04-06T03:14:50.037836+0000 mon.a (mon.0) 342 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:51 smithi153 bash[23852]: audit 2024-04-06T03:14:50.044762+0000 mon.a (mon.0) 343 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:51.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:51 smithi153 bash[23852]: audit 2024-04-06T03:14:50.046201+0000 mon.a (mon.0) 344 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:14:51.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:51 smithi153 bash[23852]: audit 2024-04-06T03:14:50.047328+0000 mon.a (mon.0) 345 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:14:51.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:51 smithi153 bash[23852]: audit 2024-04-06T03:14:50.054069+0000 mon.a (mon.0) 346 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:52.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:52 smithi050 bash[21467]: audit 2024-04-06T03:14:51.301918+0000 mon.c (mon.2) 9 : audit 1 from='osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:14:52.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:52 smithi050 bash[21467]: audit 2024-04-06T03:14:51.302370+0000 mon.a (mon.0) 347 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:14:52.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:52 smithi050 bash[29221]: audit 2024-04-06T03:14:51.301918+0000 mon.c (mon.2) 9 : audit 1 from='osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:14:52.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:52 smithi050 bash[29221]: audit 2024-04-06T03:14:51.302370+0000 mon.a (mon.0) 347 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:14:52.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:52 smithi153 bash[23852]: audit 2024-04-06T03:14:51.301918+0000 mon.c (mon.2) 9 : audit 1 from='osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:14:52.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:52 smithi153 bash[23852]: audit 2024-04-06T03:14:51.302370+0000 mon.a (mon.0) 347 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[29221]: cluster 2024-04-06T03:14:51.493560+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[29221]: audit 2024-04-06T03:14:52.046583+0000 mon.a (mon.0) 348 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[29221]: cluster 2024-04-06T03:14:52.051623+0000 mon.a (mon.0) 349 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[29221]: audit 2024-04-06T03:14:52.052361+0000 mon.a (mon.0) 350 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[29221]: audit 2024-04-06T03:14:52.053803+0000 mon.c (mon.2) 10 : audit 1 from='osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[29221]: audit 2024-04-06T03:14:52.054638+0000 mon.a (mon.0) 351 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[21467]: cluster 2024-04-06T03:14:51.493560+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:53.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[21467]: audit 2024-04-06T03:14:52.046583+0000 mon.a (mon.0) 348 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:14:53.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[21467]: cluster 2024-04-06T03:14:52.051623+0000 mon.a (mon.0) 349 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-04-06T03:14:53.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[21467]: audit 2024-04-06T03:14:52.052361+0000 mon.a (mon.0) 350 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:53.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[21467]: audit 2024-04-06T03:14:52.053803+0000 mon.c (mon.2) 10 : audit 1 from='osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:53.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:53 smithi050 bash[21467]: audit 2024-04-06T03:14:52.054638+0000 mon.a (mon.0) 351 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:53 smithi153 bash[23852]: cluster 2024-04-06T03:14:51.493560+0000 mgr.y (mgr.14152) 136 : cluster 0 pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:53 smithi153 bash[23852]: audit 2024-04-06T03:14:52.046583+0000 mon.a (mon.0) 348 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-06T03:14:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:53 smithi153 bash[23852]: cluster 2024-04-06T03:14:52.051623+0000 mon.a (mon.0) 349 : cluster 0 osdmap e11: 2 total, 1 up, 2 in 2024-04-06T03:14:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:53 smithi153 bash[23852]: audit 2024-04-06T03:14:52.052361+0000 mon.a (mon.0) 350 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:53 smithi153 bash[23852]: audit 2024-04-06T03:14:52.053803+0000 mon.c (mon.2) 10 : audit 1 from='osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:53 smithi153 bash[23852]: audit 2024-04-06T03:14:52.054638+0000 mon.a (mon.0) 351 : audit 1 from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:14:53.553 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 1 on host 'smithi050' 2024-04-06T03:14:54.262 DEBUG:teuthology.orchestra.run.smithi050:osd.1> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-06T03:14:54.265 INFO:tasks.cephadm:Deploying osd.2 on smithi050 with /dev/vg_nvme/lv_2... 2024-04-06T03:14:54.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: audit 2024-04-06T03:14:53.049067+0000 mon.a (mon.0) 352 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: cluster 2024-04-06T03:14:53.055065+0000 mon.a (mon.0) 353 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: audit 2024-04-06T03:14:53.056163+0000 mon.a (mon.0) 354 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: audit 2024-04-06T03:14:53.065776+0000 mon.a (mon.0) 355 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: audit 2024-04-06T03:14:53.532036+0000 mon.a (mon.0) 356 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: audit 2024-04-06T03:14:53.540880+0000 mon.a (mon.0) 357 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[21467]: audit 2024-04-06T03:14:53.544886+0000 mon.a (mon.0) 358 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: audit 2024-04-06T03:14:53.049067+0000 mon.a (mon.0) 352 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:14:54.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: cluster 2024-04-06T03:14:53.055065+0000 mon.a (mon.0) 353 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-04-06T03:14:54.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: audit 2024-04-06T03:14:53.056163+0000 mon.a (mon.0) 354 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:54.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: audit 2024-04-06T03:14:53.065776+0000 mon.a (mon.0) 355 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:54.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: audit 2024-04-06T03:14:53.532036+0000 mon.a (mon.0) 356 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:54.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: audit 2024-04-06T03:14:53.540880+0000 mon.a (mon.0) 357 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:54.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:54 smithi050 bash[29221]: audit 2024-04-06T03:14:53.544886+0000 mon.a (mon.0) 358 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:54.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: audit 2024-04-06T03:14:53.049067+0000 mon.a (mon.0) 352 : audit 1 from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:14:54.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: cluster 2024-04-06T03:14:53.055065+0000 mon.a (mon.0) 353 : cluster 0 osdmap e12: 2 total, 1 up, 2 in 2024-04-06T03:14:54.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: audit 2024-04-06T03:14:53.056163+0000 mon.a (mon.0) 354 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:54.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: audit 2024-04-06T03:14:53.065776+0000 mon.a (mon.0) 355 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:54.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: audit 2024-04-06T03:14:53.532036+0000 mon.a (mon.0) 356 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:54.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: audit 2024-04-06T03:14:53.540880+0000 mon.a (mon.0) 357 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:14:54.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:54 smithi153 bash[23852]: audit 2024-04-06T03:14:53.544886+0000 mon.a (mon.0) 358 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[21467]: cluster 2024-04-06T03:14:52.306461+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[21467]: cluster 2024-04-06T03:14:52.306580+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[21467]: cluster 2024-04-06T03:14:53.494012+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[21467]: cluster 2024-04-06T03:14:54.065331+0000 mon.a (mon.0) 359 : cluster 1 osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069] boot 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[21467]: cluster 2024-04-06T03:14:54.065446+0000 mon.a (mon.0) 360 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[21467]: audit 2024-04-06T03:14:54.066243+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[29221]: cluster 2024-04-06T03:14:52.306461+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[29221]: cluster 2024-04-06T03:14:52.306580+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[29221]: cluster 2024-04-06T03:14:53.494012+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[29221]: cluster 2024-04-06T03:14:54.065331+0000 mon.a (mon.0) 359 : cluster 1 osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069] boot 2024-04-06T03:14:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[29221]: cluster 2024-04-06T03:14:54.065446+0000 mon.a (mon.0) 360 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-04-06T03:14:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:55 smithi050 bash[29221]: audit 2024-04-06T03:14:54.066243+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:55.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:55 smithi153 bash[23852]: cluster 2024-04-06T03:14:52.306461+0000 osd.1 (osd.1) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:14:55.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:55 smithi153 bash[23852]: cluster 2024-04-06T03:14:52.306580+0000 osd.1 (osd.1) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:14:55.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:55 smithi153 bash[23852]: cluster 2024-04-06T03:14:53.494012+0000 mgr.y (mgr.14152) 137 : cluster 0 pgmap v102: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-06T03:14:55.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:55 smithi153 bash[23852]: cluster 2024-04-06T03:14:54.065331+0000 mon.a (mon.0) 359 : cluster 1 osd.1 [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069] boot 2024-04-06T03:14:55.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:55 smithi153 bash[23852]: cluster 2024-04-06T03:14:54.065446+0000 mon.a (mon.0) 360 : cluster 0 osdmap e13: 2 total, 2 up, 2 in 2024-04-06T03:14:55.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:55 smithi153 bash[23852]: audit 2024-04-06T03:14:54.066243+0000 mon.a (mon.0) 361 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:14:56.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:56 smithi050 bash[21467]: cluster 2024-04-06T03:14:55.075188+0000 mon.a (mon.0) 362 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-04-06T03:14:56.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:56 smithi050 bash[29221]: cluster 2024-04-06T03:14:55.075188+0000 mon.a (mon.0) 362 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-04-06T03:14:56.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:56 smithi153 bash[23852]: cluster 2024-04-06T03:14:55.075188+0000 mon.a (mon.0) 362 : cluster 0 osdmap e14: 2 total, 2 up, 2 in 2024-04-06T03:14:57.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:57 smithi050 bash[21467]: cluster 2024-04-06T03:14:55.494452+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:57.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:57 smithi050 bash[29221]: cluster 2024-04-06T03:14:55.494452+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:57.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:57 smithi153 bash[23852]: cluster 2024-04-06T03:14:55.494452+0000 mgr.y (mgr.14152) 138 : cluster 0 pgmap v105: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:58.506 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:14:58.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:14:58 smithi050 bash[21467]: cluster 2024-04-06T03:14:57.494872+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:58.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:14:58 smithi050 bash[29221]: cluster 2024-04-06T03:14:57.494872+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:14:59.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:14:58 smithi153 bash[23852]: cluster 2024-04-06T03:14:57.494872+0000 mgr.y (mgr.14152) 139 : cluster 0 pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:00.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:00 smithi050 bash[21467]: cluster 2024-04-06T03:14:59.495415+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:00 smithi050 bash[29221]: cluster 2024-04-06T03:14:59.495415+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:01.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:00 smithi153 bash[23852]: cluster 2024-04-06T03:14:59.495415+0000 mgr.y (mgr.14152) 140 : cluster 0 pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:02.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:02 smithi050 bash[29221]: cluster 2024-04-06T03:15:01.496011+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:02.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:02 smithi050 bash[21467]: cluster 2024-04-06T03:15:01.496011+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:03.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:02 smithi153 bash[23852]: cluster 2024-04-06T03:15:01.496011+0000 mgr.y (mgr.14152) 141 : cluster 0 pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T03:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T03:15:03.221 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-06T03:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-06T03:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146952 s, 714 MB/s 2024-04-06T03:15:03.222 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-06T03:15:04.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi050:vg_nvme/lv_2 2024-04-06T03:15:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: cluster 2024-04-06T03:15:03.496421+0000 mgr.y (mgr.14152) 142 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.499089+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.507842+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.510265+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.512040+0000 mon.a (mon.0) 366 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.521812+0000 mon.a (mon.0) 367 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.527423+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.529059+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[21467]: audit 2024-04-06T03:15:04.538206+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: cluster 2024-04-06T03:15:03.496421+0000 mgr.y (mgr.14152) 142 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.499089+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.507842+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.510265+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.512040+0000 mon.a (mon.0) 366 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.521812+0000 mon.a (mon.0) 367 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.527423+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.529059+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:04 smithi050 bash[29221]: audit 2024-04-06T03:15:04.538206+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: cluster 2024-04-06T03:15:03.496421+0000 mgr.y (mgr.14152) 142 : cluster 0 pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.499089+0000 mon.a (mon.0) 363 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.507842+0000 mon.a (mon.0) 364 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.510265+0000 mon.a (mon.0) 365 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.512040+0000 mon.a (mon.0) 366 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.521812+0000 mon.a (mon.0) 367 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.527423+0000 mon.a (mon.0) 368 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.529059+0000 mon.a (mon.0) 369 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:04 smithi153 bash[23852]: audit 2024-04-06T03:15:04.538206+0000 mon.a (mon.0) 370 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:05 smithi050 bash[21467]: cephadm 2024-04-06T03:15:04.490291+0000 mgr.y (mgr.14152) 143 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:15:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:05 smithi050 bash[21467]: cephadm 2024-04-06T03:15:04.513478+0000 mgr.y (mgr.14152) 144 : cephadm 1 Adjusting osd_memory_target on smithi050 to 8122M 2024-04-06T03:15:05.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:05 smithi050 bash[29221]: cephadm 2024-04-06T03:15:04.490291+0000 mgr.y (mgr.14152) 143 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:15:05.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:05 smithi050 bash[29221]: cephadm 2024-04-06T03:15:04.513478+0000 mgr.y (mgr.14152) 144 : cephadm 1 Adjusting osd_memory_target on smithi050 to 8122M 2024-04-06T03:15:06.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:05 smithi153 bash[23852]: cephadm 2024-04-06T03:15:04.490291+0000 mgr.y (mgr.14152) 143 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:15:06.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:05 smithi153 bash[23852]: cephadm 2024-04-06T03:15:04.513478+0000 mgr.y (mgr.14152) 144 : cephadm 1 Adjusting osd_memory_target on smithi050 to 8122M 2024-04-06T03:15:06.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:06 smithi050 bash[21467]: cluster 2024-04-06T03:15:05.496953+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:06.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:06 smithi050 bash[29221]: cluster 2024-04-06T03:15:05.496953+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:07.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:06 smithi153 bash[23852]: cluster 2024-04-06T03:15:05.496953+0000 mgr.y (mgr.14152) 145 : cluster 0 pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:08.838 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:15:08.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:08 smithi050 bash[21467]: cluster 2024-04-06T03:15:07.497403+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:08.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:08 smithi050 bash[29221]: cluster 2024-04-06T03:15:07.497403+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:09.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:08 smithi153 bash[23852]: cluster 2024-04-06T03:15:07.497403+0000 mgr.y (mgr.14152) 146 : cluster 0 pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:10 smithi050 bash[21467]: cluster 2024-04-06T03:15:09.497948+0000 mgr.y (mgr.14152) 147 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:10 smithi050 bash[29221]: cluster 2024-04-06T03:15:09.497948+0000 mgr.y (mgr.14152) 147 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:11.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:10 smithi153 bash[23852]: cluster 2024-04-06T03:15:09.497948+0000 mgr.y (mgr.14152) 147 : cluster 0 pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:12.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[21467]: cluster 2024-04-06T03:15:11.498325+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:12.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[21467]: audit 2024-04-06T03:15:11.746599+0000 mgr.y (mgr.14152) 149 : audit 0 from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:12.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[21467]: audit 2024-04-06T03:15:11.749333+0000 mon.a (mon.0) 371 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:12.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[21467]: audit 2024-04-06T03:15:11.754714+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:12.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[21467]: audit 2024-04-06T03:15:11.755773+0000 mon.a (mon.0) 373 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:12.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[29221]: cluster 2024-04-06T03:15:11.498325+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:12.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[29221]: audit 2024-04-06T03:15:11.746599+0000 mgr.y (mgr.14152) 149 : audit 0 from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:12.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[29221]: audit 2024-04-06T03:15:11.749333+0000 mon.a (mon.0) 371 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:12.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[29221]: audit 2024-04-06T03:15:11.754714+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:12.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:12 smithi050 bash[29221]: audit 2024-04-06T03:15:11.755773+0000 mon.a (mon.0) 373 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:12 smithi153 bash[23852]: cluster 2024-04-06T03:15:11.498325+0000 mgr.y (mgr.14152) 148 : cluster 0 pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:12 smithi153 bash[23852]: audit 2024-04-06T03:15:11.746599+0000 mgr.y (mgr.14152) 149 : audit 0 from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:15:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:12 smithi153 bash[23852]: audit 2024-04-06T03:15:11.749333+0000 mon.a (mon.0) 371 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:15:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:12 smithi153 bash[23852]: audit 2024-04-06T03:15:11.754714+0000 mon.a (mon.0) 372 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:15:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:12 smithi153 bash[23852]: audit 2024-04-06T03:15:11.755773+0000 mon.a (mon.0) 373 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:14.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:14 smithi050 bash[21467]: cluster 2024-04-06T03:15:13.498780+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:14.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:14 smithi050 bash[29221]: cluster 2024-04-06T03:15:13.498780+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:15.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:14 smithi153 bash[23852]: cluster 2024-04-06T03:15:13.498780+0000 mgr.y (mgr.14152) 150 : cluster 0 pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:16.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:16 smithi050 bash[21467]: cluster 2024-04-06T03:15:15.499248+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:16.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:16 smithi050 bash[29221]: cluster 2024-04-06T03:15:15.499248+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:17.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:16 smithi153 bash[23852]: cluster 2024-04-06T03:15:15.499248+0000 mgr.y (mgr.14152) 151 : cluster 0 pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[21467]: cluster 2024-04-06T03:15:17.141976+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:15:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[21467]: cluster 2024-04-06T03:15:17.142104+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:15:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[21467]: cluster 2024-04-06T03:15:17.499689+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[21467]: cluster 2024-04-06T03:15:17.639114+0000 mon.a (mon.0) 374 : cluster 0 osdmap e15: 2 total, 2 up, 2 in 2024-04-06T03:15:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[29221]: cluster 2024-04-06T03:15:17.141976+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:15:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[29221]: cluster 2024-04-06T03:15:17.142104+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:15:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[29221]: cluster 2024-04-06T03:15:17.499689+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:18 smithi050 bash[29221]: cluster 2024-04-06T03:15:17.639114+0000 mon.a (mon.0) 374 : cluster 0 osdmap e15: 2 total, 2 up, 2 in 2024-04-06T03:15:19.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:18 smithi153 bash[23852]: cluster 2024-04-06T03:15:17.141976+0000 osd.0 (osd.0) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:15:19.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:18 smithi153 bash[23852]: cluster 2024-04-06T03:15:17.142104+0000 osd.0 (osd.0) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:15:19.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:18 smithi153 bash[23852]: cluster 2024-04-06T03:15:17.499689+0000 mgr.y (mgr.14152) 152 : cluster 0 pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:19.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:18 smithi153 bash[23852]: cluster 2024-04-06T03:15:17.639114+0000 mon.a (mon.0) 374 : cluster 0 osdmap e15: 2 total, 2 up, 2 in 2024-04-06T03:15:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[21467]: cluster 2024-04-06T03:15:19.500165+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[21467]: audit 2024-04-06T03:15:20.351962+0000 mon.c (mon.2) 11 : audit 1 from='client.? 172.21.15.50:0/3334399302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]: dispatch 2024-04-06T03:15:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[21467]: audit 2024-04-06T03:15:20.352569+0000 mon.a (mon.0) 375 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]: dispatch 2024-04-06T03:15:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[21467]: audit 2024-04-06T03:15:20.355314+0000 mon.a (mon.0) 376 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]': finished 2024-04-06T03:15:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[21467]: cluster 2024-04-06T03:15:20.358771+0000 mon.a (mon.0) 377 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-04-06T03:15:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[21467]: audit 2024-04-06T03:15:20.359361+0000 mon.a (mon.0) 378 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[29221]: cluster 2024-04-06T03:15:19.500165+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[29221]: audit 2024-04-06T03:15:20.351962+0000 mon.c (mon.2) 11 : audit 1 from='client.? 172.21.15.50:0/3334399302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]: dispatch 2024-04-06T03:15:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[29221]: audit 2024-04-06T03:15:20.352569+0000 mon.a (mon.0) 375 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]: dispatch 2024-04-06T03:15:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[29221]: audit 2024-04-06T03:15:20.355314+0000 mon.a (mon.0) 376 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]': finished 2024-04-06T03:15:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[29221]: cluster 2024-04-06T03:15:20.358771+0000 mon.a (mon.0) 377 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-04-06T03:15:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:20 smithi050 bash[29221]: audit 2024-04-06T03:15:20.359361+0000 mon.a (mon.0) 378 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:21.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:20 smithi153 bash[23852]: cluster 2024-04-06T03:15:19.500165+0000 mgr.y (mgr.14152) 153 : cluster 0 pgmap v118: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:21.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:20 smithi153 bash[23852]: audit 2024-04-06T03:15:20.351962+0000 mon.c (mon.2) 11 : audit 1 from='client.? 172.21.15.50:0/3334399302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]: dispatch 2024-04-06T03:15:21.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:20 smithi153 bash[23852]: audit 2024-04-06T03:15:20.352569+0000 mon.a (mon.0) 375 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]: dispatch 2024-04-06T03:15:21.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:20 smithi153 bash[23852]: audit 2024-04-06T03:15:20.355314+0000 mon.a (mon.0) 376 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b"}]': finished 2024-04-06T03:15:21.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:20 smithi153 bash[23852]: cluster 2024-04-06T03:15:20.358771+0000 mon.a (mon.0) 377 : cluster 0 osdmap e16: 3 total, 2 up, 3 in 2024-04-06T03:15:21.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:20 smithi153 bash[23852]: audit 2024-04-06T03:15:20.359361+0000 mon.a (mon.0) 378 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:21 smithi050 bash[21467]: audit 2024-04-06T03:15:21.068666+0000 mon.a (mon.0) 379 : audit 0 from='client.? 172.21.15.50:0/4105262270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:21.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:21 smithi050 bash[29221]: audit 2024-04-06T03:15:21.068666+0000 mon.a (mon.0) 379 : audit 0 from='client.? 172.21.15.50:0/4105262270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:22.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:21 smithi153 bash[23852]: audit 2024-04-06T03:15:21.068666+0000 mon.a (mon.0) 379 : audit 0 from='client.? 172.21.15.50:0/4105262270' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:15:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:22 smithi050 bash[21467]: cluster 2024-04-06T03:15:21.500623+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:22.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:22 smithi050 bash[29221]: cluster 2024-04-06T03:15:21.500623+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:23.038 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:22 smithi153 bash[23852]: cluster 2024-04-06T03:15:21.500623+0000 mgr.y (mgr.14152) 154 : cluster 0 pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:24 smithi050 bash[21467]: cluster 2024-04-06T03:15:23.501137+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:24.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:24 smithi050 bash[29221]: cluster 2024-04-06T03:15:23.501137+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:25.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:24 smithi153 bash[23852]: cluster 2024-04-06T03:15:23.501137+0000 mgr.y (mgr.14152) 155 : cluster 0 pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:26.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:26 smithi050 bash[21467]: cluster 2024-04-06T03:15:25.501649+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:26.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:26 smithi050 bash[29221]: cluster 2024-04-06T03:15:25.501649+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:27.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:26 smithi153 bash[23852]: cluster 2024-04-06T03:15:25.501649+0000 mgr.y (mgr.14152) 156 : cluster 0 pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:28 smithi050 bash[21467]: cluster 2024-04-06T03:15:27.502126+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:28.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:28 smithi050 bash[29221]: cluster 2024-04-06T03:15:27.502126+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:29.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:28 smithi153 bash[23852]: cluster 2024-04-06T03:15:27.502126+0000 mgr.y (mgr.14152) 157 : cluster 0 pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:30 smithi050 bash[21467]: cluster 2024-04-06T03:15:29.502603+0000 mgr.y (mgr.14152) 158 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:30.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:30 smithi050 bash[29221]: cluster 2024-04-06T03:15:29.502603+0000 mgr.y (mgr.14152) 158 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:31.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:30 smithi153 bash[23852]: cluster 2024-04-06T03:15:29.502603+0000 mgr.y (mgr.14152) 158 : cluster 0 pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:32.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:32 smithi050 bash[29221]: cluster 2024-04-06T03:15:31.503045+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:32 smithi050 bash[21467]: cluster 2024-04-06T03:15:31.503045+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:33.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:32 smithi153 bash[23852]: cluster 2024-04-06T03:15:31.503045+0000 mgr.y (mgr.14152) 159 : cluster 0 pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:33.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:33 smithi050 bash[21467]: audit 2024-04-06T03:15:33.026509+0000 mon.a (mon.0) 380 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:15:33.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:33 smithi050 bash[21467]: audit 2024-04-06T03:15:33.028240+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:33.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:33 smithi050 bash[21467]: cephadm 2024-04-06T03:15:33.029626+0000 mgr.y (mgr.14152) 160 : cephadm 1 Deploying daemon osd.2 on smithi050 2024-04-06T03:15:33.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:33 smithi050 bash[29221]: audit 2024-04-06T03:15:33.026509+0000 mon.a (mon.0) 380 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:15:33.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:33 smithi050 bash[29221]: audit 2024-04-06T03:15:33.028240+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:33.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:33 smithi050 bash[29221]: cephadm 2024-04-06T03:15:33.029626+0000 mgr.y (mgr.14152) 160 : cephadm 1 Deploying daemon osd.2 on smithi050 2024-04-06T03:15:34.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:33 smithi153 bash[23852]: audit 2024-04-06T03:15:33.026509+0000 mon.a (mon.0) 380 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-06T03:15:34.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:33 smithi153 bash[23852]: audit 2024-04-06T03:15:33.028240+0000 mon.a (mon.0) 381 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:34.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:33 smithi153 bash[23852]: cephadm 2024-04-06T03:15:33.029626+0000 mgr.y (mgr.14152) 160 : cephadm 1 Deploying daemon osd.2 on smithi050 2024-04-06T03:15:34.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:34 smithi050 bash[21467]: cluster 2024-04-06T03:15:33.503408+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:34.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:34 smithi050 bash[29221]: cluster 2024-04-06T03:15:33.503408+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:35.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:34 smithi153 bash[23852]: cluster 2024-04-06T03:15:33.503408+0000 mgr.y (mgr.14152) 161 : cluster 0 pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:36.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:36 smithi050 bash[21467]: cluster 2024-04-06T03:15:35.503874+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:36.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:36 smithi050 bash[29221]: cluster 2024-04-06T03:15:35.503874+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:37.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:36 smithi153 bash[23852]: cluster 2024-04-06T03:15:35.503874+0000 mgr.y (mgr.14152) 162 : cluster 0 pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:37.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.429 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.429 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.429 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.722 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.723 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.723 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:37.723 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:15:37 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[21467]: cluster 2024-04-06T03:15:37.504255+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[21467]: audit 2024-04-06T03:15:37.727451+0000 mon.a (mon.0) 382 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[21467]: audit 2024-04-06T03:15:37.732442+0000 mon.a (mon.0) 383 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[21467]: audit 2024-04-06T03:15:37.739137+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[29221]: cluster 2024-04-06T03:15:37.504255+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[29221]: audit 2024-04-06T03:15:37.727451+0000 mon.a (mon.0) 382 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[29221]: audit 2024-04-06T03:15:37.732442+0000 mon.a (mon.0) 383 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:38.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:38 smithi050 bash[29221]: audit 2024-04-06T03:15:37.739137+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:39.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:38 smithi153 bash[23852]: cluster 2024-04-06T03:15:37.504255+0000 mgr.y (mgr.14152) 163 : cluster 0 pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:39.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:38 smithi153 bash[23852]: audit 2024-04-06T03:15:37.727451+0000 mon.a (mon.0) 382 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:39.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:38 smithi153 bash[23852]: audit 2024-04-06T03:15:37.732442+0000 mon.a (mon.0) 383 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:39.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:38 smithi153 bash[23852]: audit 2024-04-06T03:15:37.739137+0000 mon.a (mon.0) 384 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:40.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:40 smithi050 bash[21467]: cluster 2024-04-06T03:15:39.504704+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:40 smithi050 bash[29221]: cluster 2024-04-06T03:15:39.504704+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:41.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:40 smithi153 bash[23852]: cluster 2024-04-06T03:15:39.504704+0000 mgr.y (mgr.14152) 164 : cluster 0 pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:42.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:42 smithi050 bash[21467]: cluster 2024-04-06T03:15:41.505195+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:42.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:42 smithi050 bash[29221]: cluster 2024-04-06T03:15:41.505195+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:43.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:42 smithi153 bash[23852]: cluster 2024-04-06T03:15:41.505195+0000 mgr.y (mgr.14152) 165 : cluster 0 pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:44.412 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[21467]: audit 2024-04-06T03:15:43.079268+0000 mon.a (mon.0) 385 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.412 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[21467]: audit 2024-04-06T03:15:43.088643+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.412 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[21467]: audit 2024-04-06T03:15:43.709017+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:44.412 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[21467]: audit 2024-04-06T03:15:43.710499+0000 mon.a (mon.0) 388 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:44.412 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[21467]: audit 2024-04-06T03:15:43.717243+0000 mon.a (mon.0) 389 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[29221]: audit 2024-04-06T03:15:43.079268+0000 mon.a (mon.0) 385 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[29221]: audit 2024-04-06T03:15:43.088643+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[29221]: audit 2024-04-06T03:15:43.709017+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:44.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[29221]: audit 2024-04-06T03:15:43.710499+0000 mon.a (mon.0) 388 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:44.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:44 smithi050 bash[29221]: audit 2024-04-06T03:15:43.717243+0000 mon.a (mon.0) 389 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:44 smithi153 bash[23852]: audit 2024-04-06T03:15:43.079268+0000 mon.a (mon.0) 385 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:44 smithi153 bash[23852]: audit 2024-04-06T03:15:43.088643+0000 mon.a (mon.0) 386 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:44.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:44 smithi153 bash[23852]: audit 2024-04-06T03:15:43.709017+0000 mon.a (mon.0) 387 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:44.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:44 smithi153 bash[23852]: audit 2024-04-06T03:15:43.710499+0000 mon.a (mon.0) 388 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:44.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:44 smithi153 bash[23852]: audit 2024-04-06T03:15:43.717243+0000 mon.a (mon.0) 389 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:45.389 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:45 smithi050 bash[21467]: cluster 2024-04-06T03:15:43.505538+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:45.389 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:45 smithi050 bash[21467]: audit 2024-04-06T03:15:44.426155+0000 mon.a (mon.0) 390 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:15:45.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:45 smithi050 bash[29221]: cluster 2024-04-06T03:15:43.505538+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:45.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:45 smithi050 bash[29221]: audit 2024-04-06T03:15:44.426155+0000 mon.a (mon.0) 390 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:15:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:45 smithi153 bash[23852]: cluster 2024-04-06T03:15:43.505538+0000 mgr.y (mgr.14152) 166 : cluster 0 pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:45.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:45 smithi153 bash[23852]: audit 2024-04-06T03:15:44.426155+0000 mon.a (mon.0) 390 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[21467]: audit 2024-04-06T03:15:45.093833+0000 mon.a (mon.0) 391 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[21467]: cluster 2024-04-06T03:15:45.099181+0000 mon.a (mon.0) 392 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[21467]: audit 2024-04-06T03:15:45.099555+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[21467]: audit 2024-04-06T03:15:45.099964+0000 mon.a (mon.0) 394 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[29221]: audit 2024-04-06T03:15:45.093833+0000 mon.a (mon.0) 391 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[29221]: cluster 2024-04-06T03:15:45.099181+0000 mon.a (mon.0) 392 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-04-06T03:15:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[29221]: audit 2024-04-06T03:15:45.099555+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:46.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:46 smithi050 bash[29221]: audit 2024-04-06T03:15:45.099964+0000 mon.a (mon.0) 394 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:15:46.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:46 smithi153 bash[23852]: audit 2024-04-06T03:15:45.093833+0000 mon.a (mon.0) 391 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-06T03:15:46.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:46 smithi153 bash[23852]: cluster 2024-04-06T03:15:45.099181+0000 mon.a (mon.0) 392 : cluster 0 osdmap e17: 3 total, 2 up, 3 in 2024-04-06T03:15:46.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:46 smithi153 bash[23852]: audit 2024-04-06T03:15:45.099555+0000 mon.a (mon.0) 393 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:46.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:46 smithi153 bash[23852]: audit 2024-04-06T03:15:45.099964+0000 mon.a (mon.0) 394 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:15:46.746 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 2 on host 'smithi050' 2024-04-06T03:15:47.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: cluster 2024-04-06T03:15:45.505862+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:47.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: audit 2024-04-06T03:15:46.095444+0000 mon.a (mon.0) 395 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:15:47.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: cluster 2024-04-06T03:15:46.098126+0000 mon.a (mon.0) 396 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-04-06T03:15:47.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: audit 2024-04-06T03:15:46.098771+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:47.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: audit 2024-04-06T03:15:46.104550+0000 mon.a (mon.0) 398 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:47.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: audit 2024-04-06T03:15:46.705469+0000 mon.a (mon.0) 399 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:47.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: audit 2024-04-06T03:15:46.711226+0000 mon.a (mon.0) 400 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:47.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[21467]: audit 2024-04-06T03:15:46.719835+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:47.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: cluster 2024-04-06T03:15:45.505862+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:47.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: audit 2024-04-06T03:15:46.095444+0000 mon.a (mon.0) 395 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:15:47.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: cluster 2024-04-06T03:15:46.098126+0000 mon.a (mon.0) 396 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-04-06T03:15:47.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: audit 2024-04-06T03:15:46.098771+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:47.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: audit 2024-04-06T03:15:46.104550+0000 mon.a (mon.0) 398 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:47.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: audit 2024-04-06T03:15:46.705469+0000 mon.a (mon.0) 399 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:47.412 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: audit 2024-04-06T03:15:46.711226+0000 mon.a (mon.0) 400 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:47.412 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:47 smithi050 bash[29221]: audit 2024-04-06T03:15:46.719835+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:47.414 DEBUG:teuthology.orchestra.run.smithi050:osd.2> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-06T03:15:47.417 INFO:tasks.cephadm:Deploying osd.3 on smithi050 with /dev/vg_nvme/lv_1... 2024-04-06T03:15:47.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-06T03:15:47.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: cluster 2024-04-06T03:15:45.505862+0000 mgr.y (mgr.14152) 167 : cluster 0 pgmap v133: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:47.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: audit 2024-04-06T03:15:46.095444+0000 mon.a (mon.0) 395 : audit 1 from='osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:15:47.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: cluster 2024-04-06T03:15:46.098126+0000 mon.a (mon.0) 396 : cluster 0 osdmap e18: 3 total, 2 up, 3 in 2024-04-06T03:15:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: audit 2024-04-06T03:15:46.098771+0000 mon.a (mon.0) 397 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: audit 2024-04-06T03:15:46.104550+0000 mon.a (mon.0) 398 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: audit 2024-04-06T03:15:46.705469+0000 mon.a (mon.0) 399 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:15:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: audit 2024-04-06T03:15:46.711226+0000 mon.a (mon.0) 400 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:47 smithi153 bash[23852]: audit 2024-04-06T03:15:46.719835+0000 mon.a (mon.0) 401 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:48.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: cluster 2024-04-06T03:15:45.471694+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: cluster 2024-04-06T03:15:45.471770+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: audit 2024-04-06T03:15:47.102707+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: cluster 2024-04-06T03:15:47.109946+0000 mon.a (mon.0) 403 : cluster 1 osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904] boot 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: cluster 2024-04-06T03:15:47.110046+0000 mon.a (mon.0) 404 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: audit 2024-04-06T03:15:47.110836+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: cluster 2024-04-06T03:15:47.506294+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[21467]: audit 2024-04-06T03:15:47.641554+0000 mon.a (mon.0) 406 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: cluster 2024-04-06T03:15:45.471694+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: cluster 2024-04-06T03:15:45.471770+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: audit 2024-04-06T03:15:47.102707+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: cluster 2024-04-06T03:15:47.109946+0000 mon.a (mon.0) 403 : cluster 1 osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904] boot 2024-04-06T03:15:48.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: cluster 2024-04-06T03:15:47.110046+0000 mon.a (mon.0) 404 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-04-06T03:15:48.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: audit 2024-04-06T03:15:47.110836+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:48.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: cluster 2024-04-06T03:15:47.506294+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:48.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:48 smithi050 bash[29221]: audit 2024-04-06T03:15:47.641554+0000 mon.a (mon.0) 406 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:15:48.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: cluster 2024-04-06T03:15:45.471694+0000 osd.2 (osd.2) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:15:48.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: cluster 2024-04-06T03:15:45.471770+0000 osd.2 (osd.2) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:15:48.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: audit 2024-04-06T03:15:47.102707+0000 mon.a (mon.0) 402 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:48.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: cluster 2024-04-06T03:15:47.109946+0000 mon.a (mon.0) 403 : cluster 1 osd.2 [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904] boot 2024-04-06T03:15:48.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: cluster 2024-04-06T03:15:47.110046+0000 mon.a (mon.0) 404 : cluster 0 osdmap e19: 3 total, 3 up, 3 in 2024-04-06T03:15:48.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: audit 2024-04-06T03:15:47.110836+0000 mon.a (mon.0) 405 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:15:48.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: cluster 2024-04-06T03:15:47.506294+0000 mgr.y (mgr.14152) 168 : cluster 0 pgmap v136: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-06T03:15:48.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:48 smithi153 bash[23852]: audit 2024-04-06T03:15:47.641554+0000 mon.a (mon.0) 406 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:15:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:49 smithi050 bash[21467]: audit 2024-04-06T03:15:48.125047+0000 mon.a (mon.0) 407 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-06T03:15:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:49 smithi050 bash[21467]: cluster 2024-04-06T03:15:48.131163+0000 mon.a (mon.0) 408 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-04-06T03:15:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:49 smithi050 bash[21467]: audit 2024-04-06T03:15:48.132531+0000 mon.a (mon.0) 409 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:15:49.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:49 smithi050 bash[29221]: audit 2024-04-06T03:15:48.125047+0000 mon.a (mon.0) 407 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-06T03:15:49.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:49 smithi050 bash[29221]: cluster 2024-04-06T03:15:48.131163+0000 mon.a (mon.0) 408 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-04-06T03:15:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:49 smithi050 bash[29221]: audit 2024-04-06T03:15:48.132531+0000 mon.a (mon.0) 409 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:15:49.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:49 smithi153 bash[23852]: audit 2024-04-06T03:15:48.125047+0000 mon.a (mon.0) 407 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-06T03:15:49.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:49 smithi153 bash[23852]: cluster 2024-04-06T03:15:48.131163+0000 mon.a (mon.0) 408 : cluster 0 osdmap e20: 3 total, 3 up, 3 in 2024-04-06T03:15:49.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:49 smithi153 bash[23852]: audit 2024-04-06T03:15:48.132531+0000 mon.a (mon.0) 409 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:15:50.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:50 smithi050 bash[29221]: audit 2024-04-06T03:15:49.127331+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:15:50.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:50 smithi050 bash[29221]: cluster 2024-04-06T03:15:49.133467+0000 mon.a (mon.0) 411 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-04-06T03:15:50.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:50 smithi050 bash[29221]: cluster 2024-04-06T03:15:49.506765+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v139: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:50.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:50 smithi050 bash[21467]: audit 2024-04-06T03:15:49.127331+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:15:50.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:50 smithi050 bash[21467]: cluster 2024-04-06T03:15:49.133467+0000 mon.a (mon.0) 411 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-04-06T03:15:50.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:50 smithi050 bash[21467]: cluster 2024-04-06T03:15:49.506765+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v139: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:50.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:50 smithi153 bash[23852]: audit 2024-04-06T03:15:49.127331+0000 mon.a (mon.0) 410 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:15:50.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:50 smithi153 bash[23852]: cluster 2024-04-06T03:15:49.133467+0000 mon.a (mon.0) 411 : cluster 0 osdmap e21: 3 total, 3 up, 3 in 2024-04-06T03:15:50.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:50 smithi153 bash[23852]: cluster 2024-04-06T03:15:49.506765+0000 mgr.y (mgr.14152) 169 : cluster 0 pgmap v139: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:51 smithi050 bash[21467]: cluster 2024-04-06T03:15:50.156640+0000 mon.a (mon.0) 412 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-04-06T03:15:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:51 smithi050 bash[21467]: cluster 2024-04-06T03:15:50.156710+0000 mon.a (mon.0) 413 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-04-06T03:15:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:51 smithi050 bash[21467]: audit 2024-04-06T03:15:50.206740+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:51 smithi050 bash[29221]: cluster 2024-04-06T03:15:50.156640+0000 mon.a (mon.0) 412 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-04-06T03:15:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:51 smithi050 bash[29221]: cluster 2024-04-06T03:15:50.156710+0000 mon.a (mon.0) 413 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-04-06T03:15:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:51 smithi050 bash[29221]: audit 2024-04-06T03:15:50.206740+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:51.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:51 smithi153 bash[23852]: cluster 2024-04-06T03:15:50.156640+0000 mon.a (mon.0) 412 : cluster 0 mgrmap e15: y(active, since 4m), standbys: x 2024-04-06T03:15:51.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:51 smithi153 bash[23852]: cluster 2024-04-06T03:15:50.156710+0000 mon.a (mon.0) 413 : cluster 0 osdmap e22: 3 total, 3 up, 3 in 2024-04-06T03:15:51.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:51 smithi153 bash[23852]: audit 2024-04-06T03:15:50.206740+0000 mon.a (mon.0) 414 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: cluster 2024-04-06T03:15:51.507230+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v141: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.058328+0000 mon.a (mon.0) 415 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.966453+0000 mon.a (mon.0) 416 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.968374+0000 mon.a (mon.0) 417 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.968755+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.971935+0000 mon.a (mon.0) 419 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.974581+0000 mon.a (mon.0) 420 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.974738+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:54.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[29221]: audit 2024-04-06T03:15:52.974886+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: cluster 2024-04-06T03:15:51.507230+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v141: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.058328+0000 mon.a (mon.0) 415 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.966453+0000 mon.a (mon.0) 416 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.968374+0000 mon.a (mon.0) 417 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.968755+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.971935+0000 mon.a (mon.0) 419 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.974581+0000 mon.a (mon.0) 420 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.974738+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:54.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:53 smithi050 bash[21467]: audit 2024-04-06T03:15:52.974886+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:54.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: cluster 2024-04-06T03:15:51.507230+0000 mgr.y (mgr.14152) 170 : cluster 0 pgmap v141: 1 pgs: 1 active+clean; 449 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.058328+0000 mon.a (mon.0) 415 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.966453+0000 mon.a (mon.0) 416 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.968374+0000 mon.a (mon.0) 417 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.968755+0000 mon.a (mon.0) 418 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.971935+0000 mon.a (mon.0) 419 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.974581+0000 mon.a (mon.0) 420 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.974738+0000 mon.a (mon.0) 421 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:54.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:53 smithi153 bash[23852]: audit 2024-04-06T03:15:52.974886+0000 mon.a (mon.0) 422 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: audit 2024-04-06T03:15:52.973761+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: cluster 2024-04-06T03:15:53.507574+0000 mgr.y (mgr.14152) 171 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: audit 2024-04-06T03:15:53.884126+0000 mon.b (mon.1) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: audit 2024-04-06T03:15:53.946805+0000 mon.a (mon.0) 423 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: audit 2024-04-06T03:15:53.950395+0000 mon.a (mon.0) 424 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: audit 2024-04-06T03:15:53.950858+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:55.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[29221]: audit 2024-04-06T03:15:53.951229+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: audit 2024-04-06T03:15:52.973761+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: cluster 2024-04-06T03:15:53.507574+0000 mgr.y (mgr.14152) 171 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: audit 2024-04-06T03:15:53.884126+0000 mon.b (mon.1) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: audit 2024-04-06T03:15:53.946805+0000 mon.a (mon.0) 423 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: audit 2024-04-06T03:15:53.950395+0000 mon.a (mon.0) 424 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: audit 2024-04-06T03:15:53.950858+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:55.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:54 smithi050 bash[21467]: audit 2024-04-06T03:15:53.951229+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:55.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: audit 2024-04-06T03:15:52.973761+0000 mon.b (mon.1) 8 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:55.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: cluster 2024-04-06T03:15:53.507574+0000 mgr.y (mgr.14152) 171 : cluster 0 pgmap v142: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:15:55.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: audit 2024-04-06T03:15:53.884126+0000 mon.b (mon.1) 9 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:55.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: audit 2024-04-06T03:15:53.946805+0000 mon.a (mon.0) 423 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:55.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: audit 2024-04-06T03:15:53.950395+0000 mon.a (mon.0) 424 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:15:55.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: audit 2024-04-06T03:15:53.950858+0000 mon.a (mon.0) 425 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:15:55.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:54 smithi153 bash[23852]: audit 2024-04-06T03:15:53.951229+0000 mon.a (mon.0) 426 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:15:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-06T03:15:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-06T03:15:56.003 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-06T03:15:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-06T03:15:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00992283 s, 1.1 GB/s 2024-04-06T03:15:56.003 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-06T03:15:56.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:55 smithi050 bash[21467]: audit 2024-04-06T03:15:53.949866+0000 mon.c (mon.2) 12 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:56.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:55 smithi050 bash[21467]: audit 2024-04-06T03:15:54.898965+0000 mon.c (mon.2) 13 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:55 smithi050 bash[29221]: audit 2024-04-06T03:15:53.949866+0000 mon.c (mon.2) 12 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:55 smithi050 bash[29221]: audit 2024-04-06T03:15:54.898965+0000 mon.c (mon.2) 13 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:56.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:55 smithi153 bash[23852]: audit 2024-04-06T03:15:53.949866+0000 mon.c (mon.2) 12 : audit 1 from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-06T03:15:56.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:55 smithi153 bash[23852]: audit 2024-04-06T03:15:54.898965+0000 mon.c (mon.2) 13 : audit 1 from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-06T03:15:56.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi050:vg_nvme/lv_1 2024-04-06T03:15:57.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:56 smithi050 bash[21467]: cluster 2024-04-06T03:15:55.508089+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 243 KiB/s wr, 16 op/s 2024-04-06T03:15:57.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:56 smithi050 bash[29221]: cluster 2024-04-06T03:15:55.508089+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 243 KiB/s wr, 16 op/s 2024-04-06T03:15:57.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:56 smithi153 bash[23852]: cluster 2024-04-06T03:15:55.508089+0000 mgr.y (mgr.14152) 172 : cluster 0 pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 243 KiB/s wr, 16 op/s 2024-04-06T03:15:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: cephadm 2024-04-06T03:15:57.465763+0000 mgr.y (mgr.14152) 173 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:15:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.474925+0000 mon.a (mon.0) 427 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.485077+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.487780+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.489918+0000 mon.a (mon.0) 430 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.491932+0000 mon.a (mon.0) 431 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: cephadm 2024-04-06T03:15:57.493499+0000 mgr.y (mgr.14152) 174 : cephadm 1 Adjusting osd_memory_target on smithi050 to 5415M 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.500298+0000 mon.a (mon.0) 432 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.505234+0000 mon.a (mon.0) 433 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.507103+0000 mon.a (mon.0) 434 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: cluster 2024-04-06T03:15:57.508601+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 177 KiB/s wr, 12 op/s 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[21467]: audit 2024-04-06T03:15:57.517565+0000 mon.a (mon.0) 435 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: cephadm 2024-04-06T03:15:57.465763+0000 mgr.y (mgr.14152) 173 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.474925+0000 mon.a (mon.0) 427 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.485077+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.487780+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.489918+0000 mon.a (mon.0) 430 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.491932+0000 mon.a (mon.0) 431 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: cephadm 2024-04-06T03:15:57.493499+0000 mgr.y (mgr.14152) 174 : cephadm 1 Adjusting osd_memory_target on smithi050 to 5415M 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.500298+0000 mon.a (mon.0) 432 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.505234+0000 mon.a (mon.0) 433 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.507103+0000 mon.a (mon.0) 434 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: cluster 2024-04-06T03:15:57.508601+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 177 KiB/s wr, 12 op/s 2024-04-06T03:15:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:15:58 smithi050 bash[29221]: audit 2024-04-06T03:15:57.517565+0000 mon.a (mon.0) 435 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: cephadm 2024-04-06T03:15:57.465763+0000 mgr.y (mgr.14152) 173 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.474925+0000 mon.a (mon.0) 427 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.485077+0000 mon.a (mon.0) 428 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.487780+0000 mon.a (mon.0) 429 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.489918+0000 mon.a (mon.0) 430 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.491932+0000 mon.a (mon.0) 431 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: cephadm 2024-04-06T03:15:57.493499+0000 mgr.y (mgr.14152) 174 : cephadm 1 Adjusting osd_memory_target on smithi050 to 5415M 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.500298+0000 mon.a (mon.0) 432 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.505234+0000 mon.a (mon.0) 433 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.507103+0000 mon.a (mon.0) 434 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: cluster 2024-04-06T03:15:57.508601+0000 mgr.y (mgr.14152) 175 : cluster 0 pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 177 KiB/s wr, 12 op/s 2024-04-06T03:15:58.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:15:58 smithi153 bash[23852]: audit 2024-04-06T03:15:57.517565+0000 mon.a (mon.0) 435 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:00.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:00 smithi050 bash[21467]: cluster 2024-04-06T03:15:59.509083+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 139 KiB/s wr, 9 op/s 2024-04-06T03:16:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:00 smithi050 bash[29221]: cluster 2024-04-06T03:15:59.509083+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 139 KiB/s wr, 9 op/s 2024-04-06T03:16:01.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:00 smithi153 bash[23852]: cluster 2024-04-06T03:15:59.509083+0000 mgr.y (mgr.14152) 176 : cluster 0 pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 139 KiB/s wr, 9 op/s 2024-04-06T03:16:01.719 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:16:02.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:02 smithi050 bash[21467]: cluster 2024-04-06T03:16:01.509633+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 115 KiB/s wr, 7 op/s 2024-04-06T03:16:02.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:02 smithi050 bash[29221]: cluster 2024-04-06T03:16:01.509633+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 115 KiB/s wr, 7 op/s 2024-04-06T03:16:03.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:02 smithi153 bash[23852]: cluster 2024-04-06T03:16:01.509633+0000 mgr.y (mgr.14152) 177 : cluster 0 pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 115 KiB/s wr, 7 op/s 2024-04-06T03:16:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:04 smithi050 bash[21467]: cluster 2024-04-06T03:16:03.510190+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-06T03:16:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:04 smithi050 bash[29221]: cluster 2024-04-06T03:16:03.510190+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-06T03:16:05.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:04 smithi153 bash[23852]: cluster 2024-04-06T03:16:03.510190+0000 mgr.y (mgr.14152) 178 : cluster 0 pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-06T03:16:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[21467]: audit 2024-04-06T03:16:04.760722+0000 mgr.y (mgr.14152) 179 : audit 0 from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:16:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[21467]: audit 2024-04-06T03:16:04.767661+0000 mon.a (mon.0) 436 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:16:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[21467]: audit 2024-04-06T03:16:04.772399+0000 mon.a (mon.0) 437 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:16:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[21467]: audit 2024-04-06T03:16:04.773527+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[29221]: audit 2024-04-06T03:16:04.760722+0000 mgr.y (mgr.14152) 179 : audit 0 from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:16:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[29221]: audit 2024-04-06T03:16:04.767661+0000 mon.a (mon.0) 436 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:16:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[29221]: audit 2024-04-06T03:16:04.772399+0000 mon.a (mon.0) 437 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:16:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:05 smithi050 bash[29221]: audit 2024-04-06T03:16:04.773527+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:06.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:05 smithi153 bash[23852]: audit 2024-04-06T03:16:04.760722+0000 mgr.y (mgr.14152) 179 : audit 0 from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:16:06.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:05 smithi153 bash[23852]: audit 2024-04-06T03:16:04.767661+0000 mon.a (mon.0) 436 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:16:06.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:05 smithi153 bash[23852]: audit 2024-04-06T03:16:04.772399+0000 mon.a (mon.0) 437 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:16:06.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:05 smithi153 bash[23852]: audit 2024-04-06T03:16:04.773527+0000 mon.a (mon.0) 438 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:06.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:06 smithi050 bash[21467]: cluster 2024-04-06T03:16:05.510775+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-06T03:16:06.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:06 smithi050 bash[29221]: cluster 2024-04-06T03:16:05.510775+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-06T03:16:07.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:06 smithi153 bash[23852]: cluster 2024-04-06T03:16:05.510775+0000 mgr.y (mgr.14152) 180 : cluster 0 pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2024-04-06T03:16:08.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:08 smithi050 bash[21467]: cluster 2024-04-06T03:16:07.511426+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:08.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:08 smithi050 bash[29221]: cluster 2024-04-06T03:16:07.511426+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:09.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:08 smithi153 bash[23852]: cluster 2024-04-06T03:16:07.511426+0000 mgr.y (mgr.14152) 181 : cluster 0 pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:09.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:09 smithi050 bash[21467]: cluster 2024-04-06T03:16:09.307923+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:09.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:09 smithi050 bash[21467]: cluster 2024-04-06T03:16:09.308126+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:09.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:09 smithi050 bash[29221]: cluster 2024-04-06T03:16:09.307923+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:09.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:09 smithi050 bash[29221]: cluster 2024-04-06T03:16:09.308126+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:10.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:09 smithi153 bash[23852]: cluster 2024-04-06T03:16:09.307923+0000 osd.1 (osd.1) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:10.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:09 smithi153 bash[23852]: cluster 2024-04-06T03:16:09.308126+0000 osd.1 (osd.1) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:10.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:10 smithi050 bash[21467]: cluster 2024-04-06T03:16:09.512009+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:10.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:10 smithi050 bash[21467]: cluster 2024-04-06T03:16:09.600313+0000 mon.a (mon.0) 439 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-04-06T03:16:10.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:10 smithi050 bash[29221]: cluster 2024-04-06T03:16:09.512009+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:10.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:10 smithi050 bash[29221]: cluster 2024-04-06T03:16:09.600313+0000 mon.a (mon.0) 439 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-04-06T03:16:11.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:10 smithi153 bash[23852]: cluster 2024-04-06T03:16:09.512009+0000 mgr.y (mgr.14152) 182 : cluster 0 pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:11.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:10 smithi153 bash[23852]: cluster 2024-04-06T03:16:09.600313+0000 mon.a (mon.0) 439 : cluster 0 osdmap e23: 3 total, 3 up, 3 in 2024-04-06T03:16:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:12 smithi050 bash[21467]: cluster 2024-04-06T03:16:11.512620+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:12.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:12 smithi050 bash[29221]: cluster 2024-04-06T03:16:11.512620+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:12 smithi153 bash[23852]: cluster 2024-04-06T03:16:11.512620+0000 mgr.y (mgr.14152) 183 : cluster 0 pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:13.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[29221]: audit 2024-04-06T03:16:13.409166+0000 mon.a (mon.0) 440 : audit 1 from='client.? 172.21.15.50:0/202641457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240"}]: dispatch 2024-04-06T03:16:13.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[29221]: audit 2024-04-06T03:16:13.412606+0000 mon.a (mon.0) 441 : audit 1 from='client.? 172.21.15.50:0/202641457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240"}]': finished 2024-04-06T03:16:13.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[29221]: cluster 2024-04-06T03:16:13.418035+0000 mon.a (mon.0) 442 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-04-06T03:16:13.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[29221]: audit 2024-04-06T03:16:13.418440+0000 mon.a (mon.0) 443 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[21467]: audit 2024-04-06T03:16:13.409166+0000 mon.a (mon.0) 440 : audit 1 from='client.? 172.21.15.50:0/202641457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240"}]: dispatch 2024-04-06T03:16:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[21467]: audit 2024-04-06T03:16:13.412606+0000 mon.a (mon.0) 441 : audit 1 from='client.? 172.21.15.50:0/202641457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240"}]': finished 2024-04-06T03:16:13.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[21467]: cluster 2024-04-06T03:16:13.418035+0000 mon.a (mon.0) 442 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-04-06T03:16:13.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:13 smithi050 bash[21467]: audit 2024-04-06T03:16:13.418440+0000 mon.a (mon.0) 443 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:14.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:13 smithi153 bash[23852]: audit 2024-04-06T03:16:13.409166+0000 mon.a (mon.0) 440 : audit 1 from='client.? 172.21.15.50:0/202641457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240"}]: dispatch 2024-04-06T03:16:14.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:13 smithi153 bash[23852]: audit 2024-04-06T03:16:13.412606+0000 mon.a (mon.0) 441 : audit 1 from='client.? 172.21.15.50:0/202641457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240"}]': finished 2024-04-06T03:16:14.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:13 smithi153 bash[23852]: cluster 2024-04-06T03:16:13.418035+0000 mon.a (mon.0) 442 : cluster 0 osdmap e24: 4 total, 3 up, 4 in 2024-04-06T03:16:14.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:13 smithi153 bash[23852]: audit 2024-04-06T03:16:13.418440+0000 mon.a (mon.0) 443 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:14.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:14 smithi050 bash[29221]: cluster 2024-04-06T03:16:13.513040+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:14.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:14 smithi050 bash[29221]: audit 2024-04-06T03:16:14.160713+0000 mon.a (mon.0) 444 : audit 0 from='client.? 172.21.15.50:0/1677554074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:16:14.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:14 smithi050 bash[21467]: cluster 2024-04-06T03:16:13.513040+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:14.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:14 smithi050 bash[21467]: audit 2024-04-06T03:16:14.160713+0000 mon.a (mon.0) 444 : audit 0 from='client.? 172.21.15.50:0/1677554074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:16:15.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:14 smithi153 bash[23852]: cluster 2024-04-06T03:16:13.513040+0000 mgr.y (mgr.14152) 184 : cluster 0 pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:15.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:14 smithi153 bash[23852]: audit 2024-04-06T03:16:14.160713+0000 mon.a (mon.0) 444 : audit 0 from='client.? 172.21.15.50:0/1677554074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:16:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:16 smithi050 bash[21467]: cluster 2024-04-06T03:16:15.513553+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:16.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:16 smithi050 bash[29221]: cluster 2024-04-06T03:16:15.513553+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:17.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:16 smithi153 bash[23852]: cluster 2024-04-06T03:16:15.513553+0000 mgr.y (mgr.14152) 185 : cluster 0 pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:19.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:18 smithi050 bash[29221]: cluster 2024-04-06T03:16:17.514021+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:19.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:18 smithi050 bash[21467]: cluster 2024-04-06T03:16:17.514021+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:19.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:18 smithi153 bash[23852]: cluster 2024-04-06T03:16:17.514021+0000 mgr.y (mgr.14152) 186 : cluster 0 pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:21.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:20 smithi050 bash[29221]: cluster 2024-04-06T03:16:19.514652+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:21.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:20 smithi050 bash[21467]: cluster 2024-04-06T03:16:19.514652+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:21.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:20 smithi153 bash[23852]: cluster 2024-04-06T03:16:19.514652+0000 mgr.y (mgr.14152) 187 : cluster 0 pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:23.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:22 smithi050 bash[21467]: cluster 2024-04-06T03:16:21.515226+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:23.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:22 smithi050 bash[29221]: cluster 2024-04-06T03:16:21.515226+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:23.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:22 smithi153 bash[23852]: cluster 2024-04-06T03:16:21.515226+0000 mgr.y (mgr.14152) 188 : cluster 0 pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:25.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:24 smithi050 bash[29221]: cluster 2024-04-06T03:16:23.515701+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:25.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:24 smithi050 bash[21467]: cluster 2024-04-06T03:16:23.515701+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:25.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:24 smithi153 bash[23852]: cluster 2024-04-06T03:16:23.515701+0000 mgr.y (mgr.14152) 189 : cluster 0 pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:27.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:26 smithi153 bash[23852]: cluster 2024-04-06T03:16:25.516061+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:27.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:26 smithi153 bash[23852]: audit 2024-04-06T03:16:26.288340+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:16:27.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:26 smithi153 bash[23852]: audit 2024-04-06T03:16:26.290159+0000 mon.a (mon.0) 446 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:26 smithi050 bash[21467]: cluster 2024-04-06T03:16:25.516061+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:26 smithi050 bash[21467]: audit 2024-04-06T03:16:26.288340+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:16:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:26 smithi050 bash[21467]: audit 2024-04-06T03:16:26.290159+0000 mon.a (mon.0) 446 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:27.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:26 smithi050 bash[29221]: cluster 2024-04-06T03:16:25.516061+0000 mgr.y (mgr.14152) 190 : cluster 0 pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:27.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:26 smithi050 bash[29221]: audit 2024-04-06T03:16:26.288340+0000 mon.a (mon.0) 445 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-06T03:16:27.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:26 smithi050 bash[29221]: audit 2024-04-06T03:16:26.290159+0000 mon.a (mon.0) 446 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:28.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:27 smithi153 bash[23852]: cephadm 2024-04-06T03:16:26.291835+0000 mgr.y (mgr.14152) 191 : cephadm 1 Deploying daemon osd.3 on smithi050 2024-04-06T03:16:28.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:28 smithi050 bash[21467]: cephadm 2024-04-06T03:16:26.291835+0000 mgr.y (mgr.14152) 191 : cephadm 1 Deploying daemon osd.3 on smithi050 2024-04-06T03:16:28.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:28 smithi050 bash[29221]: cephadm 2024-04-06T03:16:26.291835+0000 mgr.y (mgr.14152) 191 : cephadm 1 Deploying daemon osd.3 on smithi050 2024-04-06T03:16:29.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:29 smithi153 bash[23852]: cluster 2024-04-06T03:16:27.516574+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:29.425 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:29 smithi050 bash[21467]: cluster 2024-04-06T03:16:27.516574+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:29.426 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:29 smithi050 bash[29221]: cluster 2024-04-06T03:16:27.516574+0000 mgr.y (mgr.14152) 192 : cluster 0 pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:30.873 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:16:30 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:30.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:30 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:30.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:30 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:30.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:16:30 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:30.875 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:16:30 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:30.875 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:16:30 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:31 smithi050 bash[21467]: cluster 2024-04-06T03:16:29.517086+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:31 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.134 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:16:31 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:31 smithi050 bash[29221]: cluster 2024-04-06T03:16:29.517086+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:31.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:31 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.135 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:16:31 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.135 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:16:31 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.136 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:16:31 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:16:31.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:31 smithi153 bash[23852]: cluster 2024-04-06T03:16:29.517086+0000 mgr.y (mgr.14152) 193 : cluster 0 pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:32.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[21467]: audit 2024-04-06T03:16:31.186399+0000 mon.a (mon.0) 447 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:32.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[21467]: audit 2024-04-06T03:16:31.189007+0000 mon.a (mon.0) 448 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:32.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[21467]: audit 2024-04-06T03:16:31.192310+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:32.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[21467]: cluster 2024-04-06T03:16:31.517484+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:32.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[29221]: audit 2024-04-06T03:16:31.186399+0000 mon.a (mon.0) 447 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:32.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[29221]: audit 2024-04-06T03:16:31.189007+0000 mon.a (mon.0) 448 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:32.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[29221]: audit 2024-04-06T03:16:31.192310+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:32.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:32 smithi050 bash[29221]: cluster 2024-04-06T03:16:31.517484+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:32.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:32 smithi153 bash[23852]: audit 2024-04-06T03:16:31.186399+0000 mon.a (mon.0) 447 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:32.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:32 smithi153 bash[23852]: audit 2024-04-06T03:16:31.189007+0000 mon.a (mon.0) 448 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:32.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:32 smithi153 bash[23852]: audit 2024-04-06T03:16:31.192310+0000 mon.a (mon.0) 449 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:32.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:32 smithi153 bash[23852]: cluster 2024-04-06T03:16:31.517484+0000 mgr.y (mgr.14152) 194 : cluster 0 pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:34.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[21467]: cluster 2024-04-06T03:16:33.190586+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:34.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[21467]: cluster 2024-04-06T03:16:33.190700+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:34.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[21467]: cluster 2024-04-06T03:16:33.201598+0000 mon.a (mon.0) 450 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-04-06T03:16:34.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[21467]: audit 2024-04-06T03:16:33.201890+0000 mon.a (mon.0) 451 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:34.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[21467]: cluster 2024-04-06T03:16:33.518072+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:34.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[29221]: cluster 2024-04-06T03:16:33.190586+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:34.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[29221]: cluster 2024-04-06T03:16:33.190700+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:34.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[29221]: cluster 2024-04-06T03:16:33.201598+0000 mon.a (mon.0) 450 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-04-06T03:16:34.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[29221]: audit 2024-04-06T03:16:33.201890+0000 mon.a (mon.0) 451 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:34.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:34 smithi050 bash[29221]: cluster 2024-04-06T03:16:33.518072+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:34.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:34 smithi153 bash[23852]: cluster 2024-04-06T03:16:33.190586+0000 osd.0 (osd.0) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:34 smithi153 bash[23852]: cluster 2024-04-06T03:16:33.190700+0000 osd.0 (osd.0) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:34 smithi153 bash[23852]: cluster 2024-04-06T03:16:33.201598+0000 mon.a (mon.0) 450 : cluster 0 osdmap e25: 4 total, 3 up, 4 in 2024-04-06T03:16:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:34 smithi153 bash[23852]: audit 2024-04-06T03:16:33.201890+0000 mon.a (mon.0) 451 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:34 smithi153 bash[23852]: cluster 2024-04-06T03:16:33.518072+0000 mgr.y (mgr.14152) 195 : cluster 0 pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:36.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:36 smithi050 bash[21467]: cluster 2024-04-06T03:16:35.518707+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:36.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:36 smithi050 bash[29221]: cluster 2024-04-06T03:16:35.518707+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:37.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:36 smithi153 bash[23852]: cluster 2024-04-06T03:16:35.518707+0000 mgr.y (mgr.14152) 196 : cluster 0 pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[21467]: audit 2024-04-06T03:16:36.581394+0000 mon.a (mon.0) 452 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[21467]: audit 2024-04-06T03:16:36.586978+0000 mon.a (mon.0) 453 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[21467]: audit 2024-04-06T03:16:36.588553+0000 mon.a (mon.0) 454 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[21467]: audit 2024-04-06T03:16:36.589476+0000 mon.a (mon.0) 455 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[21467]: audit 2024-04-06T03:16:36.594443+0000 mon.a (mon.0) 456 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:37.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[29221]: audit 2024-04-06T03:16:36.581394+0000 mon.a (mon.0) 452 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[29221]: audit 2024-04-06T03:16:36.586978+0000 mon.a (mon.0) 453 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[29221]: audit 2024-04-06T03:16:36.588553+0000 mon.a (mon.0) 454 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[29221]: audit 2024-04-06T03:16:36.589476+0000 mon.a (mon.0) 455 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:37 smithi050 bash[29221]: audit 2024-04-06T03:16:36.594443+0000 mon.a (mon.0) 456 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:38.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:37 smithi153 bash[23852]: audit 2024-04-06T03:16:36.581394+0000 mon.a (mon.0) 452 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:38.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:37 smithi153 bash[23852]: audit 2024-04-06T03:16:36.586978+0000 mon.a (mon.0) 453 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:38.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:37 smithi153 bash[23852]: audit 2024-04-06T03:16:36.588553+0000 mon.a (mon.0) 454 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:38.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:37 smithi153 bash[23852]: audit 2024-04-06T03:16:36.589476+0000 mon.a (mon.0) 455 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:38.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:37 smithi153 bash[23852]: audit 2024-04-06T03:16:36.594443+0000 mon.a (mon.0) 456 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:39.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:38 smithi153 bash[23852]: cluster 2024-04-06T03:16:37.519172+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:39.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:38 smithi153 bash[23852]: audit 2024-04-06T03:16:38.109822+0000 mon.c (mon.2) 14 : audit 1 from='osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:16:39.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:38 smithi153 bash[23852]: audit 2024-04-06T03:16:38.110146+0000 mon.a (mon.0) 457 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:16:39.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:39 smithi050 bash[21467]: cluster 2024-04-06T03:16:37.519172+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:39 smithi050 bash[21467]: audit 2024-04-06T03:16:38.109822+0000 mon.c (mon.2) 14 : audit 1 from='osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:16:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:39 smithi050 bash[21467]: audit 2024-04-06T03:16:38.110146+0000 mon.a (mon.0) 457 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:39 smithi050 bash[29221]: cluster 2024-04-06T03:16:37.519172+0000 mgr.y (mgr.14152) 197 : cluster 0 pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:39 smithi050 bash[29221]: audit 2024-04-06T03:16:38.109822+0000 mon.c (mon.2) 14 : audit 1 from='osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:16:39.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:39 smithi050 bash[29221]: audit 2024-04-06T03:16:38.110146+0000 mon.a (mon.0) 457 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-06T03:16:40.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:40 smithi153 bash[23852]: audit 2024-04-06T03:16:38.991180+0000 mon.a (mon.0) 458 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:16:40.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:40 smithi153 bash[23852]: cluster 2024-04-06T03:16:38.997529+0000 mon.a (mon.0) 459 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-04-06T03:16:40.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:40 smithi153 bash[23852]: audit 2024-04-06T03:16:38.998608+0000 mon.a (mon.0) 460 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:40.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:40 smithi153 bash[23852]: audit 2024-04-06T03:16:38.999017+0000 mon.c (mon.2) 15 : audit 1 from='osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:16:40.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:40 smithi153 bash[23852]: audit 2024-04-06T03:16:39.006694+0000 mon.a (mon.0) 461 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:16:40.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[21467]: audit 2024-04-06T03:16:38.991180+0000 mon.a (mon.0) 458 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:16:40.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[21467]: cluster 2024-04-06T03:16:38.997529+0000 mon.a (mon.0) 459 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-04-06T03:16:40.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[21467]: audit 2024-04-06T03:16:38.998608+0000 mon.a (mon.0) 460 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:40.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[21467]: audit 2024-04-06T03:16:38.999017+0000 mon.c (mon.2) 15 : audit 1 from='osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:16:40.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[21467]: audit 2024-04-06T03:16:39.006694+0000 mon.a (mon.0) 461 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:16:40.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[29221]: audit 2024-04-06T03:16:38.991180+0000 mon.a (mon.0) 458 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-06T03:16:40.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[29221]: cluster 2024-04-06T03:16:38.997529+0000 mon.a (mon.0) 459 : cluster 0 osdmap e26: 4 total, 3 up, 4 in 2024-04-06T03:16:40.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[29221]: audit 2024-04-06T03:16:38.998608+0000 mon.a (mon.0) 460 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:40.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[29221]: audit 2024-04-06T03:16:38.999017+0000 mon.c (mon.2) 15 : audit 1 from='osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:16:40.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:40 smithi050 bash[29221]: audit 2024-04-06T03:16:39.006694+0000 mon.a (mon.0) 461 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-06T03:16:40.524 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 3 on host 'smithi050' 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: cluster 2024-04-06T03:16:39.519538+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: audit 2024-04-06T03:16:40.008052+0000 mon.a (mon.0) 462 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: cluster 2024-04-06T03:16:40.015368+0000 mon.a (mon.0) 463 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: audit 2024-04-06T03:16:40.016734+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: audit 2024-04-06T03:16:40.023774+0000 mon.a (mon.0) 465 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: audit 2024-04-06T03:16:40.494390+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: audit 2024-04-06T03:16:40.504392+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:41 smithi153 bash[23852]: audit 2024-04-06T03:16:40.534789+0000 mon.a (mon.0) 468 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:41.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: cluster 2024-04-06T03:16:39.519538+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: audit 2024-04-06T03:16:40.008052+0000 mon.a (mon.0) 462 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: cluster 2024-04-06T03:16:40.015368+0000 mon.a (mon.0) 463 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: audit 2024-04-06T03:16:40.016734+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: audit 2024-04-06T03:16:40.023774+0000 mon.a (mon.0) 465 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: audit 2024-04-06T03:16:40.494390+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: audit 2024-04-06T03:16:40.504392+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[21467]: audit 2024-04-06T03:16:40.534789+0000 mon.a (mon.0) 468 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: cluster 2024-04-06T03:16:39.519538+0000 mgr.y (mgr.14152) 198 : cluster 0 pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: audit 2024-04-06T03:16:40.008052+0000 mon.a (mon.0) 462 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: cluster 2024-04-06T03:16:40.015368+0000 mon.a (mon.0) 463 : cluster 0 osdmap e27: 4 total, 3 up, 4 in 2024-04-06T03:16:41.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: audit 2024-04-06T03:16:40.016734+0000 mon.a (mon.0) 464 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:41.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: audit 2024-04-06T03:16:40.023774+0000 mon.a (mon.0) 465 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:41.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: audit 2024-04-06T03:16:40.494390+0000 mon.a (mon.0) 466 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:41.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: audit 2024-04-06T03:16:40.504392+0000 mon.a (mon.0) 467 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:41.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:41 smithi050 bash[29221]: audit 2024-04-06T03:16:40.534789+0000 mon.a (mon.0) 468 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:16:41.313 DEBUG:teuthology.orchestra.run.smithi050:osd.3> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-06T03:16:41.316 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-06T03:16:41.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-06T03:16:42.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:42 smithi153 bash[23852]: cluster 2024-04-06T03:16:39.084203+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:42.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:42 smithi153 bash[23852]: cluster 2024-04-06T03:16:39.084343+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:42.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:42 smithi153 bash[23852]: cluster 2024-04-06T03:16:41.014174+0000 mon.a (mon.0) 469 : cluster 1 osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508] boot 2024-04-06T03:16:42.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:42 smithi153 bash[23852]: cluster 2024-04-06T03:16:41.014263+0000 mon.a (mon.0) 470 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-04-06T03:16:42.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:42 smithi153 bash[23852]: audit 2024-04-06T03:16:41.014718+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:42.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[21467]: cluster 2024-04-06T03:16:39.084203+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:42.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[21467]: cluster 2024-04-06T03:16:39.084343+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:42.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[21467]: cluster 2024-04-06T03:16:41.014174+0000 mon.a (mon.0) 469 : cluster 1 osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508] boot 2024-04-06T03:16:42.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[21467]: cluster 2024-04-06T03:16:41.014263+0000 mon.a (mon.0) 470 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-04-06T03:16:42.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[21467]: audit 2024-04-06T03:16:41.014718+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[29221]: cluster 2024-04-06T03:16:39.084203+0000 osd.3 (osd.3) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:16:42.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[29221]: cluster 2024-04-06T03:16:39.084343+0000 osd.3 (osd.3) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:16:42.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[29221]: cluster 2024-04-06T03:16:41.014174+0000 mon.a (mon.0) 469 : cluster 1 osd.3 [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508] boot 2024-04-06T03:16:42.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[29221]: cluster 2024-04-06T03:16:41.014263+0000 mon.a (mon.0) 470 : cluster 0 osdmap e28: 4 total, 4 up, 4 in 2024-04-06T03:16:42.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:42 smithi050 bash[29221]: audit 2024-04-06T03:16:41.014718+0000 mon.a (mon.0) 471 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:16:43.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:43 smithi153 bash[23852]: cluster 2024-04-06T03:16:41.520102+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:43.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:43 smithi153 bash[23852]: cluster 2024-04-06T03:16:42.032107+0000 mon.a (mon.0) 472 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-04-06T03:16:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:43 smithi050 bash[21467]: cluster 2024-04-06T03:16:41.520102+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:43 smithi050 bash[21467]: cluster 2024-04-06T03:16:42.032107+0000 mon.a (mon.0) 472 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-04-06T03:16:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:43 smithi050 bash[29221]: cluster 2024-04-06T03:16:41.520102+0000 mgr.y (mgr.14152) 199 : cluster 0 pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:43 smithi050 bash[29221]: cluster 2024-04-06T03:16:42.032107+0000 mon.a (mon.0) 472 : cluster 0 osdmap e29: 4 total, 4 up, 4 in 2024-04-06T03:16:45.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:45 smithi153 bash[23852]: cluster 2024-04-06T03:16:43.520513+0000 mgr.y (mgr.14152) 200 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:45 smithi050 bash[21467]: cluster 2024-04-06T03:16:43.520513+0000 mgr.y (mgr.14152) 200 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:45.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:45 smithi050 bash[29221]: cluster 2024-04-06T03:16:43.520513+0000 mgr.y (mgr.14152) 200 : cluster 0 pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:46.024 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:16:47.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:47 smithi153 bash[23852]: cluster 2024-04-06T03:16:45.521005+0000 mgr.y (mgr.14152) 201 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:47 smithi050 bash[21467]: cluster 2024-04-06T03:16:45.521005+0000 mgr.y (mgr.14152) 201 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:47.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:47 smithi050 bash[29221]: cluster 2024-04-06T03:16:45.521005+0000 mgr.y (mgr.14152) 201 : cluster 0 pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:48.894 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-06T03:16:48.894 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-06T03:16:48.894 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-06T03:16:48.894 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-06T03:16:48.895 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136287 s, 769 MB/s 2024-04-06T03:16:48.895 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-06T03:16:49.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:49 smithi153 bash[23852]: cluster 2024-04-06T03:16:47.521551+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:49.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:49 smithi050 bash[21467]: cluster 2024-04-06T03:16:47.521551+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:49.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:49 smithi050 bash[29221]: cluster 2024-04-06T03:16:47.521551+0000 mgr.y (mgr.14152) 202 : cluster 0 pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:49.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: cluster 2024-04-06T03:16:49.522075+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.235959+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.244570+0000 mon.a (mon.0) 474 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.246567+0000 mon.a (mon.0) 475 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.248166+0000 mon.a (mon.0) 476 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.249560+0000 mon.a (mon.0) 477 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.250999+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.257577+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.261023+0000 mon.a (mon.0) 480 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:51.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.263277+0000 mon.a (mon.0) 481 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:51.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:51 smithi153 bash[23852]: audit 2024-04-06T03:16:50.271494+0000 mon.a (mon.0) 482 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: cluster 2024-04-06T03:16:49.522075+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.235959+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.244570+0000 mon.a (mon.0) 474 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.246567+0000 mon.a (mon.0) 475 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.248166+0000 mon.a (mon.0) 476 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.249560+0000 mon.a (mon.0) 477 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.250999+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.257577+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.261023+0000 mon.a (mon.0) 480 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.263277+0000 mon.a (mon.0) 481 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[21467]: audit 2024-04-06T03:16:50.271494+0000 mon.a (mon.0) 482 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: cluster 2024-04-06T03:16:49.522075+0000 mgr.y (mgr.14152) 203 : cluster 0 pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.235959+0000 mon.a (mon.0) 473 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.244570+0000 mon.a (mon.0) 474 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.246567+0000 mon.a (mon.0) 475 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.248166+0000 mon.a (mon.0) 476 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.249560+0000 mon.a (mon.0) 477 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.250999+0000 mon.a (mon.0) 478 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.257577+0000 mon.a (mon.0) 479 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:51.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.261023+0000 mon.a (mon.0) 480 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.263277+0000 mon.a (mon.0) 481 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:16:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:51 smithi050 bash[29221]: audit 2024-04-06T03:16:50.271494+0000 mon.a (mon.0) 482 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:16:52.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:52 smithi050 bash[21467]: cephadm 2024-04-06T03:16:50.227720+0000 mgr.y (mgr.14152) 204 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:16:52.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:52 smithi050 bash[21467]: cephadm 2024-04-06T03:16:50.252131+0000 mgr.y (mgr.14152) 205 : cephadm 1 Adjusting osd_memory_target on smithi050 to 4061M 2024-04-06T03:16:52.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:52 smithi050 bash[29221]: cephadm 2024-04-06T03:16:50.227720+0000 mgr.y (mgr.14152) 204 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:16:52.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:52 smithi050 bash[29221]: cephadm 2024-04-06T03:16:50.252131+0000 mgr.y (mgr.14152) 205 : cephadm 1 Adjusting osd_memory_target on smithi050 to 4061M 2024-04-06T03:16:52.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:52 smithi153 bash[23852]: cephadm 2024-04-06T03:16:50.227720+0000 mgr.y (mgr.14152) 204 : cephadm 1 Detected new or changed devices on smithi050 2024-04-06T03:16:52.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:52 smithi153 bash[23852]: cephadm 2024-04-06T03:16:50.252131+0000 mgr.y (mgr.14152) 205 : cephadm 1 Adjusting osd_memory_target on smithi050 to 4061M 2024-04-06T03:16:53.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:53 smithi050 bash[21467]: cluster 2024-04-06T03:16:51.522597+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:53.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:53 smithi050 bash[29221]: cluster 2024-04-06T03:16:51.522597+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:53.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:53 smithi153 bash[23852]: cluster 2024-04-06T03:16:51.522597+0000 mgr.y (mgr.14152) 206 : cluster 0 pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:54.270 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:16:55.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:55 smithi153 bash[23852]: cluster 2024-04-06T03:16:53.523171+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:55 smithi050 bash[21467]: cluster 2024-04-06T03:16:53.523171+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:55 smithi050 bash[29221]: cluster 2024-04-06T03:16:53.523171+0000 mgr.y (mgr.14152) 207 : cluster 0 pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[21467]: cluster 2024-04-06T03:16:55.523715+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[21467]: audit 2024-04-06T03:16:56.880493+0000 mon.a (mon.0) 483 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:16:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[21467]: audit 2024-04-06T03:16:56.887104+0000 mon.a (mon.0) 484 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:16:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[21467]: audit 2024-04-06T03:16:56.888858+0000 mon.a (mon.0) 485 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:57.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[29221]: cluster 2024-04-06T03:16:55.523715+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[29221]: audit 2024-04-06T03:16:56.880493+0000 mon.a (mon.0) 483 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:16:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[29221]: audit 2024-04-06T03:16:56.887104+0000 mon.a (mon.0) 484 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:16:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:57 smithi050 bash[29221]: audit 2024-04-06T03:16:56.888858+0000 mon.a (mon.0) 485 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:57.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:57 smithi153 bash[23852]: cluster 2024-04-06T03:16:55.523715+0000 mgr.y (mgr.14152) 208 : cluster 0 pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:57.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:57 smithi153 bash[23852]: audit 2024-04-06T03:16:56.880493+0000 mon.a (mon.0) 483 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:16:57.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:57 smithi153 bash[23852]: audit 2024-04-06T03:16:56.887104+0000 mon.a (mon.0) 484 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:16:57.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:57 smithi153 bash[23852]: audit 2024-04-06T03:16:56.888858+0000 mon.a (mon.0) 485 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:16:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:58 smithi050 bash[21467]: audit 2024-04-06T03:16:56.872880+0000 mgr.y (mgr.14152) 209 : audit 0 from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:16:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:58 smithi050 bash[29221]: audit 2024-04-06T03:16:56.872880+0000 mgr.y (mgr.14152) 209 : audit 0 from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:16:58.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:58 smithi153 bash[23852]: audit 2024-04-06T03:16:56.872880+0000 mgr.y (mgr.14152) 209 : audit 0 from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:16:59.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:16:59 smithi050 bash[21467]: cluster 2024-04-06T03:16:57.524261+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:59.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:16:59 smithi050 bash[29221]: cluster 2024-04-06T03:16:57.524261+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:16:59.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:16:59 smithi153 bash[23852]: cluster 2024-04-06T03:16:57.524261+0000 mgr.y (mgr.14152) 210 : cluster 0 pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:01.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:01 smithi050 bash[21467]: cluster 2024-04-06T03:16:59.524789+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:01.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:01 smithi050 bash[29221]: cluster 2024-04-06T03:16:59.524789+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:01.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:01 smithi153 bash[23852]: cluster 2024-04-06T03:16:59.524789+0000 mgr.y (mgr.14152) 211 : cluster 0 pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:03.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:03 smithi153 bash[23852]: cluster 2024-04-06T03:17:01.525353+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:03.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:03 smithi050 bash[21467]: cluster 2024-04-06T03:17:01.525353+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:03.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:03 smithi050 bash[29221]: cluster 2024-04-06T03:17:01.525353+0000 mgr.y (mgr.14152) 212 : cluster 0 pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:04.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:04 smithi050 bash[21467]: cluster 2024-04-06T03:17:02.805565+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:04.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:04 smithi050 bash[21467]: cluster 2024-04-06T03:17:02.805684+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:04.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:04 smithi050 bash[21467]: cluster 2024-04-06T03:17:03.094442+0000 mon.a (mon.0) 486 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-04-06T03:17:04.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:04 smithi050 bash[29221]: cluster 2024-04-06T03:17:02.805565+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:04.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:04 smithi050 bash[29221]: cluster 2024-04-06T03:17:02.805684+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:04.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:04 smithi050 bash[29221]: cluster 2024-04-06T03:17:03.094442+0000 mon.a (mon.0) 486 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-04-06T03:17:04.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:04 smithi153 bash[23852]: cluster 2024-04-06T03:17:02.805565+0000 osd.2 (osd.2) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:04.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:04 smithi153 bash[23852]: cluster 2024-04-06T03:17:02.805684+0000 osd.2 (osd.2) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:04.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:04 smithi153 bash[23852]: cluster 2024-04-06T03:17:03.094442+0000 mon.a (mon.0) 486 : cluster 0 osdmap e30: 4 total, 4 up, 4 in 2024-04-06T03:17:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[21467]: cluster 2024-04-06T03:17:03.525888+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[21467]: audit 2024-04-06T03:17:05.059102+0000 mon.b (mon.1) 10 : audit 1 from='client.? 172.21.15.153:0/2578842413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]: dispatch 2024-04-06T03:17:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[21467]: audit 2024-04-06T03:17:05.061292+0000 mon.a (mon.0) 487 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]: dispatch 2024-04-06T03:17:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[21467]: audit 2024-04-06T03:17:05.064734+0000 mon.a (mon.0) 488 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]': finished 2024-04-06T03:17:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[21467]: cluster 2024-04-06T03:17:05.070688+0000 mon.a (mon.0) 489 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[21467]: audit 2024-04-06T03:17:05.071291+0000 mon.a (mon.0) 490 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[29221]: cluster 2024-04-06T03:17:03.525888+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[29221]: audit 2024-04-06T03:17:05.059102+0000 mon.b (mon.1) 10 : audit 1 from='client.? 172.21.15.153:0/2578842413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]: dispatch 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[29221]: audit 2024-04-06T03:17:05.061292+0000 mon.a (mon.0) 487 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]: dispatch 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[29221]: audit 2024-04-06T03:17:05.064734+0000 mon.a (mon.0) 488 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]': finished 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[29221]: cluster 2024-04-06T03:17:05.070688+0000 mon.a (mon.0) 489 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-04-06T03:17:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:05 smithi050 bash[29221]: audit 2024-04-06T03:17:05.071291+0000 mon.a (mon.0) 490 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:05.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:05 smithi153 bash[23852]: cluster 2024-04-06T03:17:03.525888+0000 mgr.y (mgr.14152) 213 : cluster 0 pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:05.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:05 smithi153 bash[23852]: audit 2024-04-06T03:17:05.059102+0000 mon.b (mon.1) 10 : audit 1 from='client.? 172.21.15.153:0/2578842413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]: dispatch 2024-04-06T03:17:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:05 smithi153 bash[23852]: audit 2024-04-06T03:17:05.061292+0000 mon.a (mon.0) 487 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]: dispatch 2024-04-06T03:17:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:05 smithi153 bash[23852]: audit 2024-04-06T03:17:05.064734+0000 mon.a (mon.0) 488 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48"}]': finished 2024-04-06T03:17:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:05 smithi153 bash[23852]: cluster 2024-04-06T03:17:05.070688+0000 mon.a (mon.0) 489 : cluster 0 osdmap e31: 5 total, 4 up, 5 in 2024-04-06T03:17:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:05 smithi153 bash[23852]: audit 2024-04-06T03:17:05.071291+0000 mon.a (mon.0) 490 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:06.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:06 smithi050 bash[21467]: audit 2024-04-06T03:17:05.772606+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.153:0/3095086175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:17:06.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:06 smithi050 bash[29221]: audit 2024-04-06T03:17:05.772606+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.153:0/3095086175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:17:06.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:06 smithi153 bash[23852]: audit 2024-04-06T03:17:05.772606+0000 mon.b (mon.1) 11 : audit 0 from='client.? 172.21.15.153:0/3095086175' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:17:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:07 smithi050 bash[21467]: cluster 2024-04-06T03:17:05.526442+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:07 smithi050 bash[21467]: cluster 2024-04-06T03:17:05.781405+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-04-06T03:17:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:07 smithi050 bash[21467]: cluster 2024-04-06T03:17:05.785748+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-04-06T03:17:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:07 smithi050 bash[29221]: cluster 2024-04-06T03:17:05.526442+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:07 smithi050 bash[29221]: cluster 2024-04-06T03:17:05.781405+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-04-06T03:17:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:07 smithi050 bash[29221]: cluster 2024-04-06T03:17:05.785748+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-04-06T03:17:07.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:07 smithi153 bash[23852]: cluster 2024-04-06T03:17:05.526442+0000 mgr.y (mgr.14152) 214 : cluster 0 pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:07.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:07 smithi153 bash[23852]: cluster 2024-04-06T03:17:05.781405+0000 osd.1 (osd.1) 5 : cluster 0 1.0 scrub starts 2024-04-06T03:17:07.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:07 smithi153 bash[23852]: cluster 2024-04-06T03:17:05.785748+0000 osd.1 (osd.1) 6 : cluster 0 1.0 scrub ok 2024-04-06T03:17:08.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:08 smithi153 bash[23852]: cluster 2024-04-06T03:17:07.527005+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:08.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:08 smithi050 bash[21467]: cluster 2024-04-06T03:17:07.527005+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:08.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:08 smithi050 bash[29221]: cluster 2024-04-06T03:17:07.527005+0000 mgr.y (mgr.14152) 215 : cluster 0 pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:10 smithi050 bash[21467]: cluster 2024-04-06T03:17:09.527553+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:10.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:10 smithi050 bash[29221]: cluster 2024-04-06T03:17:09.527553+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:11.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:10 smithi153 bash[23852]: cluster 2024-04-06T03:17:09.527553+0000 mgr.y (mgr.14152) 216 : cluster 0 pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:12.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:12 smithi050 bash[21467]: cluster 2024-04-06T03:17:11.528102+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:12.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:12 smithi050 bash[29221]: cluster 2024-04-06T03:17:11.528102+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:13.039 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:12 smithi153 bash[23852]: cluster 2024-04-06T03:17:11.528102+0000 mgr.y (mgr.14152) 217 : cluster 0 pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:14.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:14 smithi153 bash[23852]: cluster 2024-04-06T03:17:13.528672+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:14.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:14 smithi050 bash[29221]: cluster 2024-04-06T03:17:13.528672+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:14.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:14 smithi050 bash[21467]: cluster 2024-04-06T03:17:13.528672+0000 mgr.y (mgr.14152) 218 : cluster 0 pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:16.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:16 smithi153 bash[23852]: cluster 2024-04-06T03:17:15.529199+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:16.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:16 smithi050 bash[29221]: cluster 2024-04-06T03:17:15.529199+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:16.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:16 smithi050 bash[21467]: cluster 2024-04-06T03:17:15.529199+0000 mgr.y (mgr.14152) 219 : cluster 0 pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:17.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:17 smithi153 bash[23852]: audit 2024-04-06T03:17:17.055698+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:17:17.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:17 smithi153 bash[23852]: audit 2024-04-06T03:17:17.057827+0000 mon.a (mon.0) 492 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:17.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:17 smithi153 bash[23852]: cephadm 2024-04-06T03:17:17.059531+0000 mgr.y (mgr.14152) 220 : cephadm 1 Deploying daemon osd.4 on smithi153 2024-04-06T03:17:17.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:17 smithi050 bash[21467]: audit 2024-04-06T03:17:17.055698+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:17:17.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:17 smithi050 bash[21467]: audit 2024-04-06T03:17:17.057827+0000 mon.a (mon.0) 492 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:17.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:17 smithi050 bash[21467]: cephadm 2024-04-06T03:17:17.059531+0000 mgr.y (mgr.14152) 220 : cephadm 1 Deploying daemon osd.4 on smithi153 2024-04-06T03:17:17.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:17 smithi050 bash[29221]: audit 2024-04-06T03:17:17.055698+0000 mon.a (mon.0) 491 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-06T03:17:17.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:17 smithi050 bash[29221]: audit 2024-04-06T03:17:17.057827+0000 mon.a (mon.0) 492 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:17.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:17 smithi050 bash[29221]: cephadm 2024-04-06T03:17:17.059531+0000 mgr.y (mgr.14152) 220 : cephadm 1 Deploying daemon osd.4 on smithi153 2024-04-06T03:17:18.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:18 smithi050 bash[21467]: cluster 2024-04-06T03:17:17.529710+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:18 smithi050 bash[29221]: cluster 2024-04-06T03:17:17.529710+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:19.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:18 smithi153 bash[23852]: cluster 2024-04-06T03:17:17.529710+0000 mgr.y (mgr.14152) 221 : cluster 0 pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:20.972 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:20 smithi153 bash[23852]: cluster 2024-04-06T03:17:19.530251+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:20 smithi050 bash[21467]: cluster 2024-04-06T03:17:19.530251+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:20.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:20 smithi050 bash[29221]: cluster 2024-04-06T03:17:19.530251+0000 mgr.y (mgr.14152) 222 : cluster 0 pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:21.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:20 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:17:21.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:21 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:17:21.290 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:17:20 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:17:21.290 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:17:21 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:17:22.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:22 smithi153 bash[23852]: audit 2024-04-06T03:17:21.359255+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:22.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:22 smithi153 bash[23852]: audit 2024-04-06T03:17:21.368229+0000 mon.a (mon.0) 494 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:22.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:22 smithi153 bash[23852]: audit 2024-04-06T03:17:21.400223+0000 mon.a (mon.0) 495 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:22.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:22 smithi153 bash[23852]: cluster 2024-04-06T03:17:21.530835+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:22.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:22 smithi153 bash[23852]: cluster 2024-04-06T03:17:22.228243+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:22.648 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:22 smithi153 bash[23852]: cluster 2024-04-06T03:17:22.228337+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[21467]: audit 2024-04-06T03:17:21.359255+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[21467]: audit 2024-04-06T03:17:21.368229+0000 mon.a (mon.0) 494 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[21467]: audit 2024-04-06T03:17:21.400223+0000 mon.a (mon.0) 495 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[21467]: cluster 2024-04-06T03:17:21.530835+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[21467]: cluster 2024-04-06T03:17:22.228243+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[21467]: cluster 2024-04-06T03:17:22.228337+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[29221]: audit 2024-04-06T03:17:21.359255+0000 mon.a (mon.0) 493 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:22.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[29221]: audit 2024-04-06T03:17:21.368229+0000 mon.a (mon.0) 494 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[29221]: audit 2024-04-06T03:17:21.400223+0000 mon.a (mon.0) 495 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[29221]: cluster 2024-04-06T03:17:21.530835+0000 mgr.y (mgr.14152) 223 : cluster 0 pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[29221]: cluster 2024-04-06T03:17:22.228243+0000 osd.1 (osd.1) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:22 smithi050 bash[29221]: cluster 2024-04-06T03:17:22.228337+0000 osd.1 (osd.1) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:23.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:23 smithi050 bash[29221]: cluster 2024-04-06T03:17:22.381311+0000 mon.a (mon.0) 496 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-04-06T03:17:23.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:23 smithi050 bash[29221]: audit 2024-04-06T03:17:22.381668+0000 mon.a (mon.0) 497 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:23.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:23 smithi050 bash[21467]: cluster 2024-04-06T03:17:22.381311+0000 mon.a (mon.0) 496 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-04-06T03:17:23.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:23 smithi050 bash[21467]: audit 2024-04-06T03:17:22.381668+0000 mon.a (mon.0) 497 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:23 smithi153 bash[23852]: cluster 2024-04-06T03:17:22.381311+0000 mon.a (mon.0) 496 : cluster 0 osdmap e32: 5 total, 4 up, 5 in 2024-04-06T03:17:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:23 smithi153 bash[23852]: audit 2024-04-06T03:17:22.381668+0000 mon.a (mon.0) 497 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:24.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:24 smithi050 bash[21467]: cluster 2024-04-06T03:17:23.531393+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:24 smithi050 bash[29221]: cluster 2024-04-06T03:17:23.531393+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:24.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:24 smithi153 bash[23852]: cluster 2024-04-06T03:17:23.531393+0000 mgr.y (mgr.14152) 224 : cluster 0 pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:26.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:26 smithi050 bash[29221]: cluster 2024-04-06T03:17:25.531937+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:26.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:26 smithi050 bash[21467]: cluster 2024-04-06T03:17:25.531937+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:27.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:26 smithi153 bash[23852]: cluster 2024-04-06T03:17:25.531937+0000 mgr.y (mgr.14152) 225 : cluster 0 pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:28.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:28 smithi153 bash[23852]: cluster 2024-04-06T03:17:27.532487+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:28.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:28 smithi153 bash[23852]: audit 2024-04-06T03:17:27.603636+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:17:28.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:28 smithi153 bash[23852]: audit 2024-04-06T03:17:27.605840+0000 mon.a (mon.0) 498 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:17:28.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:28 smithi153 bash[23852]: audit 2024-04-06T03:17:28.118556+0000 mon.a (mon.0) 499 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:28.894 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:28 smithi153 bash[23852]: audit 2024-04-06T03:17:28.129461+0000 mon.a (mon.0) 500 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[21467]: cluster 2024-04-06T03:17:27.532487+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[21467]: audit 2024-04-06T03:17:27.603636+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[21467]: audit 2024-04-06T03:17:27.605840+0000 mon.a (mon.0) 498 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[21467]: audit 2024-04-06T03:17:28.118556+0000 mon.a (mon.0) 499 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[21467]: audit 2024-04-06T03:17:28.129461+0000 mon.a (mon.0) 500 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[29221]: cluster 2024-04-06T03:17:27.532487+0000 mgr.y (mgr.14152) 226 : cluster 0 pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[29221]: audit 2024-04-06T03:17:27.603636+0000 mon.b (mon.1) 12 : audit 1 from='osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:17:28.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[29221]: audit 2024-04-06T03:17:27.605840+0000 mon.a (mon.0) 498 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-06T03:17:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[29221]: audit 2024-04-06T03:17:28.118556+0000 mon.a (mon.0) 499 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:28 smithi050 bash[29221]: audit 2024-04-06T03:17:28.129461+0000 mon.a (mon.0) 500 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.600610+0000 mon.a (mon.0) 501 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.605927+0000 mon.b (mon.1) 13 : audit 1 from='osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: cluster 2024-04-06T03:17:28.607800+0000 mon.a (mon.0) 502 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.608714+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.609418+0000 mon.a (mon.0) 504 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.809447+0000 mon.a (mon.0) 505 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:29.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.811328+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:29.620 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:29 smithi153 bash[23852]: audit 2024-04-06T03:17:28.824036+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.600610+0000 mon.a (mon.0) 501 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:17:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.605927+0000 mon.b (mon.1) 13 : audit 1 from='osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:17:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: cluster 2024-04-06T03:17:28.607800+0000 mon.a (mon.0) 502 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-04-06T03:17:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.608714+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.609418+0000 mon.a (mon.0) 504 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.809447+0000 mon.a (mon.0) 505 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.811328+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[21467]: audit 2024-04-06T03:17:28.824036+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.600610+0000 mon.a (mon.0) 501 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.605927+0000 mon.b (mon.1) 13 : audit 1 from='osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: cluster 2024-04-06T03:17:28.607800+0000 mon.a (mon.0) 502 : cluster 0 osdmap e33: 5 total, 4 up, 5 in 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.608714+0000 mon.a (mon.0) 503 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.609418+0000 mon.a (mon.0) 504 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.809447+0000 mon.a (mon.0) 505 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.811328+0000 mon.a (mon.0) 506 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:29 smithi050 bash[29221]: audit 2024-04-06T03:17:28.824036+0000 mon.a (mon.0) 507 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: cluster 2024-04-06T03:17:28.642443+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: cluster 2024-04-06T03:17:28.642510+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: cluster 2024-04-06T03:17:29.532973+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: audit 2024-04-06T03:17:29.603307+0000 mon.a (mon.0) 508 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: cluster 2024-04-06T03:17:29.609307+0000 mon.a (mon.0) 509 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: audit 2024-04-06T03:17:29.610503+0000 mon.a (mon.0) 510 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:30.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:30 smithi153 bash[23852]: audit 2024-04-06T03:17:29.620309+0000 mon.a (mon.0) 511 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: cluster 2024-04-06T03:17:28.642443+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: cluster 2024-04-06T03:17:28.642510+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: cluster 2024-04-06T03:17:29.532973+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: audit 2024-04-06T03:17:29.603307+0000 mon.a (mon.0) 508 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: cluster 2024-04-06T03:17:29.609307+0000 mon.a (mon.0) 509 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: audit 2024-04-06T03:17:29.610503+0000 mon.a (mon.0) 510 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[21467]: audit 2024-04-06T03:17:29.620309+0000 mon.a (mon.0) 511 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: cluster 2024-04-06T03:17:28.642443+0000 osd.4 (osd.4) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: cluster 2024-04-06T03:17:28.642510+0000 osd.4 (osd.4) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: cluster 2024-04-06T03:17:29.532973+0000 mgr.y (mgr.14152) 227 : cluster 0 pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: audit 2024-04-06T03:17:29.603307+0000 mon.a (mon.0) 508 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: cluster 2024-04-06T03:17:29.609307+0000 mon.a (mon.0) 509 : cluster 0 osdmap e34: 5 total, 4 up, 5 in 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: audit 2024-04-06T03:17:29.610503+0000 mon.a (mon.0) 510 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:30 smithi050 bash[29221]: audit 2024-04-06T03:17:29.620309+0000 mon.a (mon.0) 511 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:31.401 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-04-06T03:17:31.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: cluster 2024-04-06T03:17:30.622709+0000 mon.a (mon.0) 512 : cluster 1 osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473] boot 2024-04-06T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: cluster 2024-04-06T03:17:30.622826+0000 mon.a (mon.0) 513 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-04-06T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: audit 2024-04-06T03:17:30.623771+0000 mon.a (mon.0) 514 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: audit 2024-04-06T03:17:31.371623+0000 mon.a (mon.0) 515 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: audit 2024-04-06T03:17:31.372661+0000 mon.a (mon.0) 516 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: audit 2024-04-06T03:17:31.381091+0000 mon.a (mon.0) 517 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:31 smithi153 bash[23852]: cluster 2024-04-06T03:17:31.623574+0000 mon.a (mon.0) 518 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: cluster 2024-04-06T03:17:30.622709+0000 mon.a (mon.0) 512 : cluster 1 osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473] boot 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: cluster 2024-04-06T03:17:30.622826+0000 mon.a (mon.0) 513 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: audit 2024-04-06T03:17:30.623771+0000 mon.a (mon.0) 514 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: audit 2024-04-06T03:17:31.371623+0000 mon.a (mon.0) 515 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: audit 2024-04-06T03:17:31.372661+0000 mon.a (mon.0) 516 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: audit 2024-04-06T03:17:31.381091+0000 mon.a (mon.0) 517 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:31.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[21467]: cluster 2024-04-06T03:17:31.623574+0000 mon.a (mon.0) 518 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: cluster 2024-04-06T03:17:30.622709+0000 mon.a (mon.0) 512 : cluster 1 osd.4 [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473] boot 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: cluster 2024-04-06T03:17:30.622826+0000 mon.a (mon.0) 513 : cluster 0 osdmap e35: 5 total, 5 up, 5 in 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: audit 2024-04-06T03:17:30.623771+0000 mon.a (mon.0) 514 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: audit 2024-04-06T03:17:31.371623+0000 mon.a (mon.0) 515 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: audit 2024-04-06T03:17:31.372661+0000 mon.a (mon.0) 516 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: audit 2024-04-06T03:17:31.381091+0000 mon.a (mon.0) 517 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:31 smithi050 bash[29221]: cluster 2024-04-06T03:17:31.623574+0000 mon.a (mon.0) 518 : cluster 0 osdmap e36: 5 total, 5 up, 5 in 2024-04-06T03:17:32.093 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-06T03:17:32.096 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-06T03:17:32.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-06T03:17:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:32 smithi050 bash[21467]: cluster 2024-04-06T03:17:31.533488+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v204: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:32 smithi050 bash[21467]: cluster 2024-04-06T03:17:32.625707+0000 mon.a (mon.0) 519 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-04-06T03:17:32.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:32 smithi050 bash[29221]: cluster 2024-04-06T03:17:31.533488+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v204: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:32.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:32 smithi050 bash[29221]: cluster 2024-04-06T03:17:32.625707+0000 mon.a (mon.0) 519 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-04-06T03:17:33.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:32 smithi153 bash[23852]: cluster 2024-04-06T03:17:31.533488+0000 mgr.y (mgr.14152) 228 : cluster 0 pgmap v204: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-04-06T03:17:33.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:32 smithi153 bash[23852]: cluster 2024-04-06T03:17:32.625707+0000 mon.a (mon.0) 519 : cluster 0 osdmap e37: 5 total, 5 up, 5 in 2024-04-06T03:17:34.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:33 smithi153 bash[23852]: cluster 2024-04-06T03:17:33.001108+0000 mon.a (mon.0) 520 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-04-06T03:17:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:33 smithi050 bash[21467]: cluster 2024-04-06T03:17:33.001108+0000 mon.a (mon.0) 520 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-04-06T03:17:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:33 smithi050 bash[29221]: cluster 2024-04-06T03:17:33.001108+0000 mon.a (mon.0) 520 : cluster 0 osdmap e38: 5 total, 5 up, 5 in 2024-04-06T03:17:35.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:35 smithi153 bash[23852]: cluster 2024-04-06T03:17:33.534087+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:35.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:35 smithi153 bash[23852]: cluster 2024-04-06T03:17:34.017021+0000 mon.a (mon.0) 521 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-04-06T03:17:35.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:35 smithi050 bash[21467]: cluster 2024-04-06T03:17:33.534087+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:35.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:35 smithi050 bash[21467]: cluster 2024-04-06T03:17:34.017021+0000 mon.a (mon.0) 521 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-04-06T03:17:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:35 smithi050 bash[29221]: cluster 2024-04-06T03:17:33.534087+0000 mgr.y (mgr.14152) 229 : cluster 0 pgmap v208: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:35 smithi050 bash[29221]: cluster 2024-04-06T03:17:34.017021+0000 mon.a (mon.0) 521 : cluster 0 osdmap e39: 5 total, 5 up, 5 in 2024-04-06T03:17:36.341 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:17:37.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:37 smithi153 bash[23852]: cluster 2024-04-06T03:17:35.534719+0000 mgr.y (mgr.14152) 230 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-06T03:17:37.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:37 smithi050 bash[21467]: cluster 2024-04-06T03:17:35.534719+0000 mgr.y (mgr.14152) 230 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-06T03:17:37.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:37 smithi050 bash[29221]: cluster 2024-04-06T03:17:35.534719+0000 mgr.y (mgr.14152) 230 : cluster 0 pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-06T03:17:39.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:39 smithi153 bash[23852]: cluster 2024-04-06T03:17:37.535343+0000 mgr.y (mgr.14152) 231 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-06T03:17:39.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:39 smithi050 bash[29221]: cluster 2024-04-06T03:17:37.535343+0000 mgr.y (mgr.14152) 231 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-06T03:17:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:39 smithi050 bash[21467]: cluster 2024-04-06T03:17:37.535343+0000 mgr.y (mgr.14152) 231 : cluster 0 pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-06T03:17:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-06T03:17:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-06T03:17:40.205 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-06T03:17:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-06T03:17:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013796 s, 760 MB/s 2024-04-06T03:17:40.205 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-06T03:17:41.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-06T03:17:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:41 smithi153 bash[23852]: cluster 2024-04-06T03:17:39.535922+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-06T03:17:41.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:41 smithi050 bash[21467]: cluster 2024-04-06T03:17:39.535922+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-06T03:17:41.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:41 smithi050 bash[29221]: cluster 2024-04-06T03:17:39.535922+0000 mgr.y (mgr.14152) 232 : cluster 0 pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-06T03:17:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: cluster 2024-04-06T03:17:41.169828+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: cluster 2024-04-06T03:17:41.169926+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: cephadm 2024-04-06T03:17:41.471736+0000 mgr.y (mgr.14152) 233 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:17:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.480788+0000 mon.a (mon.0) 522 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.490009+0000 mon.a (mon.0) 523 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.492483+0000 mon.a (mon.0) 524 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: cephadm 2024-04-06T03:17:41.494056+0000 mgr.y (mgr.14152) 234 : cephadm 1 Adjusting osd_memory_target on smithi153 to 17265M 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.500247+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.504525+0000 mon.a (mon.0) 526 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.506438+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: audit 2024-04-06T03:17:41.516567+0000 mon.a (mon.0) 528 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: cluster 2024-04-06T03:17:41.536360+0000 mgr.y (mgr.14152) 235 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[21467]: cluster 2024-04-06T03:17:42.031829+0000 mon.a (mon.0) 529 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: cluster 2024-04-06T03:17:41.169828+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: cluster 2024-04-06T03:17:41.169926+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: cephadm 2024-04-06T03:17:41.471736+0000 mgr.y (mgr.14152) 233 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:17:42.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.480788+0000 mon.a (mon.0) 522 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.490009+0000 mon.a (mon.0) 523 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.492483+0000 mon.a (mon.0) 524 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: cephadm 2024-04-06T03:17:41.494056+0000 mgr.y (mgr.14152) 234 : cephadm 1 Adjusting osd_memory_target on smithi153 to 17265M 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.500247+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.504525+0000 mon.a (mon.0) 526 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.506438+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: audit 2024-04-06T03:17:41.516567+0000 mon.a (mon.0) 528 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: cluster 2024-04-06T03:17:41.536360+0000 mgr.y (mgr.14152) 235 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-06T03:17:42.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:42 smithi050 bash[29221]: cluster 2024-04-06T03:17:42.031829+0000 mon.a (mon.0) 529 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-04-06T03:17:42.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: cluster 2024-04-06T03:17:41.169828+0000 osd.0 (osd.0) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:42.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: cluster 2024-04-06T03:17:41.169926+0000 osd.0 (osd.0) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:42.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: cephadm 2024-04-06T03:17:41.471736+0000 mgr.y (mgr.14152) 233 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:17:42.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.480788+0000 mon.a (mon.0) 522 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.490009+0000 mon.a (mon.0) 523 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.492483+0000 mon.a (mon.0) 524 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: cephadm 2024-04-06T03:17:41.494056+0000 mgr.y (mgr.14152) 234 : cephadm 1 Adjusting osd_memory_target on smithi153 to 17265M 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.500247+0000 mon.a (mon.0) 525 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.504525+0000 mon.a (mon.0) 526 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.506438+0000 mon.a (mon.0) 527 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: audit 2024-04-06T03:17:41.516567+0000 mon.a (mon.0) 528 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: cluster 2024-04-06T03:17:41.536360+0000 mgr.y (mgr.14152) 235 : cluster 0 pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-06T03:17:42.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:42 smithi153 bash[23852]: cluster 2024-04-06T03:17:42.031829+0000 mon.a (mon.0) 529 : cluster 0 osdmap e40: 5 total, 5 up, 5 in 2024-04-06T03:17:45.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:45 smithi153 bash[23852]: cluster 2024-04-06T03:17:43.536857+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-06T03:17:45.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:45 smithi153 bash[23852]: cluster 2024-04-06T03:17:44.088129+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:45.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:45 smithi153 bash[23852]: cluster 2024-04-06T03:17:44.088240+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:45 smithi050 bash[21467]: cluster 2024-04-06T03:17:43.536857+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-06T03:17:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:45 smithi050 bash[21467]: cluster 2024-04-06T03:17:44.088129+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:45 smithi050 bash[21467]: cluster 2024-04-06T03:17:44.088240+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:45.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:45 smithi050 bash[29221]: cluster 2024-04-06T03:17:43.536857+0000 mgr.y (mgr.14152) 236 : cluster 0 pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-06T03:17:45.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:45 smithi050 bash[29221]: cluster 2024-04-06T03:17:44.088129+0000 osd.3 (osd.3) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:17:45.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:45 smithi050 bash[29221]: cluster 2024-04-06T03:17:44.088240+0000 osd.3 (osd.3) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:17:45.779 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:17:46.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:46 smithi153 bash[23852]: cluster 2024-04-06T03:17:45.015008+0000 mon.a (mon.0) 530 : cluster 0 osdmap e41: 5 total, 5 up, 5 in 2024-04-06T03:17:46.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:46 smithi050 bash[21467]: cluster 2024-04-06T03:17:45.015008+0000 mon.a (mon.0) 530 : cluster 0 osdmap e41: 5 total, 5 up, 5 in 2024-04-06T03:17:46.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:46 smithi050 bash[29221]: cluster 2024-04-06T03:17:45.015008+0000 mon.a (mon.0) 530 : cluster 0 osdmap e41: 5 total, 5 up, 5 in 2024-04-06T03:17:47.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:47 smithi153 bash[23852]: cluster 2024-04-06T03:17:45.537389+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:47.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:47 smithi050 bash[21467]: cluster 2024-04-06T03:17:45.537389+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:47.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:47 smithi050 bash[29221]: cluster 2024-04-06T03:17:45.537389+0000 mgr.y (mgr.14152) 237 : cluster 0 pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:49.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:49 smithi153 bash[23852]: cluster 2024-04-06T03:17:47.537901+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:49.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:49 smithi153 bash[23852]: audit 2024-04-06T03:17:48.364998+0000 mon.a (mon.0) 531 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:17:49.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:49 smithi153 bash[23852]: audit 2024-04-06T03:17:48.371526+0000 mon.a (mon.0) 532 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:17:49.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:49 smithi153 bash[23852]: audit 2024-04-06T03:17:48.373393+0000 mon.a (mon.0) 533 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[21467]: cluster 2024-04-06T03:17:47.537901+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[21467]: audit 2024-04-06T03:17:48.364998+0000 mon.a (mon.0) 531 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:17:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[21467]: audit 2024-04-06T03:17:48.371526+0000 mon.a (mon.0) 532 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:17:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[21467]: audit 2024-04-06T03:17:48.373393+0000 mon.a (mon.0) 533 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:49.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[29221]: cluster 2024-04-06T03:17:47.537901+0000 mgr.y (mgr.14152) 238 : cluster 0 pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:49.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[29221]: audit 2024-04-06T03:17:48.364998+0000 mon.a (mon.0) 531 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:17:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[29221]: audit 2024-04-06T03:17:48.371526+0000 mon.a (mon.0) 532 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:17:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:49 smithi050 bash[29221]: audit 2024-04-06T03:17:48.373393+0000 mon.a (mon.0) 533 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:17:50.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:50 smithi153 bash[23852]: audit 2024-04-06T03:17:48.357801+0000 mgr.y (mgr.14152) 239 : audit 0 from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:50.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:50 smithi050 bash[21467]: audit 2024-04-06T03:17:48.357801+0000 mgr.y (mgr.14152) 239 : audit 0 from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:50.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:50 smithi050 bash[29221]: audit 2024-04-06T03:17:48.357801+0000 mgr.y (mgr.14152) 239 : audit 0 from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:17:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:51 smithi153 bash[23852]: cluster 2024-04-06T03:17:49.538446+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:51.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:51 smithi050 bash[21467]: cluster 2024-04-06T03:17:49.538446+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:51 smithi050 bash[29221]: cluster 2024-04-06T03:17:49.538446+0000 mgr.y (mgr.14152) 240 : cluster 0 pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:53.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:53 smithi153 bash[23852]: cluster 2024-04-06T03:17:51.539016+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:53.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:53 smithi050 bash[21467]: cluster 2024-04-06T03:17:51.539016+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:53.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:53 smithi050 bash[29221]: cluster 2024-04-06T03:17:51.539016+0000 mgr.y (mgr.14152) 241 : cluster 0 pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:55.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:55 smithi153 bash[23852]: cluster 2024-04-06T03:17:53.539675+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:55.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:55 smithi050 bash[21467]: cluster 2024-04-06T03:17:53.539675+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:55 smithi050 bash[29221]: cluster 2024-04-06T03:17:53.539675+0000 mgr.y (mgr.14152) 242 : cluster 0 pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:57.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:57 smithi153 bash[23852]: cluster 2024-04-06T03:17:55.540388+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:57.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:57 smithi153 bash[23852]: audit 2024-04-06T03:17:56.640708+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.153:0/650229716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]: dispatch 2024-04-06T03:17:57.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:57 smithi153 bash[23852]: audit 2024-04-06T03:17:56.643409+0000 mon.a (mon.0) 534 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]: dispatch 2024-04-06T03:17:57.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:57 smithi153 bash[23852]: audit 2024-04-06T03:17:56.646837+0000 mon.a (mon.0) 535 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]': finished 2024-04-06T03:17:57.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:57 smithi153 bash[23852]: cluster 2024-04-06T03:17:56.652932+0000 mon.a (mon.0) 536 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-04-06T03:17:57.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:57 smithi153 bash[23852]: audit 2024-04-06T03:17:56.653432+0000 mon.a (mon.0) 537 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[21467]: cluster 2024-04-06T03:17:55.540388+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[21467]: audit 2024-04-06T03:17:56.640708+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.153:0/650229716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]: dispatch 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[21467]: audit 2024-04-06T03:17:56.643409+0000 mon.a (mon.0) 534 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]: dispatch 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[21467]: audit 2024-04-06T03:17:56.646837+0000 mon.a (mon.0) 535 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]': finished 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[21467]: cluster 2024-04-06T03:17:56.652932+0000 mon.a (mon.0) 536 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[21467]: audit 2024-04-06T03:17:56.653432+0000 mon.a (mon.0) 537 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:17:57.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[29221]: cluster 2024-04-06T03:17:55.540388+0000 mgr.y (mgr.14152) 243 : cluster 0 pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[29221]: audit 2024-04-06T03:17:56.640708+0000 mon.b (mon.1) 14 : audit 1 from='client.? 172.21.15.153:0/650229716' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]: dispatch 2024-04-06T03:17:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[29221]: audit 2024-04-06T03:17:56.643409+0000 mon.a (mon.0) 534 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]: dispatch 2024-04-06T03:17:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[29221]: audit 2024-04-06T03:17:56.646837+0000 mon.a (mon.0) 535 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241"}]': finished 2024-04-06T03:17:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[29221]: cluster 2024-04-06T03:17:56.652932+0000 mon.a (mon.0) 536 : cluster 0 osdmap e42: 6 total, 5 up, 6 in 2024-04-06T03:17:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:57 smithi050 bash[29221]: audit 2024-04-06T03:17:56.653432+0000 mon.a (mon.0) 537 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:17:58.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:58 smithi050 bash[21467]: audit 2024-04-06T03:17:57.403667+0000 mon.b (mon.1) 15 : audit 0 from='client.? 172.21.15.153:0/3134307035' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:17:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:58 smithi050 bash[29221]: audit 2024-04-06T03:17:57.403667+0000 mon.b (mon.1) 15 : audit 0 from='client.? 172.21.15.153:0/3134307035' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:17:58.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:58 smithi153 bash[23852]: audit 2024-04-06T03:17:57.403667+0000 mon.b (mon.1) 15 : audit 0 from='client.? 172.21.15.153:0/3134307035' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:17:59.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:17:59 smithi050 bash[21467]: cluster 2024-04-06T03:17:57.540936+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:59.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:17:59 smithi050 bash[29221]: cluster 2024-04-06T03:17:57.540936+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:17:59.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:17:59 smithi153 bash[23852]: cluster 2024-04-06T03:17:57.540936+0000 mgr.y (mgr.14152) 244 : cluster 0 pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:01.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:01 smithi050 bash[21467]: cluster 2024-04-06T03:17:59.541593+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:01.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:01 smithi050 bash[29221]: cluster 2024-04-06T03:17:59.541593+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:01.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:01 smithi153 bash[23852]: cluster 2024-04-06T03:17:59.541593+0000 mgr.y (mgr.14152) 245 : cluster 0 pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:03.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:03 smithi050 bash[21467]: cluster 2024-04-06T03:18:01.542253+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:03.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:03 smithi050 bash[29221]: cluster 2024-04-06T03:18:01.542253+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:03.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:03 smithi153 bash[23852]: cluster 2024-04-06T03:18:01.542253+0000 mgr.y (mgr.14152) 246 : cluster 0 pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:05.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:05 smithi050 bash[21467]: cluster 2024-04-06T03:18:03.542845+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:05 smithi050 bash[29221]: cluster 2024-04-06T03:18:03.542845+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:05 smithi153 bash[23852]: cluster 2024-04-06T03:18:03.542845+0000 mgr.y (mgr.14152) 247 : cluster 0 pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:07.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:07 smithi153 bash[23852]: cluster 2024-04-06T03:18:05.543410+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:07.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:07 smithi050 bash[29221]: cluster 2024-04-06T03:18:05.543410+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:07 smithi050 bash[21467]: cluster 2024-04-06T03:18:05.543410+0000 mgr.y (mgr.14152) 248 : cluster 0 pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:09.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:09 smithi050 bash[29221]: cluster 2024-04-06T03:18:07.543967+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:09 smithi050 bash[29221]: audit 2024-04-06T03:18:08.799601+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:18:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:09 smithi050 bash[29221]: audit 2024-04-06T03:18:08.801640+0000 mon.a (mon.0) 539 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:09 smithi050 bash[21467]: cluster 2024-04-06T03:18:07.543967+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:09 smithi050 bash[21467]: audit 2024-04-06T03:18:08.799601+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:18:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:09 smithi050 bash[21467]: audit 2024-04-06T03:18:08.801640+0000 mon.a (mon.0) 539 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:09.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:09 smithi153 bash[23852]: cluster 2024-04-06T03:18:07.543967+0000 mgr.y (mgr.14152) 249 : cluster 0 pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:09.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:09 smithi153 bash[23852]: audit 2024-04-06T03:18:08.799601+0000 mon.a (mon.0) 538 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-06T03:18:09.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:09 smithi153 bash[23852]: audit 2024-04-06T03:18:08.801640+0000 mon.a (mon.0) 539 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:10.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:10 smithi153 bash[23852]: cephadm 2024-04-06T03:18:08.803452+0000 mgr.y (mgr.14152) 250 : cephadm 1 Deploying daemon osd.5 on smithi153 2024-04-06T03:18:10.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:10 smithi050 bash[21467]: cephadm 2024-04-06T03:18:08.803452+0000 mgr.y (mgr.14152) 250 : cephadm 1 Deploying daemon osd.5 on smithi153 2024-04-06T03:18:10.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:10 smithi050 bash[29221]: cephadm 2024-04-06T03:18:08.803452+0000 mgr.y (mgr.14152) 250 : cephadm 1 Deploying daemon osd.5 on smithi153 2024-04-06T03:18:11.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:11 smithi153 bash[23852]: cluster 2024-04-06T03:18:09.544543+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:11.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:11 smithi050 bash[21467]: cluster 2024-04-06T03:18:09.544543+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:11.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:11 smithi050 bash[29221]: cluster 2024-04-06T03:18:09.544543+0000 mgr.y (mgr.14152) 251 : cluster 0 pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:12.857 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:18:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:18:12.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:18:12.858 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:18:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:18:13.139 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:18:13 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:18:13.139 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:13 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:18:13.139 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:13 smithi153 bash[23852]: cluster 2024-04-06T03:18:11.545214+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:13.140 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:18:13 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:18:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:13 smithi050 bash[21467]: cluster 2024-04-06T03:18:11.545214+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:13 smithi050 bash[29221]: cluster 2024-04-06T03:18:11.545214+0000 mgr.y (mgr.14152) 252 : cluster 0 pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:14.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:14 smithi153 bash[23852]: audit 2024-04-06T03:18:13.160682+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:14.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:14 smithi153 bash[23852]: audit 2024-04-06T03:18:13.169629+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:14.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:14 smithi153 bash[23852]: audit 2024-04-06T03:18:13.191082+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:14.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:14 smithi153 bash[23852]: cluster 2024-04-06T03:18:13.545792+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:14.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[21467]: audit 2024-04-06T03:18:13.160682+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[21467]: audit 2024-04-06T03:18:13.169629+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[21467]: audit 2024-04-06T03:18:13.191082+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[21467]: cluster 2024-04-06T03:18:13.545792+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[29221]: audit 2024-04-06T03:18:13.160682+0000 mon.a (mon.0) 540 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[29221]: audit 2024-04-06T03:18:13.169629+0000 mon.a (mon.0) 541 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[29221]: audit 2024-04-06T03:18:13.191082+0000 mon.a (mon.0) 542 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:14 smithi050 bash[29221]: cluster 2024-04-06T03:18:13.545792+0000 mgr.y (mgr.14152) 253 : cluster 0 pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:16.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:16 smithi153 bash[23852]: cluster 2024-04-06T03:18:15.546318+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:16.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:16 smithi050 bash[21467]: cluster 2024-04-06T03:18:15.546318+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:16.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:16 smithi050 bash[29221]: cluster 2024-04-06T03:18:15.546318+0000 mgr.y (mgr.14152) 254 : cluster 0 pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:18.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:18 smithi050 bash[21467]: cluster 2024-04-06T03:18:17.546811+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:18 smithi050 bash[21467]: cluster 2024-04-06T03:18:17.783989+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:18 smithi050 bash[21467]: cluster 2024-04-06T03:18:17.784085+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:18 smithi050 bash[29221]: cluster 2024-04-06T03:18:17.546811+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:18 smithi050 bash[29221]: cluster 2024-04-06T03:18:17.783989+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:18 smithi050 bash[29221]: cluster 2024-04-06T03:18:17.784085+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:19.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:18 smithi153 bash[23852]: cluster 2024-04-06T03:18:17.546811+0000 mgr.y (mgr.14152) 255 : cluster 0 pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:19.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:18 smithi153 bash[23852]: cluster 2024-04-06T03:18:17.783989+0000 osd.2 (osd.2) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:19.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:18 smithi153 bash[23852]: cluster 2024-04-06T03:18:17.784085+0000 osd.2 (osd.2) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:19.789 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:19 smithi153 bash[23852]: cluster 2024-04-06T03:18:18.618190+0000 mon.a (mon.0) 543 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-04-06T03:18:19.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:19 smithi153 bash[23852]: audit 2024-04-06T03:18:18.618495+0000 mon.a (mon.0) 544 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:19.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:19 smithi153 bash[23852]: cluster 2024-04-06T03:18:18.705807+0000 osd.1 (osd.1) 9 : cluster 0 1.0 scrub starts 2024-04-06T03:18:19.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:19 smithi153 bash[23852]: cluster 2024-04-06T03:18:18.709166+0000 osd.1 (osd.1) 10 : cluster 0 1.0 scrub ok 2024-04-06T03:18:19.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:19 smithi153 bash[23852]: audit 2024-04-06T03:18:19.500289+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:18:19.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:19 smithi153 bash[23852]: audit 2024-04-06T03:18:19.503035+0000 mon.a (mon.0) 545 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[21467]: cluster 2024-04-06T03:18:18.618190+0000 mon.a (mon.0) 543 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[21467]: audit 2024-04-06T03:18:18.618495+0000 mon.a (mon.0) 544 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[21467]: cluster 2024-04-06T03:18:18.705807+0000 osd.1 (osd.1) 9 : cluster 0 1.0 scrub starts 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[21467]: cluster 2024-04-06T03:18:18.709166+0000 osd.1 (osd.1) 10 : cluster 0 1.0 scrub ok 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[21467]: audit 2024-04-06T03:18:19.500289+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[21467]: audit 2024-04-06T03:18:19.503035+0000 mon.a (mon.0) 545 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[29221]: cluster 2024-04-06T03:18:18.618190+0000 mon.a (mon.0) 543 : cluster 0 osdmap e43: 6 total, 5 up, 6 in 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[29221]: audit 2024-04-06T03:18:18.618495+0000 mon.a (mon.0) 544 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:19.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[29221]: cluster 2024-04-06T03:18:18.705807+0000 osd.1 (osd.1) 9 : cluster 0 1.0 scrub starts 2024-04-06T03:18:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[29221]: cluster 2024-04-06T03:18:18.709166+0000 osd.1 (osd.1) 10 : cluster 0 1.0 scrub ok 2024-04-06T03:18:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[29221]: audit 2024-04-06T03:18:19.500289+0000 mon.b (mon.1) 16 : audit 1 from='osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:18:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:19 smithi050 bash[29221]: audit 2024-04-06T03:18:19.503035+0000 mon.a (mon.0) 545 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-06T03:18:20.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: cluster 2024-04-06T03:18:19.547342+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:20.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:19.621146+0000 mon.a (mon.0) 546 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:18:20.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:19.626718+0000 mon.b (mon.1) 17 : audit 1 from='osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:18:20.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: cluster 2024-04-06T03:18:19.629098+0000 mon.a (mon.0) 547 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:19.630100+0000 mon.a (mon.0) 548 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:19.630720+0000 mon.a (mon.0) 549 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:20.048367+0000 mon.a (mon.0) 550 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:20.057786+0000 mon.a (mon.0) 551 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:20.059890+0000 mon.a (mon.0) 552 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:20.061880+0000 mon.a (mon.0) 553 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:20 smithi153 bash[23852]: audit 2024-04-06T03:18:20.072108+0000 mon.a (mon.0) 554 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: cluster 2024-04-06T03:18:19.547342+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:19.621146+0000 mon.a (mon.0) 546 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:19.626718+0000 mon.b (mon.1) 17 : audit 1 from='osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: cluster 2024-04-06T03:18:19.629098+0000 mon.a (mon.0) 547 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:19.630100+0000 mon.a (mon.0) 548 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:19.630720+0000 mon.a (mon.0) 549 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:20.048367+0000 mon.a (mon.0) 550 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:20.057786+0000 mon.a (mon.0) 551 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:20.059890+0000 mon.a (mon.0) 552 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:20.061880+0000 mon.a (mon.0) 553 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[21467]: audit 2024-04-06T03:18:20.072108+0000 mon.a (mon.0) 554 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: cluster 2024-04-06T03:18:19.547342+0000 mgr.y (mgr.14152) 256 : cluster 0 pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:19.621146+0000 mon.a (mon.0) 546 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:19.626718+0000 mon.b (mon.1) 17 : audit 1 from='osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: cluster 2024-04-06T03:18:19.629098+0000 mon.a (mon.0) 547 : cluster 0 osdmap e44: 6 total, 5 up, 6 in 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:19.630100+0000 mon.a (mon.0) 548 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:19.630720+0000 mon.a (mon.0) 549 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:20.048367+0000 mon.a (mon.0) 550 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:20.057786+0000 mon.a (mon.0) 551 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:20.059890+0000 mon.a (mon.0) 552 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:20.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:20.061880+0000 mon.a (mon.0) 553 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:20.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:20 smithi050 bash[29221]: audit 2024-04-06T03:18:20.072108+0000 mon.a (mon.0) 554 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[21467]: audit 2024-04-06T03:18:20.623587+0000 mon.a (mon.0) 555 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[21467]: cluster 2024-04-06T03:18:20.629690+0000 mon.a (mon.0) 556 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[21467]: audit 2024-04-06T03:18:20.631913+0000 mon.a (mon.0) 557 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[21467]: audit 2024-04-06T03:18:21.632319+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[29221]: audit 2024-04-06T03:18:20.623587+0000 mon.a (mon.0) 555 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[29221]: cluster 2024-04-06T03:18:20.629690+0000 mon.a (mon.0) 556 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[29221]: audit 2024-04-06T03:18:20.631913+0000 mon.a (mon.0) 557 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:21.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:21 smithi050 bash[29221]: audit 2024-04-06T03:18:21.632319+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:22.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:21 smithi153 bash[23852]: audit 2024-04-06T03:18:20.623587+0000 mon.a (mon.0) 555 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:18:22.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:21 smithi153 bash[23852]: cluster 2024-04-06T03:18:20.629690+0000 mon.a (mon.0) 556 : cluster 0 osdmap e45: 6 total, 5 up, 6 in 2024-04-06T03:18:22.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:21 smithi153 bash[23852]: audit 2024-04-06T03:18:20.631913+0000 mon.a (mon.0) 557 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:22.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:21 smithi153 bash[23852]: audit 2024-04-06T03:18:21.632319+0000 mon.a (mon.0) 558 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[21467]: cluster 2024-04-06T03:18:20.447892+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[21467]: cluster 2024-04-06T03:18:20.448004+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[21467]: cluster 2024-04-06T03:18:21.547874+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[21467]: cluster 2024-04-06T03:18:21.650708+0000 mon.a (mon.0) 559 : cluster 1 osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117] boot 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[21467]: cluster 2024-04-06T03:18:21.650948+0000 mon.a (mon.0) 560 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[21467]: audit 2024-04-06T03:18:21.652523+0000 mon.a (mon.0) 561 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:22.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[29221]: cluster 2024-04-06T03:18:20.447892+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:22.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[29221]: cluster 2024-04-06T03:18:20.448004+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:22.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[29221]: cluster 2024-04-06T03:18:21.547874+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:22.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[29221]: cluster 2024-04-06T03:18:21.650708+0000 mon.a (mon.0) 559 : cluster 1 osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117] boot 2024-04-06T03:18:22.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[29221]: cluster 2024-04-06T03:18:21.650948+0000 mon.a (mon.0) 560 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-04-06T03:18:22.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:22 smithi050 bash[29221]: audit 2024-04-06T03:18:21.652523+0000 mon.a (mon.0) 561 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:22 smithi153 bash[23852]: cluster 2024-04-06T03:18:20.447892+0000 osd.5 (osd.5) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:22 smithi153 bash[23852]: cluster 2024-04-06T03:18:20.448004+0000 osd.5 (osd.5) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:22 smithi153 bash[23852]: cluster 2024-04-06T03:18:21.547874+0000 mgr.y (mgr.14152) 257 : cluster 0 pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-04-06T03:18:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:22 smithi153 bash[23852]: cluster 2024-04-06T03:18:21.650708+0000 mon.a (mon.0) 559 : cluster 1 osd.5 [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117] boot 2024-04-06T03:18:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:22 smithi153 bash[23852]: cluster 2024-04-06T03:18:21.650948+0000 mon.a (mon.0) 560 : cluster 0 osdmap e46: 6 total, 6 up, 6 in 2024-04-06T03:18:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:22 smithi153 bash[23852]: audit 2024-04-06T03:18:21.652523+0000 mon.a (mon.0) 561 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:18:23.420 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-04-06T03:18:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:23 smithi153 bash[23852]: cluster 2024-04-06T03:18:22.651184+0000 mon.a (mon.0) 562 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-04-06T03:18:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:23 smithi153 bash[23852]: audit 2024-04-06T03:18:23.405552+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:23 smithi153 bash[23852]: audit 2024-04-06T03:18:23.415796+0000 mon.a (mon.0) 564 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:23 smithi153 bash[23852]: audit 2024-04-06T03:18:23.417302+0000 mon.a (mon.0) 565 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:23.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:23 smithi153 bash[23852]: cluster 2024-04-06T03:18:23.648912+0000 mon.a (mon.0) 566 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-04-06T03:18:23.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[21467]: cluster 2024-04-06T03:18:22.651184+0000 mon.a (mon.0) 562 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-04-06T03:18:23.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[21467]: audit 2024-04-06T03:18:23.405552+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[21467]: audit 2024-04-06T03:18:23.415796+0000 mon.a (mon.0) 564 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[21467]: audit 2024-04-06T03:18:23.417302+0000 mon.a (mon.0) 565 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[21467]: cluster 2024-04-06T03:18:23.648912+0000 mon.a (mon.0) 566 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[29221]: cluster 2024-04-06T03:18:22.651184+0000 mon.a (mon.0) 562 : cluster 0 osdmap e47: 6 total, 6 up, 6 in 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[29221]: audit 2024-04-06T03:18:23.405552+0000 mon.a (mon.0) 563 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[29221]: audit 2024-04-06T03:18:23.415796+0000 mon.a (mon.0) 564 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[29221]: audit 2024-04-06T03:18:23.417302+0000 mon.a (mon.0) 565 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:18:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:23 smithi050 bash[29221]: cluster 2024-04-06T03:18:23.648912+0000 mon.a (mon.0) 566 : cluster 0 osdmap e48: 6 total, 6 up, 6 in 2024-04-06T03:18:24.000 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-06T03:18:24.002 INFO:tasks.cephadm:Deploying osd.6 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-06T03:18:24.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-06T03:18:24.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:24 smithi050 bash[21467]: cluster 2024-04-06T03:18:23.548304+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v242: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:24.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:24 smithi050 bash[29221]: cluster 2024-04-06T03:18:23.548304+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v242: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:25.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:24 smithi153 bash[23852]: cluster 2024-04-06T03:18:23.548304+0000 mgr.y (mgr.14152) 258 : cluster 0 pgmap v242: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:25.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:25 smithi050 bash[21467]: cluster 2024-04-06T03:18:24.662337+0000 mon.a (mon.0) 567 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-04-06T03:18:25.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:25 smithi050 bash[29221]: cluster 2024-04-06T03:18:24.662337+0000 mon.a (mon.0) 567 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-04-06T03:18:26.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:25 smithi153 bash[23852]: cluster 2024-04-06T03:18:24.662337+0000 mon.a (mon.0) 567 : cluster 0 osdmap e49: 6 total, 6 up, 6 in 2024-04-06T03:18:26.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:26 smithi050 bash[21467]: cluster 2024-04-06T03:18:25.548908+0000 mgr.y (mgr.14152) 259 : cluster 0 pgmap v245: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:26.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:26 smithi050 bash[21467]: cluster 2024-04-06T03:18:25.680149+0000 mon.a (mon.0) 568 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-04-06T03:18:26.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:26 smithi050 bash[29221]: cluster 2024-04-06T03:18:25.548908+0000 mgr.y (mgr.14152) 259 : cluster 0 pgmap v245: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:26.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:26 smithi050 bash[29221]: cluster 2024-04-06T03:18:25.680149+0000 mon.a (mon.0) 568 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-04-06T03:18:27.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:26 smithi153 bash[23852]: cluster 2024-04-06T03:18:25.548908+0000 mgr.y (mgr.14152) 259 : cluster 0 pgmap v245: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:27.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:26 smithi153 bash[23852]: cluster 2024-04-06T03:18:25.680149+0000 mon.a (mon.0) 568 : cluster 0 osdmap e50: 6 total, 6 up, 6 in 2024-04-06T03:18:27.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:27 smithi050 bash[21467]: cluster 2024-04-06T03:18:27.012021+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:27 smithi050 bash[21467]: cluster 2024-04-06T03:18:27.012138+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:27.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:27 smithi050 bash[29221]: cluster 2024-04-06T03:18:27.012021+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:27.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:27 smithi050 bash[29221]: cluster 2024-04-06T03:18:27.012138+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:28.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:27 smithi153 bash[23852]: cluster 2024-04-06T03:18:27.012021+0000 osd.1 (osd.1) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:28.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:27 smithi153 bash[23852]: cluster 2024-04-06T03:18:27.012138+0000 osd.1 (osd.1) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:28.382 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:18:28.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:28 smithi050 bash[21467]: cluster 2024-04-06T03:18:27.549442+0000 mgr.y (mgr.14152) 260 : cluster 0 pgmap v247: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:28 smithi050 bash[21467]: cluster 2024-04-06T03:18:27.690515+0000 mon.a (mon.0) 569 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-04-06T03:18:28.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:28 smithi050 bash[29221]: cluster 2024-04-06T03:18:27.549442+0000 mgr.y (mgr.14152) 260 : cluster 0 pgmap v247: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:28.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:28 smithi050 bash[29221]: cluster 2024-04-06T03:18:27.690515+0000 mon.a (mon.0) 569 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-04-06T03:18:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:28 smithi153 bash[23852]: cluster 2024-04-06T03:18:27.549442+0000 mgr.y (mgr.14152) 260 : cluster 0 pgmap v247: 1 pgs: 1 remapped; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:28 smithi153 bash[23852]: cluster 2024-04-06T03:18:27.690515+0000 mon.a (mon.0) 569 : cluster 0 osdmap e51: 6 total, 6 up, 6 in 2024-04-06T03:18:30.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:30 smithi050 bash[21467]: cluster 2024-04-06T03:18:29.549981+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-06T03:18:30.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:30 smithi050 bash[29221]: cluster 2024-04-06T03:18:29.549981+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-06T03:18:31.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:30 smithi153 bash[23852]: cluster 2024-04-06T03:18:29.549981+0000 mgr.y (mgr.14152) 261 : cluster 0 pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-06T03:18:32.541 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-06T03:18:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-06T03:18:32.542 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-06T03:18:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-06T03:18:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013485 s, 778 MB/s 2024-04-06T03:18:32.542 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-06T03:18:32.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:32 smithi153 bash[23852]: cluster 2024-04-06T03:18:31.550553+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-06T03:18:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:32 smithi050 bash[21467]: cluster 2024-04-06T03:18:31.550553+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-06T03:18:32.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:32 smithi050 bash[29221]: cluster 2024-04-06T03:18:31.550553+0000 mgr.y (mgr.14152) 262 : cluster 0 pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-06T03:18:33.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-06T03:18:35.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: cluster 2024-04-06T03:18:33.551155+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:18:35.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: cephadm 2024-04-06T03:18:33.772937+0000 mgr.y (mgr.14152) 264 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:18:35.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.782167+0000 mon.a (mon.0) 570 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.791771+0000 mon.a (mon.0) 571 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.794199+0000 mon.a (mon.0) 572 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.796267+0000 mon.a (mon.0) 573 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: cephadm 2024-04-06T03:18:33.797889+0000 mgr.y (mgr.14152) 265 : cephadm 1 Adjusting osd_memory_target on smithi153 to 8632M 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.804476+0000 mon.a (mon.0) 574 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.808736+0000 mon.a (mon.0) 575 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.810717+0000 mon.a (mon.0) 576 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:35.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:34 smithi153 bash[23852]: audit 2024-04-06T03:18:33.820457+0000 mon.a (mon.0) 577 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: cluster 2024-04-06T03:18:33.551155+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: cephadm 2024-04-06T03:18:33.772937+0000 mgr.y (mgr.14152) 264 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.782167+0000 mon.a (mon.0) 570 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.791771+0000 mon.a (mon.0) 571 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.794199+0000 mon.a (mon.0) 572 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.796267+0000 mon.a (mon.0) 573 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: cephadm 2024-04-06T03:18:33.797889+0000 mgr.y (mgr.14152) 265 : cephadm 1 Adjusting osd_memory_target on smithi153 to 8632M 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.804476+0000 mon.a (mon.0) 574 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.808736+0000 mon.a (mon.0) 575 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.810717+0000 mon.a (mon.0) 576 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:35.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[21467]: audit 2024-04-06T03:18:33.820457+0000 mon.a (mon.0) 577 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: cluster 2024-04-06T03:18:33.551155+0000 mgr.y (mgr.14152) 263 : cluster 0 pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: cephadm 2024-04-06T03:18:33.772937+0000 mgr.y (mgr.14152) 264 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.782167+0000 mon.a (mon.0) 570 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.791771+0000 mon.a (mon.0) 571 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.794199+0000 mon.a (mon.0) 572 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.796267+0000 mon.a (mon.0) 573 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: cephadm 2024-04-06T03:18:33.797889+0000 mgr.y (mgr.14152) 265 : cephadm 1 Adjusting osd_memory_target on smithi153 to 8632M 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.804476+0000 mon.a (mon.0) 574 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.808736+0000 mon.a (mon.0) 575 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.810717+0000 mon.a (mon.0) 576 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:18:35.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:34 smithi050 bash[29221]: audit 2024-04-06T03:18:33.820457+0000 mon.a (mon.0) 577 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:18:37.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:36 smithi153 bash[23852]: cluster 2024-04-06T03:18:35.551694+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:18:37.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:36 smithi050 bash[21467]: cluster 2024-04-06T03:18:35.551694+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:18:37.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:36 smithi050 bash[29221]: cluster 2024-04-06T03:18:35.551694+0000 mgr.y (mgr.14152) 266 : cluster 0 pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:18:38.159 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:18:39.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:39 smithi153 bash[23852]: cluster 2024-04-06T03:18:37.552376+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:18:39.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:39 smithi050 bash[21467]: cluster 2024-04-06T03:18:37.552376+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:18:39.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:39 smithi050 bash[29221]: cluster 2024-04-06T03:18:37.552376+0000 mgr.y (mgr.14152) 267 : cluster 0 pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:18:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:41 smithi153 bash[23852]: cluster 2024-04-06T03:18:39.552947+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:41 smithi153 bash[23852]: audit 2024-04-06T03:18:40.922630+0000 mon.a (mon.0) 578 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:18:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:41 smithi153 bash[23852]: audit 2024-04-06T03:18:40.929548+0000 mon.a (mon.0) 579 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:18:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:41 smithi153 bash[23852]: audit 2024-04-06T03:18:40.931503+0000 mon.a (mon.0) 580 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[21467]: cluster 2024-04-06T03:18:39.552947+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[21467]: audit 2024-04-06T03:18:40.922630+0000 mon.a (mon.0) 578 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[21467]: audit 2024-04-06T03:18:40.929548+0000 mon.a (mon.0) 579 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[21467]: audit 2024-04-06T03:18:40.931503+0000 mon.a (mon.0) 580 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[29221]: cluster 2024-04-06T03:18:39.552947+0000 mgr.y (mgr.14152) 268 : cluster 0 pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[29221]: audit 2024-04-06T03:18:40.922630+0000 mon.a (mon.0) 578 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[29221]: audit 2024-04-06T03:18:40.929548+0000 mon.a (mon.0) 579 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:18:41.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:41 smithi050 bash[29221]: audit 2024-04-06T03:18:40.931503+0000 mon.a (mon.0) 580 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:18:42.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:42 smithi153 bash[23852]: audit 2024-04-06T03:18:40.915356+0000 mgr.y (mgr.14152) 269 : audit 0 from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:42.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:42 smithi050 bash[21467]: audit 2024-04-06T03:18:40.915356+0000 mgr.y (mgr.14152) 269 : audit 0 from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:42 smithi050 bash[29221]: audit 2024-04-06T03:18:40.915356+0000 mgr.y (mgr.14152) 269 : audit 0 from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:18:43.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:43 smithi153 bash[23852]: cluster 2024-04-06T03:18:41.553502+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:43.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:43 smithi050 bash[21467]: cluster 2024-04-06T03:18:41.553502+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:43 smithi050 bash[29221]: cluster 2024-04-06T03:18:41.553502+0000 mgr.y (mgr.14152) 270 : cluster 0 pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:45.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:45 smithi153 bash[23852]: cluster 2024-04-06T03:18:43.554094+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:45.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:45 smithi050 bash[21467]: cluster 2024-04-06T03:18:43.554094+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:45.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:45 smithi050 bash[29221]: cluster 2024-04-06T03:18:43.554094+0000 mgr.y (mgr.14152) 271 : cluster 0 pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:47.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:47 smithi153 bash[23852]: cluster 2024-04-06T03:18:45.554617+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:47.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:47 smithi050 bash[21467]: cluster 2024-04-06T03:18:45.554617+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:47.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:47 smithi050 bash[29221]: cluster 2024-04-06T03:18:45.554617+0000 mgr.y (mgr.14152) 272 : cluster 0 pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:47.555204+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:47.759724+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:47.759849+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:48.044583+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:48.044672+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:48.063388+0000 mon.a (mon.0) 581 : cluster 0 osdmap e52: 6 total, 6 up, 6 in 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: audit 2024-04-06T03:18:48.981100+0000 mon.b (mon.1) 18 : audit 1 from='client.? 172.21.15.153:0/1603177151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]: dispatch 2024-04-06T03:18:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: audit 2024-04-06T03:18:48.983943+0000 mon.a (mon.0) 582 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]: dispatch 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: audit 2024-04-06T03:18:48.987509+0000 mon.a (mon.0) 583 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]': finished 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: cluster 2024-04-06T03:18:48.995067+0000 mon.a (mon.0) 584 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[21467]: audit 2024-04-06T03:18:48.995910+0000 mon.a (mon.0) 585 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:47.555204+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:47.759724+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:47.759849+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:48.044583+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:48.044672+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:48.063388+0000 mon.a (mon.0) 581 : cluster 0 osdmap e52: 6 total, 6 up, 6 in 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: audit 2024-04-06T03:18:48.981100+0000 mon.b (mon.1) 18 : audit 1 from='client.? 172.21.15.153:0/1603177151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]: dispatch 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: audit 2024-04-06T03:18:48.983943+0000 mon.a (mon.0) 582 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]: dispatch 2024-04-06T03:18:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: audit 2024-04-06T03:18:48.987509+0000 mon.a (mon.0) 583 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]': finished 2024-04-06T03:18:49.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: cluster 2024-04-06T03:18:48.995067+0000 mon.a (mon.0) 584 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-04-06T03:18:49.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:49 smithi050 bash[29221]: audit 2024-04-06T03:18:48.995910+0000 mon.a (mon.0) 585 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:18:49.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:47.555204+0000 mgr.y (mgr.14152) 273 : cluster 0 pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:49.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:47.759724+0000 osd.4 (osd.4) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:49.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:47.759849+0000 osd.4 (osd.4) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:48.044583+0000 osd.0 (osd.0) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:48.044672+0000 osd.0 (osd.0) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:48.063388+0000 mon.a (mon.0) 581 : cluster 0 osdmap e52: 6 total, 6 up, 6 in 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: audit 2024-04-06T03:18:48.981100+0000 mon.b (mon.1) 18 : audit 1 from='client.? 172.21.15.153:0/1603177151' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]: dispatch 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: audit 2024-04-06T03:18:48.983943+0000 mon.a (mon.0) 582 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]: dispatch 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: audit 2024-04-06T03:18:48.987509+0000 mon.a (mon.0) 583 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98027d53-4324-44d1-89ce-8da85be58efb"}]': finished 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: cluster 2024-04-06T03:18:48.995067+0000 mon.a (mon.0) 584 : cluster 0 osdmap e53: 7 total, 6 up, 7 in 2024-04-06T03:18:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:49 smithi153 bash[23852]: audit 2024-04-06T03:18:48.995910+0000 mon.a (mon.0) 585 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:18:50.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:50 smithi050 bash[21467]: audit 2024-04-06T03:18:49.727783+0000 mon.b (mon.1) 19 : audit 0 from='client.? 172.21.15.153:0/1167036262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:18:50.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:50 smithi050 bash[29221]: audit 2024-04-06T03:18:49.727783+0000 mon.b (mon.1) 19 : audit 0 from='client.? 172.21.15.153:0/1167036262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:18:50.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:50 smithi153 bash[23852]: audit 2024-04-06T03:18:49.727783+0000 mon.b (mon.1) 19 : audit 0 from='client.? 172.21.15.153:0/1167036262' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:18:51.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:51 smithi050 bash[21467]: cluster 2024-04-06T03:18:49.555732+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:51 smithi050 bash[29221]: cluster 2024-04-06T03:18:49.555732+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:51.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:51 smithi153 bash[23852]: cluster 2024-04-06T03:18:49.555732+0000 mgr.y (mgr.14152) 274 : cluster 0 pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:53.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:53 smithi050 bash[21467]: cluster 2024-04-06T03:18:51.556296+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:53.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:53 smithi050 bash[29221]: cluster 2024-04-06T03:18:51.556296+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:53.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:53 smithi153 bash[23852]: cluster 2024-04-06T03:18:51.556296+0000 mgr.y (mgr.14152) 275 : cluster 0 pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:55.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:55 smithi050 bash[21467]: cluster 2024-04-06T03:18:53.556844+0000 mgr.y (mgr.14152) 276 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:55 smithi050 bash[29221]: cluster 2024-04-06T03:18:53.556844+0000 mgr.y (mgr.14152) 276 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:55.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:55 smithi153 bash[23852]: cluster 2024-04-06T03:18:53.556844+0000 mgr.y (mgr.14152) 276 : cluster 0 pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:57.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:57 smithi050 bash[21467]: cluster 2024-04-06T03:18:55.557355+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:57.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:57 smithi050 bash[29221]: cluster 2024-04-06T03:18:55.557355+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:57.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:57 smithi153 bash[23852]: cluster 2024-04-06T03:18:55.557355+0000 mgr.y (mgr.14152) 277 : cluster 0 pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:59.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:18:59 smithi153 bash[23852]: cluster 2024-04-06T03:18:57.557899+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:59.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:18:59 smithi050 bash[21467]: cluster 2024-04-06T03:18:57.557899+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:18:59.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:18:59 smithi050 bash[29221]: cluster 2024-04-06T03:18:57.557899+0000 mgr.y (mgr.14152) 278 : cluster 0 pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:01.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:01 smithi050 bash[21467]: cluster 2024-04-06T03:18:59.558460+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:01.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:01 smithi050 bash[29221]: cluster 2024-04-06T03:18:59.558460+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:01.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:01 smithi153 bash[23852]: cluster 2024-04-06T03:18:59.558460+0000 mgr.y (mgr.14152) 279 : cluster 0 pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:02.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:02 smithi050 bash[21467]: audit 2024-04-06T03:19:01.419708+0000 mon.a (mon.0) 586 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:19:02.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:02 smithi050 bash[21467]: audit 2024-04-06T03:19:01.421615+0000 mon.a (mon.0) 587 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:02.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:02 smithi050 bash[29221]: audit 2024-04-06T03:19:01.419708+0000 mon.a (mon.0) 586 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:19:02.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:02 smithi050 bash[29221]: audit 2024-04-06T03:19:01.421615+0000 mon.a (mon.0) 587 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:02.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:02 smithi153 bash[23852]: audit 2024-04-06T03:19:01.419708+0000 mon.a (mon.0) 586 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-06T03:19:02.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:02 smithi153 bash[23852]: audit 2024-04-06T03:19:01.421615+0000 mon.a (mon.0) 587 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:03.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:03 smithi153 bash[23852]: cephadm 2024-04-06T03:19:01.423258+0000 mgr.y (mgr.14152) 280 : cephadm 1 Deploying daemon osd.6 on smithi153 2024-04-06T03:19:03.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:03 smithi153 bash[23852]: cluster 2024-04-06T03:19:01.558980+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:03.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:03 smithi050 bash[21467]: cephadm 2024-04-06T03:19:01.423258+0000 mgr.y (mgr.14152) 280 : cephadm 1 Deploying daemon osd.6 on smithi153 2024-04-06T03:19:03.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:03 smithi050 bash[21467]: cluster 2024-04-06T03:19:01.558980+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:03.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:03 smithi050 bash[29221]: cephadm 2024-04-06T03:19:01.423258+0000 mgr.y (mgr.14152) 280 : cephadm 1 Deploying daemon osd.6 on smithi153 2024-04-06T03:19:03.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:03 smithi050 bash[29221]: cluster 2024-04-06T03:19:01.558980+0000 mgr.y (mgr.14152) 281 : cluster 0 pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:05.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 bash[23852]: cluster 2024-04-06T03:19:03.559574+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:05.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 bash[23852]: cluster 2024-04-06T03:19:04.024486+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:05.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 bash[23852]: cluster 2024-04-06T03:19:04.024575+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:05.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 bash[23852]: cluster 2024-04-06T03:19:04.114348+0000 mon.a (mon.0) 588 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-04-06T03:19:05.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 bash[23852]: audit 2024-04-06T03:19:04.114727+0000 mon.a (mon.0) 589 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:05.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[29221]: cluster 2024-04-06T03:19:03.559574+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[29221]: cluster 2024-04-06T03:19:04.024486+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[29221]: cluster 2024-04-06T03:19:04.024575+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[29221]: cluster 2024-04-06T03:19:04.114348+0000 mon.a (mon.0) 588 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[29221]: audit 2024-04-06T03:19:04.114727+0000 mon.a (mon.0) 589 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[21467]: cluster 2024-04-06T03:19:03.559574+0000 mgr.y (mgr.14152) 282 : cluster 0 pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[21467]: cluster 2024-04-06T03:19:04.024486+0000 osd.3 (osd.3) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[21467]: cluster 2024-04-06T03:19:04.024575+0000 osd.3 (osd.3) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[21467]: cluster 2024-04-06T03:19:04.114348+0000 mon.a (mon.0) 588 : cluster 0 osdmap e54: 7 total, 6 up, 7 in 2024-04-06T03:19:05.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:05 smithi050 bash[21467]: audit 2024-04-06T03:19:04.114727+0000 mon.a (mon.0) 589 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:05.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:05.731 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:05.732 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:05.732 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:06.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:06.020 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:06.020 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:06.020 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:19:05 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:07.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:07 smithi153 bash[23852]: cluster 2024-04-06T03:19:05.560129+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:07.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:07 smithi153 bash[23852]: audit 2024-04-06T03:19:06.038918+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:07.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:07 smithi153 bash[23852]: audit 2024-04-06T03:19:06.048900+0000 mon.a (mon.0) 591 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:07.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:07 smithi153 bash[23852]: audit 2024-04-06T03:19:06.053169+0000 mon.a (mon.0) 592 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[21467]: cluster 2024-04-06T03:19:05.560129+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[21467]: audit 2024-04-06T03:19:06.038918+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[21467]: audit 2024-04-06T03:19:06.048900+0000 mon.a (mon.0) 591 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[21467]: audit 2024-04-06T03:19:06.053169+0000 mon.a (mon.0) 592 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[29221]: cluster 2024-04-06T03:19:05.560129+0000 mgr.y (mgr.14152) 283 : cluster 0 pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[29221]: audit 2024-04-06T03:19:06.038918+0000 mon.a (mon.0) 590 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[29221]: audit 2024-04-06T03:19:06.048900+0000 mon.a (mon.0) 591 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:07 smithi050 bash[29221]: audit 2024-04-06T03:19:06.053169+0000 mon.a (mon.0) 592 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:09.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:09 smithi153 bash[23852]: cluster 2024-04-06T03:19:07.560760+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:09 smithi050 bash[21467]: cluster 2024-04-06T03:19:07.560760+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:09.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:09 smithi050 bash[29221]: cluster 2024-04-06T03:19:07.560760+0000 mgr.y (mgr.14152) 284 : cluster 0 pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:11.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:11 smithi153 bash[23852]: cluster 2024-04-06T03:19:09.561313+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:11.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:11 smithi050 bash[21467]: cluster 2024-04-06T03:19:09.561313+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:11.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:11 smithi050 bash[29221]: cluster 2024-04-06T03:19:09.561313+0000 mgr.y (mgr.14152) 285 : cluster 0 pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:13.423 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:13 smithi153 bash[23852]: cluster 2024-04-06T03:19:11.561878+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:13.423 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:13 smithi153 bash[23852]: audit 2024-04-06T03:19:12.408491+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:19:13.423 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:13 smithi153 bash[23852]: audit 2024-04-06T03:19:12.411446+0000 mon.a (mon.0) 593 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:19:13.423 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:13 smithi153 bash[23852]: audit 2024-04-06T03:19:13.007579+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:13.423 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:13 smithi153 bash[23852]: audit 2024-04-06T03:19:13.018019+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:13.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[29221]: cluster 2024-04-06T03:19:11.561878+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:13.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[29221]: audit 2024-04-06T03:19:12.408491+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:19:13.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[29221]: audit 2024-04-06T03:19:12.411446+0000 mon.a (mon.0) 593 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:19:13.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[29221]: audit 2024-04-06T03:19:13.007579+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:13.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[29221]: audit 2024-04-06T03:19:13.018019+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:13.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[21467]: cluster 2024-04-06T03:19:11.561878+0000 mgr.y (mgr.14152) 286 : cluster 0 pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:13.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[21467]: audit 2024-04-06T03:19:12.408491+0000 mon.b (mon.1) 20 : audit 1 from='osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:19:13.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[21467]: audit 2024-04-06T03:19:12.411446+0000 mon.a (mon.0) 593 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-06T03:19:13.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[21467]: audit 2024-04-06T03:19:13.007579+0000 mon.a (mon.0) 594 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:13.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:13 smithi050 bash[21467]: audit 2024-04-06T03:19:13.018019+0000 mon.a (mon.0) 595 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.046023+0000 mon.a (mon.0) 596 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: cluster 2024-04-06T03:19:13.052042+0000 mon.a (mon.0) 597 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.053013+0000 mon.a (mon.0) 598 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.053296+0000 mon.b (mon.1) 21 : audit 1 from='osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.065379+0000 mon.a (mon.0) 599 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.701764+0000 mon.a (mon.0) 600 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:14.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.703869+0000 mon.a (mon.0) 601 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[29221]: audit 2024-04-06T03:19:13.714497+0000 mon.a (mon.0) 602 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.046023+0000 mon.a (mon.0) 596 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: cluster 2024-04-06T03:19:13.052042+0000 mon.a (mon.0) 597 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.053013+0000 mon.a (mon.0) 598 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.053296+0000 mon.b (mon.1) 21 : audit 1 from='osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.065379+0000 mon.a (mon.0) 599 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.701764+0000 mon.a (mon.0) 600 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.703869+0000 mon.a (mon.0) 601 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:14.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:14 smithi050 bash[21467]: audit 2024-04-06T03:19:13.714497+0000 mon.a (mon.0) 602 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.046023+0000 mon.a (mon.0) 596 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: cluster 2024-04-06T03:19:13.052042+0000 mon.a (mon.0) 597 : cluster 0 osdmap e55: 7 total, 6 up, 7 in 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.053013+0000 mon.a (mon.0) 598 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.053296+0000 mon.b (mon.1) 21 : audit 1 from='osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.065379+0000 mon.a (mon.0) 599 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.701764+0000 mon.a (mon.0) 600 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.703869+0000 mon.a (mon.0) 601 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:14.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:14 smithi153 bash[23852]: audit 2024-04-06T03:19:13.714497+0000 mon.a (mon.0) 602 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:15.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:15 smithi153 bash[23852]: cluster 2024-04-06T03:19:13.562495+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:15.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:15 smithi153 bash[23852]: audit 2024-04-06T03:19:14.065291+0000 mon.a (mon.0) 603 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:19:15.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:15 smithi153 bash[23852]: cluster 2024-04-06T03:19:14.073678+0000 mon.a (mon.0) 604 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-04-06T03:19:15.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:15 smithi153 bash[23852]: audit 2024-04-06T03:19:14.074999+0000 mon.a (mon.0) 605 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:15.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:15 smithi153 bash[23852]: audit 2024-04-06T03:19:14.080954+0000 mon.a (mon.0) 606 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:15.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:15 smithi153 bash[23852]: audit 2024-04-06T03:19:14.625285+0000 mon.a (mon.0) 607 : audit 1 from='osd.6 ' entity='osd.6' 2024-04-06T03:19:15.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[29221]: cluster 2024-04-06T03:19:13.562495+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:15.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[29221]: audit 2024-04-06T03:19:14.065291+0000 mon.a (mon.0) 603 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[29221]: cluster 2024-04-06T03:19:14.073678+0000 mon.a (mon.0) 604 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[29221]: audit 2024-04-06T03:19:14.074999+0000 mon.a (mon.0) 605 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[29221]: audit 2024-04-06T03:19:14.080954+0000 mon.a (mon.0) 606 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[29221]: audit 2024-04-06T03:19:14.625285+0000 mon.a (mon.0) 607 : audit 1 from='osd.6 ' entity='osd.6' 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[21467]: cluster 2024-04-06T03:19:13.562495+0000 mgr.y (mgr.14152) 287 : cluster 0 pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[21467]: audit 2024-04-06T03:19:14.065291+0000 mon.a (mon.0) 603 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[21467]: cluster 2024-04-06T03:19:14.073678+0000 mon.a (mon.0) 604 : cluster 0 osdmap e56: 7 total, 6 up, 7 in 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[21467]: audit 2024-04-06T03:19:14.074999+0000 mon.a (mon.0) 605 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[21467]: audit 2024-04-06T03:19:14.080954+0000 mon.a (mon.0) 606 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:15.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:15 smithi050 bash[21467]: audit 2024-04-06T03:19:14.625285+0000 mon.a (mon.0) 607 : audit 1 from='osd.6 ' entity='osd.6' 2024-04-06T03:19:16.341 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 6 on host 'smithi153' 2024-04-06T03:19:16.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:16 smithi153 bash[23852]: cluster 2024-04-06T03:19:13.429135+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:16.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:16 smithi153 bash[23852]: cluster 2024-04-06T03:19:13.429200+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:16.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:16 smithi153 bash[23852]: cluster 2024-04-06T03:19:15.074795+0000 mon.a (mon.0) 608 : cluster 1 osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208] boot 2024-04-06T03:19:16.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:16 smithi153 bash[23852]: cluster 2024-04-06T03:19:15.074957+0000 mon.a (mon.0) 609 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-04-06T03:19:16.359 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:16 smithi153 bash[23852]: audit 2024-04-06T03:19:15.075977+0000 mon.a (mon.0) 610 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:16.359 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:16 smithi153 bash[23852]: cluster 2024-04-06T03:19:16.075918+0000 mon.a (mon.0) 611 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-04-06T03:19:16.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[21467]: cluster 2024-04-06T03:19:13.429135+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:16.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[21467]: cluster 2024-04-06T03:19:13.429200+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[21467]: cluster 2024-04-06T03:19:15.074795+0000 mon.a (mon.0) 608 : cluster 1 osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208] boot 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[21467]: cluster 2024-04-06T03:19:15.074957+0000 mon.a (mon.0) 609 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[21467]: audit 2024-04-06T03:19:15.075977+0000 mon.a (mon.0) 610 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[21467]: cluster 2024-04-06T03:19:16.075918+0000 mon.a (mon.0) 611 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[29221]: cluster 2024-04-06T03:19:13.429135+0000 osd.6 (osd.6) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[29221]: cluster 2024-04-06T03:19:13.429200+0000 osd.6 (osd.6) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[29221]: cluster 2024-04-06T03:19:15.074795+0000 mon.a (mon.0) 608 : cluster 1 osd.6 [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208] boot 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[29221]: cluster 2024-04-06T03:19:15.074957+0000 mon.a (mon.0) 609 : cluster 0 osdmap e57: 7 total, 7 up, 7 in 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[29221]: audit 2024-04-06T03:19:15.075977+0000 mon.a (mon.0) 610 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:19:16.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:16 smithi050 bash[29221]: cluster 2024-04-06T03:19:16.075918+0000 mon.a (mon.0) 611 : cluster 0 osdmap e58: 7 total, 7 up, 7 in 2024-04-06T03:19:17.020 DEBUG:teuthology.orchestra.run.smithi153:osd.6> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.6.service 2024-04-06T03:19:17.023 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_1... 2024-04-06T03:19:17.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-06T03:19:17.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:17 smithi153 bash[23852]: cluster 2024-04-06T03:19:15.563162+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 561 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:17.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:17 smithi153 bash[23852]: audit 2024-04-06T03:19:16.300005+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:17.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:17 smithi153 bash[23852]: audit 2024-04-06T03:19:16.309840+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:17.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:17 smithi153 bash[23852]: audit 2024-04-06T03:19:16.310974+0000 mon.a (mon.0) 614 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:17.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:17 smithi153 bash[23852]: cluster 2024-04-06T03:19:17.078451+0000 mon.a (mon.0) 615 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-04-06T03:19:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[21467]: cluster 2024-04-06T03:19:15.563162+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 561 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[21467]: audit 2024-04-06T03:19:16.300005+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[21467]: audit 2024-04-06T03:19:16.309840+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[21467]: audit 2024-04-06T03:19:16.310974+0000 mon.a (mon.0) 614 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[21467]: cluster 2024-04-06T03:19:17.078451+0000 mon.a (mon.0) 615 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[29221]: cluster 2024-04-06T03:19:15.563162+0000 mgr.y (mgr.14152) 288 : cluster 0 pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 561 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[29221]: audit 2024-04-06T03:19:16.300005+0000 mon.a (mon.0) 612 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[29221]: audit 2024-04-06T03:19:16.309840+0000 mon.a (mon.0) 613 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[29221]: audit 2024-04-06T03:19:16.310974+0000 mon.a (mon.0) 614 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:19:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:17 smithi050 bash[29221]: cluster 2024-04-06T03:19:17.078451+0000 mon.a (mon.0) 615 : cluster 0 osdmap e59: 7 total, 7 up, 7 in 2024-04-06T03:19:19.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:19 smithi153 bash[23852]: cluster 2024-04-06T03:19:17.563774+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v281: 1 pgs: 1 remapped+peering; 577 KiB data, 561 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:19.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:19 smithi153 bash[23852]: cluster 2024-04-06T03:19:18.031612+0000 mon.a (mon.0) 616 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-04-06T03:19:19.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:19 smithi050 bash[21467]: cluster 2024-04-06T03:19:17.563774+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v281: 1 pgs: 1 remapped+peering; 577 KiB data, 561 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:19.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:19 smithi050 bash[21467]: cluster 2024-04-06T03:19:18.031612+0000 mon.a (mon.0) 616 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-04-06T03:19:19.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:19 smithi050 bash[29221]: cluster 2024-04-06T03:19:17.563774+0000 mgr.y (mgr.14152) 289 : cluster 0 pgmap v281: 1 pgs: 1 remapped+peering; 577 KiB data, 561 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:19.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:19 smithi050 bash[29221]: cluster 2024-04-06T03:19:18.031612+0000 mon.a (mon.0) 616 : cluster 0 osdmap e60: 7 total, 7 up, 7 in 2024-04-06T03:19:20.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:20 smithi153 bash[23852]: cluster 2024-04-06T03:19:19.033885+0000 mon.a (mon.0) 617 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-04-06T03:19:20.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:20 smithi153 bash[23852]: cluster 2024-04-06T03:19:19.388285+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-04-06T03:19:20.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:20 smithi153 bash[23852]: cluster 2024-04-06T03:19:19.392129+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-04-06T03:19:20.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:20 smithi050 bash[21467]: cluster 2024-04-06T03:19:19.033885+0000 mon.a (mon.0) 617 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-04-06T03:19:20.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:20 smithi050 bash[21467]: cluster 2024-04-06T03:19:19.388285+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-04-06T03:19:20.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:20 smithi050 bash[21467]: cluster 2024-04-06T03:19:19.392129+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-04-06T03:19:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:20 smithi050 bash[29221]: cluster 2024-04-06T03:19:19.033885+0000 mon.a (mon.0) 617 : cluster 0 osdmap e61: 7 total, 7 up, 7 in 2024-04-06T03:19:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:20 smithi050 bash[29221]: cluster 2024-04-06T03:19:19.388285+0000 osd.6 (osd.6) 3 : cluster 0 1.0 scrub starts 2024-04-06T03:19:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:20 smithi050 bash[29221]: cluster 2024-04-06T03:19:19.392129+0000 osd.6 (osd.6) 4 : cluster 0 1.0 scrub ok 2024-04-06T03:19:21.287 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:19:21.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:21 smithi153 bash[23852]: cluster 2024-04-06T03:19:19.564324+0000 mgr.y (mgr.14152) 290 : cluster 0 pgmap v284: 1 pgs: 1 remapped+peering; 577 KiB data, 562 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:21.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:21 smithi050 bash[21467]: cluster 2024-04-06T03:19:19.564324+0000 mgr.y (mgr.14152) 290 : cluster 0 pgmap v284: 1 pgs: 1 remapped+peering; 577 KiB data, 562 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:21.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:21 smithi050 bash[29221]: cluster 2024-04-06T03:19:19.564324+0000 mgr.y (mgr.14152) 290 : cluster 0 pgmap v284: 1 pgs: 1 remapped+peering; 577 KiB data, 562 MiB used, 625 GiB / 626 GiB avail 2024-04-06T03:19:23.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:23 smithi153 bash[23852]: cluster 2024-04-06T03:19:21.564899+0000 mgr.y (mgr.14152) 291 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T03:19:23.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:23 smithi050 bash[29221]: cluster 2024-04-06T03:19:21.564899+0000 mgr.y (mgr.14152) 291 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T03:19:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:23 smithi050 bash[21467]: cluster 2024-04-06T03:19:21.564899+0000 mgr.y (mgr.14152) 291 : cluster 0 pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 199 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T03:19:25.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:25 smithi153 bash[23852]: cluster 2024-04-06T03:19:23.565522+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-06T03:19:25.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:25 smithi050 bash[21467]: cluster 2024-04-06T03:19:23.565522+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-06T03:19:25.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:25 smithi050 bash[29221]: cluster 2024-04-06T03:19:23.565522+0000 mgr.y (mgr.14152) 292 : cluster 0 pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-06T03:19:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-06T03:19:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-06T03:19:25.689 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-06T03:19:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-06T03:19:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0139049 s, 754 MB/s 2024-04-06T03:19:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-06T03:19:26.586 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-04-06T03:19:27.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: cluster 2024-04-06T03:19:25.566070+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.918334+0000 mon.a (mon.0) 618 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.928484+0000 mon.a (mon.0) 619 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.931104+0000 mon.a (mon.0) 620 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.932838+0000 mon.a (mon.0) 621 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.934537+0000 mon.a (mon.0) 622 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.943005+0000 mon.a (mon.0) 623 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.947733+0000 mon.a (mon.0) 624 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.949696+0000 mon.a (mon.0) 625 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:27.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[21467]: audit 2024-04-06T03:19:26.959798+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: cluster 2024-04-06T03:19:25.566070+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.918334+0000 mon.a (mon.0) 618 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.928484+0000 mon.a (mon.0) 619 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.931104+0000 mon.a (mon.0) 620 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.932838+0000 mon.a (mon.0) 621 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.934537+0000 mon.a (mon.0) 622 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.943005+0000 mon.a (mon.0) 623 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.947733+0000 mon.a (mon.0) 624 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.949696+0000 mon.a (mon.0) 625 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:27.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:27 smithi050 bash[29221]: audit 2024-04-06T03:19:26.959798+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: cluster 2024-04-06T03:19:25.566070+0000 mgr.y (mgr.14152) 293 : cluster 0 pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:19:27.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.918334+0000 mon.a (mon.0) 618 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.928484+0000 mon.a (mon.0) 619 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.931104+0000 mon.a (mon.0) 620 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.932838+0000 mon.a (mon.0) 621 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.934537+0000 mon.a (mon.0) 622 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.943005+0000 mon.a (mon.0) 623 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.947733+0000 mon.a (mon.0) 624 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.949696+0000 mon.a (mon.0) 625 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:19:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:27 smithi153 bash[23852]: audit 2024-04-06T03:19:26.959798+0000 mon.a (mon.0) 626 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:19:28.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:28 smithi050 bash[21467]: cephadm 2024-04-06T03:19:26.909092+0000 mgr.y (mgr.14152) 294 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:19:28.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:28 smithi050 bash[21467]: cephadm 2024-04-06T03:19:26.935850+0000 mgr.y (mgr.14152) 295 : cephadm 1 Adjusting osd_memory_target on smithi153 to 5755M 2024-04-06T03:19:28.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:28 smithi050 bash[29221]: cephadm 2024-04-06T03:19:26.909092+0000 mgr.y (mgr.14152) 294 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:19:28.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:28 smithi050 bash[29221]: cephadm 2024-04-06T03:19:26.935850+0000 mgr.y (mgr.14152) 295 : cephadm 1 Adjusting osd_memory_target on smithi153 to 5755M 2024-04-06T03:19:28.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:28 smithi153 bash[23852]: cephadm 2024-04-06T03:19:26.909092+0000 mgr.y (mgr.14152) 294 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:19:28.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:28 smithi153 bash[23852]: cephadm 2024-04-06T03:19:26.935850+0000 mgr.y (mgr.14152) 295 : cephadm 1 Adjusting osd_memory_target on smithi153 to 5755M 2024-04-06T03:19:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[21467]: cluster 2024-04-06T03:19:27.566648+0000 mgr.y (mgr.14152) 296 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-06T03:19:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[21467]: cluster 2024-04-06T03:19:27.819962+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[21467]: cluster 2024-04-06T03:19:27.820060+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:29.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[21467]: cluster 2024-04-06T03:19:28.076352+0000 mon.a (mon.0) 627 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-04-06T03:19:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[29221]: cluster 2024-04-06T03:19:27.566648+0000 mgr.y (mgr.14152) 296 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-06T03:19:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[29221]: cluster 2024-04-06T03:19:27.819962+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[29221]: cluster 2024-04-06T03:19:27.820060+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:29 smithi050 bash[29221]: cluster 2024-04-06T03:19:28.076352+0000 mon.a (mon.0) 627 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-04-06T03:19:29.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:29 smithi153 bash[23852]: cluster 2024-04-06T03:19:27.566648+0000 mgr.y (mgr.14152) 296 : cluster 0 pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-06T03:19:29.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:29 smithi153 bash[23852]: cluster 2024-04-06T03:19:27.819962+0000 osd.2 (osd.2) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:29.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:29 smithi153 bash[23852]: cluster 2024-04-06T03:19:27.820060+0000 osd.2 (osd.2) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:29.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:29 smithi153 bash[23852]: cluster 2024-04-06T03:19:28.076352+0000 mon.a (mon.0) 627 : cluster 0 osdmap e62: 7 total, 7 up, 7 in 2024-04-06T03:19:31.324 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:19:31.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:31 smithi153 bash[23852]: cluster 2024-04-06T03:19:29.567204+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:19:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:31 smithi050 bash[21467]: cluster 2024-04-06T03:19:29.567204+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:19:31.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:31 smithi050 bash[29221]: cluster 2024-04-06T03:19:29.567204+0000 mgr.y (mgr.14152) 297 : cluster 0 pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-06T03:19:33.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:33 smithi153 bash[23852]: cluster 2024-04-06T03:19:31.344626+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:33.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:33 smithi153 bash[23852]: cluster 2024-04-06T03:19:31.344716+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:33.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:33 smithi153 bash[23852]: cluster 2024-04-06T03:19:31.567804+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:33.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:33 smithi153 bash[23852]: cluster 2024-04-06T03:19:32.088505+0000 mon.a (mon.0) 628 : cluster 0 osdmap e63: 7 total, 7 up, 7 in 2024-04-06T03:19:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[21467]: cluster 2024-04-06T03:19:31.344626+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[21467]: cluster 2024-04-06T03:19:31.344716+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[21467]: cluster 2024-04-06T03:19:31.567804+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[21467]: cluster 2024-04-06T03:19:32.088505+0000 mon.a (mon.0) 628 : cluster 0 osdmap e63: 7 total, 7 up, 7 in 2024-04-06T03:19:33.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[29221]: cluster 2024-04-06T03:19:31.344626+0000 osd.1 (osd.1) 13 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:33.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[29221]: cluster 2024-04-06T03:19:31.344716+0000 osd.1 (osd.1) 14 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:33.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[29221]: cluster 2024-04-06T03:19:31.567804+0000 mgr.y (mgr.14152) 298 : cluster 0 pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:33.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:33 smithi050 bash[29221]: cluster 2024-04-06T03:19:32.088505+0000 mon.a (mon.0) 628 : cluster 0 osdmap e63: 7 total, 7 up, 7 in 2024-04-06T03:19:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:34 smithi050 bash[21467]: audit 2024-04-06T03:19:34.032683+0000 mon.a (mon.0) 629 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:19:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:34 smithi050 bash[21467]: audit 2024-04-06T03:19:34.039039+0000 mon.a (mon.0) 630 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:19:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:34 smithi050 bash[21467]: audit 2024-04-06T03:19:34.040772+0000 mon.a (mon.0) 631 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:34 smithi050 bash[29221]: audit 2024-04-06T03:19:34.032683+0000 mon.a (mon.0) 629 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:19:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:34 smithi050 bash[29221]: audit 2024-04-06T03:19:34.039039+0000 mon.a (mon.0) 630 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:19:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:34 smithi050 bash[29221]: audit 2024-04-06T03:19:34.040772+0000 mon.a (mon.0) 631 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:34 smithi153 bash[23852]: audit 2024-04-06T03:19:34.032683+0000 mon.a (mon.0) 629 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-06T03:19:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:34 smithi153 bash[23852]: audit 2024-04-06T03:19:34.039039+0000 mon.a (mon.0) 630 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-06T03:19:34.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:34 smithi153 bash[23852]: audit 2024-04-06T03:19:34.040772+0000 mon.a (mon.0) 631 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:35 smithi050 bash[21467]: cluster 2024-04-06T03:19:33.568361+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:35 smithi050 bash[21467]: audit 2024-04-06T03:19:34.025259+0000 mgr.y (mgr.14152) 300 : audit 0 from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:35.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:35 smithi050 bash[29221]: cluster 2024-04-06T03:19:33.568361+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:35 smithi050 bash[29221]: audit 2024-04-06T03:19:34.025259+0000 mgr.y (mgr.14152) 300 : audit 0 from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:35.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:35 smithi153 bash[23852]: cluster 2024-04-06T03:19:33.568361+0000 mgr.y (mgr.14152) 299 : cluster 0 pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:35.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:35 smithi153 bash[23852]: audit 2024-04-06T03:19:34.025259+0000 mgr.y (mgr.14152) 300 : audit 0 from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:19:37.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:37 smithi050 bash[21467]: cluster 2024-04-06T03:19:35.568911+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:37.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:37 smithi050 bash[29221]: cluster 2024-04-06T03:19:35.568911+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:37.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:37 smithi153 bash[23852]: cluster 2024-04-06T03:19:35.568911+0000 mgr.y (mgr.14152) 301 : cluster 0 pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:39.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:39 smithi153 bash[23852]: cluster 2024-04-06T03:19:37.569434+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:39.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:39 smithi153 bash[23852]: cluster 2024-04-06T03:19:38.254355+0000 osd.5 (osd.5) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:39.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:39 smithi153 bash[23852]: cluster 2024-04-06T03:19:38.254482+0000 osd.5 (osd.5) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:39.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:39 smithi050 bash[21467]: cluster 2024-04-06T03:19:37.569434+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:39.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:39 smithi050 bash[21467]: cluster 2024-04-06T03:19:38.254355+0000 osd.5 (osd.5) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:39.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:39 smithi050 bash[21467]: cluster 2024-04-06T03:19:38.254482+0000 osd.5 (osd.5) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:39.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:39 smithi050 bash[29221]: cluster 2024-04-06T03:19:37.569434+0000 mgr.y (mgr.14152) 302 : cluster 0 pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:39.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:39 smithi050 bash[29221]: cluster 2024-04-06T03:19:38.254355+0000 osd.5 (osd.5) 3 : cluster 0 purged_snaps scrub starts 2024-04-06T03:19:39.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:39 smithi050 bash[29221]: cluster 2024-04-06T03:19:38.254482+0000 osd.5 (osd.5) 4 : cluster 0 purged_snaps scrub ok 2024-04-06T03:19:40.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:40 smithi153 bash[23852]: cluster 2024-04-06T03:19:39.120149+0000 mon.a (mon.0) 632 : cluster 0 osdmap e64: 7 total, 7 up, 7 in 2024-04-06T03:19:40.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:40 smithi050 bash[21467]: cluster 2024-04-06T03:19:39.120149+0000 mon.a (mon.0) 632 : cluster 0 osdmap e64: 7 total, 7 up, 7 in 2024-04-06T03:19:40.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:40 smithi050 bash[29221]: cluster 2024-04-06T03:19:39.120149+0000 mon.a (mon.0) 632 : cluster 0 osdmap e64: 7 total, 7 up, 7 in 2024-04-06T03:19:41.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:41 smithi153 bash[23852]: cluster 2024-04-06T03:19:39.570009+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:41.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:41 smithi050 bash[21467]: cluster 2024-04-06T03:19:39.570009+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:41.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:41 smithi050 bash[29221]: cluster 2024-04-06T03:19:39.570009+0000 mgr.y (mgr.14152) 303 : cluster 0 pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:43.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[21467]: cluster 2024-04-06T03:19:41.570604+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:43.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[21467]: audit 2024-04-06T03:19:42.540722+0000 mon.b (mon.1) 22 : audit 1 from='client.? 172.21.15.153:0/2904395567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]: dispatch 2024-04-06T03:19:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[21467]: audit 2024-04-06T03:19:42.544150+0000 mon.a (mon.0) 633 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]: dispatch 2024-04-06T03:19:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[21467]: audit 2024-04-06T03:19:42.547674+0000 mon.a (mon.0) 634 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]': finished 2024-04-06T03:19:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[21467]: cluster 2024-04-06T03:19:42.555481+0000 mon.a (mon.0) 635 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-04-06T03:19:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[21467]: audit 2024-04-06T03:19:42.556076+0000 mon.a (mon.0) 636 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:19:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[29221]: cluster 2024-04-06T03:19:41.570604+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:43.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[29221]: audit 2024-04-06T03:19:42.540722+0000 mon.b (mon.1) 22 : audit 1 from='client.? 172.21.15.153:0/2904395567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]: dispatch 2024-04-06T03:19:43.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[29221]: audit 2024-04-06T03:19:42.544150+0000 mon.a (mon.0) 633 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]: dispatch 2024-04-06T03:19:43.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[29221]: audit 2024-04-06T03:19:42.547674+0000 mon.a (mon.0) 634 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]': finished 2024-04-06T03:19:43.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[29221]: cluster 2024-04-06T03:19:42.555481+0000 mon.a (mon.0) 635 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-04-06T03:19:43.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:43 smithi050 bash[29221]: audit 2024-04-06T03:19:42.556076+0000 mon.a (mon.0) 636 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:19:43.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:43 smithi153 bash[23852]: cluster 2024-04-06T03:19:41.570604+0000 mgr.y (mgr.14152) 304 : cluster 0 pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:43.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:43 smithi153 bash[23852]: audit 2024-04-06T03:19:42.540722+0000 mon.b (mon.1) 22 : audit 1 from='client.? 172.21.15.153:0/2904395567' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]: dispatch 2024-04-06T03:19:43.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:43 smithi153 bash[23852]: audit 2024-04-06T03:19:42.544150+0000 mon.a (mon.0) 633 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]: dispatch 2024-04-06T03:19:43.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:43 smithi153 bash[23852]: audit 2024-04-06T03:19:42.547674+0000 mon.a (mon.0) 634 : audit 1 from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd"}]': finished 2024-04-06T03:19:43.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:43 smithi153 bash[23852]: cluster 2024-04-06T03:19:42.555481+0000 mon.a (mon.0) 635 : cluster 0 osdmap e65: 8 total, 7 up, 8 in 2024-04-06T03:19:43.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:43 smithi153 bash[23852]: audit 2024-04-06T03:19:42.556076+0000 mon.a (mon.0) 636 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:19:44.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:44 smithi050 bash[29221]: audit 2024-04-06T03:19:43.287418+0000 mon.b (mon.1) 23 : audit 0 from='client.? 172.21.15.153:0/1914181794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:19:44.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:44 smithi050 bash[21467]: audit 2024-04-06T03:19:43.287418+0000 mon.b (mon.1) 23 : audit 0 from='client.? 172.21.15.153:0/1914181794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:19:44.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:44 smithi153 bash[23852]: audit 2024-04-06T03:19:43.287418+0000 mon.b (mon.1) 23 : audit 0 from='client.? 172.21.15.153:0/1914181794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-06T03:19:45.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:45 smithi050 bash[21467]: cluster 2024-04-06T03:19:43.571176+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:45.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:45 smithi050 bash[29221]: cluster 2024-04-06T03:19:43.571176+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:45 smithi153 bash[23852]: cluster 2024-04-06T03:19:43.571176+0000 mgr.y (mgr.14152) 305 : cluster 0 pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:47.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:47 smithi050 bash[21467]: cluster 2024-04-06T03:19:45.571737+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:47.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:47 smithi050 bash[29221]: cluster 2024-04-06T03:19:45.571737+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:47 smithi153 bash[23852]: cluster 2024-04-06T03:19:45.571737+0000 mgr.y (mgr.14152) 306 : cluster 0 pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:48.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:48 smithi050 bash[21467]: cluster 2024-04-06T03:19:47.572315+0000 mgr.y (mgr.14152) 307 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:48.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:48 smithi050 bash[29221]: cluster 2024-04-06T03:19:47.572315+0000 mgr.y (mgr.14152) 307 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:48.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:48 smithi153 bash[23852]: cluster 2024-04-06T03:19:47.572315+0000 mgr.y (mgr.14152) 307 : cluster 0 pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:50.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:50 smithi050 bash[21467]: cluster 2024-04-06T03:19:49.572883+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:50.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:50 smithi050 bash[29221]: cluster 2024-04-06T03:19:49.572883+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:51.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:50 smithi153 bash[23852]: cluster 2024-04-06T03:19:49.572883+0000 mgr.y (mgr.14152) 308 : cluster 0 pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:52 smithi050 bash[21467]: cluster 2024-04-06T03:19:51.573454+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:52.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:52 smithi050 bash[29221]: cluster 2024-04-06T03:19:51.573454+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:53.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:52 smithi153 bash[23852]: cluster 2024-04-06T03:19:51.573454+0000 mgr.y (mgr.14152) 309 : cluster 0 pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:54.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:54 smithi050 bash[21467]: cluster 2024-04-06T03:19:53.574027+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:54.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:54 smithi050 bash[29221]: cluster 2024-04-06T03:19:53.574027+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:55.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:54 smithi153 bash[23852]: cluster 2024-04-06T03:19:53.574027+0000 mgr.y (mgr.14152) 310 : cluster 0 pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:55.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:55 smithi153 bash[23852]: audit 2024-04-06T03:19:55.009310+0000 mon.a (mon.0) 637 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:19:55.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:55 smithi153 bash[23852]: audit 2024-04-06T03:19:55.011084+0000 mon.a (mon.0) 638 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:55.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:55 smithi153 bash[23852]: cephadm 2024-04-06T03:19:55.012753+0000 mgr.y (mgr.14152) 311 : cephadm 1 Deploying daemon osd.7 on smithi153 2024-04-06T03:19:55.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:55 smithi050 bash[21467]: audit 2024-04-06T03:19:55.009310+0000 mon.a (mon.0) 637 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:19:55.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:55 smithi050 bash[21467]: audit 2024-04-06T03:19:55.011084+0000 mon.a (mon.0) 638 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:55.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:55 smithi050 bash[21467]: cephadm 2024-04-06T03:19:55.012753+0000 mgr.y (mgr.14152) 311 : cephadm 1 Deploying daemon osd.7 on smithi153 2024-04-06T03:19:55.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:55 smithi050 bash[29221]: audit 2024-04-06T03:19:55.009310+0000 mon.a (mon.0) 637 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-06T03:19:55.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:55 smithi050 bash[29221]: audit 2024-04-06T03:19:55.011084+0000 mon.a (mon.0) 638 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:19:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:55 smithi050 bash[29221]: cephadm 2024-04-06T03:19:55.012753+0000 mgr.y (mgr.14152) 311 : cephadm 1 Deploying daemon osd.7 on smithi153 2024-04-06T03:19:56.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:56 smithi050 bash[21467]: cluster 2024-04-06T03:19:55.574583+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:56.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:56 smithi050 bash[29221]: cluster 2024-04-06T03:19:55.574583+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:57.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:56 smithi153 bash[23852]: cluster 2024-04-06T03:19:55.574583+0000 mgr.y (mgr.14152) 312 : cluster 0 pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:58.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:19:58 smithi050 bash[21467]: cluster 2024-04-06T03:19:57.575117+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:58.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:19:58 smithi050 bash[29221]: cluster 2024-04-06T03:19:57.575117+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:59.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:58 smithi153 bash[23852]: cluster 2024-04-06T03:19:57.575117+0000 mgr.y (mgr.14152) 313 : cluster 0 pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:19:59.319 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.319 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.320 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.320 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.320 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.579 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.580 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.580 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:19:59.581 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:19:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:00.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:00 smithi153 bash[23852]: cluster 2024-04-06T03:19:59.575678+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:00.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:00 smithi153 bash[23852]: audit 2024-04-06T03:19:59.639591+0000 mon.a (mon.0) 639 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:00.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:00 smithi153 bash[23852]: audit 2024-04-06T03:19:59.640361+0000 mon.a (mon.0) 640 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:00.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:00 smithi153 bash[23852]: audit 2024-04-06T03:19:59.650444+0000 mon.a (mon.0) 641 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:00.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:00 smithi153 bash[23852]: cluster 2024-04-06T03:20:00.000173+0000 mon.a (mon.0) 642 : cluster 1 overall HEALTH_OK 2024-04-06T03:20:00.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[21467]: cluster 2024-04-06T03:19:59.575678+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:00.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[21467]: audit 2024-04-06T03:19:59.639591+0000 mon.a (mon.0) 639 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:00.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[21467]: audit 2024-04-06T03:19:59.640361+0000 mon.a (mon.0) 640 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:00.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[21467]: audit 2024-04-06T03:19:59.650444+0000 mon.a (mon.0) 641 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:00.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[21467]: cluster 2024-04-06T03:20:00.000173+0000 mon.a (mon.0) 642 : cluster 1 overall HEALTH_OK 2024-04-06T03:20:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[29221]: cluster 2024-04-06T03:19:59.575678+0000 mgr.y (mgr.14152) 314 : cluster 0 pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[29221]: audit 2024-04-06T03:19:59.639591+0000 mon.a (mon.0) 639 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[29221]: audit 2024-04-06T03:19:59.640361+0000 mon.a (mon.0) 640 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[29221]: audit 2024-04-06T03:19:59.650444+0000 mon.a (mon.0) 641 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:00.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:00 smithi050 bash[29221]: cluster 2024-04-06T03:20:00.000173+0000 mon.a (mon.0) 642 : cluster 1 overall HEALTH_OK 2024-04-06T03:20:02.790 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:02 smithi153 bash[23852]: cluster 2024-04-06T03:20:01.576382+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:02.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:02 smithi050 bash[21467]: cluster 2024-04-06T03:20:01.576382+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:02.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:02 smithi050 bash[29221]: cluster 2024-04-06T03:20:01.576382+0000 mgr.y (mgr.14152) 315 : cluster 0 pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:04.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:04 smithi153 bash[23852]: cluster 2024-04-06T03:20:03.425137+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:04.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:04 smithi153 bash[23852]: cluster 2024-04-06T03:20:03.425246+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:04.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:04 smithi050 bash[29221]: cluster 2024-04-06T03:20:03.425137+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:04.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:04 smithi050 bash[29221]: cluster 2024-04-06T03:20:03.425246+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:04.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:04 smithi050 bash[21467]: cluster 2024-04-06T03:20:03.425137+0000 osd.4 (osd.4) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:04.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:04 smithi050 bash[21467]: cluster 2024-04-06T03:20:03.425246+0000 osd.4 (osd.4) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:05.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[21467]: cluster 2024-04-06T03:20:03.315920+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:05.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[21467]: cluster 2024-04-06T03:20:03.316016+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:05.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[21467]: cluster 2024-04-06T03:20:03.577071+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:05.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[21467]: cluster 2024-04-06T03:20:04.055855+0000 mon.a (mon.0) 643 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-04-06T03:20:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[21467]: audit 2024-04-06T03:20:04.056195+0000 mon.a (mon.0) 644 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[29221]: cluster 2024-04-06T03:20:03.315920+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[29221]: cluster 2024-04-06T03:20:03.316016+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[29221]: cluster 2024-04-06T03:20:03.577071+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[29221]: cluster 2024-04-06T03:20:04.055855+0000 mon.a (mon.0) 643 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-04-06T03:20:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:05 smithi050 bash[29221]: audit 2024-04-06T03:20:04.056195+0000 mon.a (mon.0) 644 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:05 smithi153 bash[23852]: cluster 2024-04-06T03:20:03.315920+0000 osd.0 (osd.0) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:05 smithi153 bash[23852]: cluster 2024-04-06T03:20:03.316016+0000 osd.0 (osd.0) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:05.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:05 smithi153 bash[23852]: cluster 2024-04-06T03:20:03.577071+0000 mgr.y (mgr.14152) 316 : cluster 0 pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:05.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:05 smithi153 bash[23852]: cluster 2024-04-06T03:20:04.055855+0000 mon.a (mon.0) 643 : cluster 0 osdmap e66: 8 total, 7 up, 8 in 2024-04-06T03:20:05.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:05 smithi153 bash[23852]: audit 2024-04-06T03:20:04.056195+0000 mon.a (mon.0) 644 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:07.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: cluster 2024-04-06T03:20:05.577710+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.058995+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.062550+0000 mon.a (mon.0) 645 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.700027+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.710263+0000 mon.a (mon.0) 647 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.712662+0000 mon.a (mon.0) 648 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.714751+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:07.324 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:07 smithi153 bash[23852]: audit 2024-04-06T03:20:06.725035+0000 mon.a (mon.0) 650 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: cluster 2024-04-06T03:20:05.577710+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.058995+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:20:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.062550+0000 mon.a (mon.0) 645 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:20:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.700027+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.710263+0000 mon.a (mon.0) 647 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.712662+0000 mon.a (mon.0) 648 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.714751+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[21467]: audit 2024-04-06T03:20:06.725035+0000 mon.a (mon.0) 650 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: cluster 2024-04-06T03:20:05.577710+0000 mgr.y (mgr.14152) 317 : cluster 0 pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.058995+0000 mon.b (mon.1) 24 : audit 1 from='osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.062550+0000 mon.a (mon.0) 645 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.700027+0000 mon.a (mon.0) 646 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.710263+0000 mon.a (mon.0) 647 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.712662+0000 mon.a (mon.0) 648 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.714751+0000 mon.a (mon.0) 649 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:07.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:07 smithi050 bash[29221]: audit 2024-04-06T03:20:06.725035+0000 mon.a (mon.0) 650 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:08.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[21467]: audit 2024-04-06T03:20:07.064572+0000 mon.a (mon.0) 651 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:20:08.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[21467]: audit 2024-04-06T03:20:07.068177+0000 mon.b (mon.1) 25 : audit 1 from='osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:20:08.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[21467]: cluster 2024-04-06T03:20:07.070480+0000 mon.a (mon.0) 652 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-04-06T03:20:08.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[21467]: audit 2024-04-06T03:20:07.071089+0000 mon.a (mon.0) 653 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:08.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[21467]: audit 2024-04-06T03:20:07.071830+0000 mon.a (mon.0) 654 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:20:08.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[29221]: audit 2024-04-06T03:20:07.064572+0000 mon.a (mon.0) 651 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:20:08.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[29221]: audit 2024-04-06T03:20:07.068177+0000 mon.b (mon.1) 25 : audit 1 from='osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:20:08.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[29221]: cluster 2024-04-06T03:20:07.070480+0000 mon.a (mon.0) 652 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-04-06T03:20:08.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[29221]: audit 2024-04-06T03:20:07.071089+0000 mon.a (mon.0) 653 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:08.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:08 smithi050 bash[29221]: audit 2024-04-06T03:20:07.071830+0000 mon.a (mon.0) 654 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:20:08.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:08 smithi153 bash[23852]: audit 2024-04-06T03:20:07.064572+0000 mon.a (mon.0) 651 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-06T03:20:08.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:08 smithi153 bash[23852]: audit 2024-04-06T03:20:07.068177+0000 mon.b (mon.1) 25 : audit 1 from='osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:20:08.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:08 smithi153 bash[23852]: cluster 2024-04-06T03:20:07.070480+0000 mon.a (mon.0) 652 : cluster 0 osdmap e67: 8 total, 7 up, 8 in 2024-04-06T03:20:08.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:08 smithi153 bash[23852]: audit 2024-04-06T03:20:07.071089+0000 mon.a (mon.0) 653 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:08.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:08 smithi153 bash[23852]: audit 2024-04-06T03:20:07.071830+0000 mon.a (mon.0) 654 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-06T03:20:09.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: cluster 2024-04-06T03:20:07.074306+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:09.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: cluster 2024-04-06T03:20:07.074398+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:09.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: cluster 2024-04-06T03:20:07.578365+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:09.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: audit 2024-04-06T03:20:08.072925+0000 mon.a (mon.0) 655 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:20:09.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: cluster 2024-04-06T03:20:08.082191+0000 mon.a (mon.0) 656 : cluster 0 osdmap e68: 8 total, 7 up, 8 in 2024-04-06T03:20:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: audit 2024-04-06T03:20:08.083071+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: audit 2024-04-06T03:20:08.085460+0000 mon.a (mon.0) 658 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:09 smithi153 bash[23852]: audit 2024-04-06T03:20:08.630836+0000 mon.a (mon.0) 659 : audit 1 from='osd.7 ' entity='osd.7' 2024-04-06T03:20:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: cluster 2024-04-06T03:20:07.074306+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: cluster 2024-04-06T03:20:07.074398+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: cluster 2024-04-06T03:20:07.578365+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: audit 2024-04-06T03:20:08.072925+0000 mon.a (mon.0) 655 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:20:09.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: cluster 2024-04-06T03:20:08.082191+0000 mon.a (mon.0) 656 : cluster 0 osdmap e68: 8 total, 7 up, 8 in 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: audit 2024-04-06T03:20:08.083071+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: audit 2024-04-06T03:20:08.085460+0000 mon.a (mon.0) 658 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[21467]: audit 2024-04-06T03:20:08.630836+0000 mon.a (mon.0) 659 : audit 1 from='osd.7 ' entity='osd.7' 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: cluster 2024-04-06T03:20:07.074306+0000 osd.7 (osd.7) 1 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: cluster 2024-04-06T03:20:07.074398+0000 osd.7 (osd.7) 2 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: cluster 2024-04-06T03:20:07.578365+0000 mgr.y (mgr.14152) 318 : cluster 0 pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: audit 2024-04-06T03:20:08.072925+0000 mon.a (mon.0) 655 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: cluster 2024-04-06T03:20:08.082191+0000 mon.a (mon.0) 656 : cluster 0 osdmap e68: 8 total, 7 up, 8 in 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: audit 2024-04-06T03:20:08.083071+0000 mon.a (mon.0) 657 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:09.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: audit 2024-04-06T03:20:08.085460+0000 mon.a (mon.0) 658 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:09 smithi050 bash[29221]: audit 2024-04-06T03:20:08.630836+0000 mon.a (mon.0) 659 : audit 1 from='osd.7 ' entity='osd.7' 2024-04-06T03:20:10.257 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-04-06T03:20:10.337 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:10 smithi153 bash[23852]: cluster 2024-04-06T03:20:09.082233+0000 mon.a (mon.0) 660 : cluster 1 osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924] boot 2024-04-06T03:20:10.337 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:10 smithi153 bash[23852]: cluster 2024-04-06T03:20:09.082299+0000 mon.a (mon.0) 661 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-04-06T03:20:10.337 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:10 smithi153 bash[23852]: audit 2024-04-06T03:20:09.082944+0000 mon.a (mon.0) 662 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:10.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:10 smithi050 bash[21467]: cluster 2024-04-06T03:20:09.082233+0000 mon.a (mon.0) 660 : cluster 1 osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924] boot 2024-04-06T03:20:10.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:10 smithi050 bash[21467]: cluster 2024-04-06T03:20:09.082299+0000 mon.a (mon.0) 661 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-04-06T03:20:10.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:10 smithi050 bash[21467]: audit 2024-04-06T03:20:09.082944+0000 mon.a (mon.0) 662 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:10.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:10 smithi050 bash[29221]: cluster 2024-04-06T03:20:09.082233+0000 mon.a (mon.0) 660 : cluster 1 osd.7 [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924] boot 2024-04-06T03:20:10.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:10 smithi050 bash[29221]: cluster 2024-04-06T03:20:09.082299+0000 mon.a (mon.0) 661 : cluster 0 osdmap e69: 8 total, 8 up, 8 in 2024-04-06T03:20:10.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:10 smithi050 bash[29221]: audit 2024-04-06T03:20:09.082944+0000 mon.a (mon.0) 662 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:20:10.928 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.7.service 2024-04-06T03:20:10.931 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-06T03:20:10.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-06T03:20:11.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[21467]: cluster 2024-04-06T03:20:09.579061+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:11.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[21467]: cluster 2024-04-06T03:20:10.104681+0000 mon.a (mon.0) 663 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-04-06T03:20:11.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[21467]: audit 2024-04-06T03:20:10.248566+0000 mon.a (mon.0) 664 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:11.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[21467]: audit 2024-04-06T03:20:10.261046+0000 mon.a (mon.0) 665 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:11.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[21467]: audit 2024-04-06T03:20:10.274439+0000 mon.a (mon.0) 666 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:11.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[29221]: cluster 2024-04-06T03:20:09.579061+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:11.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[29221]: cluster 2024-04-06T03:20:10.104681+0000 mon.a (mon.0) 663 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-04-06T03:20:11.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[29221]: audit 2024-04-06T03:20:10.248566+0000 mon.a (mon.0) 664 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:11.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[29221]: audit 2024-04-06T03:20:10.261046+0000 mon.a (mon.0) 665 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:11.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:11 smithi050 bash[29221]: audit 2024-04-06T03:20:10.274439+0000 mon.a (mon.0) 666 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:11.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:11 smithi153 bash[23852]: cluster 2024-04-06T03:20:09.579061+0000 mgr.y (mgr.14152) 319 : cluster 0 pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:11.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:11 smithi153 bash[23852]: cluster 2024-04-06T03:20:10.104681+0000 mon.a (mon.0) 663 : cluster 0 osdmap e70: 8 total, 8 up, 8 in 2024-04-06T03:20:11.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:11 smithi153 bash[23852]: audit 2024-04-06T03:20:10.248566+0000 mon.a (mon.0) 664 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:11.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:11 smithi153 bash[23852]: audit 2024-04-06T03:20:10.261046+0000 mon.a (mon.0) 665 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:11.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:11 smithi153 bash[23852]: audit 2024-04-06T03:20:10.274439+0000 mon.a (mon.0) 666 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:12.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:12 smithi050 bash[21467]: cluster 2024-04-06T03:20:11.110981+0000 mon.a (mon.0) 667 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-04-06T03:20:12.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:12 smithi050 bash[29221]: cluster 2024-04-06T03:20:11.110981+0000 mon.a (mon.0) 667 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-04-06T03:20:12.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:12 smithi153 bash[23852]: cluster 2024-04-06T03:20:11.110981+0000 mon.a (mon.0) 667 : cluster 0 osdmap e71: 8 total, 8 up, 8 in 2024-04-06T03:20:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:13 smithi050 bash[21467]: cluster 2024-04-06T03:20:11.579686+0000 mgr.y (mgr.14152) 320 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:13.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:13 smithi050 bash[21467]: cluster 2024-04-06T03:20:13.046185+0000 mon.a (mon.0) 668 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:20:13.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:13 smithi050 bash[29221]: cluster 2024-04-06T03:20:11.579686+0000 mgr.y (mgr.14152) 320 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:13.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:13 smithi050 bash[29221]: cluster 2024-04-06T03:20:13.046185+0000 mon.a (mon.0) 668 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:20:13.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:13 smithi153 bash[23852]: cluster 2024-04-06T03:20:11.579686+0000 mgr.y (mgr.14152) 320 : cluster 0 pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:13.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:13 smithi153 bash[23852]: cluster 2024-04-06T03:20:13.046185+0000 mon.a (mon.0) 668 : cluster 0 osdmap e72: 8 total, 8 up, 8 in 2024-04-06T03:20:15.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:15 smithi050 bash[21467]: cluster 2024-04-06T03:20:13.580456+0000 mgr.y (mgr.14152) 321 : cluster 0 pgmap v322: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-06T03:20:15.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:15 smithi050 bash[21467]: cluster 2024-04-06T03:20:14.056133+0000 mon.a (mon.0) 669 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:20:15.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:15 smithi050 bash[21467]: cluster 2024-04-06T03:20:14.116207+0000 mon.a (mon.0) 670 : cluster 3 Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:20:15.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:15 smithi050 bash[29221]: cluster 2024-04-06T03:20:13.580456+0000 mgr.y (mgr.14152) 321 : cluster 0 pgmap v322: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-06T03:20:15.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:15 smithi050 bash[29221]: cluster 2024-04-06T03:20:14.056133+0000 mon.a (mon.0) 669 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:20:15.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:15 smithi050 bash[29221]: cluster 2024-04-06T03:20:14.116207+0000 mon.a (mon.0) 670 : cluster 3 Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:20:15.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:15 smithi153 bash[23852]: cluster 2024-04-06T03:20:13.580456+0000 mgr.y (mgr.14152) 321 : cluster 0 pgmap v322: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-04-06T03:20:15.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:15 smithi153 bash[23852]: cluster 2024-04-06T03:20:14.056133+0000 mon.a (mon.0) 669 : cluster 0 osdmap e73: 8 total, 8 up, 8 in 2024-04-06T03:20:15.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:15 smithi153 bash[23852]: cluster 2024-04-06T03:20:14.116207+0000 mon.a (mon.0) 670 : cluster 3 Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-06T03:20:15.698 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:20:16.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:16 smithi153 bash[23852]: cluster 2024-04-06T03:20:16.052210+0000 mon.a (mon.0) 671 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-06T03:20:16.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:16 smithi153 bash[23852]: cluster 2024-04-06T03:20:16.052288+0000 mon.a (mon.0) 672 : cluster 1 Cluster is now healthy 2024-04-06T03:20:16.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:16 smithi153 bash[23852]: cluster 2024-04-06T03:20:16.068122+0000 mon.a (mon.0) 673 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-04-06T03:20:16.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:16 smithi050 bash[21467]: cluster 2024-04-06T03:20:16.052210+0000 mon.a (mon.0) 671 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-06T03:20:16.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:16 smithi050 bash[21467]: cluster 2024-04-06T03:20:16.052288+0000 mon.a (mon.0) 672 : cluster 1 Cluster is now healthy 2024-04-06T03:20:16.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:16 smithi050 bash[21467]: cluster 2024-04-06T03:20:16.068122+0000 mon.a (mon.0) 673 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-04-06T03:20:16.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:16 smithi050 bash[29221]: cluster 2024-04-06T03:20:16.052210+0000 mon.a (mon.0) 671 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-06T03:20:16.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:16 smithi050 bash[29221]: cluster 2024-04-06T03:20:16.052288+0000 mon.a (mon.0) 672 : cluster 1 Cluster is now healthy 2024-04-06T03:20:16.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:16 smithi050 bash[29221]: cluster 2024-04-06T03:20:16.068122+0000 mon.a (mon.0) 673 : cluster 0 osdmap e74: 8 total, 8 up, 8 in 2024-04-06T03:20:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:17 smithi050 bash[21467]: cluster 2024-04-06T03:20:15.408315+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:17 smithi050 bash[21467]: cluster 2024-04-06T03:20:15.408447+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:17 smithi050 bash[21467]: cluster 2024-04-06T03:20:15.581072+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T03:20:17.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:17 smithi050 bash[29221]: cluster 2024-04-06T03:20:15.408315+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:17 smithi050 bash[29221]: cluster 2024-04-06T03:20:15.408447+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:17 smithi050 bash[29221]: cluster 2024-04-06T03:20:15.581072+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T03:20:17.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:17 smithi153 bash[23852]: cluster 2024-04-06T03:20:15.408315+0000 osd.3 (osd.3) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:17.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:17 smithi153 bash[23852]: cluster 2024-04-06T03:20:15.408447+0000 osd.3 (osd.3) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:17.533 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:17 smithi153 bash[23852]: cluster 2024-04-06T03:20:15.581072+0000 mgr.y (mgr.14152) 322 : cluster 0 pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-06T03:20:18.846 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:20:19.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:19 smithi050 bash[21467]: cluster 2024-04-06T03:20:17.581646+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-06T03:20:19.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:19 smithi050 bash[21467]: audit 2024-04-06T03:20:18.856311+0000 mon.a (mon.0) 674 : audit 0 from='client.? 172.21.15.50:0/1903435004' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T03:20:19.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:19 smithi050 bash[29221]: cluster 2024-04-06T03:20:17.581646+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-06T03:20:19.214 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:19 smithi050 bash[29221]: audit 2024-04-06T03:20:18.856311+0000 mon.a (mon.0) 674 : audit 0 from='client.? 172.21.15.50:0/1903435004' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T03:20:19.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:19 smithi153 bash[23852]: cluster 2024-04-06T03:20:17.581646+0000 mgr.y (mgr.14152) 323 : cluster 0 pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-06T03:20:19.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:19 smithi153 bash[23852]: audit 2024-04-06T03:20:18.856311+0000 mon.a (mon.0) 674 : audit 0 from='client.? 172.21.15.50:0/1903435004' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-06T03:20:19.748 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":74,"num_osds":8,"num_up_osds":8,"osd_up_since":1712373609,"num_in_osds":8,"osd_in_since":1712373582,"num_remapped_pgs":0} 2024-04-06T03:20:19.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:20:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: cephadm 2024-04-06T03:20:19.466225+0000 mgr.y (mgr.14152) 324 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:20:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.475985+0000 mon.a (mon.0) 675 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.489142+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.491970+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.493816+0000 mon.a (mon.0) 678 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.495520+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.497249+0000 mon.a (mon.0) 680 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: cephadm 2024-04-06T03:20:19.498652+0000 mgr.y (mgr.14152) 325 : cephadm 1 Adjusting osd_memory_target on smithi153 to 4316M 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.505366+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.509438+0000 mon.a (mon.0) 682 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.511158+0000 mon.a (mon.0) 683 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: audit 2024-04-06T03:20:19.518971+0000 mon.a (mon.0) 684 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[21467]: cluster 2024-04-06T03:20:19.582262+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-06T03:20:20.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: cephadm 2024-04-06T03:20:19.466225+0000 mgr.y (mgr.14152) 324 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.475985+0000 mon.a (mon.0) 675 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.489142+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.491970+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.493816+0000 mon.a (mon.0) 678 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.495520+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.497249+0000 mon.a (mon.0) 680 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: cephadm 2024-04-06T03:20:19.498652+0000 mgr.y (mgr.14152) 325 : cephadm 1 Adjusting osd_memory_target on smithi153 to 4316M 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.505366+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.509438+0000 mon.a (mon.0) 682 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.511158+0000 mon.a (mon.0) 683 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: audit 2024-04-06T03:20:19.518971+0000 mon.a (mon.0) 684 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:20 smithi050 bash[29221]: cluster 2024-04-06T03:20:19.582262+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: cephadm 2024-04-06T03:20:19.466225+0000 mgr.y (mgr.14152) 324 : cephadm 1 Detected new or changed devices on smithi153 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.475985+0000 mon.a (mon.0) 675 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.489142+0000 mon.a (mon.0) 676 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.491970+0000 mon.a (mon.0) 677 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.493816+0000 mon.a (mon.0) 678 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.495520+0000 mon.a (mon.0) 679 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.497249+0000 mon.a (mon.0) 680 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:20:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: cephadm 2024-04-06T03:20:19.498652+0000 mgr.y (mgr.14152) 325 : cephadm 1 Adjusting osd_memory_target on smithi153 to 4316M 2024-04-06T03:20:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.505366+0000 mon.a (mon.0) 681 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.509438+0000 mon.a (mon.0) 682 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.511158+0000 mon.a (mon.0) 683 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: audit 2024-04-06T03:20:19.518971+0000 mon.a (mon.0) 684 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:20 smithi153 bash[23852]: cluster 2024-04-06T03:20:19.582262+0000 mgr.y (mgr.14152) 326 : cluster 0 pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-06T03:20:22.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:22 smithi050 bash[21467]: cluster 2024-04-06T03:20:21.582802+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:20:22.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:22 smithi050 bash[29221]: cluster 2024-04-06T03:20:21.582802+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:20:23.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:22 smithi153 bash[23852]: cluster 2024-04-06T03:20:21.582802+0000 mgr.y (mgr.14152) 327 : cluster 0 pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:20:24.511 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:20:25.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:25 smithi153 bash[23852]: cluster 2024-04-06T03:20:23.583387+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-06T03:20:25.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:25 smithi050 bash[21467]: cluster 2024-04-06T03:20:23.583387+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-06T03:20:25.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:25 smithi050 bash[29221]: cluster 2024-04-06T03:20:23.583387+0000 mgr.y (mgr.14152) 328 : cluster 0 pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-06T03:20:27.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:27 smithi050 bash[29221]: cluster 2024-04-06T03:20:25.583791+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:27.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:27 smithi050 bash[21467]: cluster 2024-04-06T03:20:25.583791+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:27.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:27 smithi153 bash[23852]: cluster 2024-04-06T03:20:25.583791+0000 mgr.y (mgr.14152) 329 : cluster 0 pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:27.622 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:20:27.622 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":74,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:10:26.969128+0000","modified":"2024-04-06T03:20:16.052911+0000","last_up_change":"2024-04-06T03:20:09.072730+0000","last_in_change":"2024-04-06T03:19:42.545394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:15:48.096900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e607f527-4592-4acd-a47f-76575615ae1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6803","nonce":3531754646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6805","nonce":3531754646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6809","nonce":3531754646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6807","nonce":3531754646}]},"public_addr":"172.21.15.50:6803/3531754646","cluster_addr":"172.21.15.50:6805/3531754646","heartbeat_back_addr":"172.21.15.50:6809/3531754646","heartbeat_front_addr":"172.21.15.50:6807/3531754646","state":["exists","up"]},{"osd":1,"uuid":"2ae8f784-d36a-4633-8630-2aed04e4ff10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6811","nonce":4128410069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6813","nonce":4128410069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6817","nonce":4128410069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6815","nonce":4128410069}]},"public_addr":"172.21.15.50:6811/4128410069","cluster_addr":"172.21.15.50:6813/4128410069","heartbeat_back_addr":"172.21.15.50:6817/4128410069","heartbeat_front_addr":"172.21.15.50:6815/4128410069","state":["exists","up"]},{"osd":2,"uuid":"43f31d09-5c16-4aea-8822-e0d877cc931b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6819","nonce":1307247904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6821","nonce":1307247904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6825","nonce":1307247904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6823","nonce":1307247904}]},"public_addr":"172.21.15.50:6819/1307247904","cluster_addr":"172.21.15.50:6821/1307247904","heartbeat_back_addr":"172.21.15.50:6825/1307247904","heartbeat_front_addr":"172.21.15.50:6823/1307247904","state":["exists","up"]},{"osd":3,"uuid":"4c913325-442e-4ee5-94f1-8b5bca0c3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6827","nonce":1397888508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6829","nonce":1397888508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6833","nonce":1397888508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6831","nonce":1397888508}]},"public_addr":"172.21.15.50:6827/1397888508","cluster_addr":"172.21.15.50:6829/1397888508","heartbeat_back_addr":"172.21.15.50:6833/1397888508","heartbeat_front_addr":"172.21.15.50:6831/1397888508","state":["exists","up"]},{"osd":4,"uuid":"50c02bfc-c2f2-4aae-b1d5-c38a86726b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6801","nonce":3377683473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6803","nonce":3377683473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6807","nonce":3377683473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6805","nonce":3377683473}]},"public_addr":"172.21.15.153:6801/3377683473","cluster_addr":"172.21.15.153:6803/3377683473","heartbeat_back_addr":"172.21.15.153:6807/3377683473","heartbeat_front_addr":"172.21.15.153:6805/3377683473","state":["exists","up"]},{"osd":5,"uuid":"c3e8194b-2c20-4ffb-859d-00c33902b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6809","nonce":357179117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6811","nonce":357179117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6815","nonce":357179117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6813","nonce":357179117}]},"public_addr":"172.21.15.153:6809/357179117","cluster_addr":"172.21.15.153:6811/357179117","heartbeat_back_addr":"172.21.15.153:6815/357179117","heartbeat_front_addr":"172.21.15.153:6813/357179117","state":["exists","up"]},{"osd":6,"uuid":"98027d53-4324-44d1-89ce-8da85be58efb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6817","nonce":1552538208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6819","nonce":1552538208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6823","nonce":1552538208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6821","nonce":1552538208}]},"public_addr":"172.21.15.153:6817/1552538208","cluster_addr":"172.21.15.153:6819/1552538208","heartbeat_back_addr":"172.21.15.153:6823/1552538208","heartbeat_front_addr":"172.21.15.153:6821/1552538208","state":["exists","up"]},{"osd":7,"uuid":"6b47d95a-1228-4fb1-a315-5371d9b42bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6825","nonce":2479193924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6827","nonce":2479193924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6831","nonce":2479193924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6829","nonce":2479193924}]},"public_addr":"172.21.15.153:6825/2479193924","cluster_addr":"172.21.15.153:6827/2479193924","heartbeat_back_addr":"172.21.15.153:6831/2479193924","heartbeat_front_addr":"172.21.15.153:6829/2479193924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:20:03.316021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:19:31.344721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:19:27.820067+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:20:15.408454+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:20:03.425253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:19:38.254487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:19:13.429203+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:20:07.074403+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:6801/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2170219983":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2891059051":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6801/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1268273764":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/2880368026":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6800/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/567103210":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/2738019433":"2024-04-07T03:11:33.442338+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:20:28.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:28 smithi050 bash[21467]: audit 2024-04-06T03:20:27.630978+0000 mon.a (mon.0) 685 : audit 0 from='client.? 172.21.15.50:0/2203779273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:20:28.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:28 smithi050 bash[29221]: audit 2024-04-06T03:20:27.630978+0000 mon.a (mon.0) 685 : audit 0 from='client.? 172.21.15.50:0/2203779273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:20:28.411 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:15:48.096900+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-06T03:20:28.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-06T03:20:28.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:28 smithi153 bash[23852]: audit 2024-04-06T03:20:27.630978+0000 mon.a (mon.0) 685 : audit 0 from='client.? 172.21.15.50:0/2203779273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:20:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:29 smithi050 bash[21467]: cluster 2024-04-06T03:20:27.584296+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:29.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:29 smithi050 bash[29221]: cluster 2024-04-06T03:20:27.584296+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:29.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:29 smithi153 bash[23852]: cluster 2024-04-06T03:20:27.584296+0000 mgr.y (mgr.14152) 330 : cluster 0 pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[21467]: cluster 2024-04-06T03:20:29.584903+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[21467]: cluster 2024-04-06T03:20:29.800755+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[21467]: cluster 2024-04-06T03:20:29.800867+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[21467]: cluster 2024-04-06T03:20:30.077274+0000 mon.a (mon.0) 686 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[29221]: cluster 2024-04-06T03:20:29.584903+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[29221]: cluster 2024-04-06T03:20:29.800755+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:31.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[29221]: cluster 2024-04-06T03:20:29.800867+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:31.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:31 smithi050 bash[29221]: cluster 2024-04-06T03:20:30.077274+0000 mon.a (mon.0) 686 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-04-06T03:20:31.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:31 smithi153 bash[23852]: cluster 2024-04-06T03:20:29.584903+0000 mgr.y (mgr.14152) 331 : cluster 0 pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:31.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:31 smithi153 bash[23852]: cluster 2024-04-06T03:20:29.800755+0000 osd.6 (osd.6) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:31.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:31 smithi153 bash[23852]: cluster 2024-04-06T03:20:29.800867+0000 osd.6 (osd.6) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:31.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:31 smithi153 bash[23852]: cluster 2024-04-06T03:20:30.077274+0000 mon.a (mon.0) 686 : cluster 0 osdmap e75: 8 total, 8 up, 8 in 2024-04-06T03:20:33.167 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:20:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:33 smithi050 bash[21467]: cluster 2024-04-06T03:20:31.585469+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:33.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:33 smithi050 bash[29221]: cluster 2024-04-06T03:20:31.585469+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:33.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:33 smithi153 bash[23852]: cluster 2024-04-06T03:20:31.585469+0000 mgr.y (mgr.14152) 332 : cluster 0 pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:35 smithi050 bash[21467]: cluster 2024-04-06T03:20:33.586149+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:35.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:35 smithi050 bash[29221]: cluster 2024-04-06T03:20:33.586149+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:35.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:35 smithi153 bash[23852]: cluster 2024-04-06T03:20:33.586149+0000 mgr.y (mgr.14152) 333 : cluster 0 pgmap v335: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:36.517 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-06T03:20:37.250 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi050 2024-04-06T03:20:37.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply rgw foo.a --placement '1;smithi050=foo.a' 2024-04-06T03:20:37.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:37 smithi050 bash[21467]: cluster 2024-04-06T03:20:35.586779+0000 mgr.y (mgr.14152) 334 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:37.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:37 smithi050 bash[21467]: audit 2024-04-06T03:20:36.527579+0000 mon.a (mon.0) 687 : audit 0 from='client.? 172.21.15.50:0/1188074021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T03:20:37.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:37 smithi050 bash[29221]: cluster 2024-04-06T03:20:35.586779+0000 mgr.y (mgr.14152) 334 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:37.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:37 smithi050 bash[29221]: audit 2024-04-06T03:20:36.527579+0000 mon.a (mon.0) 687 : audit 0 from='client.? 172.21.15.50:0/1188074021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T03:20:37.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:37 smithi153 bash[23852]: cluster 2024-04-06T03:20:35.586779+0000 mgr.y (mgr.14152) 334 : cluster 0 pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:37.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:37 smithi153 bash[23852]: audit 2024-04-06T03:20:36.527579+0000 mon.a (mon.0) 687 : audit 0 from='client.? 172.21.15.50:0/1188074021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-06T03:20:39.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:39 smithi050 bash[21467]: cluster 2024-04-06T03:20:37.587354+0000 mgr.y (mgr.14152) 335 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:39.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:39 smithi050 bash[29221]: cluster 2024-04-06T03:20:37.587354+0000 mgr.y (mgr.14152) 335 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:39.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:39 smithi153 bash[23852]: cluster 2024-04-06T03:20:37.587354+0000 mgr.y (mgr.14152) 335 : cluster 0 pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:41.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:41 smithi050 bash[21467]: cluster 2024-04-06T03:20:39.587940+0000 mgr.y (mgr.14152) 336 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:41.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:41 smithi050 bash[29221]: cluster 2024-04-06T03:20:39.587940+0000 mgr.y (mgr.14152) 336 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:41.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:41 smithi153 bash[23852]: cluster 2024-04-06T03:20:39.587940+0000 mgr.y (mgr.14152) 336 : cluster 0 pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:41.997 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:20:43.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:43 smithi050 bash[21467]: cluster 2024-04-06T03:20:41.588558+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:43.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:43 smithi050 bash[29221]: cluster 2024-04-06T03:20:41.588558+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:43.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:43 smithi153 bash[23852]: cluster 2024-04-06T03:20:41.588558+0000 mgr.y (mgr.14152) 337 : cluster 0 pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:44.728 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled rgw.foo.a update... 2024-04-06T03:20:45.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:45 smithi153 bash[23852]: cluster 2024-04-06T03:20:43.589201+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:45.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:45 smithi153 bash[23852]: cluster 2024-04-06T03:20:44.108719+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-04-06T03:20:45.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:45 smithi153 bash[23852]: cluster 2024-04-06T03:20:44.112641+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-04-06T03:20:45.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:45 smithi153 bash[23852]: audit 2024-04-06T03:20:44.737878+0000 mon.a (mon.0) 688 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:45.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:45 smithi153 bash[23852]: audit 2024-04-06T03:20:44.768862+0000 mon.a (mon.0) 689 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:45.388 DEBUG:teuthology.orchestra.run.smithi050:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@rgw.foo.a.service 2024-04-06T03:20:45.391 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi153 2024-04-06T03:20:45.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool create datapool 3 3 replicated 2024-04-06T03:20:45.399 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[29221]: cluster 2024-04-06T03:20:43.589201+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:45.399 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[29221]: cluster 2024-04-06T03:20:44.108719+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[29221]: cluster 2024-04-06T03:20:44.112641+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[29221]: audit 2024-04-06T03:20:44.737878+0000 mon.a (mon.0) 688 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[29221]: audit 2024-04-06T03:20:44.768862+0000 mon.a (mon.0) 689 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[21467]: cluster 2024-04-06T03:20:43.589201+0000 mgr.y (mgr.14152) 338 : cluster 0 pgmap v340: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[21467]: cluster 2024-04-06T03:20:44.108719+0000 osd.7 (osd.7) 3 : cluster 0 1.0 scrub starts 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[21467]: cluster 2024-04-06T03:20:44.112641+0000 osd.7 (osd.7) 4 : cluster 0 1.0 scrub ok 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[21467]: audit 2024-04-06T03:20:44.737878+0000 mon.a (mon.0) 688 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:45.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:45 smithi050 bash[21467]: audit 2024-04-06T03:20:44.768862+0000 mon.a (mon.0) 689 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:44.726250+0000 mgr.y (mgr.14152) 339 : audit 0 from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: cephadm 2024-04-06T03:20:44.730283+0000 mgr.y (mgr.14152) 340 : cephadm 1 Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.451246+0000 mon.a (mon.0) 690 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.453105+0000 mon.a (mon.0) 691 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.463210+0000 mon.a (mon.0) 692 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.469553+0000 mon.a (mon.0) 693 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.472739+0000 mon.a (mon.0) 694 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:20:46.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.484539+0000 mon.a (mon.0) 695 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[21467]: audit 2024-04-06T03:20:45.488436+0000 mon.a (mon.0) 696 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:44.726250+0000 mgr.y (mgr.14152) 339 : audit 0 from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: cephadm 2024-04-06T03:20:44.730283+0000 mgr.y (mgr.14152) 340 : cephadm 1 Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.451246+0000 mon.a (mon.0) 690 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.453105+0000 mon.a (mon.0) 691 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.463210+0000 mon.a (mon.0) 692 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.469553+0000 mon.a (mon.0) 693 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.472739+0000 mon.a (mon.0) 694 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.484539+0000 mon.a (mon.0) 695 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:46.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:46 smithi050 bash[29221]: audit 2024-04-06T03:20:45.488436+0000 mon.a (mon.0) 696 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:44.726250+0000 mgr.y (mgr.14152) 339 : audit 0 from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: cephadm 2024-04-06T03:20:44.730283+0000 mgr.y (mgr.14152) 340 : cephadm 1 Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.451246+0000 mon.a (mon.0) 690 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.453105+0000 mon.a (mon.0) 691 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.463210+0000 mon.a (mon.0) 692 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.469553+0000 mon.a (mon.0) 693 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-06T03:20:46.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.472739+0000 mon.a (mon.0) 694 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-06T03:20:46.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.484539+0000 mon.a (mon.0) 695 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:46.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:46 smithi153 bash[23852]: audit 2024-04-06T03:20:45.488436+0000 mon.a (mon.0) 696 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:47.426 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:47 smithi050 bash[21467]: cephadm 2024-04-06T03:20:45.490319+0000 mgr.y (mgr.14152) 341 : cephadm 1 Deploying daemon rgw.foo.a on smithi050 2024-04-06T03:20:47.426 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:47 smithi050 bash[21467]: cluster 2024-04-06T03:20:45.589831+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:47.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:47 smithi050 bash[29221]: cephadm 2024-04-06T03:20:45.490319+0000 mgr.y (mgr.14152) 341 : cephadm 1 Deploying daemon rgw.foo.a on smithi050 2024-04-06T03:20:47.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:47 smithi050 bash[29221]: cluster 2024-04-06T03:20:45.589831+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:47.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:47 smithi153 bash[23852]: cephadm 2024-04-06T03:20:45.490319+0000 mgr.y (mgr.14152) 341 : cephadm 1 Deploying daemon rgw.foo.a on smithi050 2024-04-06T03:20:47.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:47 smithi153 bash[23852]: cluster 2024-04-06T03:20:45.589831+0000 mgr.y (mgr.14152) 342 : cluster 0 pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:48.243 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.243 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.244 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.244 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.244 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[29221]: cluster 2024-04-06T03:20:46.719514+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[29221]: cluster 2024-04-06T03:20:46.719607+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[29221]: cluster 2024-04-06T03:20:47.404256+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[29221]: cluster 2024-04-06T03:20:47.404345+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[29221]: cluster 2024-04-06T03:20:47.490395+0000 mon.a (mon.0) 697 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[21467]: cluster 2024-04-06T03:20:46.719514+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[21467]: cluster 2024-04-06T03:20:46.719607+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[21467]: cluster 2024-04-06T03:20:47.404256+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:48.245 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[21467]: cluster 2024-04-06T03:20:47.404345+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:48.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 bash[21467]: cluster 2024-04-06T03:20:47.490395+0000 mon.a (mon.0) 697 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-04-06T03:20:48.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.499 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.500 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.500 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.500 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.500 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:20:48.501 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:20:48 smithi050 systemd[1]: Started Ceph rgw.foo.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:20:48.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:48 smithi153 bash[23852]: cluster 2024-04-06T03:20:46.719514+0000 osd.2 (osd.2) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:48.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:48 smithi153 bash[23852]: cluster 2024-04-06T03:20:46.719607+0000 osd.2 (osd.2) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:48.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:48 smithi153 bash[23852]: cluster 2024-04-06T03:20:47.404256+0000 osd.5 (osd.5) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:48.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:48 smithi153 bash[23852]: cluster 2024-04-06T03:20:47.404345+0000 osd.5 (osd.5) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:48.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:48 smithi153 bash[23852]: cluster 2024-04-06T03:20:47.490395+0000 mon.a (mon.0) 697 : cluster 0 osdmap e76: 8 total, 8 up, 8 in 2024-04-06T03:20:49.459 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: cluster 2024-04-06T03:20:47.590423+0000 mgr.y (mgr.14152) 343 : cluster 0 pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: audit 2024-04-06T03:20:48.554370+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: audit 2024-04-06T03:20:48.560803+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: audit 2024-04-06T03:20:48.617689+0000 mon.a (mon.0) 700 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: audit 2024-04-06T03:20:48.624913+0000 mon.a (mon.0) 701 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: audit 2024-04-06T03:20:48.631223+0000 mon.a (mon.0) 702 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[21467]: audit 2024-04-06T03:20:48.696199+0000 mon.a (mon.0) 703 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: cluster 2024-04-06T03:20:47.590423+0000 mgr.y (mgr.14152) 343 : cluster 0 pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: audit 2024-04-06T03:20:48.554370+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: audit 2024-04-06T03:20:48.560803+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: audit 2024-04-06T03:20:48.617689+0000 mon.a (mon.0) 700 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: audit 2024-04-06T03:20:48.624913+0000 mon.a (mon.0) 701 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: audit 2024-04-06T03:20:48.631223+0000 mon.a (mon.0) 702 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:49 smithi050 bash[29221]: audit 2024-04-06T03:20:48.696199+0000 mon.a (mon.0) 703 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: cluster 2024-04-06T03:20:47.590423+0000 mgr.y (mgr.14152) 343 : cluster 0 pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: audit 2024-04-06T03:20:48.554370+0000 mon.a (mon.0) 698 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: audit 2024-04-06T03:20:48.560803+0000 mon.a (mon.0) 699 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: audit 2024-04-06T03:20:48.617689+0000 mon.a (mon.0) 700 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: audit 2024-04-06T03:20:48.624913+0000 mon.a (mon.0) 701 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: audit 2024-04-06T03:20:48.631223+0000 mon.a (mon.0) 702 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:49.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:49 smithi153 bash[23852]: audit 2024-04-06T03:20:48.696199+0000 mon.a (mon.0) 703 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:20:50.126 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:20:50.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:50 smithi050 bash[21467]: cephadm 2024-04-06T03:20:48.618673+0000 mgr.y (mgr.14152) 344 : cephadm 1 Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-06T03:20:50.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:50 smithi050 bash[29221]: cephadm 2024-04-06T03:20:48.618673+0000 mgr.y (mgr.14152) 344 : cephadm 1 Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-06T03:20:50.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:50 smithi153 bash[23852]: cephadm 2024-04-06T03:20:48.618673+0000 mgr.y (mgr.14152) 344 : cephadm 1 Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-06T03:20:51.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:51 smithi153 bash[23852]: cluster 2024-04-06T03:20:49.590792+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:51.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:51 smithi153 bash[23852]: cluster 2024-04-06T03:20:50.120560+0000 mon.a (mon.0) 704 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-04-06T03:20:51.280 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:51 smithi153 bash[23852]: audit 2024-04-06T03:20:50.123749+0000 mon.a (mon.0) 705 : audit 1 from='client.? 172.21.15.50:0/1103877100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-06T03:20:51.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:51 smithi050 bash[21467]: cluster 2024-04-06T03:20:49.590792+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:51.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:51 smithi050 bash[21467]: cluster 2024-04-06T03:20:50.120560+0000 mon.a (mon.0) 704 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-04-06T03:20:51.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:51 smithi050 bash[21467]: audit 2024-04-06T03:20:50.123749+0000 mon.a (mon.0) 705 : audit 1 from='client.? 172.21.15.50:0/1103877100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-06T03:20:51.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:51 smithi050 bash[29221]: cluster 2024-04-06T03:20:49.590792+0000 mgr.y (mgr.14152) 345 : cluster 0 pgmap v344: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:51.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:51 smithi050 bash[29221]: cluster 2024-04-06T03:20:50.120560+0000 mon.a (mon.0) 704 : cluster 0 osdmap e77: 8 total, 8 up, 8 in 2024-04-06T03:20:51.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:51 smithi050 bash[29221]: audit 2024-04-06T03:20:50.123749+0000 mon.a (mon.0) 705 : audit 1 from='client.? 172.21.15.50:0/1103877100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-06T03:20:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:52 smithi153 bash[23852]: audit 2024-04-06T03:20:51.126767+0000 mon.a (mon.0) 706 : audit 1 from='client.? 172.21.15.50:0/1103877100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-06T03:20:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:52 smithi153 bash[23852]: cluster 2024-04-06T03:20:51.134653+0000 mon.a (mon.0) 707 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-04-06T03:20:52.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:52 smithi050 bash[21467]: audit 2024-04-06T03:20:51.126767+0000 mon.a (mon.0) 706 : audit 1 from='client.? 172.21.15.50:0/1103877100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-06T03:20:52.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:52 smithi050 bash[21467]: cluster 2024-04-06T03:20:51.134653+0000 mon.a (mon.0) 707 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-04-06T03:20:52.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:52 smithi050 bash[29221]: audit 2024-04-06T03:20:51.126767+0000 mon.a (mon.0) 706 : audit 1 from='client.? 172.21.15.50:0/1103877100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-06T03:20:52.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:52 smithi050 bash[29221]: cluster 2024-04-06T03:20:51.134653+0000 mon.a (mon.0) 707 : cluster 0 osdmap e78: 8 total, 8 up, 8 in 2024-04-06T03:20:53.336 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'datapool' created 2024-04-06T03:20:53.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:53 smithi153 bash[23852]: cluster 2024-04-06T03:20:51.591304+0000 mgr.y (mgr.14152) 346 : cluster 0 pgmap v347: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:53.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:53 smithi153 bash[23852]: cluster 2024-04-06T03:20:52.146992+0000 mon.a (mon.0) 708 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-04-06T03:20:53.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:53 smithi153 bash[23852]: audit 2024-04-06T03:20:52.150416+0000 mon.c (mon.2) 16 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-06T03:20:53.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:53 smithi153 bash[23852]: audit 2024-04-06T03:20:52.151857+0000 mon.a (mon.0) 709 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-06T03:20:53.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:53 smithi153 bash[23852]: cluster 2024-04-06T03:20:52.332907+0000 osd.1 (osd.1) 15 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:53.572 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:53 smithi153 bash[23852]: cluster 2024-04-06T03:20:52.332999+0000 osd.1 (osd.1) 16 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:53.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[29221]: cluster 2024-04-06T03:20:51.591304+0000 mgr.y (mgr.14152) 346 : cluster 0 pgmap v347: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[29221]: cluster 2024-04-06T03:20:52.146992+0000 mon.a (mon.0) 708 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[29221]: audit 2024-04-06T03:20:52.150416+0000 mon.c (mon.2) 16 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[29221]: audit 2024-04-06T03:20:52.151857+0000 mon.a (mon.0) 709 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[29221]: cluster 2024-04-06T03:20:52.332907+0000 osd.1 (osd.1) 15 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[29221]: cluster 2024-04-06T03:20:52.332999+0000 osd.1 (osd.1) 16 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[21467]: cluster 2024-04-06T03:20:51.591304+0000 mgr.y (mgr.14152) 346 : cluster 0 pgmap v347: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[21467]: cluster 2024-04-06T03:20:52.146992+0000 mon.a (mon.0) 708 : cluster 0 osdmap e79: 8 total, 8 up, 8 in 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[21467]: audit 2024-04-06T03:20:52.150416+0000 mon.c (mon.2) 16 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-06T03:20:53.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[21467]: audit 2024-04-06T03:20:52.151857+0000 mon.a (mon.0) 709 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-06T03:20:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[21467]: cluster 2024-04-06T03:20:52.332907+0000 osd.1 (osd.1) 15 : cluster 0 purged_snaps scrub starts 2024-04-06T03:20:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:53 smithi050 bash[21467]: cluster 2024-04-06T03:20:52.332999+0000 osd.1 (osd.1) 16 : cluster 0 purged_snaps scrub ok 2024-04-06T03:20:54.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- rbd pool init datapool 2024-04-06T03:20:54.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:52.838544+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.153:0/1437203425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-06T03:20:54.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:52.842965+0000 mon.a (mon.0) 710 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-06T03:20:54.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:53.317660+0000 mon.a (mon.0) 711 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-06T03:20:54.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:53.317779+0000 mon.a (mon.0) 712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-06T03:20:54.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: cluster 2024-04-06T03:20:53.322428+0000 mon.a (mon.0) 713 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: cluster 2024-04-06T03:20:53.591977+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v350: 68 pgs: 8 creating+peering, 13 active+clean, 47 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:53.814555+0000 mon.a (mon.0) 714 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:54.082888+0000 mon.a (mon.0) 715 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:54.087213+0000 mon.a (mon.0) 716 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:54.088155+0000 mon.a (mon.0) 717 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: audit 2024-04-06T03:20:54.088854+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[29221]: cephadm 2024-04-06T03:20:54.093083+0000 mgr.y (mgr.14152) 348 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:52.838544+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.153:0/1437203425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:52.842965+0000 mon.a (mon.0) 710 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:53.317660+0000 mon.a (mon.0) 711 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-06T03:20:54.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:53.317779+0000 mon.a (mon.0) 712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: cluster 2024-04-06T03:20:53.322428+0000 mon.a (mon.0) 713 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: cluster 2024-04-06T03:20:53.591977+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v350: 68 pgs: 8 creating+peering, 13 active+clean, 47 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:53.814555+0000 mon.a (mon.0) 714 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:54.082888+0000 mon.a (mon.0) 715 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:54.087213+0000 mon.a (mon.0) 716 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:54.088155+0000 mon.a (mon.0) 717 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: audit 2024-04-06T03:20:54.088854+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:54.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:54 smithi050 bash[21467]: cephadm 2024-04-06T03:20:54.093083+0000 mgr.y (mgr.14152) 348 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-06T03:20:55.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:52.838544+0000 mon.b (mon.1) 26 : audit 1 from='client.? 172.21.15.153:0/1437203425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-06T03:20:55.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:52.842965+0000 mon.a (mon.0) 710 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-06T03:20:55.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:53.317660+0000 mon.a (mon.0) 711 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-06T03:20:55.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:53.317779+0000 mon.a (mon.0) 712 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: cluster 2024-04-06T03:20:53.322428+0000 mon.a (mon.0) 713 : cluster 0 osdmap e80: 8 total, 8 up, 8 in 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: cluster 2024-04-06T03:20:53.591977+0000 mgr.y (mgr.14152) 347 : cluster 0 pgmap v350: 68 pgs: 8 creating+peering, 13 active+clean, 47 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 511 B/s wr, 1 op/s 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:53.814555+0000 mon.a (mon.0) 714 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:54.082888+0000 mon.a (mon.0) 715 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:54.087213+0000 mon.a (mon.0) 716 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:54.088155+0000 mon.a (mon.0) 717 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: audit 2024-04-06T03:20:54.088854+0000 mon.a (mon.0) 718 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:20:55.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:54 smithi153 bash[23852]: cephadm 2024-04-06T03:20:54.093083+0000 mgr.y (mgr.14152) 348 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-06T03:20:56.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:55 smithi153 bash[23852]: cluster 2024-04-06T03:20:54.713792+0000 mon.a (mon.0) 719 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:20:56.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:55 smithi153 bash[23852]: cluster 2024-04-06T03:20:54.727060+0000 mon.a (mon.0) 720 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-04-06T03:20:56.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:55 smithi153 bash[23852]: audit 2024-04-06T03:20:54.741268+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:55 smithi153 bash[23852]: audit 2024-04-06T03:20:54.743794+0000 mon.a (mon.0) 721 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:55 smithi153 bash[23852]: audit 2024-04-06T03:20:54.744870+0000 mon.c (mon.2) 18 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:55 smithi153 bash[23852]: audit 2024-04-06T03:20:54.745459+0000 mon.a (mon.0) 722 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[21467]: cluster 2024-04-06T03:20:54.713792+0000 mon.a (mon.0) 719 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[21467]: cluster 2024-04-06T03:20:54.727060+0000 mon.a (mon.0) 720 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[21467]: audit 2024-04-06T03:20:54.741268+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[21467]: audit 2024-04-06T03:20:54.743794+0000 mon.a (mon.0) 721 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[21467]: audit 2024-04-06T03:20:54.744870+0000 mon.c (mon.2) 18 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[21467]: audit 2024-04-06T03:20:54.745459+0000 mon.a (mon.0) 722 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[29221]: cluster 2024-04-06T03:20:54.713792+0000 mon.a (mon.0) 719 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:20:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[29221]: cluster 2024-04-06T03:20:54.727060+0000 mon.a (mon.0) 720 : cluster 0 osdmap e81: 8 total, 8 up, 8 in 2024-04-06T03:20:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[29221]: audit 2024-04-06T03:20:54.741268+0000 mon.c (mon.2) 17 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[29221]: audit 2024-04-06T03:20:54.743794+0000 mon.a (mon.0) 721 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[29221]: audit 2024-04-06T03:20:54.744870+0000 mon.c (mon.2) 18 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:55 smithi050 bash[29221]: audit 2024-04-06T03:20:54.745459+0000 mon.a (mon.0) 722 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-06T03:20:57.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:56 smithi153 bash[23852]: audit 2024-04-06T03:20:55.725015+0000 mon.a (mon.0) 723 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-06T03:20:57.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:56 smithi153 bash[23852]: audit 2024-04-06T03:20:55.725180+0000 mon.a (mon.0) 724 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-06T03:20:57.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:56 smithi153 bash[23852]: cluster 2024-04-06T03:20:55.732993+0000 mon.a (mon.0) 725 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-04-06T03:20:57.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:56 smithi050 bash[21467]: audit 2024-04-06T03:20:55.725015+0000 mon.a (mon.0) 723 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-06T03:20:57.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:56 smithi050 bash[21467]: audit 2024-04-06T03:20:55.725180+0000 mon.a (mon.0) 724 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-06T03:20:57.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:56 smithi050 bash[21467]: cluster 2024-04-06T03:20:55.732993+0000 mon.a (mon.0) 725 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-04-06T03:20:57.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:56 smithi050 bash[29221]: audit 2024-04-06T03:20:55.725015+0000 mon.a (mon.0) 723 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-06T03:20:57.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:56 smithi050 bash[29221]: audit 2024-04-06T03:20:55.725180+0000 mon.a (mon.0) 724 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-06T03:20:57.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:56 smithi050 bash[29221]: cluster 2024-04-06T03:20:55.732993+0000 mon.a (mon.0) 725 : cluster 0 osdmap e82: 8 total, 8 up, 8 in 2024-04-06T03:20:58.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:57 smithi153 bash[23852]: cluster 2024-04-06T03:20:55.592927+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v352: 100 pgs: 15 creating+peering, 42 active+clean, 43 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-06T03:20:58.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:57 smithi153 bash[23852]: cluster 2024-04-06T03:20:56.785773+0000 mon.a (mon.0) 726 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-04-06T03:20:58.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:57 smithi153 bash[23852]: audit 2024-04-06T03:20:56.794203+0000 mon.c (mon.2) 19 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:57 smithi153 bash[23852]: audit 2024-04-06T03:20:56.795249+0000 mon.c (mon.2) 20 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:57 smithi153 bash[23852]: audit 2024-04-06T03:20:56.798387+0000 mon.a (mon.0) 727 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:57 smithi153 bash[23852]: audit 2024-04-06T03:20:56.799513+0000 mon.a (mon.0) 728 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[21467]: cluster 2024-04-06T03:20:55.592927+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v352: 100 pgs: 15 creating+peering, 42 active+clean, 43 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-06T03:20:58.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[21467]: cluster 2024-04-06T03:20:56.785773+0000 mon.a (mon.0) 726 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-04-06T03:20:58.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[21467]: audit 2024-04-06T03:20:56.794203+0000 mon.c (mon.2) 19 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[21467]: audit 2024-04-06T03:20:56.795249+0000 mon.c (mon.2) 20 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[21467]: audit 2024-04-06T03:20:56.798387+0000 mon.a (mon.0) 727 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[21467]: audit 2024-04-06T03:20:56.799513+0000 mon.a (mon.0) 728 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[29221]: cluster 2024-04-06T03:20:55.592927+0000 mgr.y (mgr.14152) 349 : cluster 0 pgmap v352: 100 pgs: 15 creating+peering, 42 active+clean, 43 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-06T03:20:58.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[29221]: cluster 2024-04-06T03:20:56.785773+0000 mon.a (mon.0) 726 : cluster 0 osdmap e83: 8 total, 8 up, 8 in 2024-04-06T03:20:58.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[29221]: audit 2024-04-06T03:20:56.794203+0000 mon.c (mon.2) 19 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[29221]: audit 2024-04-06T03:20:56.795249+0000 mon.c (mon.2) 20 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[29221]: audit 2024-04-06T03:20:56.798387+0000 mon.a (mon.0) 727 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:57 smithi050 bash[29221]: audit 2024-04-06T03:20:56.799513+0000 mon.a (mon.0) 728 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-06T03:20:58.812 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: cluster 2024-04-06T03:20:57.593847+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v355: 132 pgs: 17 creating+peering, 74 active+clean, 41 unknown; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 958 B/s wr, 3 op/s 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: audit 2024-04-06T03:20:57.795674+0000 mon.a (mon.0) 729 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: audit 2024-04-06T03:20:57.795890+0000 mon.a (mon.0) 730 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: cluster 2024-04-06T03:20:57.804201+0000 mon.a (mon.0) 731 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: audit 2024-04-06T03:20:57.806754+0000 mon.c (mon.2) 21 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: audit 2024-04-06T03:20:57.812557+0000 mon.a (mon.0) 732 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: audit 2024-04-06T03:20:57.815688+0000 mon.c (mon.2) 22 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:58 smithi153 bash[23852]: audit 2024-04-06T03:20:57.816619+0000 mon.a (mon.0) 733 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: cluster 2024-04-06T03:20:57.593847+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v355: 132 pgs: 17 creating+peering, 74 active+clean, 41 unknown; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 958 B/s wr, 3 op/s 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: audit 2024-04-06T03:20:57.795674+0000 mon.a (mon.0) 729 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: audit 2024-04-06T03:20:57.795890+0000 mon.a (mon.0) 730 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: cluster 2024-04-06T03:20:57.804201+0000 mon.a (mon.0) 731 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: audit 2024-04-06T03:20:57.806754+0000 mon.c (mon.2) 21 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: audit 2024-04-06T03:20:57.812557+0000 mon.a (mon.0) 732 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: audit 2024-04-06T03:20:57.815688+0000 mon.c (mon.2) 22 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[29221]: audit 2024-04-06T03:20:57.816619+0000 mon.a (mon.0) 733 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: cluster 2024-04-06T03:20:57.593847+0000 mgr.y (mgr.14152) 350 : cluster 0 pgmap v355: 132 pgs: 17 creating+peering, 74 active+clean, 41 unknown; 578 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 958 B/s wr, 3 op/s 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: audit 2024-04-06T03:20:57.795674+0000 mon.a (mon.0) 729 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: audit 2024-04-06T03:20:57.795890+0000 mon.a (mon.0) 730 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-06T03:20:59.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: cluster 2024-04-06T03:20:57.804201+0000 mon.a (mon.0) 731 : cluster 0 osdmap e84: 8 total, 8 up, 8 in 2024-04-06T03:20:59.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: audit 2024-04-06T03:20:57.806754+0000 mon.c (mon.2) 21 : audit 1 from='client.? 172.21.15.50:0/2847340914' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: audit 2024-04-06T03:20:57.812557+0000 mon.a (mon.0) 732 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: audit 2024-04-06T03:20:57.815688+0000 mon.c (mon.2) 22 : audit 1 from='client.? 172.21.15.50:0/492749749' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:20:59.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:58 smithi050 bash[21467]: audit 2024-04-06T03:20:57.816619+0000 mon.a (mon.0) 733 : audit 1 from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-06T03:21:00.153 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:58.807480+0000 mon.a (mon.0) 734 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:58.807682+0000 mon.a (mon.0) 735 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: cluster 2024-04-06T03:20:58.814691+0000 mon.a (mon.0) 736 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.129047+0000 mon.a (mon.0) 737 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.134960+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.145428+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.210442+0000 mon.a (mon.0) 740 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:00.154 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.212282+0000 mon.a (mon.0) 741 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:00.155 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.213055+0000 mon.a (mon.0) 742 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:00.155 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:20:59 smithi153 bash[23852]: audit 2024-04-06T03:20:59.501923+0000 mon.a (mon.0) 743 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:58.807480+0000 mon.a (mon.0) 734 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T03:21:00.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:58.807682+0000 mon.a (mon.0) 735 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T03:21:00.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: cluster 2024-04-06T03:20:58.814691+0000 mon.a (mon.0) 736 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-04-06T03:21:00.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.129047+0000 mon.a (mon.0) 737 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.134960+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.145428+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.210442+0000 mon.a (mon.0) 740 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:00.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.212282+0000 mon.a (mon.0) 741 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:00.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.213055+0000 mon.a (mon.0) 742 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:00.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[21467]: audit 2024-04-06T03:20:59.501923+0000 mon.a (mon.0) 743 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:58.807480+0000 mon.a (mon.0) 734 : audit 1 from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T03:21:00.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:58.807682+0000 mon.a (mon.0) 735 : audit 1 from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-06T03:21:00.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: cluster 2024-04-06T03:20:58.814691+0000 mon.a (mon.0) 736 : cluster 0 osdmap e85: 8 total, 8 up, 8 in 2024-04-06T03:21:00.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.129047+0000 mon.a (mon.0) 737 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.134960+0000 mon.a (mon.0) 738 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.145428+0000 mon.a (mon.0) 739 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:00.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.210442+0000 mon.a (mon.0) 740 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:00.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.212282+0000 mon.a (mon.0) 741 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:00.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.213055+0000 mon.a (mon.0) 742 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:00.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:20:59 smithi050 bash[29221]: audit 2024-04-06T03:20:59.501923+0000 mon.a (mon.0) 743 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:01.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:00 smithi153 bash[23852]: cephadm 2024-04-06T03:20:59.218100+0000 mgr.y (mgr.14152) 351 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-06T03:21:01.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:00 smithi153 bash[23852]: cluster 2024-04-06T03:20:59.594776+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v358: 132 pgs: 16 creating+peering, 101 active+clean, 15 unknown; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.2 KiB/s wr, 45 op/s 2024-04-06T03:21:01.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:00 smithi153 bash[23852]: cluster 2024-04-06T03:21:00.500207+0000 mon.a (mon.0) 744 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:21:01.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:00 smithi050 bash[21467]: cephadm 2024-04-06T03:20:59.218100+0000 mgr.y (mgr.14152) 351 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-06T03:21:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:00 smithi050 bash[21467]: cluster 2024-04-06T03:20:59.594776+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v358: 132 pgs: 16 creating+peering, 101 active+clean, 15 unknown; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.2 KiB/s wr, 45 op/s 2024-04-06T03:21:01.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:00 smithi050 bash[21467]: cluster 2024-04-06T03:21:00.500207+0000 mon.a (mon.0) 744 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:21:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:00 smithi050 bash[29221]: cephadm 2024-04-06T03:20:59.218100+0000 mgr.y (mgr.14152) 351 : cephadm 1 Checking dashboard <-> RGW credentials 2024-04-06T03:21:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:00 smithi050 bash[29221]: cluster 2024-04-06T03:20:59.594776+0000 mgr.y (mgr.14152) 352 : cluster 0 pgmap v358: 132 pgs: 16 creating+peering, 101 active+clean, 15 unknown; 579 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 2.2 KiB/s wr, 45 op/s 2024-04-06T03:21:01.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:00 smithi050 bash[29221]: cluster 2024-04-06T03:21:00.500207+0000 mon.a (mon.0) 744 : cluster 3 Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:21:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:01 smithi050 bash[21467]: audit 2024-04-06T03:21:01.561457+0000 mon.b (mon.1) 27 : audit 1 from='client.? 172.21.15.153:0/3088108480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-06T03:21:02.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:01 smithi050 bash[21467]: audit 2024-04-06T03:21:01.565937+0000 mon.a (mon.0) 745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-06T03:21:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:01 smithi050 bash[29221]: audit 2024-04-06T03:21:01.561457+0000 mon.b (mon.1) 27 : audit 1 from='client.? 172.21.15.153:0/3088108480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-06T03:21:02.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:01 smithi050 bash[29221]: audit 2024-04-06T03:21:01.565937+0000 mon.a (mon.0) 745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-06T03:21:02.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:01 smithi153 bash[23852]: audit 2024-04-06T03:21:01.561457+0000 mon.b (mon.1) 27 : audit 1 from='client.? 172.21.15.153:0/3088108480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-06T03:21:02.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:01 smithi153 bash[23852]: audit 2024-04-06T03:21:01.565937+0000 mon.a (mon.0) 745 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-06T03:21:03.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:02 smithi050 bash[21467]: cluster 2024-04-06T03:21:01.595930+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v359: 132 pgs: 9 creating+peering, 118 active+clean, 5 unknown; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 3.9 KiB/s wr, 143 op/s 2024-04-06T03:21:03.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:02 smithi050 bash[21467]: audit 2024-04-06T03:21:01.823281+0000 mon.a (mon.0) 746 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-06T03:21:03.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:02 smithi050 bash[21467]: cluster 2024-04-06T03:21:01.830898+0000 mon.a (mon.0) 747 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-04-06T03:21:03.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:02 smithi050 bash[29221]: cluster 2024-04-06T03:21:01.595930+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v359: 132 pgs: 9 creating+peering, 118 active+clean, 5 unknown; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 3.9 KiB/s wr, 143 op/s 2024-04-06T03:21:03.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:02 smithi050 bash[29221]: audit 2024-04-06T03:21:01.823281+0000 mon.a (mon.0) 746 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-06T03:21:03.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:02 smithi050 bash[29221]: cluster 2024-04-06T03:21:01.830898+0000 mon.a (mon.0) 747 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-04-06T03:21:03.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:02 smithi153 bash[23852]: cluster 2024-04-06T03:21:01.595930+0000 mgr.y (mgr.14152) 353 : cluster 0 pgmap v359: 132 pgs: 9 creating+peering, 118 active+clean, 5 unknown; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 3.9 KiB/s wr, 143 op/s 2024-04-06T03:21:03.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:02 smithi153 bash[23852]: audit 2024-04-06T03:21:01.823281+0000 mon.a (mon.0) 746 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-06T03:21:03.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:02 smithi153 bash[23852]: cluster 2024-04-06T03:21:01.830898+0000 mon.a (mon.0) 747 : cluster 0 osdmap e86: 8 total, 8 up, 8 in 2024-04-06T03:21:04.111 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:03 smithi153 bash[23852]: cluster 2024-04-06T03:21:02.833189+0000 mon.a (mon.0) 748 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-04-06T03:21:04.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:03 smithi050 bash[21467]: cluster 2024-04-06T03:21:02.833189+0000 mon.a (mon.0) 748 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-04-06T03:21:04.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:03 smithi050 bash[29221]: cluster 2024-04-06T03:21:02.833189+0000 mon.a (mon.0) 748 : cluster 0 osdmap e87: 8 total, 8 up, 8 in 2024-04-06T03:21:04.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.153 --placement '1;smithi153=iscsi.a' 2024-04-06T03:21:05.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:04 smithi153 bash[23852]: cluster 2024-04-06T03:21:03.596772+0000 mgr.y (mgr.14152) 354 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 6.0 KiB/s wr, 261 op/s 2024-04-06T03:21:05.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:04 smithi153 bash[23852]: cluster 2024-04-06T03:21:03.852207+0000 mon.a (mon.0) 749 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-04-06T03:21:05.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:04 smithi153 bash[23852]: cluster 2024-04-06T03:21:04.308509+0000 mon.a (mon.0) 750 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:21:05.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:04 smithi153 bash[23852]: cluster 2024-04-06T03:21:04.308586+0000 mon.a (mon.0) 751 : cluster 1 Cluster is now healthy 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[21467]: cluster 2024-04-06T03:21:03.596772+0000 mgr.y (mgr.14152) 354 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 6.0 KiB/s wr, 261 op/s 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[21467]: cluster 2024-04-06T03:21:03.852207+0000 mon.a (mon.0) 749 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[21467]: cluster 2024-04-06T03:21:04.308509+0000 mon.a (mon.0) 750 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[21467]: cluster 2024-04-06T03:21:04.308586+0000 mon.a (mon.0) 751 : cluster 1 Cluster is now healthy 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[29221]: cluster 2024-04-06T03:21:03.596772+0000 mgr.y (mgr.14152) 354 : cluster 0 pgmap v362: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 6.0 KiB/s wr, 261 op/s 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[29221]: cluster 2024-04-06T03:21:03.852207+0000 mon.a (mon.0) 749 : cluster 0 osdmap e88: 8 total, 8 up, 8 in 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[29221]: cluster 2024-04-06T03:21:04.308509+0000 mon.a (mon.0) 750 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:21:05.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:04 smithi050 bash[29221]: cluster 2024-04-06T03:21:04.308586+0000 mon.a (mon.0) 751 : cluster 1 Cluster is now healthy 2024-04-06T03:21:07.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:06 smithi050 bash[21467]: cluster 2024-04-06T03:21:05.597828+0000 mgr.y (mgr.14152) 355 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 5.0 KiB/s wr, 241 op/s 2024-04-06T03:21:07.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:06 smithi050 bash[29221]: cluster 2024-04-06T03:21:05.597828+0000 mgr.y (mgr.14152) 355 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 5.0 KiB/s wr, 241 op/s 2024-04-06T03:21:07.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:06 smithi153 bash[23852]: cluster 2024-04-06T03:21:05.597828+0000 mgr.y (mgr.14152) 355 : cluster 0 pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 5.0 KiB/s wr, 241 op/s 2024-04-06T03:21:09.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:08 smithi050 bash[21467]: cluster 2024-04-06T03:21:07.598928+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 2.7 KiB/s wr, 132 op/s 2024-04-06T03:21:09.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:08 smithi050 bash[21467]: cluster 2024-04-06T03:21:08.318240+0000 mon.a (mon.0) 752 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-04-06T03:21:09.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:08 smithi050 bash[29221]: cluster 2024-04-06T03:21:07.598928+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 2.7 KiB/s wr, 132 op/s 2024-04-06T03:21:09.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:08 smithi050 bash[29221]: cluster 2024-04-06T03:21:08.318240+0000 mon.a (mon.0) 752 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-04-06T03:21:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:08 smithi153 bash[23852]: cluster 2024-04-06T03:21:07.598928+0000 mgr.y (mgr.14152) 356 : cluster 0 pgmap v365: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 2.7 KiB/s wr, 132 op/s 2024-04-06T03:21:09.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:08 smithi153 bash[23852]: cluster 2024-04-06T03:21:08.318240+0000 mon.a (mon.0) 752 : cluster 0 osdmap e89: 8 total, 8 up, 8 in 2024-04-06T03:21:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:21:11.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:10 smithi050 bash[21467]: cluster 2024-04-06T03:21:09.599870+0000 mgr.y (mgr.14152) 357 : cluster 0 pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 453 B/s wr, 15 op/s 2024-04-06T03:21:11.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:10 smithi050 bash[29221]: cluster 2024-04-06T03:21:09.599870+0000 mgr.y (mgr.14152) 357 : cluster 0 pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 453 B/s wr, 15 op/s 2024-04-06T03:21:11.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:10 smithi153 bash[23852]: cluster 2024-04-06T03:21:09.599870+0000 mgr.y (mgr.14152) 357 : cluster 0 pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 453 B/s wr, 15 op/s 2024-04-06T03:21:12.352 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled iscsi.datapool update... 2024-04-06T03:21:13.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:12 smithi153 bash[23852]: cluster 2024-04-06T03:21:11.600891+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-06T03:21:13.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:12 smithi153 bash[23852]: audit 2024-04-06T03:21:12.361606+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:13.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:12 smithi153 bash[23852]: audit 2024-04-06T03:21:12.404611+0000 mon.a (mon.0) 754 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:13.103 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-04-06T03:21:13.103 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:21:13.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-06T03:21:13.114 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:12 smithi050 bash[29221]: cluster 2024-04-06T03:21:11.600891+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-06T03:21:13.114 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:12 smithi050 bash[29221]: audit 2024-04-06T03:21:12.361606+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:13.114 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:12 smithi050 bash[29221]: audit 2024-04-06T03:21:12.404611+0000 mon.a (mon.0) 754 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:13.115 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:12 smithi050 bash[21467]: cluster 2024-04-06T03:21:11.600891+0000 mgr.y (mgr.14152) 358 : cluster 0 pgmap v368: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-06T03:21:13.115 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:12 smithi050 bash[21467]: audit 2024-04-06T03:21:12.361606+0000 mon.a (mon.0) 753 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:13.115 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:12 smithi050 bash[21467]: audit 2024-04-06T03:21:12.404611+0000 mon.a (mon.0) 754 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:13.117 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:21:13.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-06T03:21:13.131 DEBUG:teuthology.orchestra.run.smithi153:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@iscsi.iscsi.a.service 2024-04-06T03:21:13.177 INFO:tasks.cephadm:Adding prometheus.a on smithi153 2024-04-06T03:21:13.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply prometheus '1;smithi153=a' 2024-04-06T03:21:14.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:12.347012+0000 mgr.y (mgr.14152) 359 : audit 0 from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.153", "placement": "1;smithi153=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:14.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: cephadm 2024-04-06T03:21:12.353433+0000 mgr.y (mgr.14152) 360 : cephadm 1 Saving service iscsi.datapool spec with placement smithi153=iscsi.a;count:1 2024-04-06T03:21:14.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:13.033214+0000 mon.a (mon.0) 755 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:14.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:13.035006+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:14.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:13.044812+0000 mon.a (mon.0) 757 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:13.050223+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:13.053480+0000 mon.a (mon.0) 759 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: audit 2024-04-06T03:21:13.075252+0000 mon.a (mon.0) 760 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: cephadm 2024-04-06T03:21:13.077213+0000 mgr.y (mgr.14152) 361 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi153 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[21467]: cluster 2024-04-06T03:21:13.320154+0000 mon.a (mon.0) 761 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:12.347012+0000 mgr.y (mgr.14152) 359 : audit 0 from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.153", "placement": "1;smithi153=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: cephadm 2024-04-06T03:21:12.353433+0000 mgr.y (mgr.14152) 360 : cephadm 1 Saving service iscsi.datapool spec with placement smithi153=iscsi.a;count:1 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:13.033214+0000 mon.a (mon.0) 755 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:13.035006+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:13.044812+0000 mon.a (mon.0) 757 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:13.050223+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T03:21:14.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:13.053480+0000 mon.a (mon.0) 759 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-06T03:21:14.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: audit 2024-04-06T03:21:13.075252+0000 mon.a (mon.0) 760 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:14.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: cephadm 2024-04-06T03:21:13.077213+0000 mgr.y (mgr.14152) 361 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi153 2024-04-06T03:21:14.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:13 smithi050 bash[29221]: cluster 2024-04-06T03:21:13.320154+0000 mon.a (mon.0) 761 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-04-06T03:21:14.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:12.347012+0000 mgr.y (mgr.14152) 359 : audit 0 from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.153", "placement": "1;smithi153=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:14.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: cephadm 2024-04-06T03:21:12.353433+0000 mgr.y (mgr.14152) 360 : cephadm 1 Saving service iscsi.datapool spec with placement smithi153=iscsi.a;count:1 2024-04-06T03:21:14.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:13.033214+0000 mon.a (mon.0) 755 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:14.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:13.035006+0000 mon.a (mon.0) 756 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:14.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:13.044812+0000 mon.a (mon.0) 757 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:14.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:13.050223+0000 mon.a (mon.0) 758 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-06T03:21:14.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:13.053480+0000 mon.a (mon.0) 759 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-06T03:21:14.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: audit 2024-04-06T03:21:13.075252+0000 mon.a (mon.0) 760 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:14.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: cephadm 2024-04-06T03:21:13.077213+0000 mgr.y (mgr.14152) 361 : cephadm 1 Deploying daemon iscsi.iscsi.a on smithi153 2024-04-06T03:21:14.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:13 smithi153 bash[23852]: cluster 2024-04-06T03:21:13.320154+0000 mon.a (mon.0) 761 : cluster 0 osdmap e90: 8 total, 8 up, 8 in 2024-04-06T03:21:15.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:14 smithi050 bash[21467]: cluster 2024-04-06T03:21:13.601718+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:15.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:14 smithi050 bash[29221]: cluster 2024-04-06T03:21:13.601718+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:15.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:14 smithi153 bash[23852]: cluster 2024-04-06T03:21:13.601718+0000 mgr.y (mgr.14152) 362 : cluster 0 pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:15.686 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.687 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.687 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.687 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.688 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.688 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.952 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.952 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.952 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.953 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.953 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:15.953 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:21:15 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:17.273 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: cluster 2024-04-06T03:21:15.602858+0000 mgr.y (mgr.14152) 363 : cluster 0 pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:17.273 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: audit 2024-04-06T03:21:16.026963+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.273 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: audit 2024-04-06T03:21:16.037577+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.274 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: audit 2024-04-06T03:21:16.056306+0000 mon.a (mon.0) 764 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.274 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: cephadm 2024-04-06T03:21:16.058112+0000 mgr.y (mgr.14152) 364 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-04-06T03:21:17.274 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: audit 2024-04-06T03:21:16.075197+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.274 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[23852]: audit 2024-04-06T03:21:16.140525+0000 mon.a (mon.0) 766 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: cluster 2024-04-06T03:21:15.602858+0000 mgr.y (mgr.14152) 363 : cluster 0 pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: audit 2024-04-06T03:21:16.026963+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: audit 2024-04-06T03:21:16.037577+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: audit 2024-04-06T03:21:16.056306+0000 mon.a (mon.0) 764 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: cephadm 2024-04-06T03:21:16.058112+0000 mgr.y (mgr.14152) 364 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: audit 2024-04-06T03:21:16.075197+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[21467]: audit 2024-04-06T03:21:16.140525+0000 mon.a (mon.0) 766 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: cluster 2024-04-06T03:21:15.602858+0000 mgr.y (mgr.14152) 363 : cluster 0 pgmap v371: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: audit 2024-04-06T03:21:16.026963+0000 mon.a (mon.0) 762 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: audit 2024-04-06T03:21:16.037577+0000 mon.a (mon.0) 763 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: audit 2024-04-06T03:21:16.056306+0000 mon.a (mon.0) 764 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: cephadm 2024-04-06T03:21:16.058112+0000 mgr.y (mgr.14152) 364 : cephadm 1 Checking pool "datapool" exists for service iscsi.datapool 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: audit 2024-04-06T03:21:16.075197+0000 mon.a (mon.0) 765 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:17.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:17 smithi050 bash[29221]: audit 2024-04-06T03:21:16.140525+0000 mon.a (mon.0) 766 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:17.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[48791]: debug Processing osd blocklist entries for this node 2024-04-06T03:21:17.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[48791]: debug Reading the configuration object to update local LIO configuration 2024-04-06T03:21:17.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[48791]: debug Configuration does not have an entry for this host(smithi153.front.sepia.ceph.com) - nothing to define to LIO 2024-04-06T03:21:18.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:18 smithi153 bash[23852]: cluster 2024-04-06T03:21:17.090044+0000 mon.a (mon.0) 767 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-04-06T03:21:18.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:18 smithi153 bash[23852]: audit 2024-04-06T03:21:17.966542+0000 mon.b (mon.1) 28 : audit 0 from='client.? 172.21.15.153:0/2338714852' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T03:21:18.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:17 smithi153 bash[48791]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-04-06T03:21:18.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:18 smithi050 bash[21467]: cluster 2024-04-06T03:21:17.090044+0000 mon.a (mon.0) 767 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-04-06T03:21:18.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:18 smithi050 bash[21467]: audit 2024-04-06T03:21:17.966542+0000 mon.b (mon.1) 28 : audit 0 from='client.? 172.21.15.153:0/2338714852' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T03:21:18.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:18 smithi050 bash[29221]: cluster 2024-04-06T03:21:17.090044+0000 mon.a (mon.0) 767 : cluster 0 mgrmap e16: y(active, since 9m), standbys: x 2024-04-06T03:21:18.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:18 smithi050 bash[29221]: audit 2024-04-06T03:21:17.966542+0000 mon.b (mon.1) 28 : audit 0 from='client.? 172.21.15.153:0/2338714852' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-06T03:21:19.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:19 smithi050 bash[21467]: cluster 2024-04-06T03:21:17.603943+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:19.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:19 smithi050 bash[21467]: cluster 2024-04-06T03:21:18.323324+0000 mon.a (mon.0) 768 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-04-06T03:21:19.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:19 smithi050 bash[21467]: audit 2024-04-06T03:21:18.825254+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:19.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:19 smithi050 bash[29221]: cluster 2024-04-06T03:21:17.603943+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:19.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:19 smithi050 bash[29221]: cluster 2024-04-06T03:21:18.323324+0000 mon.a (mon.0) 768 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-04-06T03:21:19.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:19 smithi050 bash[29221]: audit 2024-04-06T03:21:18.825254+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:19.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:19 smithi153 bash[23852]: cluster 2024-04-06T03:21:17.603943+0000 mgr.y (mgr.14152) 365 : cluster 0 pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:19.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:19 smithi153 bash[23852]: cluster 2024-04-06T03:21:18.323324+0000 mon.a (mon.0) 768 : cluster 0 osdmap e91: 8 total, 8 up, 8 in 2024-04-06T03:21:19.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:19 smithi153 bash[23852]: audit 2024-04-06T03:21:18.825254+0000 mon.a (mon.0) 769 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:19.979 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:21:21.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:21 smithi153 bash[23852]: cluster 2024-04-06T03:21:19.604904+0000 mgr.y (mgr.14152) 366 : cluster 0 pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:21.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:21 smithi050 bash[21467]: cluster 2024-04-06T03:21:19.604904+0000 mgr.y (mgr.14152) 366 : cluster 0 pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:21.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:21 smithi050 bash[29221]: cluster 2024-04-06T03:21:19.604904+0000 mgr.y (mgr.14152) 366 : cluster 0 pgmap v374: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:22.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:22 smithi050 bash[21467]: cluster 2024-04-06T03:21:20.183280+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:22.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:22 smithi050 bash[21467]: cluster 2024-04-06T03:21:20.183394+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:22.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:22 smithi050 bash[21467]: cluster 2024-04-06T03:21:21.097803+0000 mon.a (mon.0) 770 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-04-06T03:21:22.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:22 smithi050 bash[29221]: cluster 2024-04-06T03:21:20.183280+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:22.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:22 smithi050 bash[29221]: cluster 2024-04-06T03:21:20.183394+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:22.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:22 smithi050 bash[29221]: cluster 2024-04-06T03:21:21.097803+0000 mon.a (mon.0) 770 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-04-06T03:21:22.501 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:22 smithi153 bash[23852]: cluster 2024-04-06T03:21:20.183280+0000 osd.3 (osd.3) 9 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:22.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:22 smithi153 bash[23852]: cluster 2024-04-06T03:21:20.183394+0000 osd.3 (osd.3) 10 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:22.502 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:22 smithi153 bash[23852]: cluster 2024-04-06T03:21:21.097803+0000 mon.a (mon.0) 770 : cluster 0 osdmap e92: 8 total, 8 up, 8 in 2024-04-06T03:21:22.930 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled prometheus update... 2024-04-06T03:21:23.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: cluster 2024-04-06T03:21:21.605594+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-04-06T03:21:23.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.844227+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.855309+0000 mon.a (mon.0) 772 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.857763+0000 mon.a (mon.0) 773 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.859784+0000 mon.a (mon.0) 774 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.872288+0000 mon.a (mon.0) 775 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.940104+0000 mon.a (mon.0) 776 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.959419+0000 mon.a (mon.0) 777 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.962766+0000 mon.a (mon.0) 778 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:22.972171+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:23.005298+0000 mon.a (mon.0) 780 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi153"}]: dispatch 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:23.016505+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:23 smithi153 bash[23852]: audit 2024-04-06T03:21:23.020393+0000 mon.a (mon.0) 782 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: cluster 2024-04-06T03:21:21.605594+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.844227+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.855309+0000 mon.a (mon.0) 772 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.857763+0000 mon.a (mon.0) 773 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.859784+0000 mon.a (mon.0) 774 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.872288+0000 mon.a (mon.0) 775 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.940104+0000 mon.a (mon.0) 776 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.959419+0000 mon.a (mon.0) 777 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T03:21:23.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.962766+0000 mon.a (mon.0) 778 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:22.972171+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:23.005298+0000 mon.a (mon.0) 780 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi153"}]: dispatch 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:23.016505+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[21467]: audit 2024-04-06T03:21:23.020393+0000 mon.a (mon.0) 782 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: cluster 2024-04-06T03:21:21.605594+0000 mgr.y (mgr.14152) 367 : cluster 0 pgmap v376: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 1 op/s 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.844227+0000 mon.a (mon.0) 771 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.855309+0000 mon.a (mon.0) 772 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.857763+0000 mon.a (mon.0) 773 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.859784+0000 mon.a (mon.0) 774 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.872288+0000 mon.a (mon.0) 775 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.940104+0000 mon.a (mon.0) 776 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.959419+0000 mon.a (mon.0) 777 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T03:21:23.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.962766+0000 mon.a (mon.0) 778 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T03:21:23.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:22.972171+0000 mon.a (mon.0) 779 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:23.005298+0000 mon.a (mon.0) 780 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi153"}]: dispatch 2024-04-06T03:21:23.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:23.016505+0000 mon.a (mon.0) 781 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:23.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:23 smithi050 bash[29221]: audit 2024-04-06T03:21:23.020393+0000 mon.a (mon.0) 782 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:23.642 DEBUG:teuthology.orchestra.run.smithi153:prometheus.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@prometheus.a.service 2024-04-06T03:21:23.645 INFO:tasks.cephadm:Adding node-exporter.a on smithi050 2024-04-06T03:21:23.645 INFO:tasks.cephadm:Adding node-exporter.b on smithi153 2024-04-06T03:21:23.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply node-exporter '2;smithi050=a;smithi153=b' 2024-04-06T03:21:24.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:22.926477+0000 mgr.y (mgr.14152) 368 : audit 0 from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:24.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: cephadm 2024-04-06T03:21:22.930314+0000 mgr.y (mgr.14152) 369 : cephadm 1 Saving service prometheus spec with placement smithi153=a;count:1 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:22.960338+0000 mgr.y (mgr.14152) 370 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: cephadm 2024-04-06T03:21:22.962391+0000 mgr.y (mgr.14152) 371 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.153:5000 to Dashboard 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:22.963339+0000 mgr.y (mgr.14152) 372 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:23.006695+0000 mgr.y (mgr.14152) 373 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi153"}]: dispatch 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: cluster 2024-04-06T03:21:23.326619+0000 mon.a (mon.0) 783 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:23.545569+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:23.551808+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:23.553280+0000 mon.a (mon.0) 786 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:23.554142+0000 mon.a (mon.0) 787 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:24.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[29221]: audit 2024-04-06T03:21:23.559678+0000 mon.a (mon.0) 788 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:22.926477+0000 mgr.y (mgr.14152) 368 : audit 0 from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: cephadm 2024-04-06T03:21:22.930314+0000 mgr.y (mgr.14152) 369 : cephadm 1 Saving service prometheus spec with placement smithi153=a;count:1 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:22.960338+0000 mgr.y (mgr.14152) 370 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: cephadm 2024-04-06T03:21:22.962391+0000 mgr.y (mgr.14152) 371 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.153:5000 to Dashboard 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:22.963339+0000 mgr.y (mgr.14152) 372 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:23.006695+0000 mgr.y (mgr.14152) 373 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi153"}]: dispatch 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: cluster 2024-04-06T03:21:23.326619+0000 mon.a (mon.0) 783 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:23.545569+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:23.551808+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:23.553280+0000 mon.a (mon.0) 786 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:23.554142+0000 mon.a (mon.0) 787 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:24.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:24 smithi050 bash[21467]: audit 2024-04-06T03:21:23.559678+0000 mon.a (mon.0) 788 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.541 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:24 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:24.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:22.926477+0000 mgr.y (mgr.14152) 368 : audit 0 from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: cephadm 2024-04-06T03:21:22.930314+0000 mgr.y (mgr.14152) 369 : cephadm 1 Saving service prometheus spec with placement smithi153=a;count:1 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:22.960338+0000 mgr.y (mgr.14152) 370 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: cephadm 2024-04-06T03:21:22.962391+0000 mgr.y (mgr.14152) 371 : cephadm 1 Adding iSCSI gateway http://:@172.21.15.153:5000 to Dashboard 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:22.963339+0000 mgr.y (mgr.14152) 372 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:23.006695+0000 mgr.y (mgr.14152) 373 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi153"}]: dispatch 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: cluster 2024-04-06T03:21:23.326619+0000 mon.a (mon.0) 783 : cluster 0 osdmap e93: 8 total, 8 up, 8 in 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:23.545569+0000 mon.a (mon.0) 784 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:23.551808+0000 mon.a (mon.0) 785 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:23.553280+0000 mon.a (mon.0) 786 : audit 0 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:23.554142+0000 mon.a (mon.0) 787 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:24.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:24 smithi153 bash[23852]: audit 2024-04-06T03:21:23.559678+0000 mon.a (mon.0) 788 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:25.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:25 smithi050 bash[21467]: cluster 2024-04-06T03:21:23.606077+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-04-06T03:21:25.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:25 smithi050 bash[21467]: cephadm 2024-04-06T03:21:23.833793+0000 mgr.y (mgr.14152) 375 : cephadm 1 Deploying daemon prometheus.a on smithi153 2024-04-06T03:21:25.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:25 smithi050 bash[29221]: cluster 2024-04-06T03:21:23.606077+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-04-06T03:21:25.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:25 smithi050 bash[29221]: cephadm 2024-04-06T03:21:23.833793+0000 mgr.y (mgr.14152) 375 : cephadm 1 Deploying daemon prometheus.a on smithi153 2024-04-06T03:21:25.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:25 smithi153 bash[23852]: cluster 2024-04-06T03:21:23.606077+0000 mgr.y (mgr.14152) 374 : cluster 0 pgmap v378: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-04-06T03:21:25.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:25 smithi153 bash[23852]: cephadm 2024-04-06T03:21:23.833793+0000 mgr.y (mgr.14152) 375 : cephadm 1 Deploying daemon prometheus.a on smithi153 2024-04-06T03:21:26.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:26 smithi050 bash[21467]: cluster 2024-04-06T03:21:24.863064+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:26.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:26 smithi050 bash[21467]: cluster 2024-04-06T03:21:24.863182+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:26.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:26 smithi050 bash[21467]: cluster 2024-04-06T03:21:25.568642+0000 mon.a (mon.0) 789 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-04-06T03:21:26.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:26 smithi050 bash[29221]: cluster 2024-04-06T03:21:24.863064+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:26.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:26 smithi050 bash[29221]: cluster 2024-04-06T03:21:24.863182+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:26.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:26 smithi050 bash[29221]: cluster 2024-04-06T03:21:25.568642+0000 mon.a (mon.0) 789 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-04-06T03:21:26.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:26 smithi153 bash[23852]: cluster 2024-04-06T03:21:24.863064+0000 osd.0 (osd.0) 13 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:26.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:26 smithi153 bash[23852]: cluster 2024-04-06T03:21:24.863182+0000 osd.0 (osd.0) 14 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:26.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:26 smithi153 bash[23852]: cluster 2024-04-06T03:21:25.568642+0000 mon.a (mon.0) 789 : cluster 0 osdmap e94: 8 total, 8 up, 8 in 2024-04-06T03:21:27.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:27 smithi050 bash[21467]: cluster 2024-04-06T03:21:25.606764+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-04-06T03:21:27.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:27 smithi050 bash[29221]: cluster 2024-04-06T03:21:25.606764+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-04-06T03:21:27.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:27 smithi153 bash[23852]: cluster 2024-04-06T03:21:25.606764+0000 mgr.y (mgr.14152) 376 : cluster 0 pgmap v380: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-04-06T03:21:28.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:27 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:21:28.449 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:21:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:29 smithi050 bash[21467]: cluster 2024-04-06T03:21:27.607852+0000 mgr.y (mgr.14152) 377 : cluster 0 pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-04-06T03:21:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:29 smithi050 bash[21467]: audit 2024-04-06T03:21:27.671296+0000 mgr.y (mgr.14152) 378 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:29.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:29 smithi050 bash[21467]: cluster 2024-04-06T03:21:28.327973+0000 mon.a (mon.0) 790 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-04-06T03:21:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:29 smithi050 bash[29221]: cluster 2024-04-06T03:21:27.607852+0000 mgr.y (mgr.14152) 377 : cluster 0 pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-04-06T03:21:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:29 smithi050 bash[29221]: audit 2024-04-06T03:21:27.671296+0000 mgr.y (mgr.14152) 378 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:29.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:29 smithi050 bash[29221]: cluster 2024-04-06T03:21:28.327973+0000 mon.a (mon.0) 790 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-04-06T03:21:29.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:29 smithi153 bash[23852]: cluster 2024-04-06T03:21:27.607852+0000 mgr.y (mgr.14152) 377 : cluster 0 pgmap v381: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 785 B/s rd, 0 op/s 2024-04-06T03:21:29.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:29 smithi153 bash[23852]: audit 2024-04-06T03:21:27.671296+0000 mgr.y (mgr.14152) 378 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:29.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:29 smithi153 bash[23852]: cluster 2024-04-06T03:21:28.327973+0000 mon.a (mon.0) 790 : cluster 0 osdmap e95: 8 total, 8 up, 8 in 2024-04-06T03:21:31.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:31 smithi050 bash[21467]: cluster 2024-04-06T03:21:29.608719+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-04-06T03:21:31.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:31 smithi050 bash[29221]: cluster 2024-04-06T03:21:29.608719+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-04-06T03:21:31.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:31 smithi153 bash[23852]: cluster 2024-04-06T03:21:29.608719+0000 mgr.y (mgr.14152) 379 : cluster 0 pgmap v383: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 814 B/s rd, 0 op/s 2024-04-06T03:21:33.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:33 smithi050 bash[21467]: cluster 2024-04-06T03:21:31.609783+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:33.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:33 smithi050 bash[29221]: cluster 2024-04-06T03:21:31.609783+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:33.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:33 smithi153 bash[23852]: cluster 2024-04-06T03:21:31.609783+0000 mgr.y (mgr.14152) 380 : cluster 0 pgmap v384: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[21467]: cluster 2024-04-06T03:21:33.219412+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[21467]: cluster 2024-04-06T03:21:33.219561+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[21467]: audit 2024-04-06T03:21:33.534817+0000 mon.a (mon.0) 791 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-06T03:21:34.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[21467]: cluster 2024-04-06T03:21:34.039543+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[21467]: cluster 2024-04-06T03:21:34.039657+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[29221]: cluster 2024-04-06T03:21:33.219412+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[29221]: cluster 2024-04-06T03:21:33.219561+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[29221]: audit 2024-04-06T03:21:33.534817+0000 mon.a (mon.0) 791 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[29221]: cluster 2024-04-06T03:21:34.039543+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:34.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:34 smithi050 bash[29221]: cluster 2024-04-06T03:21:34.039657+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:34.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:34 smithi153 bash[23852]: cluster 2024-04-06T03:21:33.219412+0000 osd.4 (osd.4) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:34.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:34 smithi153 bash[23852]: cluster 2024-04-06T03:21:33.219561+0000 osd.4 (osd.4) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:34.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:34 smithi153 bash[23852]: audit 2024-04-06T03:21:33.534817+0000 mon.a (mon.0) 791 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-06T03:21:34.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:34 smithi153 bash[23852]: cluster 2024-04-06T03:21:34.039543+0000 osd.7 (osd.7) 5 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:34.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:34 smithi153 bash[23852]: cluster 2024-04-06T03:21:34.039657+0000 osd.7 (osd.7) 6 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:34.909 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled node-exporter update... 2024-04-06T03:21:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[21467]: cluster 2024-04-06T03:21:33.610877+0000 mgr.y (mgr.14152) 381 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[21467]: audit 2024-04-06T03:21:34.143405+0000 mon.a (mon.0) 792 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-06T03:21:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[21467]: cluster 2024-04-06T03:21:34.152687+0000 mon.a (mon.0) 793 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-04-06T03:21:35.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[21467]: audit 2024-04-06T03:21:34.919062+0000 mon.a (mon.0) 794 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[29221]: cluster 2024-04-06T03:21:33.610877+0000 mgr.y (mgr.14152) 381 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[29221]: audit 2024-04-06T03:21:34.143405+0000 mon.a (mon.0) 792 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-06T03:21:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[29221]: cluster 2024-04-06T03:21:34.152687+0000 mon.a (mon.0) 793 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-04-06T03:21:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:35 smithi050 bash[29221]: audit 2024-04-06T03:21:34.919062+0000 mon.a (mon.0) 794 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:35.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:35 smithi153 bash[23852]: cluster 2024-04-06T03:21:33.610877+0000 mgr.y (mgr.14152) 381 : cluster 0 pgmap v385: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:35.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:35 smithi153 bash[23852]: audit 2024-04-06T03:21:34.143405+0000 mon.a (mon.0) 792 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-06T03:21:35.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:35 smithi153 bash[23852]: cluster 2024-04-06T03:21:34.152687+0000 mon.a (mon.0) 793 : cluster 0 osdmap e96: 8 total, 8 up, 8 in 2024-04-06T03:21:35.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:35 smithi153 bash[23852]: audit 2024-04-06T03:21:34.919062+0000 mon.a (mon.0) 794 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:36.160 DEBUG:teuthology.orchestra.run.smithi050:node-exporter.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@node-exporter.a.service 2024-04-06T03:21:36.163 DEBUG:teuthology.orchestra.run.smithi153:node-exporter.b> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@node-exporter.b.service 2024-04-06T03:21:36.166 INFO:tasks.cephadm:Adding alertmanager.a on smithi050 2024-04-06T03:21:36.166 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply alertmanager '1;smithi050=a' 2024-04-06T03:21:36.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:36 smithi153 bash[23852]: cluster 2024-04-06T03:21:34.605287+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:36.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:36 smithi153 bash[23852]: cluster 2024-04-06T03:21:34.605362+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:36.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:36 smithi153 bash[23852]: audit 2024-04-06T03:21:34.905540+0000 mgr.y (mgr.14152) 382 : audit 0 from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:36.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:36 smithi153 bash[23852]: cephadm 2024-04-06T03:21:34.909418+0000 mgr.y (mgr.14152) 383 : cephadm 1 Saving service node-exporter spec with placement smithi050=a;smithi153=b;count:2 2024-04-06T03:21:36.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:36 smithi153 bash[23852]: cluster 2024-04-06T03:21:35.152397+0000 mon.a (mon.0) 795 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-04-06T03:21:36.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[21467]: cluster 2024-04-06T03:21:34.605287+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:36.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[21467]: cluster 2024-04-06T03:21:34.605362+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:36.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[21467]: audit 2024-04-06T03:21:34.905540+0000 mgr.y (mgr.14152) 382 : audit 0 from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:36.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[21467]: cephadm 2024-04-06T03:21:34.909418+0000 mgr.y (mgr.14152) 383 : cephadm 1 Saving service node-exporter spec with placement smithi050=a;smithi153=b;count:2 2024-04-06T03:21:36.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[21467]: cluster 2024-04-06T03:21:35.152397+0000 mon.a (mon.0) 795 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-04-06T03:21:36.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[29221]: cluster 2024-04-06T03:21:34.605287+0000 osd.6 (osd.6) 7 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:36.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[29221]: cluster 2024-04-06T03:21:34.605362+0000 osd.6 (osd.6) 8 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:36.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[29221]: audit 2024-04-06T03:21:34.905540+0000 mgr.y (mgr.14152) 382 : audit 0 from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:36.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[29221]: cephadm 2024-04-06T03:21:34.909418+0000 mgr.y (mgr.14152) 383 : cephadm 1 Saving service node-exporter spec with placement smithi050=a;smithi153=b;count:2 2024-04-06T03:21:36.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:36 smithi050 bash[29221]: cluster 2024-04-06T03:21:35.152397+0000 mon.a (mon.0) 795 : cluster 0 osdmap e97: 8 total, 8 up, 8 in 2024-04-06T03:21:37.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:37 smithi153 bash[23852]: cluster 2024-04-06T03:21:35.611576+0000 mgr.y (mgr.14152) 384 : cluster 0 pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-04-06T03:21:37.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:37 smithi050 bash[21467]: cluster 2024-04-06T03:21:35.611576+0000 mgr.y (mgr.14152) 384 : cluster 0 pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-04-06T03:21:37.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:37 smithi050 bash[29221]: cluster 2024-04-06T03:21:35.611576+0000 mgr.y (mgr.14152) 384 : cluster 0 pgmap v388: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 702 B/s rd, 0 op/s 2024-04-06T03:21:37.948 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:37 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:21:38.210 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.211 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:38 smithi153 bash[23852]: cluster 2024-04-06T03:21:37.612758+0000 mgr.y (mgr.14152) 385 : cluster 0 pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:38.212 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:38 smithi153 bash[23852]: audit 2024-04-06T03:21:37.682447+0000 mgr.y (mgr.14152) 386 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:38.212 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.212 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.212 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.213 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.213 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.213 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.213 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:21:37 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:38 smithi050 bash[29221]: cluster 2024-04-06T03:21:37.612758+0000 mgr.y (mgr.14152) 385 : cluster 0 pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:38.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:38 smithi050 bash[29221]: audit 2024-04-06T03:21:37.682447+0000 mgr.y (mgr.14152) 386 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:38.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:38 smithi050 bash[21467]: cluster 2024-04-06T03:21:37.612758+0000 mgr.y (mgr.14152) 385 : cluster 0 pgmap v389: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:21:38.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:38 smithi050 bash[21467]: audit 2024-04-06T03:21:37.682447+0000 mgr.y (mgr.14152) 386 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:38.542 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.542 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.542 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.543 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.543 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.544 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:38.544 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:38 smithi153 systemd[1]: Started Ceph prometheus.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:21:39.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[23852]: audit 2024-04-06T03:21:38.390989+0000 mon.a (mon.0) 796 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[23852]: audit 2024-04-06T03:21:38.401953+0000 mon.a (mon.0) 797 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[23852]: audit 2024-04-06T03:21:38.454963+0000 mon.a (mon.0) 798 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[23852]: audit 2024-04-06T03:21:38.462888+0000 mon.a (mon.0) 799 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T03:21:39.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[23852]: audit 2024-04-06T03:21:38.840394+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.366Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.366Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.366Z caller=main.go:570 level=info host_details="(Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 smithi153 (none))" 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.366Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.366Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.367Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.368Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.369Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.369Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.371Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.371Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.703µs 2024-04-06T03:21:39.525 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.371Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.372Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.372Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=41.731µs wal_replay_duration=205.031µs wbl_replay_duration=97ns total_replay_duration=266.998µs 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.373Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.373Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.373Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.395Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.125942ms db_storage=895ns remote_storage=1.682µs web_handler=381ns query_engine=725ns scrape=266.783µs scrape_sd=90.185µs notify=1.088µs notify_sd=2.658µs rules=21.524316ms tracing=6.107µs 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.395Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-06T03:21:39.526 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[49872]: ts=2024-04-06T03:21:39.395Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21725]: ignoring --setuser ceph since I am not root 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21725]: ignoring --setgroup ceph since I am not root 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[29221]: audit 2024-04-06T03:21:38.390989+0000 mon.a (mon.0) 796 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[29221]: audit 2024-04-06T03:21:38.401953+0000 mon.a (mon.0) 797 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[29221]: audit 2024-04-06T03:21:38.454963+0000 mon.a (mon.0) 798 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[29221]: audit 2024-04-06T03:21:38.462888+0000 mon.a (mon.0) 799 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T03:21:39.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[29221]: audit 2024-04-06T03:21:38.840394+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21467]: audit 2024-04-06T03:21:38.390989+0000 mon.a (mon.0) 796 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21467]: audit 2024-04-06T03:21:38.401953+0000 mon.a (mon.0) 797 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21467]: audit 2024-04-06T03:21:38.454963+0000 mon.a (mon.0) 798 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21467]: audit 2024-04-06T03:21:38.462888+0000 mon.a (mon.0) 799 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-06T03:21:39.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21467]: audit 2024-04-06T03:21:38.840394+0000 mon.a (mon.0) 800 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' 2024-04-06T03:21:39.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[24761]: ignoring --setuser ceph since I am not root 2024-04-06T03:21:39.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[24761]: ignoring --setgroup ceph since I am not root 2024-04-06T03:21:39.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[24761]: debug 2024-04-06T03:21:39.530+0000 7f48063c0700 1 -- 172.21.15.153:0/3879875760 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca9273a5a0 con 0x55ca934b8c00 2024-04-06T03:21:39.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:39 smithi153 bash[24761]: debug 2024-04-06T03:21:39.678+0000 7f48101a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T03:21:39.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:39 smithi050 bash[21725]: debug 2024-04-06T03:21:39.668+0000 7f6510178200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-06T03:21:40.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:40 smithi050 bash[21467]: audit 2024-04-06T03:21:39.455439+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T03:21:40.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:40 smithi050 bash[21467]: cluster 2024-04-06T03:21:39.468148+0000 mon.a (mon.0) 802 : cluster 0 mgrmap e17: y(active, since 10m), standbys: x 2024-04-06T03:21:40.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:40 smithi050 bash[29221]: audit 2024-04-06T03:21:39.455439+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T03:21:40.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:40 smithi050 bash[29221]: cluster 2024-04-06T03:21:39.468148+0000 mon.a (mon.0) 802 : cluster 0 mgrmap e17: y(active, since 10m), standbys: x 2024-04-06T03:21:40.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:40 smithi153 bash[23852]: audit 2024-04-06T03:21:39.455439+0000 mon.a (mon.0) 801 : audit 1 from='mgr.14152 172.21.15.50:0/701988539' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-06T03:21:40.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:40 smithi153 bash[23852]: cluster 2024-04-06T03:21:39.468148+0000 mon.a (mon.0) 802 : cluster 0 mgrmap e17: y(active, since 10m), standbys: x 2024-04-06T03:21:40.969 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:21:41.541 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:41 smithi153 bash[24761]: debug 2024-04-06T03:21:41.266+0000 7f48101a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T03:21:41.541 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:41 smithi153 bash[24761]: debug 2024-04-06T03:21:41.370+0000 7f48101a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T03:21:41.661 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:41 smithi050 bash[21725]: debug 2024-04-06T03:21:41.304+0000 7f6510178200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-06T03:21:41.661 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:41 smithi050 bash[21725]: debug 2024-04-06T03:21:41.408+0000 7f6510178200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-06T03:21:41.916 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:41 smithi153 bash[24761]: debug 2024-04-06T03:21:41.630+0000 7f48101a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T03:21:41.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:41 smithi050 bash[21725]: debug 2024-04-06T03:21:41.664+0000 7f6510178200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-06T03:21:42.541 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:42 smithi153 bash[24761]: debug 2024-04-06T03:21:42.442+0000 7f48101a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T03:21:42.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:42 smithi050 bash[21725]: debug 2024-04-06T03:21:42.492+0000 7f6510178200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-06T03:21:42.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:42 smithi050 bash[21725]: debug 2024-04-06T03:21:42.608+0000 7f6510178200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T03:21:42.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:42 smithi050 bash[21725]: debug 2024-04-06T03:21:42.716+0000 7f6510178200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T03:21:43.026 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:42 smithi153 bash[24761]: debug 2024-04-06T03:21:42.554+0000 7f48101a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-06T03:21:43.026 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:42 smithi153 bash[24761]: debug 2024-04-06T03:21:42.662+0000 7f48101a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-06T03:21:43.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:43 smithi153 bash[24761]: debug 2024-04-06T03:21:43.030+0000 7f48101a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T03:21:43.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:43 smithi153 bash[24761]: debug 2024-04-06T03:21:43.142+0000 7f48101a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T03:21:43.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:43 smithi153 bash[24761]: debug 2024-04-06T03:21:43.250+0000 7f48101a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T03:21:43.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:43 smithi050 bash[21725]: debug 2024-04-06T03:21:43.084+0000 7f6510178200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-06T03:21:43.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:43 smithi050 bash[21725]: debug 2024-04-06T03:21:43.204+0000 7f6510178200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-06T03:21:43.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:43 smithi050 bash[21725]: debug 2024-04-06T03:21:43.312+0000 7f6510178200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-06T03:21:43.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:43 smithi153 bash[24761]: debug 2024-04-06T03:21:43.618+0000 7f48101a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T03:21:43.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:43 smithi050 bash[21725]: debug 2024-04-06T03:21:43.688+0000 7f6510178200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-06T03:21:43.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:43 smithi050 bash[21725]: debug 2024-04-06T03:21:43.936+0000 7f6510178200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T03:21:44.125 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:43 smithi153 bash[24761]: debug 2024-04-06T03:21:43.870+0000 7f48101a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-06T03:21:44.482 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:44 smithi050 bash[21725]: debug 2024-04-06T03:21:44.212+0000 7f6510178200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T03:21:44.483 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:44 smithi050 bash[21725]: debug 2024-04-06T03:21:44.332+0000 7f6510178200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T03:21:44.515 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:44 smithi153 bash[24761]: debug 2024-04-06T03:21:44.130+0000 7f48101a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-06T03:21:44.515 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:44 smithi153 bash[24761]: debug 2024-04-06T03:21:44.245+0000 7f48101a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-06T03:21:44.515 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:44 smithi153 bash[24761]: debug 2024-04-06T03:21:44.517+0000 7f48101a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T03:21:44.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:44 smithi050 bash[21725]: debug 2024-04-06T03:21:44.620+0000 7f6510178200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-06T03:21:45.289 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:44 smithi153 bash[24761]: debug 2024-04-06T03:21:44.937+0000 7f48101a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:21:45.289 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:45 smithi153 bash[24761]: debug 2024-04-06T03:21:45.173+0000 7f48101a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T03:21:45.291 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:45 smithi050 bash[21725]: debug 2024-04-06T03:21:45.044+0000 7f6510178200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:21:45.541 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:45 smithi153 bash[24761]: debug 2024-04-06T03:21:45.293+0000 7f48101a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T03:21:45.541 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:45 smithi153 bash[24761]: debug 2024-04-06T03:21:45.537+0000 7f48101a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:21:45.674 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:45 smithi050 bash[21725]: debug 2024-04-06T03:21:45.296+0000 7f6510178200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-06T03:21:45.675 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:45 smithi050 bash[21725]: debug 2024-04-06T03:21:45.420+0000 7f6510178200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-06T03:21:45.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:45 smithi050 bash[21725]: debug 2024-04-06T03:21:45.680+0000 7f6510178200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-06T03:21:46.957 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:46 smithi153 bash[24761]: debug 2024-04-06T03:21:46.549+0000 7f48101a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T03:21:46.957 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:46 smithi153 bash[24761]: debug 2024-04-06T03:21:46.665+0000 7f48101a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T03:21:46.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:46 smithi050 bash[21725]: debug 2024-04-06T03:21:46.728+0000 7f6510178200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-06T03:21:46.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:46 smithi050 bash[21725]: debug 2024-04-06T03:21:46.848+0000 7f6510178200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-06T03:21:47.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:46 smithi153 bash[24761]: debug 2024-04-06T03:21:46.961+0000 7f48101a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T03:21:47.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: debug 2024-04-06T03:21:47.073+0000 7f48101a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T03:21:47.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21725]: debug 2024-04-06T03:21:47.160+0000 7f6510178200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-06T03:21:47.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21725]: debug 2024-04-06T03:21:47.280+0000 7f6510178200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-06T03:21:47.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: debug 2024-04-06T03:21:47.525+0000 7f48101a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T03:21:47.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: debug 2024-04-06T03:21:47.645+0000 7f48101a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T03:21:47.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: [06/Apr/2024:03:21:47] ENGINE Bus STARTING 2024-04-06T03:21:47.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: CherryPy Checker: 2024-04-06T03:21:47.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: The Application mounted at '' has an empty config. 2024-04-06T03:21:47.791 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: [06/Apr/2024:03:21:47] ENGINE Serving on http://:::9283 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[24761]: [06/Apr/2024:03:21:47] ENGINE Bus STARTED 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[23852]: audit 2024-04-06T03:21:47.672216+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[23852]: audit 2024-04-06T03:21:47.673134+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.674034+0000 mon.a (mon.0) 803 : cluster 0 Standby manager daemon x restarted 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.674301+0000 mon.a (mon.0) 804 : cluster 0 Standby manager daemon x started 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[23852]: audit 2024-04-06T03:21:47.675932+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T03:21:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:47 smithi153 bash[23852]: audit 2024-04-06T03:21:47.676409+0000 mon.b (mon.1) 32 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:21:47.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21725]: debug 2024-04-06T03:21:47.760+0000 7f6510178200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-06T03:21:47.981 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21725]: debug 2024-04-06T03:21:47.888+0000 7f6510178200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-06T03:21:47.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21467]: audit 2024-04-06T03:21:47.672216+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T03:21:47.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21467]: audit 2024-04-06T03:21:47.673134+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:21:47.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.674034+0000 mon.a (mon.0) 803 : cluster 0 Standby manager daemon x restarted 2024-04-06T03:21:47.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.674301+0000 mon.a (mon.0) 804 : cluster 0 Standby manager daemon x started 2024-04-06T03:21:47.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21467]: audit 2024-04-06T03:21:47.675932+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T03:21:47.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[21467]: audit 2024-04-06T03:21:47.676409+0000 mon.b (mon.1) 32 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:21:47.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[29221]: audit 2024-04-06T03:21:47.672216+0000 mon.b (mon.1) 29 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-06T03:21:47.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[29221]: audit 2024-04-06T03:21:47.673134+0000 mon.b (mon.1) 30 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-06T03:21:47.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.674034+0000 mon.a (mon.0) 803 : cluster 0 Standby manager daemon x restarted 2024-04-06T03:21:47.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.674301+0000 mon.a (mon.0) 804 : cluster 0 Standby manager daemon x started 2024-04-06T03:21:47.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[29221]: audit 2024-04-06T03:21:47.675932+0000 mon.b (mon.1) 31 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-06T03:21:47.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:47 smithi050 bash[29221]: audit 2024-04-06T03:21:47.676409+0000 mon.b (mon.1) 32 : audit 0 from='mgr.? 172.21.15.153:0/2204545601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-06T03:21:48.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21725]: [06/Apr/2024:03:21:48] ENGINE Bus STARTING 2024-04-06T03:21:48.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21725]: CherryPy Checker: 2024-04-06T03:21:48.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21725]: The Application mounted at '' has an empty config. 2024-04-06T03:21:48.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21725]: [06/Apr/2024:03:21:48] ENGINE Serving on http://:::9283 2024-04-06T03:21:48.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21725]: [06/Apr/2024:03:21:48] ENGINE Bus STARTED 2024-04-06T03:21:48.962 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled alertmanager update... 2024-04-06T03:21:49.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.738176+0000 mon.a (mon.0) 805 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.915739+0000 mon.a (mon.0) 806 : cluster 1 Active manager daemon y restarted 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.916043+0000 mon.a (mon.0) 807 : cluster 1 Activating manager daemon y 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.930512+0000 mon.a (mon.0) 808 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.932046+0000 mon.a (mon.0) 809 : cluster 0 mgrmap e19: y(active, starting, since 0.0160819s), standbys: x 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.935942+0000 mon.a (mon.0) 810 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.936170+0000 mon.a (mon.0) 811 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.936284+0000 mon.a (mon.0) 812 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.938883+0000 mon.a (mon.0) 813 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.939053+0000 mon.a (mon.0) 814 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T03:21:49.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.939690+0000 mon.a (mon.0) 815 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.939964+0000 mon.a (mon.0) 816 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.940207+0000 mon.a (mon.0) 817 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.940384+0000 mon.a (mon.0) 818 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.940578+0000 mon.a (mon.0) 819 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.940772+0000 mon.a (mon.0) 820 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.941009+0000 mon.a (mon.0) 821 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.941206+0000 mon.a (mon.0) 822 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.941883+0000 mon.a (mon.0) 823 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.942058+0000 mon.a (mon.0) 824 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:47.942691+0000 mon.a (mon.0) 825 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:47.966800+0000 mon.a (mon.0) 826 : cluster 1 Manager daemon y is now available 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:48.004953+0000 mon.a (mon.0) 827 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:49.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:48.032195+0000 mon.a (mon.0) 828 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:49.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:48.036050+0000 mon.a (mon.0) 829 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:49.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:48.042276+0000 mon.a (mon.0) 830 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:21:49.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: audit 2024-04-06T03:21:48.071245+0000 mon.a (mon.0) 831 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T03:21:49.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:48 smithi153 bash[23852]: cluster 2024-04-06T03:21:48.329204+0000 mon.a (mon.0) 832 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.738176+0000 mon.a (mon.0) 805 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.915739+0000 mon.a (mon.0) 806 : cluster 1 Active manager daemon y restarted 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.916043+0000 mon.a (mon.0) 807 : cluster 1 Activating manager daemon y 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.930512+0000 mon.a (mon.0) 808 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.932046+0000 mon.a (mon.0) 809 : cluster 0 mgrmap e19: y(active, starting, since 0.0160819s), standbys: x 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.935942+0000 mon.a (mon.0) 810 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.936170+0000 mon.a (mon.0) 811 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.936284+0000 mon.a (mon.0) 812 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.938883+0000 mon.a (mon.0) 813 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T03:21:49.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.939053+0000 mon.a (mon.0) 814 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.939690+0000 mon.a (mon.0) 815 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.939964+0000 mon.a (mon.0) 816 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.940207+0000 mon.a (mon.0) 817 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.940384+0000 mon.a (mon.0) 818 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.940578+0000 mon.a (mon.0) 819 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.940772+0000 mon.a (mon.0) 820 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.941009+0000 mon.a (mon.0) 821 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.941206+0000 mon.a (mon.0) 822 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.941883+0000 mon.a (mon.0) 823 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:21:49.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.942058+0000 mon.a (mon.0) 824 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.738176+0000 mon.a (mon.0) 805 : cluster 0 mgrmap e18: y(active, since 10m), standbys: x 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.915739+0000 mon.a (mon.0) 806 : cluster 1 Active manager daemon y restarted 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.916043+0000 mon.a (mon.0) 807 : cluster 1 Activating manager daemon y 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.930512+0000 mon.a (mon.0) 808 : cluster 0 osdmap e98: 8 total, 8 up, 8 in 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.932046+0000 mon.a (mon.0) 809 : cluster 0 mgrmap e19: y(active, starting, since 0.0160819s), standbys: x 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.935942+0000 mon.a (mon.0) 810 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.936170+0000 mon.a (mon.0) 811 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.936284+0000 mon.a (mon.0) 812 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.938883+0000 mon.a (mon.0) 813 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.939053+0000 mon.a (mon.0) 814 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-06T03:21:49.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.939690+0000 mon.a (mon.0) 815 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.939964+0000 mon.a (mon.0) 816 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.940207+0000 mon.a (mon.0) 817 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.940384+0000 mon.a (mon.0) 818 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.940578+0000 mon.a (mon.0) 819 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.940772+0000 mon.a (mon.0) 820 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.941009+0000 mon.a (mon.0) 821 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.941206+0000 mon.a (mon.0) 822 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-06T03:21:49.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.941883+0000 mon.a (mon.0) 823 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-06T03:21:49.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.942058+0000 mon.a (mon.0) 824 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-06T03:21:49.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:47.942691+0000 mon.a (mon.0) 825 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:21:49.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:47.966800+0000 mon.a (mon.0) 826 : cluster 1 Manager daemon y is now available 2024-04-06T03:21:49.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:48.004953+0000 mon.a (mon.0) 827 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:49.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:48.032195+0000 mon.a (mon.0) 828 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:49.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:48.036050+0000 mon.a (mon.0) 829 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:48.042276+0000 mon.a (mon.0) 830 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: audit 2024-04-06T03:21:48.071245+0000 mon.a (mon.0) 831 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[21467]: cluster 2024-04-06T03:21:48.329204+0000 mon.a (mon.0) 832 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:47.942691+0000 mon.a (mon.0) 825 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:47.966800+0000 mon.a (mon.0) 826 : cluster 1 Manager daemon y is now available 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:48.004953+0000 mon.a (mon.0) 827 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:48.032195+0000 mon.a (mon.0) 828 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:48.036050+0000 mon.a (mon.0) 829 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:48.042276+0000 mon.a (mon.0) 830 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: audit 2024-04-06T03:21:48.071245+0000 mon.a (mon.0) 831 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-06T03:21:49.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:48 smithi050 bash[29221]: cluster 2024-04-06T03:21:48.329204+0000 mon.a (mon.0) 832 : cluster 0 osdmap e99: 8 total, 8 up, 8 in 2024-04-06T03:21:49.731 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:21:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:21:49.774 DEBUG:teuthology.orchestra.run.smithi050:alertmanager.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@alertmanager.a.service 2024-04-06T03:21:49.777 INFO:tasks.cephadm:Adding grafana.a on smithi153 2024-04-06T03:21:49.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph orch apply grafana '1;smithi153=a' 2024-04-06T03:21:50.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cephadm 2024-04-06T03:21:48.538843+0000 mgr.y (mgr.24368) 1 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Bus STARTING 2024-04-06T03:21:50.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cephadm 2024-04-06T03:21:48.641261+0000 mgr.y (mgr.24368) 2 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Serving on http://172.21.15.50:8765 2024-04-06T03:21:50.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cephadm 2024-04-06T03:21:48.760257+0000 mgr.y (mgr.24368) 3 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Serving on https://172.21.15.50:7150 2024-04-06T03:21:50.041 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cephadm 2024-04-06T03:21:48.760447+0000 mgr.y (mgr.24368) 4 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Bus STARTED 2024-04-06T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cluster 2024-04-06T03:21:48.940862+0000 mon.a (mon.0) 833 : cluster 0 mgrmap e20: y(active, since 1.0249s), standbys: x 2024-04-06T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: audit 2024-04-06T03:21:48.951651+0000 mgr.y (mgr.24368) 5 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: audit 2024-04-06T03:21:48.954225+0000 mgr.y (mgr.24368) 6 : audit 0 from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cephadm 2024-04-06T03:21:48.961195+0000 mgr.y (mgr.24368) 7 : cephadm 1 Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-06T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: cluster 2024-04-06T03:21:48.962357+0000 mgr.y (mgr.24368) 8 : cluster 0 pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:50.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:49 smithi153 bash[23852]: audit 2024-04-06T03:21:48.971318+0000 mon.a (mon.0) 834 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cephadm 2024-04-06T03:21:48.538843+0000 mgr.y (mgr.24368) 1 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Bus STARTING 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cephadm 2024-04-06T03:21:48.641261+0000 mgr.y (mgr.24368) 2 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Serving on http://172.21.15.50:8765 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cephadm 2024-04-06T03:21:48.760257+0000 mgr.y (mgr.24368) 3 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Serving on https://172.21.15.50:7150 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cephadm 2024-04-06T03:21:48.760447+0000 mgr.y (mgr.24368) 4 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Bus STARTED 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cluster 2024-04-06T03:21:48.940862+0000 mon.a (mon.0) 833 : cluster 0 mgrmap e20: y(active, since 1.0249s), standbys: x 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: audit 2024-04-06T03:21:48.951651+0000 mgr.y (mgr.24368) 5 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:50.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: audit 2024-04-06T03:21:48.954225+0000 mgr.y (mgr.24368) 6 : audit 0 from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cephadm 2024-04-06T03:21:48.961195+0000 mgr.y (mgr.24368) 7 : cephadm 1 Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: cluster 2024-04-06T03:21:48.962357+0000 mgr.y (mgr.24368) 8 : cluster 0 pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[21467]: audit 2024-04-06T03:21:48.971318+0000 mon.a (mon.0) 834 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cephadm 2024-04-06T03:21:48.538843+0000 mgr.y (mgr.24368) 1 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Bus STARTING 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cephadm 2024-04-06T03:21:48.641261+0000 mgr.y (mgr.24368) 2 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Serving on http://172.21.15.50:8765 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cephadm 2024-04-06T03:21:48.760257+0000 mgr.y (mgr.24368) 3 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Serving on https://172.21.15.50:7150 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cephadm 2024-04-06T03:21:48.760447+0000 mgr.y (mgr.24368) 4 : cephadm 1 [06/Apr/2024:03:21:48] ENGINE Bus STARTED 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cluster 2024-04-06T03:21:48.940862+0000 mon.a (mon.0) 833 : cluster 0 mgrmap e20: y(active, since 1.0249s), standbys: x 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: audit 2024-04-06T03:21:48.951651+0000 mgr.y (mgr.24368) 5 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: audit 2024-04-06T03:21:48.954225+0000 mgr.y (mgr.24368) 6 : audit 0 from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:50.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cephadm 2024-04-06T03:21:48.961195+0000 mgr.y (mgr.24368) 7 : cephadm 1 Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-06T03:21:50.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: cluster 2024-04-06T03:21:48.962357+0000 mgr.y (mgr.24368) 8 : cluster 0 pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:50.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:49 smithi050 bash[29221]: audit 2024-04-06T03:21:48.971318+0000 mon.a (mon.0) 834 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:51.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:50 smithi050 bash[21467]: cluster 2024-04-06T03:21:49.942322+0000 mgr.y (mgr.24368) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:51.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:50 smithi050 bash[29221]: cluster 2024-04-06T03:21:49.942322+0000 mgr.y (mgr.24368) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:51.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:50 smithi153 bash[23852]: cluster 2024-04-06T03:21:49.942322+0000 mgr.y (mgr.24368) 9 : cluster 0 pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:52.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:51 smithi050 bash[21467]: cluster 2024-04-06T03:21:50.948617+0000 mon.a (mon.0) 835 : cluster 0 mgrmap e21: y(active, since 3s), standbys: x 2024-04-06T03:21:52.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:51 smithi050 bash[29221]: cluster 2024-04-06T03:21:50.948617+0000 mon.a (mon.0) 835 : cluster 0 mgrmap e21: y(active, since 3s), standbys: x 2024-04-06T03:21:52.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:51 smithi153 bash[23852]: cluster 2024-04-06T03:21:50.948617+0000 mon.a (mon.0) 835 : cluster 0 mgrmap e21: y(active, since 3s), standbys: x 2024-04-06T03:21:53.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:52 smithi050 bash[29221]: cluster 2024-04-06T03:21:51.943100+0000 mgr.y (mgr.24368) 10 : cluster 0 pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:53.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:52 smithi050 bash[21467]: cluster 2024-04-06T03:21:51.943100+0000 mgr.y (mgr.24368) 10 : cluster 0 pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:53.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:52 smithi153 bash[23852]: cluster 2024-04-06T03:21:51.943100+0000 mgr.y (mgr.24368) 10 : cluster 0 pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: cluster 2024-04-06T03:21:52.625299+0000 osd.3 (osd.3) 11 : cluster 0 2.1e deep-scrub starts 2024-04-06T03:21:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: cluster 2024-04-06T03:21:52.627769+0000 osd.3 (osd.3) 12 : cluster 0 2.1e deep-scrub ok 2024-04-06T03:21:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: cluster 2024-04-06T03:21:52.846243+0000 osd.1 (osd.1) 17 : cluster 0 2.15 scrub starts 2024-04-06T03:21:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: cluster 2024-04-06T03:21:52.848725+0000 osd.1 (osd.1) 18 : cluster 0 2.15 scrub ok 2024-04-06T03:21:54.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: cluster 2024-04-06T03:21:52.970371+0000 mon.a (mon.0) 836 : cluster 0 mgrmap e22: y(active, since 5s), standbys: x 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: audit 2024-04-06T03:21:53.670272+0000 mon.a (mon.0) 837 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[21467]: audit 2024-04-06T03:21:53.680469+0000 mon.a (mon.0) 838 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: cluster 2024-04-06T03:21:52.625299+0000 osd.3 (osd.3) 11 : cluster 0 2.1e deep-scrub starts 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: cluster 2024-04-06T03:21:52.627769+0000 osd.3 (osd.3) 12 : cluster 0 2.1e deep-scrub ok 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: cluster 2024-04-06T03:21:52.846243+0000 osd.1 (osd.1) 17 : cluster 0 2.15 scrub starts 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: cluster 2024-04-06T03:21:52.848725+0000 osd.1 (osd.1) 18 : cluster 0 2.15 scrub ok 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: cluster 2024-04-06T03:21:52.970371+0000 mon.a (mon.0) 836 : cluster 0 mgrmap e22: y(active, since 5s), standbys: x 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: audit 2024-04-06T03:21:53.670272+0000 mon.a (mon.0) 837 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:54.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:53 smithi050 bash[29221]: audit 2024-04-06T03:21:53.680469+0000 mon.a (mon.0) 838 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:54.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: cluster 2024-04-06T03:21:52.625299+0000 osd.3 (osd.3) 11 : cluster 0 2.1e deep-scrub starts 2024-04-06T03:21:54.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: cluster 2024-04-06T03:21:52.627769+0000 osd.3 (osd.3) 12 : cluster 0 2.1e deep-scrub ok 2024-04-06T03:21:54.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: cluster 2024-04-06T03:21:52.846243+0000 osd.1 (osd.1) 17 : cluster 0 2.15 scrub starts 2024-04-06T03:21:54.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: cluster 2024-04-06T03:21:52.848725+0000 osd.1 (osd.1) 18 : cluster 0 2.15 scrub ok 2024-04-06T03:21:54.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: cluster 2024-04-06T03:21:52.970371+0000 mon.a (mon.0) 836 : cluster 0 mgrmap e22: y(active, since 5s), standbys: x 2024-04-06T03:21:54.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: audit 2024-04-06T03:21:53.670272+0000 mon.a (mon.0) 837 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:54.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:53 smithi153 bash[23852]: audit 2024-04-06T03:21:53.680469+0000 mon.a (mon.0) 838 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:54.448 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: cluster 2024-04-06T03:21:53.943654+0000 mgr.y (mgr.24368) 11 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.124563+0000 mon.a (mon.0) 839 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.132942+0000 mon.a (mon.0) 840 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: cluster 2024-04-06T03:21:54.255282+0000 osd.7 (osd.7) 7 : cluster 0 2.1d scrub starts 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: cluster 2024-04-06T03:21:54.257292+0000 osd.7 (osd.7) 8 : cluster 0 2.1d scrub ok 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.843474+0000 mon.a (mon.0) 841 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.850008+0000 mon.a (mon.0) 842 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.851620+0000 mon.a (mon.0) 843 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.852616+0000 mon.a (mon.0) 844 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.853457+0000 mon.a (mon.0) 845 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.854277+0000 mon.a (mon.0) 846 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.383 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:55 smithi153 bash[23852]: audit 2024-04-06T03:21:54.859963+0000 mon.a (mon.0) 847 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: cluster 2024-04-06T03:21:53.943654+0000 mgr.y (mgr.24368) 11 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.124563+0000 mon.a (mon.0) 839 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.132942+0000 mon.a (mon.0) 840 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: cluster 2024-04-06T03:21:54.255282+0000 osd.7 (osd.7) 7 : cluster 0 2.1d scrub starts 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: cluster 2024-04-06T03:21:54.257292+0000 osd.7 (osd.7) 8 : cluster 0 2.1d scrub ok 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.843474+0000 mon.a (mon.0) 841 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.850008+0000 mon.a (mon.0) 842 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.851620+0000 mon.a (mon.0) 843 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.852616+0000 mon.a (mon.0) 844 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.853457+0000 mon.a (mon.0) 845 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.854277+0000 mon.a (mon.0) 846 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[21467]: audit 2024-04-06T03:21:54.859963+0000 mon.a (mon.0) 847 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: cluster 2024-04-06T03:21:53.943654+0000 mgr.y (mgr.24368) 11 : cluster 0 pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.124563+0000 mon.a (mon.0) 839 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.132942+0000 mon.a (mon.0) 840 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: cluster 2024-04-06T03:21:54.255282+0000 osd.7 (osd.7) 7 : cluster 0 2.1d scrub starts 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: cluster 2024-04-06T03:21:54.257292+0000 osd.7 (osd.7) 8 : cluster 0 2.1d scrub ok 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.843474+0000 mon.a (mon.0) 841 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.850008+0000 mon.a (mon.0) 842 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.851620+0000 mon.a (mon.0) 843 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.852616+0000 mon.a (mon.0) 844 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.853457+0000 mon.a (mon.0) 845 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.854277+0000 mon.a (mon.0) 846 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:55.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:55 smithi050 bash[29221]: audit 2024-04-06T03:21:54.859963+0000 mon.a (mon.0) 847 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cluster 2024-04-06T03:21:54.624591+0000 osd.5 (osd.5) 7 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cluster 2024-04-06T03:21:54.626048+0000 osd.5 (osd.5) 8 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cluster 2024-04-06T03:21:54.845035+0000 osd.1 (osd.1) 19 : cluster 0 2.a scrub starts 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cluster 2024-04-06T03:21:54.846716+0000 osd.1 (osd.1) 20 : cluster 0 2.a scrub ok 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cephadm 2024-04-06T03:21:54.854948+0000 mgr.y (mgr.24368) 12 : cephadm 1 Adjusting osd_memory_target on smithi050 to 3805M 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.200018+0000 mon.a (mon.0) 848 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.211138+0000 mon.a (mon.0) 849 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.214223+0000 mon.a (mon.0) 850 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.216123+0000 mon.a (mon.0) 851 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.217959+0000 mon.a (mon.0) 852 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.219911+0000 mon.a (mon.0) 853 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cephadm 2024-04-06T03:21:55.221457+0000 mgr.y (mgr.24368) 13 : cephadm 1 Adjusting osd_memory_target on smithi153 to 3804M 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.228584+0000 mon.a (mon.0) 854 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.233010+0000 mon.a (mon.0) 855 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: audit 2024-04-06T03:21:55.234701+0000 mon.a (mon.0) 856 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[21467]: cephadm 2024-04-06T03:21:55.237134+0000 mgr.y (mgr.24368) 14 : cephadm 1 Updating smithi050:/etc/ceph/ceph.conf 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cluster 2024-04-06T03:21:54.624591+0000 osd.5 (osd.5) 7 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cluster 2024-04-06T03:21:54.626048+0000 osd.5 (osd.5) 8 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:21:56.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cluster 2024-04-06T03:21:54.845035+0000 osd.1 (osd.1) 19 : cluster 0 2.a scrub starts 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cluster 2024-04-06T03:21:54.846716+0000 osd.1 (osd.1) 20 : cluster 0 2.a scrub ok 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cephadm 2024-04-06T03:21:54.854948+0000 mgr.y (mgr.24368) 12 : cephadm 1 Adjusting osd_memory_target on smithi050 to 3805M 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.200018+0000 mon.a (mon.0) 848 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.211138+0000 mon.a (mon.0) 849 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.214223+0000 mon.a (mon.0) 850 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.216123+0000 mon.a (mon.0) 851 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.217959+0000 mon.a (mon.0) 852 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.219911+0000 mon.a (mon.0) 853 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cephadm 2024-04-06T03:21:55.221457+0000 mgr.y (mgr.24368) 13 : cephadm 1 Adjusting osd_memory_target on smithi153 to 3804M 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.228584+0000 mon.a (mon.0) 854 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.233010+0000 mon.a (mon.0) 855 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: audit 2024-04-06T03:21:55.234701+0000 mon.a (mon.0) 856 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:56.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:56 smithi050 bash[29221]: cephadm 2024-04-06T03:21:55.237134+0000 mgr.y (mgr.24368) 14 : cephadm 1 Updating smithi050:/etc/ceph/ceph.conf 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cluster 2024-04-06T03:21:54.624591+0000 osd.5 (osd.5) 7 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cluster 2024-04-06T03:21:54.626048+0000 osd.5 (osd.5) 8 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cluster 2024-04-06T03:21:54.845035+0000 osd.1 (osd.1) 19 : cluster 0 2.a scrub starts 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cluster 2024-04-06T03:21:54.846716+0000 osd.1 (osd.1) 20 : cluster 0 2.a scrub ok 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cephadm 2024-04-06T03:21:54.854948+0000 mgr.y (mgr.24368) 12 : cephadm 1 Adjusting osd_memory_target on smithi050 to 3805M 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.200018+0000 mon.a (mon.0) 848 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.211138+0000 mon.a (mon.0) 849 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.214223+0000 mon.a (mon.0) 850 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.216123+0000 mon.a (mon.0) 851 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.217959+0000 mon.a (mon.0) 852 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.219911+0000 mon.a (mon.0) 853 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:21:56.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cephadm 2024-04-06T03:21:55.221457+0000 mgr.y (mgr.24368) 13 : cephadm 1 Adjusting osd_memory_target on smithi153 to 3804M 2024-04-06T03:21:56.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.228584+0000 mon.a (mon.0) 854 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:56.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.233010+0000 mon.a (mon.0) 855 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:21:56.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: audit 2024-04-06T03:21:55.234701+0000 mon.a (mon.0) 856 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:21:56.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:56 smithi153 bash[23852]: cephadm 2024-04-06T03:21:55.237134+0000 mgr.y (mgr.24368) 14 : cephadm 1 Updating smithi050:/etc/ceph/ceph.conf 2024-04-06T03:21:57.350 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.350 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.351 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.351 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.351 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.351 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.352 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:55.572316+0000 mgr.y (mgr.24368) 15 : cephadm 1 Updating smithi153:/etc/ceph/ceph.conf 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:55.595957+0000 osd.5 (osd.5) 9 : cluster 0 2.12 scrub starts 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:55.597440+0000 osd.5 (osd.5) 10 : cluster 0 2.12 scrub ok 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:55.773512+0000 osd.2 (osd.2) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:55.773593+0000 osd.2 (osd.2) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:55.788221+0000 mgr.y (mgr.24368) 16 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:21:57.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:55.839109+0000 mgr.y (mgr.24368) 17 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:55.944975+0000 mgr.y (mgr.24368) 18 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:55.954036+0000 mgr.y (mgr.24368) 19 : cephadm 1 Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:56.004994+0000 mgr.y (mgr.24368) 20 : cephadm 1 Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:56.119097+0000 mgr.y (mgr.24368) 21 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:56.170502+0000 mgr.y (mgr.24368) 22 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:56.233176+0000 mon.a (mon.0) 857 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: audit 2024-04-06T03:21:56.291372+0000 mon.a (mon.0) 858 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: audit 2024-04-06T03:21:56.297877+0000 mon.a (mon.0) 859 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: audit 2024-04-06T03:21:56.342058+0000 mon.a (mon.0) 860 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: audit 2024-04-06T03:21:56.352183+0000 mon.a (mon.0) 861 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: audit 2024-04-06T03:21:56.361911+0000 mon.a (mon.0) 862 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cephadm 2024-04-06T03:21:56.370044+0000 mgr.y (mgr.24368) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi050 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:56.410542+0000 osd.2 (osd.2) 13 : cluster 0 2.e deep-scrub starts 2024-04-06T03:21:57.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[21467]: cluster 2024-04-06T03:21:56.412864+0000 osd.2 (osd.2) 14 : cluster 0 2.e deep-scrub ok 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:55.572316+0000 mgr.y (mgr.24368) 15 : cephadm 1 Updating smithi153:/etc/ceph/ceph.conf 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:55.595957+0000 osd.5 (osd.5) 9 : cluster 0 2.12 scrub starts 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:55.597440+0000 osd.5 (osd.5) 10 : cluster 0 2.12 scrub ok 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:55.773512+0000 osd.2 (osd.2) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:55.773593+0000 osd.2 (osd.2) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:57.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:55.788221+0000 mgr.y (mgr.24368) 16 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:55.839109+0000 mgr.y (mgr.24368) 17 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:55.944975+0000 mgr.y (mgr.24368) 18 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:55.954036+0000 mgr.y (mgr.24368) 19 : cephadm 1 Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:56.004994+0000 mgr.y (mgr.24368) 20 : cephadm 1 Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:56.119097+0000 mgr.y (mgr.24368) 21 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:56.170502+0000 mgr.y (mgr.24368) 22 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:56.233176+0000 mon.a (mon.0) 857 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: audit 2024-04-06T03:21:56.291372+0000 mon.a (mon.0) 858 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: audit 2024-04-06T03:21:56.297877+0000 mon.a (mon.0) 859 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: audit 2024-04-06T03:21:56.342058+0000 mon.a (mon.0) 860 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: audit 2024-04-06T03:21:56.352183+0000 mon.a (mon.0) 861 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: audit 2024-04-06T03:21:56.361911+0000 mon.a (mon.0) 862 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cephadm 2024-04-06T03:21:56.370044+0000 mgr.y (mgr.24368) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi050 2024-04-06T03:21:57.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:56.410542+0000 osd.2 (osd.2) 13 : cluster 0 2.e deep-scrub starts 2024-04-06T03:21:57.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[29221]: cluster 2024-04-06T03:21:56.412864+0000 osd.2 (osd.2) 14 : cluster 0 2.e deep-scrub ok 2024-04-06T03:21:57.536 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled grafana update... 2024-04-06T03:21:57.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:55.572316+0000 mgr.y (mgr.24368) 15 : cephadm 1 Updating smithi153:/etc/ceph/ceph.conf 2024-04-06T03:21:57.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:55.595957+0000 osd.5 (osd.5) 9 : cluster 0 2.12 scrub starts 2024-04-06T03:21:57.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:55.597440+0000 osd.5 (osd.5) 10 : cluster 0 2.12 scrub ok 2024-04-06T03:21:57.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:55.773512+0000 osd.2 (osd.2) 11 : cluster 0 purged_snaps scrub starts 2024-04-06T03:21:57.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:55.773593+0000 osd.2 (osd.2) 12 : cluster 0 purged_snaps scrub ok 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:55.788221+0000 mgr.y (mgr.24368) 16 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:55.839109+0000 mgr.y (mgr.24368) 17 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:55.944975+0000 mgr.y (mgr.24368) 18 : cluster 0 pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:55.954036+0000 mgr.y (mgr.24368) 19 : cephadm 1 Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:56.004994+0000 mgr.y (mgr.24368) 20 : cephadm 1 Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:56.119097+0000 mgr.y (mgr.24368) 21 : cephadm 1 Updating smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:56.170502+0000 mgr.y (mgr.24368) 22 : cephadm 1 Updating smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:56.233176+0000 mon.a (mon.0) 857 : cluster 0 osdmap e100: 8 total, 8 up, 8 in 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: audit 2024-04-06T03:21:56.291372+0000 mon.a (mon.0) 858 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: audit 2024-04-06T03:21:56.297877+0000 mon.a (mon.0) 859 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: audit 2024-04-06T03:21:56.342058+0000 mon.a (mon.0) 860 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: audit 2024-04-06T03:21:56.352183+0000 mon.a (mon.0) 861 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: audit 2024-04-06T03:21:56.361911+0000 mon.a (mon.0) 862 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:57.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cephadm 2024-04-06T03:21:56.370044+0000 mgr.y (mgr.24368) 23 : cephadm 1 Deploying daemon node-exporter.a on smithi050 2024-04-06T03:21:57.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:56.410542+0000 osd.2 (osd.2) 13 : cluster 0 2.e deep-scrub starts 2024-04-06T03:21:57.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:57 smithi153 bash[23852]: cluster 2024-04-06T03:21:56.412864+0000 osd.2 (osd.2) 14 : cluster 0 2.e deep-scrub ok 2024-04-06T03:21:57.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.732 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.732 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.733 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.733 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.733 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:57.733 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:57 smithi050 systemd[1]: Started Ceph node-exporter.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:21:57.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:57 smithi050 bash[57670]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-04-06T03:21:58.201 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:57 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.317 DEBUG:teuthology.orchestra.run.smithi153:grafana.a> sudo journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@grafana.a.service 2024-04-06T03:21:58.319 INFO:tasks.cephadm:Setting up client nodes... 2024-04-06T03:21:58.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T03:21:58.477 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.477 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.478 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.478 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.478 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.479 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.479 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: cluster 2024-04-06T03:21:56.774241+0000 osd.1 (osd.1) 21 : cluster 0 3.4 scrub starts 2024-04-06T03:21:58.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: cluster 2024-04-06T03:21:56.778523+0000 osd.1 (osd.1) 22 : cluster 0 3.4 scrub ok 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: cluster 2024-04-06T03:21:57.272619+0000 osd.7 (osd.7) 9 : cluster 0 3.11 scrub starts 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: cluster 2024-04-06T03:21:57.275182+0000 osd.7 (osd.7) 10 : cluster 0 3.11 scrub ok 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: audit 2024-04-06T03:21:57.514269+0000 mon.a (mon.0) 863 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: audit 2024-04-06T03:21:57.519505+0000 mon.a (mon.0) 864 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: audit 2024-04-06T03:21:57.540255+0000 mon.a (mon.0) 865 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: audit 2024-04-06T03:21:57.545231+0000 mon.a (mon.0) 866 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.098837+0000 osd.0 (osd.0) 15 : cluster 0 3.1b scrub starts 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.103960+0000 osd.0 (osd.0) 16 : cluster 0 3.1b scrub ok 2024-04-06T03:21:58.480 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.481 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: v1.5.0: Pulling from prometheus/node-exporter 2024-04-06T03:21:58.481 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 22b70bddd3ac: Pulling fs layer 2024-04-06T03:21:58.481 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 5c12815fee55: Pulling fs layer 2024-04-06T03:21:58.481 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: c0e87333d380: Pulling fs layer 2024-04-06T03:21:58.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: cluster 2024-04-06T03:21:56.774241+0000 osd.1 (osd.1) 21 : cluster 0 3.4 scrub starts 2024-04-06T03:21:58.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: cluster 2024-04-06T03:21:56.778523+0000 osd.1 (osd.1) 22 : cluster 0 3.4 scrub ok 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: cluster 2024-04-06T03:21:57.272619+0000 osd.7 (osd.7) 9 : cluster 0 3.11 scrub starts 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: cluster 2024-04-06T03:21:57.275182+0000 osd.7 (osd.7) 10 : cluster 0 3.11 scrub ok 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: audit 2024-04-06T03:21:57.514269+0000 mon.a (mon.0) 863 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: audit 2024-04-06T03:21:57.519505+0000 mon.a (mon.0) 864 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: audit 2024-04-06T03:21:57.540255+0000 mon.a (mon.0) 865 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: audit 2024-04-06T03:21:57.545231+0000 mon.a (mon.0) 866 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.098837+0000 osd.0 (osd.0) 15 : cluster 0 3.1b scrub starts 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.103960+0000 osd.0 (osd.0) 16 : cluster 0 3.1b scrub ok 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: cluster 2024-04-06T03:21:56.774241+0000 osd.1 (osd.1) 21 : cluster 0 3.4 scrub starts 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: cluster 2024-04-06T03:21:56.778523+0000 osd.1 (osd.1) 22 : cluster 0 3.4 scrub ok 2024-04-06T03:21:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: cluster 2024-04-06T03:21:57.272619+0000 osd.7 (osd.7) 9 : cluster 0 3.11 scrub starts 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: cluster 2024-04-06T03:21:57.275182+0000 osd.7 (osd.7) 10 : cluster 0 3.11 scrub ok 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: audit 2024-04-06T03:21:57.514269+0000 mon.a (mon.0) 863 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: audit 2024-04-06T03:21:57.519505+0000 mon.a (mon.0) 864 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: audit 2024-04-06T03:21:57.540255+0000 mon.a (mon.0) 865 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: audit 2024-04-06T03:21:57.545231+0000 mon.a (mon.0) 866 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.098837+0000 osd.0 (osd.0) 15 : cluster 0 3.1b scrub starts 2024-04-06T03:21:58.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.103960+0000 osd.0 (osd.0) 16 : cluster 0 3.1b scrub ok 2024-04-06T03:21:58.622 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.622 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.742 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.742 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.742 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.743 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.743 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.744 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.744 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:58.744 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:58 smithi153 systemd[1]: Started Ceph node-exporter.b for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 22b70bddd3ac: Verifying Checksum 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 22b70bddd3ac: Download complete 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 5c12815fee55: Verifying Checksum 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 5c12815fee55: Download complete 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: c0e87333d380: Verifying Checksum 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: c0e87333d380: Download complete 2024-04-06T03:21:58.981 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:58 smithi050 bash[57670]: 22b70bddd3ac: Pull complete 2024-04-06T03:21:59.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:21:59.041 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:58 smithi153 bash[50896]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-04-06T03:21:59.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:21:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:21:59.482 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[57670]: 5c12815fee55: Pull complete 2024-04-06T03:21:59.482 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:21:59 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:21:59.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: audit 2024-04-06T03:21:57.535854+0000 mgr.y (mgr.24368) 24 : audit 0 from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:59.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cephadm 2024-04-06T03:21:57.538140+0000 mgr.y (mgr.24368) 25 : cephadm 1 Saving service grafana spec with placement smithi153=a;count:1 2024-04-06T03:21:59.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cephadm 2024-04-06T03:21:57.541207+0000 mgr.y (mgr.24368) 26 : cephadm 1 Deploying daemon node-exporter.b on smithi153 2024-04-06T03:21:59.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:57.945687+0000 mgr.y (mgr.24368) 27 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T03:21:59.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.305492+0000 osd.4 (osd.4) 9 : cluster 0 2.f scrub starts 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.309494+0000 osd.4 (osd.4) 10 : cluster 0 2.f scrub ok 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.337775+0000 mon.a (mon.0) 867 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.421654+0000 osd.2 (osd.2) 15 : cluster 0 6.14 scrub starts 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.423933+0000 osd.2 (osd.2) 16 : cluster 0 6.14 scrub ok 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.638817+0000 osd.3 (osd.3) 13 : cluster 0 4.0 scrub starts 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: cluster 2024-04-06T03:21:58.641292+0000 osd.3 (osd.3) 14 : cluster 0 4.0 scrub ok 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: audit 2024-04-06T03:21:58.805994+0000 mon.a (mon.0) 868 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: audit 2024-04-06T03:21:58.817188+0000 mon.a (mon.0) 869 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: audit 2024-04-06T03:21:58.862441+0000 mon.a (mon.0) 870 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: audit 2024-04-06T03:21:58.868691+0000 mon.a (mon.0) 871 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[21467]: audit 2024-04-06T03:21:58.879543+0000 mon.a (mon.0) 872 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: audit 2024-04-06T03:21:57.535854+0000 mgr.y (mgr.24368) 24 : audit 0 from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cephadm 2024-04-06T03:21:57.538140+0000 mgr.y (mgr.24368) 25 : cephadm 1 Saving service grafana spec with placement smithi153=a;count:1 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cephadm 2024-04-06T03:21:57.541207+0000 mgr.y (mgr.24368) 26 : cephadm 1 Deploying daemon node-exporter.b on smithi153 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:57.945687+0000 mgr.y (mgr.24368) 27 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.305492+0000 osd.4 (osd.4) 9 : cluster 0 2.f scrub starts 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.309494+0000 osd.4 (osd.4) 10 : cluster 0 2.f scrub ok 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.337775+0000 mon.a (mon.0) 867 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.421654+0000 osd.2 (osd.2) 15 : cluster 0 6.14 scrub starts 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.423933+0000 osd.2 (osd.2) 16 : cluster 0 6.14 scrub ok 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.638817+0000 osd.3 (osd.3) 13 : cluster 0 4.0 scrub starts 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: cluster 2024-04-06T03:21:58.641292+0000 osd.3 (osd.3) 14 : cluster 0 4.0 scrub ok 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: audit 2024-04-06T03:21:58.805994+0000 mon.a (mon.0) 868 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: audit 2024-04-06T03:21:58.817188+0000 mon.a (mon.0) 869 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: audit 2024-04-06T03:21:58.862441+0000 mon.a (mon.0) 870 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: audit 2024-04-06T03:21:58.868691+0000 mon.a (mon.0) 871 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[29221]: audit 2024-04-06T03:21:58.879543+0000 mon.a (mon.0) 872 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.624 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: audit 2024-04-06T03:21:57.535854+0000 mgr.y (mgr.24368) 24 : audit 0 from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-06T03:21:59.624 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cephadm 2024-04-06T03:21:57.538140+0000 mgr.y (mgr.24368) 25 : cephadm 1 Saving service grafana spec with placement smithi153=a;count:1 2024-04-06T03:21:59.624 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cephadm 2024-04-06T03:21:57.541207+0000 mgr.y (mgr.24368) 26 : cephadm 1 Deploying daemon node-exporter.b on smithi153 2024-04-06T03:21:59.624 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:57.945687+0000 mgr.y (mgr.24368) 27 : cluster 0 pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-06T03:21:59.624 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.305492+0000 osd.4 (osd.4) 9 : cluster 0 2.f scrub starts 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.309494+0000 osd.4 (osd.4) 10 : cluster 0 2.f scrub ok 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.337775+0000 mon.a (mon.0) 867 : cluster 0 osdmap e101: 8 total, 8 up, 8 in 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.421654+0000 osd.2 (osd.2) 15 : cluster 0 6.14 scrub starts 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.423933+0000 osd.2 (osd.2) 16 : cluster 0 6.14 scrub ok 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.638817+0000 osd.3 (osd.3) 13 : cluster 0 4.0 scrub starts 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: cluster 2024-04-06T03:21:58.641292+0000 osd.3 (osd.3) 14 : cluster 0 4.0 scrub ok 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: audit 2024-04-06T03:21:58.805994+0000 mon.a (mon.0) 868 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: audit 2024-04-06T03:21:58.817188+0000 mon.a (mon.0) 869 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: audit 2024-04-06T03:21:58.862441+0000 mon.a (mon.0) 870 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: audit 2024-04-06T03:21:58.868691+0000 mon.a (mon.0) 871 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[23852]: audit 2024-04-06T03:21:58.879543+0000 mon.a (mon.0) 872 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:21:59.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: v1.5.0: Pulling from prometheus/node-exporter 2024-04-06T03:21:59.793 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[57670]: c0e87333d380: Pull complete 2024-04-06T03:21:59.793 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[57670]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-04-06T03:21:59.958 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 22b70bddd3ac: Pulling fs layer 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 5c12815fee55: Pulling fs layer 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: c0e87333d380: Pulling fs layer 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 22b70bddd3ac: Verifying Checksum 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 22b70bddd3ac: Download complete 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 5c12815fee55: Verifying Checksum 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 5c12815fee55: Download complete 2024-04-06T03:21:59.959 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:21:59 smithi153 bash[50896]: 22b70bddd3ac: Pull complete 2024-04-06T03:22:00.231 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:21:59 smithi050 bash[57670]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-04-06T03:22:00.291 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:00 smithi153 bash[50896]: c0e87333d380: Download complete 2024-04-06T03:22:00.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:00 smithi050 bash[21467]: cephadm 2024-04-06T03:21:58.897215+0000 mgr.y (mgr.24368) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi050 2024-04-06T03:22:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:00 smithi050 bash[21467]: audit 2024-04-06T03:21:58.962798+0000 mgr.y (mgr.24368) 29 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:00.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:00 smithi050 bash[29221]: cephadm 2024-04-06T03:21:58.897215+0000 mgr.y (mgr.24368) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi050 2024-04-06T03:22:00.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:00 smithi050 bash[29221]: audit 2024-04-06T03:21:58.962798+0000 mgr.y (mgr.24368) 29 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:00.791 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:00 smithi153 bash[50896]: 5c12815fee55: Pull complete 2024-04-06T03:22:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:00 smithi153 bash[23852]: cephadm 2024-04-06T03:21:58.897215+0000 mgr.y (mgr.24368) 28 : cephadm 1 Deploying daemon alertmanager.a on smithi050 2024-04-06T03:22:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:00 smithi153 bash[23852]: audit 2024-04-06T03:21:58.962798+0000 mgr.y (mgr.24368) 29 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:01.293 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:00 smithi153 bash[50896]: c0e87333d380: Pull complete 2024-04-06T03:22:01.294 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:00 smithi153 bash[50896]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-04-06T03:22:01.294 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:01 smithi153 bash[50896]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-04-06T03:22:01.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[21467]: cluster 2024-04-06T03:21:59.656300+0000 osd.6 (osd.6) 9 : cluster 0 2.17 scrub starts 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[21467]: cluster 2024-04-06T03:21:59.658608+0000 osd.6 (osd.6) 10 : cluster 0 2.17 scrub ok 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[21467]: cluster 2024-04-06T03:21:59.682246+0000 osd.5 (osd.5) 11 : cluster 0 5.f scrub starts 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[21467]: cluster 2024-04-06T03:21:59.684764+0000 osd.5 (osd.5) 12 : cluster 0 5.f scrub ok 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[21467]: cluster 2024-04-06T03:21:59.947166+0000 mgr.y (mgr.24368) 30 : cluster 0 pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[29221]: cluster 2024-04-06T03:21:59.656300+0000 osd.6 (osd.6) 9 : cluster 0 2.17 scrub starts 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[29221]: cluster 2024-04-06T03:21:59.658608+0000 osd.6 (osd.6) 10 : cluster 0 2.17 scrub ok 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[29221]: cluster 2024-04-06T03:21:59.682246+0000 osd.5 (osd.5) 11 : cluster 0 5.f scrub starts 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[29221]: cluster 2024-04-06T03:21:59.684764+0000 osd.5 (osd.5) 12 : cluster 0 5.f scrub ok 2024-04-06T03:22:01.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:01 smithi050 bash[29221]: cluster 2024-04-06T03:21:59.947166+0000 mgr.y (mgr.24368) 30 : cluster 0 pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-06T03:22:01.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:01 smithi153 bash[23852]: cluster 2024-04-06T03:21:59.656300+0000 osd.6 (osd.6) 9 : cluster 0 2.17 scrub starts 2024-04-06T03:22:01.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:01 smithi153 bash[23852]: cluster 2024-04-06T03:21:59.658608+0000 osd.6 (osd.6) 10 : cluster 0 2.17 scrub ok 2024-04-06T03:22:01.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:01 smithi153 bash[23852]: cluster 2024-04-06T03:21:59.682246+0000 osd.5 (osd.5) 11 : cluster 0 5.f scrub starts 2024-04-06T03:22:01.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:01 smithi153 bash[23852]: cluster 2024-04-06T03:21:59.684764+0000 osd.5 (osd.5) 12 : cluster 0 5.f scrub ok 2024-04-06T03:22:01.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:01 smithi153 bash[23852]: cluster 2024-04-06T03:21:59.947166+0000 mgr.y (mgr.24368) 30 : cluster 0 pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-06T03:22:02.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:00.686704+0000 osd.6 (osd.6) 11 : cluster 0 2.7 scrub starts 2024-04-06T03:22:02.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:00.689054+0000 osd.6 (osd.6) 12 : cluster 0 2.7 scrub ok 2024-04-06T03:22:02.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:00.731346+0000 osd.5 (osd.5) 13 : cluster 0 6.7 scrub starts 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:00.733872+0000 osd.5 (osd.5) 14 : cluster 0 6.7 scrub ok 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:01.179346+0000 osd.0 (osd.0) 17 : cluster 0 3.17 scrub starts 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:01.183614+0000 osd.0 (osd.0) 18 : cluster 0 3.17 scrub ok 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:01.289757+0000 osd.7 (osd.7) 11 : cluster 0 2.b scrub starts 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[21467]: cluster 2024-04-06T03:22:01.292175+0000 osd.7 (osd.7) 12 : cluster 0 2.b scrub ok 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:00.686704+0000 osd.6 (osd.6) 11 : cluster 0 2.7 scrub starts 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:00.689054+0000 osd.6 (osd.6) 12 : cluster 0 2.7 scrub ok 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:00.731346+0000 osd.5 (osd.5) 13 : cluster 0 6.7 scrub starts 2024-04-06T03:22:02.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:00.733872+0000 osd.5 (osd.5) 14 : cluster 0 6.7 scrub ok 2024-04-06T03:22:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:01.179346+0000 osd.0 (osd.0) 17 : cluster 0 3.17 scrub starts 2024-04-06T03:22:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:01.183614+0000 osd.0 (osd.0) 18 : cluster 0 3.17 scrub ok 2024-04-06T03:22:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:01.289757+0000 osd.7 (osd.7) 11 : cluster 0 2.b scrub starts 2024-04-06T03:22:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[29221]: cluster 2024-04-06T03:22:01.292175+0000 osd.7 (osd.7) 12 : cluster 0 2.b scrub ok 2024-04-06T03:22:02.733 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.490Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-06T03:22:02.733 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.490Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-06T03:22:02.733 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.491Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-06T03:22:02.733 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.491Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-06T03:22:02.733 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.491Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.491Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=arp 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-06T03:22:02.734 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=edac 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-06T03:22:02.737 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=os 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-06T03:22:02.738 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=stat 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=time 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=uname 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-06T03:22:02.739 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-06T03:22:02.740 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-06T03:22:02.740 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:02 smithi050 bash[57670]: ts=2024-04-06T03:22:02.492Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-06T03:22:02.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:00.686704+0000 osd.6 (osd.6) 11 : cluster 0 2.7 scrub starts 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:00.689054+0000 osd.6 (osd.6) 12 : cluster 0 2.7 scrub ok 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:00.731346+0000 osd.5 (osd.5) 13 : cluster 0 6.7 scrub starts 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:00.733872+0000 osd.5 (osd.5) 14 : cluster 0 6.7 scrub ok 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:01.179346+0000 osd.0 (osd.0) 17 : cluster 0 3.17 scrub starts 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:01.183614+0000 osd.0 (osd.0) 18 : cluster 0 3.17 scrub ok 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:01.289757+0000 osd.7 (osd.7) 11 : cluster 0 2.b scrub starts 2024-04-06T03:22:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[23852]: cluster 2024-04-06T03:22:01.292175+0000 osd.7 (osd.7) 12 : cluster 0 2.b scrub ok 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.533Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.534Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-06T03:22:02.793 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=arp 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=edac 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-06T03:22:02.794 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=os 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-06T03:22:02.795 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.535Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=stat 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=time 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=uname 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-06T03:22:02.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-06T03:22:02.797 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-06T03:22:02.797 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:02 smithi153 bash[50896]: ts=2024-04-06T03:22:02.536Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: cluster 2024-04-06T03:22:01.947682+0000 mgr.y (mgr.24368) 31 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.330705+0000 osd.4 (osd.4) 11 : cluster 0 6.2 deep-scrub starts 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.332817+0000 osd.4 (osd.4) 12 : cluster 0 6.2 deep-scrub ok 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.676472+0000 osd.3 (osd.3) 15 : cluster 0 3.1e scrub starts 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.680030+0000 osd.3 (osd.3) 16 : cluster 0 3.1e scrub ok 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: audit 2024-04-06T03:22:03.041188+0000 mon.a (mon.0) 873 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[29221]: audit 2024-04-06T03:22:03.065261+0000 mon.a (mon.0) 874 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: cluster 2024-04-06T03:22:01.947682+0000 mgr.y (mgr.24368) 31 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.330705+0000 osd.4 (osd.4) 11 : cluster 0 6.2 deep-scrub starts 2024-04-06T03:22:03.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.332817+0000 osd.4 (osd.4) 12 : cluster 0 6.2 deep-scrub ok 2024-04-06T03:22:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.676472+0000 osd.3 (osd.3) 15 : cluster 0 3.1e scrub starts 2024-04-06T03:22:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.680030+0000 osd.3 (osd.3) 16 : cluster 0 3.1e scrub ok 2024-04-06T03:22:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: audit 2024-04-06T03:22:03.041188+0000 mon.a (mon.0) 873 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:03 smithi050 bash[21467]: audit 2024-04-06T03:22:03.065261+0000 mon.a (mon.0) 874 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:03.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: cluster 2024-04-06T03:22:01.947682+0000 mgr.y (mgr.24368) 31 : cluster 0 pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-06T03:22:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.330705+0000 osd.4 (osd.4) 11 : cluster 0 6.2 deep-scrub starts 2024-04-06T03:22:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.332817+0000 osd.4 (osd.4) 12 : cluster 0 6.2 deep-scrub ok 2024-04-06T03:22:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.676472+0000 osd.3 (osd.3) 15 : cluster 0 3.1e scrub starts 2024-04-06T03:22:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.680030+0000 osd.3 (osd.3) 16 : cluster 0 3.1e scrub ok 2024-04-06T03:22:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: audit 2024-04-06T03:22:03.041188+0000 mon.a (mon.0) 873 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:03 smithi153 bash[23852]: audit 2024-04-06T03:22:03.065261+0000 mon.a (mon.0) 874 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:04.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.757269+0000 osd.5 (osd.5) 15 : cluster 0 2.2 scrub starts 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.759518+0000 osd.5 (osd.5) 16 : cluster 0 2.2 scrub ok 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.769903+0000 osd.1 (osd.1) 23 : cluster 0 2.d deep-scrub starts 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[29221]: cluster 2024-04-06T03:22:02.771468+0000 osd.1 (osd.1) 24 : cluster 0 2.d deep-scrub ok 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.757269+0000 osd.5 (osd.5) 15 : cluster 0 2.2 scrub starts 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.759518+0000 osd.5 (osd.5) 16 : cluster 0 2.2 scrub ok 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.769903+0000 osd.1 (osd.1) 23 : cluster 0 2.d deep-scrub starts 2024-04-06T03:22:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:04 smithi050 bash[21467]: cluster 2024-04-06T03:22:02.771468+0000 osd.1 (osd.1) 24 : cluster 0 2.d deep-scrub ok 2024-04-06T03:22:04.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:04 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.757269+0000 osd.5 (osd.5) 15 : cluster 0 2.2 scrub starts 2024-04-06T03:22:04.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:04 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.759518+0000 osd.5 (osd.5) 16 : cluster 0 2.2 scrub ok 2024-04-06T03:22:04.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:04 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.769903+0000 osd.1 (osd.1) 23 : cluster 0 2.d deep-scrub starts 2024-04-06T03:22:04.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:04 smithi153 bash[23852]: cluster 2024-04-06T03:22:02.771468+0000 osd.1 (osd.1) 24 : cluster 0 2.d deep-scrub ok 2024-04-06T03:22:05.104 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:03.494945+0000 osd.2 (osd.2) 17 : cluster 0 5.b scrub starts 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:03.496874+0000 osd.2 (osd.2) 18 : cluster 0 5.b scrub ok 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:03.747413+0000 osd.5 (osd.5) 17 : cluster 0 5.7 scrub starts 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:03.750171+0000 osd.5 (osd.5) 18 : cluster 0 5.7 scrub ok 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:03.948423+0000 mgr.y (mgr.24368) 32 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:04.727648+0000 osd.3 (osd.3) 17 : cluster 0 3.b scrub starts 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[21467]: cluster 2024-04-06T03:22:04.731945+0000 osd.3 (osd.3) 18 : cluster 0 3.b scrub ok 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:03.494945+0000 osd.2 (osd.2) 17 : cluster 0 5.b scrub starts 2024-04-06T03:22:05.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:03.496874+0000 osd.2 (osd.2) 18 : cluster 0 5.b scrub ok 2024-04-06T03:22:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:03.747413+0000 osd.5 (osd.5) 17 : cluster 0 5.7 scrub starts 2024-04-06T03:22:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:03.750171+0000 osd.5 (osd.5) 18 : cluster 0 5.7 scrub ok 2024-04-06T03:22:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:03.948423+0000 mgr.y (mgr.24368) 32 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:04.727648+0000 osd.3 (osd.3) 17 : cluster 0 3.b scrub starts 2024-04-06T03:22:05.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:05 smithi050 bash[29221]: cluster 2024-04-06T03:22:04.731945+0000 osd.3 (osd.3) 18 : cluster 0 3.b scrub ok 2024-04-06T03:22:05.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:03.494945+0000 osd.2 (osd.2) 17 : cluster 0 5.b scrub starts 2024-04-06T03:22:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:03.496874+0000 osd.2 (osd.2) 18 : cluster 0 5.b scrub ok 2024-04-06T03:22:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:03.747413+0000 osd.5 (osd.5) 17 : cluster 0 5.7 scrub starts 2024-04-06T03:22:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:03.750171+0000 osd.5 (osd.5) 18 : cluster 0 5.7 scrub ok 2024-04-06T03:22:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:03.948423+0000 mgr.y (mgr.24368) 32 : cluster 0 pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:04.727648+0000 osd.3 (osd.3) 17 : cluster 0 3.b scrub starts 2024-04-06T03:22:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:05 smithi153 bash[23852]: cluster 2024-04-06T03:22:04.731945+0000 osd.3 (osd.3) 18 : cluster 0 3.b scrub ok 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[21467]: cluster 2024-04-06T03:22:04.612119+0000 osd.6 (osd.6) 13 : cluster 0 3.10 scrub starts 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[21467]: cluster 2024-04-06T03:22:04.615459+0000 osd.6 (osd.6) 14 : cluster 0 3.10 scrub ok 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[21467]: cluster 2024-04-06T03:22:05.461226+0000 osd.2 (osd.2) 19 : cluster 0 5.d scrub starts 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[21467]: cluster 2024-04-06T03:22:05.464761+0000 osd.2 (osd.2) 20 : cluster 0 5.d scrub ok 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[29221]: cluster 2024-04-06T03:22:04.612119+0000 osd.6 (osd.6) 13 : cluster 0 3.10 scrub starts 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[29221]: cluster 2024-04-06T03:22:04.615459+0000 osd.6 (osd.6) 14 : cluster 0 3.10 scrub ok 2024-04-06T03:22:06.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[29221]: cluster 2024-04-06T03:22:05.461226+0000 osd.2 (osd.2) 19 : cluster 0 5.d scrub starts 2024-04-06T03:22:06.643 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:06 smithi050 bash[29221]: cluster 2024-04-06T03:22:05.464761+0000 osd.2 (osd.2) 20 : cluster 0 5.d scrub ok 2024-04-06T03:22:06.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:06 smithi153 bash[23852]: cluster 2024-04-06T03:22:04.612119+0000 osd.6 (osd.6) 13 : cluster 0 3.10 scrub starts 2024-04-06T03:22:06.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:06 smithi153 bash[23852]: cluster 2024-04-06T03:22:04.615459+0000 osd.6 (osd.6) 14 : cluster 0 3.10 scrub ok 2024-04-06T03:22:06.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:06 smithi153 bash[23852]: cluster 2024-04-06T03:22:05.461226+0000 osd.2 (osd.2) 19 : cluster 0 5.d scrub starts 2024-04-06T03:22:06.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:06 smithi153 bash[23852]: cluster 2024-04-06T03:22:05.464761+0000 osd.2 (osd.2) 20 : cluster 0 5.d scrub ok 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:05.607198+0000 osd.6 (osd.6) 15 : cluster 0 5.1f deep-scrub starts 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:05.612823+0000 osd.6 (osd.6) 16 : cluster 0 5.1f deep-scrub ok 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:05.949680+0000 mgr.y (mgr.24368) 33 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.145148+0000 osd.0 (osd.0) 19 : cluster 0 3.1 scrub starts 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.148741+0000 osd.0 (osd.0) 20 : cluster 0 3.1 scrub ok 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.359184+0000 osd.4 (osd.4) 13 : cluster 0 3.1a scrub starts 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.363802+0000 osd.4 (osd.4) 14 : cluster 0 3.1a scrub ok 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.376268+0000 osd.7 (osd.7) 13 : cluster 0 3.e scrub starts 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.380547+0000 osd.7 (osd.7) 14 : cluster 0 3.e scrub ok 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.730686+0000 osd.3 (osd.3) 19 : cluster 0 2.19 scrub starts 2024-04-06T03:22:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.732597+0000 osd.3 (osd.3) 20 : cluster 0 2.19 scrub ok 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:05.607198+0000 osd.6 (osd.6) 15 : cluster 0 5.1f deep-scrub starts 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:05.612823+0000 osd.6 (osd.6) 16 : cluster 0 5.1f deep-scrub ok 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:05.949680+0000 mgr.y (mgr.24368) 33 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.145148+0000 osd.0 (osd.0) 19 : cluster 0 3.1 scrub starts 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.148741+0000 osd.0 (osd.0) 20 : cluster 0 3.1 scrub ok 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.359184+0000 osd.4 (osd.4) 13 : cluster 0 3.1a scrub starts 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.363802+0000 osd.4 (osd.4) 14 : cluster 0 3.1a scrub ok 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.376268+0000 osd.7 (osd.7) 13 : cluster 0 3.e scrub starts 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.380547+0000 osd.7 (osd.7) 14 : cluster 0 3.e scrub ok 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.730686+0000 osd.3 (osd.3) 19 : cluster 0 2.19 scrub starts 2024-04-06T03:22:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:07 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.732597+0000 osd.3 (osd.3) 20 : cluster 0 2.19 scrub ok 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:05.607198+0000 osd.6 (osd.6) 15 : cluster 0 5.1f deep-scrub starts 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:05.612823+0000 osd.6 (osd.6) 16 : cluster 0 5.1f deep-scrub ok 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:05.949680+0000 mgr.y (mgr.24368) 33 : cluster 0 pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.145148+0000 osd.0 (osd.0) 19 : cluster 0 3.1 scrub starts 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.148741+0000 osd.0 (osd.0) 20 : cluster 0 3.1 scrub ok 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.359184+0000 osd.4 (osd.4) 13 : cluster 0 3.1a scrub starts 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.363802+0000 osd.4 (osd.4) 14 : cluster 0 3.1a scrub ok 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.376268+0000 osd.7 (osd.7) 13 : cluster 0 3.e scrub starts 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.380547+0000 osd.7 (osd.7) 14 : cluster 0 3.e scrub ok 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.730686+0000 osd.3 (osd.3) 19 : cluster 0 2.19 scrub starts 2024-04-06T03:22:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:07 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.732597+0000 osd.3 (osd.3) 20 : cluster 0 2.19 scrub ok 2024-04-06T03:22:08.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:08 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.638296+0000 osd.6 (osd.6) 17 : cluster 0 2.1a scrub starts 2024-04-06T03:22:08.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:08 smithi050 bash[29221]: cluster 2024-04-06T03:22:06.640931+0000 osd.6 (osd.6) 18 : cluster 0 2.1a scrub ok 2024-04-06T03:22:08.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:08 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.638296+0000 osd.6 (osd.6) 17 : cluster 0 2.1a scrub starts 2024-04-06T03:22:08.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:08 smithi050 bash[21467]: cluster 2024-04-06T03:22:06.640931+0000 osd.6 (osd.6) 18 : cluster 0 2.1a scrub ok 2024-04-06T03:22:08.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:08 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.638296+0000 osd.6 (osd.6) 17 : cluster 0 2.1a scrub starts 2024-04-06T03:22:08.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:08 smithi153 bash[23852]: cluster 2024-04-06T03:22:06.640931+0000 osd.6 (osd.6) 18 : cluster 0 2.1a scrub ok 2024-04-06T03:22:09.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:08 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:22:09.731 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:22:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:07.770448+0000 osd.5 (osd.5) 19 : cluster 0 2.3 scrub starts 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:07.773332+0000 osd.5 (osd.5) 20 : cluster 0 2.3 scrub ok 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:07.950411+0000 mgr.y (mgr.24368) 34 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:08.435345+0000 osd.7 (osd.7) 15 : cluster 0 3.f scrub starts 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:08.439863+0000 osd.7 (osd.7) 16 : cluster 0 3.f scrub ok 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:08.454927+0000 osd.2 (osd.2) 21 : cluster 0 2.10 scrub starts 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[21467]: cluster 2024-04-06T03:22:08.456220+0000 osd.2 (osd.2) 22 : cluster 0 2.10 scrub ok 2024-04-06T03:22:09.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:07.770448+0000 osd.5 (osd.5) 19 : cluster 0 2.3 scrub starts 2024-04-06T03:22:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:07.773332+0000 osd.5 (osd.5) 20 : cluster 0 2.3 scrub ok 2024-04-06T03:22:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:07.950411+0000 mgr.y (mgr.24368) 34 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:08.435345+0000 osd.7 (osd.7) 15 : cluster 0 3.f scrub starts 2024-04-06T03:22:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:08.439863+0000 osd.7 (osd.7) 16 : cluster 0 3.f scrub ok 2024-04-06T03:22:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:08.454927+0000 osd.2 (osd.2) 21 : cluster 0 2.10 scrub starts 2024-04-06T03:22:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:09 smithi050 bash[29221]: cluster 2024-04-06T03:22:08.456220+0000 osd.2 (osd.2) 22 : cluster 0 2.10 scrub ok 2024-04-06T03:22:09.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:07.770448+0000 osd.5 (osd.5) 19 : cluster 0 2.3 scrub starts 2024-04-06T03:22:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:07.773332+0000 osd.5 (osd.5) 20 : cluster 0 2.3 scrub ok 2024-04-06T03:22:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:07.950411+0000 mgr.y (mgr.24368) 34 : cluster 0 pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:08.435345+0000 osd.7 (osd.7) 15 : cluster 0 3.f scrub starts 2024-04-06T03:22:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:08.439863+0000 osd.7 (osd.7) 16 : cluster 0 3.f scrub ok 2024-04-06T03:22:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:08.454927+0000 osd.2 (osd.2) 21 : cluster 0 2.10 scrub starts 2024-04-06T03:22:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:09 smithi153 bash[23852]: cluster 2024-04-06T03:22:08.456220+0000 osd.2 (osd.2) 22 : cluster 0 2.10 scrub ok 2024-04-06T03:22:10.361 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-04-06T03:22:10.361 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDivxBm1xr+FRAAWZWT3BQzeN8rAbeWDaBIlw== 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: cluster 2024-04-06T03:22:08.587630+0000 osd.6 (osd.6) 19 : cluster 0 2.14 deep-scrub starts 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: cluster 2024-04-06T03:22:08.589827+0000 osd.6 (osd.6) 20 : cluster 0 2.14 deep-scrub ok 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: audit 2024-04-06T03:22:08.971019+0000 mgr.y (mgr.24368) 35 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: cluster 2024-04-06T03:22:09.390483+0000 osd.4 (osd.4) 15 : cluster 0 6.17 scrub starts 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: cluster 2024-04-06T03:22:09.392817+0000 osd.4 (osd.4) 16 : cluster 0 6.17 scrub ok 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: audit 2024-04-06T03:22:10.368765+0000 mon.a (mon.0) 875 : audit 1 from='client.? 172.21.15.50:0/3794246482' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[29221]: audit 2024-04-06T03:22:10.371146+0000 mon.a (mon.0) 876 : audit 1 from='client.? 172.21.15.50:0/3794246482' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: cluster 2024-04-06T03:22:08.587630+0000 osd.6 (osd.6) 19 : cluster 0 2.14 deep-scrub starts 2024-04-06T03:22:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: cluster 2024-04-06T03:22:08.589827+0000 osd.6 (osd.6) 20 : cluster 0 2.14 deep-scrub ok 2024-04-06T03:22:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: audit 2024-04-06T03:22:08.971019+0000 mgr.y (mgr.24368) 35 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: cluster 2024-04-06T03:22:09.390483+0000 osd.4 (osd.4) 15 : cluster 0 6.17 scrub starts 2024-04-06T03:22:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: cluster 2024-04-06T03:22:09.392817+0000 osd.4 (osd.4) 16 : cluster 0 6.17 scrub ok 2024-04-06T03:22:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: audit 2024-04-06T03:22:10.368765+0000 mon.a (mon.0) 875 : audit 1 from='client.? 172.21.15.50:0/3794246482' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:10 smithi050 bash[21467]: audit 2024-04-06T03:22:10.371146+0000 mon.a (mon.0) 876 : audit 1 from='client.? 172.21.15.50:0/3794246482' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:22:10.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: cluster 2024-04-06T03:22:08.587630+0000 osd.6 (osd.6) 19 : cluster 0 2.14 deep-scrub starts 2024-04-06T03:22:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: cluster 2024-04-06T03:22:08.589827+0000 osd.6 (osd.6) 20 : cluster 0 2.14 deep-scrub ok 2024-04-06T03:22:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: audit 2024-04-06T03:22:08.971019+0000 mgr.y (mgr.24368) 35 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: cluster 2024-04-06T03:22:09.390483+0000 osd.4 (osd.4) 15 : cluster 0 6.17 scrub starts 2024-04-06T03:22:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: cluster 2024-04-06T03:22:09.392817+0000 osd.4 (osd.4) 16 : cluster 0 6.17 scrub ok 2024-04-06T03:22:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: audit 2024-04-06T03:22:10.368765+0000 mon.a (mon.0) 875 : audit 1 from='client.? 172.21.15.50:0/3794246482' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:10 smithi153 bash[23852]: audit 2024-04-06T03:22:10.371146+0000 mon.a (mon.0) 876 : audit 1 from='client.? 172.21.15.50:0/3794246482' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:22:11.556 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-06T03:22:11.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-06T03:22:11.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-06T03:22:11.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-06T03:22:11.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:09.951199+0000 mgr.y (mgr.24368) 36 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:22:11.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.217049+0000 osd.0 (osd.0) 21 : cluster 0 6.9 scrub starts 2024-04-06T03:22:11.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.218935+0000 osd.0 (osd.0) 22 : cluster 0 6.9 scrub ok 2024-04-06T03:22:11.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.394852+0000 osd.4 (osd.4) 17 : cluster 0 5.e scrub starts 2024-04-06T03:22:11.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.397941+0000 osd.4 (osd.4) 18 : cluster 0 5.e scrub ok 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.420973+0000 osd.2 (osd.2) 23 : cluster 0 5.a scrub starts 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.422472+0000 osd.2 (osd.2) 24 : cluster 0 5.a scrub ok 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.495688+0000 osd.7 (osd.7) 17 : cluster 0 2.5 scrub starts 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.497758+0000 osd.7 (osd.7) 18 : cluster 0 2.5 scrub ok 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.763593+0000 osd.1 (osd.1) 25 : cluster 0 3.0 scrub starts 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[21467]: cluster 2024-04-06T03:22:10.769345+0000 osd.1 (osd.1) 26 : cluster 0 3.0 scrub ok 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:09.951199+0000 mgr.y (mgr.24368) 36 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.217049+0000 osd.0 (osd.0) 21 : cluster 0 6.9 scrub starts 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.218935+0000 osd.0 (osd.0) 22 : cluster 0 6.9 scrub ok 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.394852+0000 osd.4 (osd.4) 17 : cluster 0 5.e scrub starts 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.397941+0000 osd.4 (osd.4) 18 : cluster 0 5.e scrub ok 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.420973+0000 osd.2 (osd.2) 23 : cluster 0 5.a scrub starts 2024-04-06T03:22:11.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.422472+0000 osd.2 (osd.2) 24 : cluster 0 5.a scrub ok 2024-04-06T03:22:11.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.495688+0000 osd.7 (osd.7) 17 : cluster 0 2.5 scrub starts 2024-04-06T03:22:11.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.497758+0000 osd.7 (osd.7) 18 : cluster 0 2.5 scrub ok 2024-04-06T03:22:11.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.763593+0000 osd.1 (osd.1) 25 : cluster 0 3.0 scrub starts 2024-04-06T03:22:11.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 bash[29221]: cluster 2024-04-06T03:22:10.769345+0000 osd.1 (osd.1) 26 : cluster 0 3.0 scrub ok 2024-04-06T03:22:11.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:09.951199+0000 mgr.y (mgr.24368) 36 : cluster 0 pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:22:11.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.217049+0000 osd.0 (osd.0) 21 : cluster 0 6.9 scrub starts 2024-04-06T03:22:11.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.218935+0000 osd.0 (osd.0) 22 : cluster 0 6.9 scrub ok 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.394852+0000 osd.4 (osd.4) 17 : cluster 0 5.e scrub starts 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.397941+0000 osd.4 (osd.4) 18 : cluster 0 5.e scrub ok 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.420973+0000 osd.2 (osd.2) 23 : cluster 0 5.a scrub starts 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.422472+0000 osd.2 (osd.2) 24 : cluster 0 5.a scrub ok 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.495688+0000 osd.7 (osd.7) 17 : cluster 0 2.5 scrub starts 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.497758+0000 osd.7 (osd.7) 18 : cluster 0 2.5 scrub ok 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.763593+0000 osd.1 (osd.1) 25 : cluster 0 3.0 scrub starts 2024-04-06T03:22:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:11 smithi153 bash[23852]: cluster 2024-04-06T03:22:10.769345+0000 osd.1 (osd.1) 26 : cluster 0 3.0 scrub ok 2024-04-06T03:22:11.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.924 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.925 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.925 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.925 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.926 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.926 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.926 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.926 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.926 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.927 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:11.927 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.232 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.232 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.233 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.233 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.233 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.234 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.234 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:11 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:12.234 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:12 smithi050 systemd[1]: Started Ceph alertmanager.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:13.333 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.340Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-06T03:22:13.333 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.340Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.557507+0000 osd.5 (osd.5) 21 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.557643+0000 osd.5 (osd.5) 22 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.581036+0000 osd.6 (osd.6) 21 : cluster 0 5.2 scrub starts 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.582993+0000 osd.6 (osd.6) 22 : cluster 0 5.2 scrub ok 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.812578+0000 osd.5 (osd.5) 23 : cluster 0 2.16 scrub starts 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.813694+0000 osd.5 (osd.5) 24 : cluster 0 2.16 scrub ok 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:11.951847+0000 mgr.y (mgr.24368) 37 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: audit 2024-04-06T03:22:12.090249+0000 mon.a (mon.0) 877 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: audit 2024-04-06T03:22:12.096489+0000 mon.a (mon.0) 878 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: audit 2024-04-06T03:22:12.147298+0000 mon.a (mon.0) 879 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: audit 2024-04-06T03:22:12.153941+0000 mon.a (mon.0) 880 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: audit 2024-04-06T03:22:12.202352+0000 mon.a (mon.0) 881 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: cluster 2024-04-06T03:22:12.410000+0000 mon.a (mon.0) 882 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[29221]: audit 2024-04-06T03:22:13.050170+0000 mon.a (mon.0) 883 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.557507+0000 osd.5 (osd.5) 21 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.557643+0000 osd.5 (osd.5) 22 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.581036+0000 osd.6 (osd.6) 21 : cluster 0 5.2 scrub starts 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.582993+0000 osd.6 (osd.6) 22 : cluster 0 5.2 scrub ok 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.812578+0000 osd.5 (osd.5) 23 : cluster 0 2.16 scrub starts 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.813694+0000 osd.5 (osd.5) 24 : cluster 0 2.16 scrub ok 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:11.951847+0000 mgr.y (mgr.24368) 37 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:13.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: audit 2024-04-06T03:22:12.090249+0000 mon.a (mon.0) 877 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: audit 2024-04-06T03:22:12.096489+0000 mon.a (mon.0) 878 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: audit 2024-04-06T03:22:12.147298+0000 mon.a (mon.0) 879 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: audit 2024-04-06T03:22:12.153941+0000 mon.a (mon.0) 880 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: audit 2024-04-06T03:22:12.202352+0000 mon.a (mon.0) 881 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:13.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: cluster 2024-04-06T03:22:12.410000+0000 mon.a (mon.0) 882 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-04-06T03:22:13.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[21467]: audit 2024-04-06T03:22:13.050170+0000 mon.a (mon.0) 883 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.557507+0000 osd.5 (osd.5) 21 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.557643+0000 osd.5 (osd.5) 22 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.581036+0000 osd.6 (osd.6) 21 : cluster 0 5.2 scrub starts 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.582993+0000 osd.6 (osd.6) 22 : cluster 0 5.2 scrub ok 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.812578+0000 osd.5 (osd.5) 23 : cluster 0 2.16 scrub starts 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.813694+0000 osd.5 (osd.5) 24 : cluster 0 2.16 scrub ok 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:11.951847+0000 mgr.y (mgr.24368) 37 : cluster 0 pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: audit 2024-04-06T03:22:12.090249+0000 mon.a (mon.0) 877 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: audit 2024-04-06T03:22:12.096489+0000 mon.a (mon.0) 878 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: audit 2024-04-06T03:22:12.147298+0000 mon.a (mon.0) 879 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: audit 2024-04-06T03:22:12.153941+0000 mon.a (mon.0) 880 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: audit 2024-04-06T03:22:12.202352+0000 mon.a (mon.0) 881 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:13.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: cluster 2024-04-06T03:22:12.410000+0000 mon.a (mon.0) 882 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2024-04-06T03:22:13.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:13 smithi153 bash[23852]: audit 2024-04-06T03:22:13.050170+0000 mon.a (mon.0) 883 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:13.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.343Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.50 port=9094 2024-04-06T03:22:13.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.344Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-06T03:22:13.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.380Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T03:22:13.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.381Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T03:22:13.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.385Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-06T03:22:13.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:13 smithi050 bash[58229]: ts=2024-04-06T03:22:13.385Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-06T03:22:14.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:14 smithi050 bash[21467]: cluster 2024-04-06T03:22:13.338222+0000 mon.a (mon.0) 884 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-04-06T03:22:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:14 smithi050 bash[21467]: cluster 2024-04-06T03:22:13.619677+0000 osd.3 (osd.3) 21 : cluster 0 6.c deep-scrub starts 2024-04-06T03:22:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:14 smithi050 bash[21467]: cluster 2024-04-06T03:22:13.622241+0000 osd.3 (osd.3) 22 : cluster 0 6.c deep-scrub ok 2024-04-06T03:22:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:14 smithi050 bash[29221]: cluster 2024-04-06T03:22:13.338222+0000 mon.a (mon.0) 884 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-04-06T03:22:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:14 smithi050 bash[29221]: cluster 2024-04-06T03:22:13.619677+0000 osd.3 (osd.3) 21 : cluster 0 6.c deep-scrub starts 2024-04-06T03:22:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:14 smithi050 bash[29221]: cluster 2024-04-06T03:22:13.622241+0000 osd.3 (osd.3) 22 : cluster 0 6.c deep-scrub ok 2024-04-06T03:22:14.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:14 smithi153 bash[23852]: cluster 2024-04-06T03:22:13.338222+0000 mon.a (mon.0) 884 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2024-04-06T03:22:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:14 smithi153 bash[23852]: cluster 2024-04-06T03:22:13.619677+0000 osd.3 (osd.3) 21 : cluster 0 6.c deep-scrub starts 2024-04-06T03:22:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:14 smithi153 bash[23852]: cluster 2024-04-06T03:22:13.622241+0000 osd.3 (osd.3) 22 : cluster 0 6.c deep-scrub ok 2024-04-06T03:22:15.731 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[58229]: ts=2024-04-06T03:22:15.345Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000373434s 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[21467]: cluster 2024-04-06T03:22:13.846783+0000 osd.5 (osd.5) 25 : cluster 0 3.1c scrub starts 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[21467]: cluster 2024-04-06T03:22:13.849784+0000 osd.5 (osd.5) 26 : cluster 0 3.1c scrub ok 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[21467]: cluster 2024-04-06T03:22:13.952972+0000 mgr.y (mgr.24368) 38 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[21467]: cluster 2024-04-06T03:22:14.501449+0000 osd.7 (osd.7) 19 : cluster 0 6.15 scrub starts 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[21467]: cluster 2024-04-06T03:22:14.504088+0000 osd.7 (osd.7) 20 : cluster 0 6.15 scrub ok 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[29221]: cluster 2024-04-06T03:22:13.846783+0000 osd.5 (osd.5) 25 : cluster 0 3.1c scrub starts 2024-04-06T03:22:15.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[29221]: cluster 2024-04-06T03:22:13.849784+0000 osd.5 (osd.5) 26 : cluster 0 3.1c scrub ok 2024-04-06T03:22:15.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[29221]: cluster 2024-04-06T03:22:13.952972+0000 mgr.y (mgr.24368) 38 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:15.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[29221]: cluster 2024-04-06T03:22:14.501449+0000 osd.7 (osd.7) 19 : cluster 0 6.15 scrub starts 2024-04-06T03:22:15.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:15 smithi050 bash[29221]: cluster 2024-04-06T03:22:14.504088+0000 osd.7 (osd.7) 20 : cluster 0 6.15 scrub ok 2024-04-06T03:22:15.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:15 smithi153 bash[23852]: cluster 2024-04-06T03:22:13.846783+0000 osd.5 (osd.5) 25 : cluster 0 3.1c scrub starts 2024-04-06T03:22:15.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:15 smithi153 bash[23852]: cluster 2024-04-06T03:22:13.849784+0000 osd.5 (osd.5) 26 : cluster 0 3.1c scrub ok 2024-04-06T03:22:15.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:15 smithi153 bash[23852]: cluster 2024-04-06T03:22:13.952972+0000 mgr.y (mgr.24368) 38 : cluster 0 pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:15.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:15 smithi153 bash[23852]: cluster 2024-04-06T03:22:14.501449+0000 osd.7 (osd.7) 19 : cluster 0 6.15 scrub starts 2024-04-06T03:22:15.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:15 smithi153 bash[23852]: cluster 2024-04-06T03:22:14.504088+0000 osd.7 (osd.7) 20 : cluster 0 6.15 scrub ok 2024-04-06T03:22:16.350 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:22:16.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:14.859046+0000 osd.5 (osd.5) 27 : cluster 0 5.16 scrub starts 2024-04-06T03:22:16.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:14.861760+0000 osd.5 (osd.5) 28 : cluster 0 5.16 scrub ok 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.241141+0000 osd.1 (osd.1) 27 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.241303+0000 osd.1 (osd.1) 28 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.361988+0000 mon.a (mon.0) 885 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.458998+0000 osd.7 (osd.7) 21 : cluster 0 3.13 scrub starts 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.465549+0000 osd.7 (osd.7) 22 : cluster 0 3.13 scrub ok 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.604822+0000 osd.3 (osd.3) 23 : cluster 0 3.2 scrub starts 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.609381+0000 osd.3 (osd.3) 24 : cluster 0 3.2 scrub ok 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:14.859046+0000 osd.5 (osd.5) 27 : cluster 0 5.16 scrub starts 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:14.861760+0000 osd.5 (osd.5) 28 : cluster 0 5.16 scrub ok 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.241141+0000 osd.1 (osd.1) 27 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:16.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.241303+0000 osd.1 (osd.1) 28 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.361988+0000 mon.a (mon.0) 885 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-04-06T03:22:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.458998+0000 osd.7 (osd.7) 21 : cluster 0 3.13 scrub starts 2024-04-06T03:22:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.465549+0000 osd.7 (osd.7) 22 : cluster 0 3.13 scrub ok 2024-04-06T03:22:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.604822+0000 osd.3 (osd.3) 23 : cluster 0 3.2 scrub starts 2024-04-06T03:22:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:16 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.609381+0000 osd.3 (osd.3) 24 : cluster 0 3.2 scrub ok 2024-04-06T03:22:16.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:14.859046+0000 osd.5 (osd.5) 27 : cluster 0 5.16 scrub starts 2024-04-06T03:22:16.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:14.861760+0000 osd.5 (osd.5) 28 : cluster 0 5.16 scrub ok 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.241141+0000 osd.1 (osd.1) 27 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.241303+0000 osd.1 (osd.1) 28 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.361988+0000 mon.a (mon.0) 885 : cluster 0 osdmap e104: 8 total, 8 up, 8 in 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.458998+0000 osd.7 (osd.7) 21 : cluster 0 3.13 scrub starts 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.465549+0000 osd.7 (osd.7) 22 : cluster 0 3.13 scrub ok 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.604822+0000 osd.3 (osd.3) 23 : cluster 0 3.2 scrub starts 2024-04-06T03:22:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:16 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.609381+0000 osd.3 (osd.3) 24 : cluster 0 3.2 scrub ok 2024-04-06T03:22:17.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:17 smithi153 bash[23852]: cluster 2024-04-06T03:22:15.953681+0000 mgr.y (mgr.24368) 39 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:17.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:17 smithi153 bash[23852]: cluster 2024-04-06T03:22:16.419039+0000 osd.2 (osd.2) 25 : cluster 0 5.8 scrub starts 2024-04-06T03:22:17.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:17 smithi153 bash[23852]: cluster 2024-04-06T03:22:16.421433+0000 osd.2 (osd.2) 26 : cluster 0 5.8 scrub ok 2024-04-06T03:22:17.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:17 smithi153 bash[23852]: audit 2024-04-06T03:22:17.149386+0000 mon.a (mon.0) 886 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:17.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:17 smithi153 bash[23852]: audit 2024-04-06T03:22:17.161312+0000 mon.a (mon.0) 887 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:17.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21467]: cluster 2024-04-06T03:22:15.953681+0000 mgr.y (mgr.24368) 39 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:17.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21467]: cluster 2024-04-06T03:22:16.419039+0000 osd.2 (osd.2) 25 : cluster 0 5.8 scrub starts 2024-04-06T03:22:17.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21467]: cluster 2024-04-06T03:22:16.421433+0000 osd.2 (osd.2) 26 : cluster 0 5.8 scrub ok 2024-04-06T03:22:17.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21467]: audit 2024-04-06T03:22:17.149386+0000 mon.a (mon.0) 886 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21467]: audit 2024-04-06T03:22:17.161312+0000 mon.a (mon.0) 887 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:17.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[29221]: cluster 2024-04-06T03:22:15.953681+0000 mgr.y (mgr.24368) 39 : cluster 0 pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:17.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[29221]: cluster 2024-04-06T03:22:16.419039+0000 osd.2 (osd.2) 25 : cluster 0 5.8 scrub starts 2024-04-06T03:22:17.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[29221]: cluster 2024-04-06T03:22:16.421433+0000 osd.2 (osd.2) 26 : cluster 0 5.8 scrub ok 2024-04-06T03:22:17.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[29221]: audit 2024-04-06T03:22:17.149386+0000 mon.a (mon.0) 886 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:17.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[29221]: audit 2024-04-06T03:22:17.161312+0000 mon.a (mon.0) 887 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.231 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21725]: [06/Apr/2024:03:22:17] ENGINE Bus STOPPING 2024-04-06T03:22:18.232 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21725]: [06/Apr/2024:03:22:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:22:18.232 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21725]: [06/Apr/2024:03:22:17] ENGINE Bus STOPPED 2024-04-06T03:22:18.232 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:17 smithi050 bash[21725]: [06/Apr/2024:03:22:17] ENGINE Bus STARTING 2024-04-06T03:22:18.232 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21725]: [06/Apr/2024:03:22:18] ENGINE Serving on http://:::9283 2024-04-06T03:22:18.232 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21725]: [06/Apr/2024:03:22:18] ENGINE Bus STARTED 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:16.721972+0000 osd.1 (osd.1) 29 : cluster 0 2.6 scrub starts 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:16.724021+0000 osd.1 (osd.1) 30 : cluster 0 2.6 scrub ok 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:16.834071+0000 osd.5 (osd.5) 29 : cluster 0 5.15 scrub starts 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:16.837222+0000 osd.5 (osd.5) 30 : cluster 0 5.15 scrub ok 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:17.420488+0000 osd.7 (osd.7) 23 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:17.421729+0000 osd.7 (osd.7) 24 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:22:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.792708+0000 mon.a (mon.0) 888 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.797710+0000 mon.a (mon.0) 889 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.798752+0000 mon.a (mon.0) 890 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.799469+0000 mon.a (mon.0) 891 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.804627+0000 mon.a (mon.0) 892 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.889929+0000 mon.a (mon.0) 893 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.895375+0000 mon.a (mon.0) 894 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.899224+0000 mon.a (mon.0) 895 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:17.904374+0000 mon.a (mon.0) 896 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: audit 2024-04-06T03:22:18.086739+0000 mon.a (mon.0) 897 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:18.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.343597+0000 mon.a (mon.0) 898 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-04-06T03:22:18.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:16.721972+0000 osd.1 (osd.1) 29 : cluster 0 2.6 scrub starts 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:16.724021+0000 osd.1 (osd.1) 30 : cluster 0 2.6 scrub ok 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:16.834071+0000 osd.5 (osd.5) 29 : cluster 0 5.15 scrub starts 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:16.837222+0000 osd.5 (osd.5) 30 : cluster 0 5.15 scrub ok 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:17.420488+0000 osd.7 (osd.7) 23 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:17.421729+0000 osd.7 (osd.7) 24 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.792708+0000 mon.a (mon.0) 888 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.797710+0000 mon.a (mon.0) 889 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.798752+0000 mon.a (mon.0) 890 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.799469+0000 mon.a (mon.0) 891 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.804627+0000 mon.a (mon.0) 892 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.889929+0000 mon.a (mon.0) 893 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.895375+0000 mon.a (mon.0) 894 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.899224+0000 mon.a (mon.0) 895 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:22:18.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:17.904374+0000 mon.a (mon.0) 896 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: audit 2024-04-06T03:22:18.086739+0000 mon.a (mon.0) 897 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.343597+0000 mon.a (mon.0) 898 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:16.721972+0000 osd.1 (osd.1) 29 : cluster 0 2.6 scrub starts 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:16.724021+0000 osd.1 (osd.1) 30 : cluster 0 2.6 scrub ok 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:16.834071+0000 osd.5 (osd.5) 29 : cluster 0 5.15 scrub starts 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:16.837222+0000 osd.5 (osd.5) 30 : cluster 0 5.15 scrub ok 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:17.420488+0000 osd.7 (osd.7) 23 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:17.421729+0000 osd.7 (osd.7) 24 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.792708+0000 mon.a (mon.0) 888 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.797710+0000 mon.a (mon.0) 889 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.798752+0000 mon.a (mon.0) 890 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.799469+0000 mon.a (mon.0) 891 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.804627+0000 mon.a (mon.0) 892 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.889929+0000 mon.a (mon.0) 893 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.895375+0000 mon.a (mon.0) 894 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.899224+0000 mon.a (mon.0) 895 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:17.904374+0000 mon.a (mon.0) 896 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: audit 2024-04-06T03:22:18.086739+0000 mon.a (mon.0) 897 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:18.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:18 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.343597+0000 mon.a (mon.0) 898 : cluster 0 osdmap e105: 8 total, 8 up, 8 in 2024-04-06T03:22:19.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:18 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:22:19.332 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-04-06T03:22:19.332 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQDrvxBm7QALFBAAzsThX2ALzQlZvPOqr+K14w== 2024-04-06T03:22:19.731 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:22:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:22:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:17.476178+0000 osd.6 (osd.6) 23 : cluster 0 3.a scrub starts 2024-04-06T03:22:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:17.478670+0000 osd.6 (osd.6) 24 : cluster 0 3.a scrub ok 2024-04-06T03:22:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cephadm 2024-04-06T03:22:17.818955+0000 mgr.y (mgr.24368) 40 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T03:22:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: audit 2024-04-06T03:22:17.899709+0000 mgr.y (mgr.24368) 41 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cephadm 2024-04-06T03:22:17.926959+0000 mgr.y (mgr.24368) 42 : cephadm 1 Deploying daemon grafana.a on smithi153 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:17.954039+0000 mgr.y (mgr.24368) 43 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.270555+0000 osd.0 (osd.0) 23 : cluster 0 2.13 scrub starts 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.272528+0000 osd.0 (osd.0) 24 : cluster 0 2.13 scrub ok 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.377419+0000 osd.4 (osd.4) 19 : cluster 0 3.14 scrub starts 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.381002+0000 osd.4 (osd.4) 20 : cluster 0 3.14 scrub ok 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: audit 2024-04-06T03:22:19.331725+0000 mon.b (mon.1) 33 : audit 1 from='client.? 172.21.15.153:0/1234218138' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: audit 2024-04-06T03:22:19.335865+0000 mon.a (mon.0) 899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[21467]: audit 2024-04-06T03:22:19.339453+0000 mon.a (mon.0) 900 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:17.476178+0000 osd.6 (osd.6) 23 : cluster 0 3.a scrub starts 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:17.478670+0000 osd.6 (osd.6) 24 : cluster 0 3.a scrub ok 2024-04-06T03:22:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cephadm 2024-04-06T03:22:17.818955+0000 mgr.y (mgr.24368) 40 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: audit 2024-04-06T03:22:17.899709+0000 mgr.y (mgr.24368) 41 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cephadm 2024-04-06T03:22:17.926959+0000 mgr.y (mgr.24368) 42 : cephadm 1 Deploying daemon grafana.a on smithi153 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:17.954039+0000 mgr.y (mgr.24368) 43 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.270555+0000 osd.0 (osd.0) 23 : cluster 0 2.13 scrub starts 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.272528+0000 osd.0 (osd.0) 24 : cluster 0 2.13 scrub ok 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.377419+0000 osd.4 (osd.4) 19 : cluster 0 3.14 scrub starts 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.381002+0000 osd.4 (osd.4) 20 : cluster 0 3.14 scrub ok 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: audit 2024-04-06T03:22:19.331725+0000 mon.b (mon.1) 33 : audit 1 from='client.? 172.21.15.153:0/1234218138' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: audit 2024-04-06T03:22:19.335865+0000 mon.a (mon.0) 899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:19 smithi050 bash[29221]: audit 2024-04-06T03:22:19.339453+0000 mon.a (mon.0) 900 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:17.476178+0000 osd.6 (osd.6) 23 : cluster 0 3.a scrub starts 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:17.478670+0000 osd.6 (osd.6) 24 : cluster 0 3.a scrub ok 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cephadm 2024-04-06T03:22:17.818955+0000 mgr.y (mgr.24368) 40 : cephadm 1 Regenerating cephadm self-signed grafana TLS certificates 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: audit 2024-04-06T03:22:17.899709+0000 mgr.y (mgr.24368) 41 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cephadm 2024-04-06T03:22:17.926959+0000 mgr.y (mgr.24368) 42 : cephadm 1 Deploying daemon grafana.a on smithi153 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:17.954039+0000 mgr.y (mgr.24368) 43 : cluster 0 pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.270555+0000 osd.0 (osd.0) 23 : cluster 0 2.13 scrub starts 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.272528+0000 osd.0 (osd.0) 24 : cluster 0 2.13 scrub ok 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.377419+0000 osd.4 (osd.4) 19 : cluster 0 3.14 scrub starts 2024-04-06T03:22:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.381002+0000 osd.4 (osd.4) 20 : cluster 0 3.14 scrub ok 2024-04-06T03:22:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: audit 2024-04-06T03:22:19.331725+0000 mon.b (mon.1) 33 : audit 1 from='client.? 172.21.15.153:0/1234218138' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: audit 2024-04-06T03:22:19.335865+0000 mon.a (mon.0) 899 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-06T03:22:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:19 smithi153 bash[23852]: audit 2024-04-06T03:22:19.339453+0000 mon.a (mon.0) 900 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-06T03:22:20.107 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-06T03:22:20.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-06T03:22:20.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-06T03:22:20.128 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-06T03:22:20.129 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-06T03:22:20.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-06T03:22:20.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.707563+0000 osd.1 (osd.1) 31 : cluster 0 2.4 scrub starts 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[21467]: cluster 2024-04-06T03:22:18.710326+0000 osd.1 (osd.1) 32 : cluster 0 2.4 scrub ok 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[21467]: audit 2024-04-06T03:22:18.978855+0000 mgr.y (mgr.24368) 44 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[21467]: cluster 2024-04-06T03:22:19.650166+0000 osd.3 (osd.3) 25 : cluster 0 6.1f scrub starts 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[21467]: cluster 2024-04-06T03:22:19.652630+0000 osd.3 (osd.3) 26 : cluster 0 6.1f scrub ok 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.707563+0000 osd.1 (osd.1) 31 : cluster 0 2.4 scrub starts 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[29221]: cluster 2024-04-06T03:22:18.710326+0000 osd.1 (osd.1) 32 : cluster 0 2.4 scrub ok 2024-04-06T03:22:20.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[29221]: audit 2024-04-06T03:22:18.978855+0000 mgr.y (mgr.24368) 44 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:20.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[29221]: cluster 2024-04-06T03:22:19.650166+0000 osd.3 (osd.3) 25 : cluster 0 6.1f scrub starts 2024-04-06T03:22:20.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:20 smithi050 bash[29221]: cluster 2024-04-06T03:22:19.652630+0000 osd.3 (osd.3) 26 : cluster 0 6.1f scrub ok 2024-04-06T03:22:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:20 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.707563+0000 osd.1 (osd.1) 31 : cluster 0 2.4 scrub starts 2024-04-06T03:22:20.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:20 smithi153 bash[23852]: cluster 2024-04-06T03:22:18.710326+0000 osd.1 (osd.1) 32 : cluster 0 2.4 scrub ok 2024-04-06T03:22:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:20 smithi153 bash[23852]: audit 2024-04-06T03:22:18.978855+0000 mgr.y (mgr.24368) 44 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:20 smithi153 bash[23852]: cluster 2024-04-06T03:22:19.650166+0000 osd.3 (osd.3) 25 : cluster 0 6.1f scrub starts 2024-04-06T03:22:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:20 smithi153 bash[23852]: cluster 2024-04-06T03:22:19.652630+0000 osd.3 (osd.3) 26 : cluster 0 6.1f scrub ok 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:19.955460+0000 mgr.y (mgr.24368) 45 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.322669+0000 osd.0 (osd.0) 25 : cluster 0 6.16 scrub starts 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.324550+0000 osd.0 (osd.0) 26 : cluster 0 6.16 scrub ok 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.412911+0000 osd.2 (osd.2) 27 : cluster 0 2.1 scrub starts 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.415255+0000 osd.2 (osd.2) 28 : cluster 0 2.1 scrub ok 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.717409+0000 osd.1 (osd.1) 33 : cluster 0 4.2 deep-scrub starts 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.722135+0000 osd.1 (osd.1) 34 : cluster 0 4.2 deep-scrub ok 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:19.955460+0000 mgr.y (mgr.24368) 45 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:22:21.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.322669+0000 osd.0 (osd.0) 25 : cluster 0 6.16 scrub starts 2024-04-06T03:22:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.324550+0000 osd.0 (osd.0) 26 : cluster 0 6.16 scrub ok 2024-04-06T03:22:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.412911+0000 osd.2 (osd.2) 27 : cluster 0 2.1 scrub starts 2024-04-06T03:22:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.415255+0000 osd.2 (osd.2) 28 : cluster 0 2.1 scrub ok 2024-04-06T03:22:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.717409+0000 osd.1 (osd.1) 33 : cluster 0 4.2 deep-scrub starts 2024-04-06T03:22:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:21 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.722135+0000 osd.1 (osd.1) 34 : cluster 0 4.2 deep-scrub ok 2024-04-06T03:22:21.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:19.955460+0000 mgr.y (mgr.24368) 45 : cluster 0 pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:22:21.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.322669+0000 osd.0 (osd.0) 25 : cluster 0 6.16 scrub starts 2024-04-06T03:22:21.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.324550+0000 osd.0 (osd.0) 26 : cluster 0 6.16 scrub ok 2024-04-06T03:22:21.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.412911+0000 osd.2 (osd.2) 27 : cluster 0 2.1 scrub starts 2024-04-06T03:22:21.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.415255+0000 osd.2 (osd.2) 28 : cluster 0 2.1 scrub ok 2024-04-06T03:22:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.717409+0000 osd.1 (osd.1) 33 : cluster 0 4.2 deep-scrub starts 2024-04-06T03:22:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:21 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.722135+0000 osd.1 (osd.1) 34 : cluster 0 4.2 deep-scrub ok 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.524644+0000 osd.6 (osd.6) 25 : cluster 0 5.11 scrub starts 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:20.526166+0000 osd.6 (osd.6) 26 : cluster 0 5.11 scrub ok 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.317048+0000 osd.7 (osd.7) 25 : cluster 0 5.9 scrub starts 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.320040+0000 osd.7 (osd.7) 26 : cluster 0 5.9 scrub ok 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.388543+0000 osd.4 (osd.4) 21 : cluster 0 4.1 scrub starts 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.389810+0000 osd.4 (osd.4) 22 : cluster 0 4.1 scrub ok 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.597996+0000 osd.3 (osd.3) 27 : cluster 0 6.b scrub starts 2024-04-06T03:22:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.600103+0000 osd.3 (osd.3) 28 : cluster 0 6.b scrub ok 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.524644+0000 osd.6 (osd.6) 25 : cluster 0 5.11 scrub starts 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:20.526166+0000 osd.6 (osd.6) 26 : cluster 0 5.11 scrub ok 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.317048+0000 osd.7 (osd.7) 25 : cluster 0 5.9 scrub starts 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.320040+0000 osd.7 (osd.7) 26 : cluster 0 5.9 scrub ok 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.388543+0000 osd.4 (osd.4) 21 : cluster 0 4.1 scrub starts 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.389810+0000 osd.4 (osd.4) 22 : cluster 0 4.1 scrub ok 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.597996+0000 osd.3 (osd.3) 27 : cluster 0 6.b scrub starts 2024-04-06T03:22:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:22 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.600103+0000 osd.3 (osd.3) 28 : cluster 0 6.b scrub ok 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.524644+0000 osd.6 (osd.6) 25 : cluster 0 5.11 scrub starts 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:20.526166+0000 osd.6 (osd.6) 26 : cluster 0 5.11 scrub ok 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.317048+0000 osd.7 (osd.7) 25 : cluster 0 5.9 scrub starts 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.320040+0000 osd.7 (osd.7) 26 : cluster 0 5.9 scrub ok 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.388543+0000 osd.4 (osd.4) 21 : cluster 0 4.1 scrub starts 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.389810+0000 osd.4 (osd.4) 22 : cluster 0 4.1 scrub ok 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.597996+0000 osd.3 (osd.3) 27 : cluster 0 6.b scrub starts 2024-04-06T03:22:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:22 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.600103+0000 osd.3 (osd.3) 28 : cluster 0 6.b scrub ok 2024-04-06T03:22:23.732 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[58229]: ts=2024-04-06T03:22:23.347Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002586385s 2024-04-06T03:22:23.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.775629+0000 osd.5 (osd.5) 31 : cluster 0 3.1d deep-scrub starts 2024-04-06T03:22:23.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.783153+0000 osd.5 (osd.5) 32 : cluster 0 3.1d deep-scrub ok 2024-04-06T03:22:23.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[21467]: cluster 2024-04-06T03:22:21.956220+0000 mgr.y (mgr.24368) 46 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:23.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.775629+0000 osd.5 (osd.5) 31 : cluster 0 3.1d deep-scrub starts 2024-04-06T03:22:23.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.783153+0000 osd.5 (osd.5) 32 : cluster 0 3.1d deep-scrub ok 2024-04-06T03:22:23.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:23 smithi050 bash[29221]: cluster 2024-04-06T03:22:21.956220+0000 mgr.y (mgr.24368) 46 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:23.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:23 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.775629+0000 osd.5 (osd.5) 31 : cluster 0 3.1d deep-scrub starts 2024-04-06T03:22:23.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:23 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.783153+0000 osd.5 (osd.5) 32 : cluster 0 3.1d deep-scrub ok 2024-04-06T03:22:23.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:23 smithi153 bash[23852]: cluster 2024-04-06T03:22:21.956220+0000 mgr.y (mgr.24368) 46 : cluster 0 pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[21467]: cluster 2024-04-06T03:22:22.549983+0000 osd.6 (osd.6) 27 : cluster 0 2.11 scrub starts 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[21467]: cluster 2024-04-06T03:22:22.552254+0000 osd.6 (osd.6) 28 : cluster 0 2.11 scrub ok 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[21467]: cluster 2024-04-06T03:22:23.373147+0000 osd.0 (osd.0) 27 : cluster 0 3.1f scrub starts 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[21467]: cluster 2024-04-06T03:22:23.375792+0000 osd.0 (osd.0) 28 : cluster 0 3.1f scrub ok 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[29221]: cluster 2024-04-06T03:22:22.549983+0000 osd.6 (osd.6) 27 : cluster 0 2.11 scrub starts 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[29221]: cluster 2024-04-06T03:22:22.552254+0000 osd.6 (osd.6) 28 : cluster 0 2.11 scrub ok 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[29221]: cluster 2024-04-06T03:22:23.373147+0000 osd.0 (osd.0) 27 : cluster 0 3.1f scrub starts 2024-04-06T03:22:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:24 smithi050 bash[29221]: cluster 2024-04-06T03:22:23.375792+0000 osd.0 (osd.0) 28 : cluster 0 3.1f scrub ok 2024-04-06T03:22:24.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:24 smithi153 bash[23852]: cluster 2024-04-06T03:22:22.549983+0000 osd.6 (osd.6) 27 : cluster 0 2.11 scrub starts 2024-04-06T03:22:24.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:24 smithi153 bash[23852]: cluster 2024-04-06T03:22:22.552254+0000 osd.6 (osd.6) 28 : cluster 0 2.11 scrub ok 2024-04-06T03:22:24.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:24 smithi153 bash[23852]: cluster 2024-04-06T03:22:23.373147+0000 osd.0 (osd.0) 27 : cluster 0 3.1f scrub starts 2024-04-06T03:22:24.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:24 smithi153 bash[23852]: cluster 2024-04-06T03:22:23.375792+0000 osd.0 (osd.0) 28 : cluster 0 3.1f scrub ok 2024-04-06T03:22:24.919 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:25.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:25 smithi050 bash[21467]: cluster 2024-04-06T03:22:23.957528+0000 mgr.y (mgr.24368) 47 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-04-06T03:22:25.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:25 smithi050 bash[21467]: cluster 2024-04-06T03:22:24.408360+0000 osd.4 (osd.4) 23 : cluster 0 6.e scrub starts 2024-04-06T03:22:25.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:25 smithi050 bash[21467]: cluster 2024-04-06T03:22:24.410378+0000 osd.4 (osd.4) 24 : cluster 0 6.e scrub ok 2024-04-06T03:22:25.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:25 smithi050 bash[29221]: cluster 2024-04-06T03:22:23.957528+0000 mgr.y (mgr.24368) 47 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-04-06T03:22:25.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:25 smithi050 bash[29221]: cluster 2024-04-06T03:22:24.408360+0000 osd.4 (osd.4) 23 : cluster 0 6.e scrub starts 2024-04-06T03:22:25.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:25 smithi050 bash[29221]: cluster 2024-04-06T03:22:24.410378+0000 osd.4 (osd.4) 24 : cluster 0 6.e scrub ok 2024-04-06T03:22:25.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:25 smithi153 bash[23852]: cluster 2024-04-06T03:22:23.957528+0000 mgr.y (mgr.24368) 47 : cluster 0 pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-04-06T03:22:25.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:25 smithi153 bash[23852]: cluster 2024-04-06T03:22:24.408360+0000 osd.4 (osd.4) 23 : cluster 0 6.e scrub starts 2024-04-06T03:22:25.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:25 smithi153 bash[23852]: cluster 2024-04-06T03:22:24.410378+0000 osd.4 (osd.4) 24 : cluster 0 6.e scrub ok 2024-04-06T03:22:26.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.361487+0000 osd.4 (osd.4) 25 : cluster 0 3.9 scrub starts 2024-04-06T03:22:26.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.366096+0000 osd.4 (osd.4) 26 : cluster 0 3.9 scrub ok 2024-04-06T03:22:26.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.479229+0000 osd.2 (osd.2) 29 : cluster 0 2.c scrub starts 2024-04-06T03:22:26.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.481331+0000 osd.2 (osd.2) 30 : cluster 0 2.c scrub ok 2024-04-06T03:22:26.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.580293+0000 osd.3 (osd.3) 29 : cluster 0 3.8 scrub starts 2024-04-06T03:22:26.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.585283+0000 osd.3 (osd.3) 30 : cluster 0 3.8 scrub ok 2024-04-06T03:22:26.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.361487+0000 osd.4 (osd.4) 25 : cluster 0 3.9 scrub starts 2024-04-06T03:22:26.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.366096+0000 osd.4 (osd.4) 26 : cluster 0 3.9 scrub ok 2024-04-06T03:22:26.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.479229+0000 osd.2 (osd.2) 29 : cluster 0 2.c scrub starts 2024-04-06T03:22:26.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.481331+0000 osd.2 (osd.2) 30 : cluster 0 2.c scrub ok 2024-04-06T03:22:26.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.580293+0000 osd.3 (osd.3) 29 : cluster 0 3.8 scrub starts 2024-04-06T03:22:26.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:26 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.585283+0000 osd.3 (osd.3) 30 : cluster 0 3.8 scrub ok 2024-04-06T03:22:26.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:26 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.361487+0000 osd.4 (osd.4) 25 : cluster 0 3.9 scrub starts 2024-04-06T03:22:26.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:26 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.366096+0000 osd.4 (osd.4) 26 : cluster 0 3.9 scrub ok 2024-04-06T03:22:26.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:26 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.479229+0000 osd.2 (osd.2) 29 : cluster 0 2.c scrub starts 2024-04-06T03:22:26.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:26 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.481331+0000 osd.2 (osd.2) 30 : cluster 0 2.c scrub ok 2024-04-06T03:22:26.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:26 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.580293+0000 osd.3 (osd.3) 29 : cluster 0 3.8 scrub starts 2024-04-06T03:22:26.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:26 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.585283+0000 osd.3 (osd.3) 30 : cluster 0 3.8 scrub ok 2024-04-06T03:22:27.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[21467]: cluster 2024-04-06T03:22:25.958604+0000 mgr.y (mgr.24368) 48 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:27.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[21467]: cluster 2024-04-06T03:22:26.391315+0000 osd.0 (osd.0) 29 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[21467]: cluster 2024-04-06T03:22:26.393700+0000 osd.0 (osd.0) 30 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[21467]: cluster 2024-04-06T03:22:26.401240+0000 osd.7 (osd.7) 27 : cluster 0 5.4 scrub starts 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[21467]: cluster 2024-04-06T03:22:26.403490+0000 osd.7 (osd.7) 28 : cluster 0 5.4 scrub ok 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[29221]: cluster 2024-04-06T03:22:25.958604+0000 mgr.y (mgr.24368) 48 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[29221]: cluster 2024-04-06T03:22:26.391315+0000 osd.0 (osd.0) 29 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[29221]: cluster 2024-04-06T03:22:26.393700+0000 osd.0 (osd.0) 30 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[29221]: cluster 2024-04-06T03:22:26.401240+0000 osd.7 (osd.7) 27 : cluster 0 5.4 scrub starts 2024-04-06T03:22:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:27 smithi050 bash[29221]: cluster 2024-04-06T03:22:26.403490+0000 osd.7 (osd.7) 28 : cluster 0 5.4 scrub ok 2024-04-06T03:22:27.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:27 smithi153 bash[23852]: cluster 2024-04-06T03:22:25.958604+0000 mgr.y (mgr.24368) 48 : cluster 0 pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:27.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:27 smithi153 bash[23852]: cluster 2024-04-06T03:22:26.391315+0000 osd.0 (osd.0) 29 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:22:27.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:27 smithi153 bash[23852]: cluster 2024-04-06T03:22:26.393700+0000 osd.0 (osd.0) 30 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:22:27.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:27 smithi153 bash[23852]: cluster 2024-04-06T03:22:26.401240+0000 osd.7 (osd.7) 27 : cluster 0 5.4 scrub starts 2024-04-06T03:22:27.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:27 smithi153 bash[23852]: cluster 2024-04-06T03:22:26.403490+0000 osd.7 (osd.7) 28 : cluster 0 5.4 scrub ok 2024-04-06T03:22:28.192 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:22:28.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:28 smithi050 bash[21467]: cluster 2024-04-06T03:22:27.455307+0000 osd.0 (osd.0) 31 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:28.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:28 smithi050 bash[21467]: cluster 2024-04-06T03:22:27.455460+0000 osd.0 (osd.0) 32 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:28.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:28 smithi050 bash[21467]: audit 2024-04-06T03:22:28.190565+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.50:0/4047049864' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T03:22:28.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:28 smithi050 bash[29221]: cluster 2024-04-06T03:22:27.455307+0000 osd.0 (osd.0) 31 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:28.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:28 smithi050 bash[29221]: cluster 2024-04-06T03:22:27.455460+0000 osd.0 (osd.0) 32 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:28.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:28 smithi050 bash[29221]: audit 2024-04-06T03:22:28.190565+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.50:0/4047049864' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T03:22:28.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:28 smithi153 bash[23852]: cluster 2024-04-06T03:22:27.455307+0000 osd.0 (osd.0) 31 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:28.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:28 smithi153 bash[23852]: cluster 2024-04-06T03:22:27.455460+0000 osd.0 (osd.0) 32 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:28.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:28 smithi153 bash[23852]: audit 2024-04-06T03:22:28.190565+0000 mon.c (mon.2) 23 : audit 0 from='client.? 172.21.15.50:0/4047049864' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-06T03:22:29.014 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":22,"flags":0,"active_gid":24368,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":611757411},{"type":"v1","addr":"172.21.15.50:6801","nonce":611757411}]},"active_addr":"172.21.15.50:6801/611757411","active_change":"2024-04-06T03:21:47.915935+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24445,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.50:8443/","prometheus":"http://172.21.15.50:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":98,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1246186531}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3231578014}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2079777484}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":77561911}]}]} 2024-04-06T03:22:29.020 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-06T03:22:29.020 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-06T03:22:29.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:22:29.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:28 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:22:29.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:29 smithi050 bash[21467]: cluster 2024-04-06T03:22:27.959119+0000 mgr.y (mgr.24368) 49 : cluster 0 pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:29.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:29 smithi050 bash[21467]: cluster 2024-04-06T03:22:28.431053+0000 mon.a (mon.0) 901 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-04-06T03:22:29.733 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:22:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:22:29.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:29 smithi050 bash[29221]: cluster 2024-04-06T03:22:27.959119+0000 mgr.y (mgr.24368) 49 : cluster 0 pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:29.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:29 smithi050 bash[29221]: cluster 2024-04-06T03:22:28.431053+0000 mon.a (mon.0) 901 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-04-06T03:22:29.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:29 smithi153 bash[23852]: cluster 2024-04-06T03:22:27.959119+0000 mgr.y (mgr.24368) 49 : cluster 0 pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:29.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:29 smithi153 bash[23852]: cluster 2024-04-06T03:22:28.431053+0000 mon.a (mon.0) 901 : cluster 0 osdmap e106: 8 total, 8 up, 8 in 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:28.512288+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:28.514613+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: audit 2024-04-06T03:22:28.989995+0000 mgr.y (mgr.24368) 50 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.381834+0000 osd.4 (osd.4) 27 : cluster 0 5.1 scrub starts 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.384309+0000 osd.4 (osd.4) 28 : cluster 0 5.1 scrub ok 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.393282+0000 osd.7 (osd.7) 29 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.395490+0000 osd.7 (osd.7) 30 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.786981+0000 osd.1 (osd.1) 35 : cluster 0 6.1 scrub starts 2024-04-06T03:22:30.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.789319+0000 osd.1 (osd.1) 36 : cluster 0 6.1 scrub ok 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:28.512288+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:28.514613+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: audit 2024-04-06T03:22:28.989995+0000 mgr.y (mgr.24368) 50 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.381834+0000 osd.4 (osd.4) 27 : cluster 0 5.1 scrub starts 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.384309+0000 osd.4 (osd.4) 28 : cluster 0 5.1 scrub ok 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.393282+0000 osd.7 (osd.7) 29 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.395490+0000 osd.7 (osd.7) 30 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.786981+0000 osd.1 (osd.1) 35 : cluster 0 6.1 scrub starts 2024-04-06T03:22:30.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:30 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.789319+0000 osd.1 (osd.1) 36 : cluster 0 6.1 scrub ok 2024-04-06T03:22:30.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:28.512288+0000 osd.2 (osd.2) 31 : cluster 0 6.f scrub starts 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:28.514613+0000 osd.2 (osd.2) 32 : cluster 0 6.f scrub ok 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: audit 2024-04-06T03:22:28.989995+0000 mgr.y (mgr.24368) 50 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.381834+0000 osd.4 (osd.4) 27 : cluster 0 5.1 scrub starts 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.384309+0000 osd.4 (osd.4) 28 : cluster 0 5.1 scrub ok 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.393282+0000 osd.7 (osd.7) 29 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.395490+0000 osd.7 (osd.7) 30 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.786981+0000 osd.1 (osd.1) 35 : cluster 0 6.1 scrub starts 2024-04-06T03:22:30.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:30 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.789319+0000 osd.1 (osd.1) 36 : cluster 0 6.1 scrub ok 2024-04-06T03:22:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:31 smithi153 bash[23852]: cluster 2024-04-06T03:22:29.960610+0000 mgr.y (mgr.24368) 51 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:31 smithi153 bash[23852]: cluster 2024-04-06T03:22:30.389181+0000 osd.7 (osd.7) 31 : cluster 0 6.5 deep-scrub starts 2024-04-06T03:22:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:31 smithi153 bash[23852]: cluster 2024-04-06T03:22:30.391398+0000 osd.7 (osd.7) 32 : cluster 0 6.5 deep-scrub ok 2024-04-06T03:22:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:31 smithi153 bash[23852]: cluster 2024-04-06T03:22:30.410311+0000 osd.0 (osd.0) 33 : cluster 0 5.5 scrub starts 2024-04-06T03:22:31.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:31 smithi153 bash[23852]: cluster 2024-04-06T03:22:30.412650+0000 osd.0 (osd.0) 34 : cluster 0 5.5 scrub ok 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[21467]: cluster 2024-04-06T03:22:29.960610+0000 mgr.y (mgr.24368) 51 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[21467]: cluster 2024-04-06T03:22:30.389181+0000 osd.7 (osd.7) 31 : cluster 0 6.5 deep-scrub starts 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[21467]: cluster 2024-04-06T03:22:30.391398+0000 osd.7 (osd.7) 32 : cluster 0 6.5 deep-scrub ok 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[21467]: cluster 2024-04-06T03:22:30.410311+0000 osd.0 (osd.0) 33 : cluster 0 5.5 scrub starts 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[21467]: cluster 2024-04-06T03:22:30.412650+0000 osd.0 (osd.0) 34 : cluster 0 5.5 scrub ok 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[29221]: cluster 2024-04-06T03:22:29.960610+0000 mgr.y (mgr.24368) 51 : cluster 0 pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[29221]: cluster 2024-04-06T03:22:30.389181+0000 osd.7 (osd.7) 31 : cluster 0 6.5 deep-scrub starts 2024-04-06T03:22:31.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[29221]: cluster 2024-04-06T03:22:30.391398+0000 osd.7 (osd.7) 32 : cluster 0 6.5 deep-scrub ok 2024-04-06T03:22:31.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[29221]: cluster 2024-04-06T03:22:30.410311+0000 osd.0 (osd.0) 33 : cluster 0 5.5 scrub starts 2024-04-06T03:22:31.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:31 smithi050 bash[29221]: cluster 2024-04-06T03:22:30.412650+0000 osd.0 (osd.0) 34 : cluster 0 5.5 scrub ok 2024-04-06T03:22:32.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:30.584837+0000 osd.2 (osd.2) 33 : cluster 0 5.6 scrub starts 2024-04-06T03:22:32.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:30.587304+0000 osd.2 (osd.2) 34 : cluster 0 5.6 scrub ok 2024-04-06T03:22:32.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.377923+0000 osd.4 (osd.4) 29 : cluster 0 3.3 deep-scrub starts 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.389218+0000 osd.4 (osd.4) 30 : cluster 0 3.3 deep-scrub ok 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.404824+0000 osd.0 (osd.0) 35 : cluster 0 2.1f scrub starts 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.407179+0000 osd.0 (osd.0) 36 : cluster 0 2.1f scrub ok 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.421076+0000 osd.7 (osd.7) 33 : cluster 0 5.1a deep-scrub starts 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.422606+0000 osd.7 (osd.7) 34 : cluster 0 5.1a deep-scrub ok 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.782246+0000 osd.1 (osd.1) 37 : cluster 0 2.9 scrub starts 2024-04-06T03:22:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:32 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.784000+0000 osd.1 (osd.1) 38 : cluster 0 2.9 scrub ok 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:30.584837+0000 osd.2 (osd.2) 33 : cluster 0 5.6 scrub starts 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:30.587304+0000 osd.2 (osd.2) 34 : cluster 0 5.6 scrub ok 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.377923+0000 osd.4 (osd.4) 29 : cluster 0 3.3 deep-scrub starts 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.389218+0000 osd.4 (osd.4) 30 : cluster 0 3.3 deep-scrub ok 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.404824+0000 osd.0 (osd.0) 35 : cluster 0 2.1f scrub starts 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.407179+0000 osd.0 (osd.0) 36 : cluster 0 2.1f scrub ok 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.421076+0000 osd.7 (osd.7) 33 : cluster 0 5.1a deep-scrub starts 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.422606+0000 osd.7 (osd.7) 34 : cluster 0 5.1a deep-scrub ok 2024-04-06T03:22:32.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.782246+0000 osd.1 (osd.1) 37 : cluster 0 2.9 scrub starts 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.784000+0000 osd.1 (osd.1) 38 : cluster 0 2.9 scrub ok 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:30.584837+0000 osd.2 (osd.2) 33 : cluster 0 5.6 scrub starts 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:30.587304+0000 osd.2 (osd.2) 34 : cluster 0 5.6 scrub ok 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.377923+0000 osd.4 (osd.4) 29 : cluster 0 3.3 deep-scrub starts 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.389218+0000 osd.4 (osd.4) 30 : cluster 0 3.3 deep-scrub ok 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.404824+0000 osd.0 (osd.0) 35 : cluster 0 2.1f scrub starts 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.407179+0000 osd.0 (osd.0) 36 : cluster 0 2.1f scrub ok 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.421076+0000 osd.7 (osd.7) 33 : cluster 0 5.1a deep-scrub starts 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.422606+0000 osd.7 (osd.7) 34 : cluster 0 5.1a deep-scrub ok 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.782246+0000 osd.1 (osd.1) 37 : cluster 0 2.9 scrub starts 2024-04-06T03:22:32.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:32 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.784000+0000 osd.1 (osd.1) 38 : cluster 0 2.9 scrub ok 2024-04-06T03:22:33.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:33 smithi153 bash[23852]: cluster 2024-04-06T03:22:31.961281+0000 mgr.y (mgr.24368) 52 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:33.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:33 smithi153 bash[23852]: audit 2024-04-06T03:22:33.170303+0000 mon.a (mon.0) 902 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:33.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:33 smithi050 bash[21467]: cluster 2024-04-06T03:22:31.961281+0000 mgr.y (mgr.24368) 52 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:33.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:33 smithi050 bash[21467]: audit 2024-04-06T03:22:33.170303+0000 mon.a (mon.0) 902 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:33.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:33 smithi050 bash[29221]: cluster 2024-04-06T03:22:31.961281+0000 mgr.y (mgr.24368) 52 : cluster 0 pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:33.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:33 smithi050 bash[29221]: audit 2024-04-06T03:22:33.170303+0000 mon.a (mon.0) 902 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:33.823 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:33.992 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.993 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.993 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.993 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.994 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.994 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.994 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:33.995 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:33 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.251 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.252 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.252 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.252 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.253 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.253 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.253 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.254 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.254 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:22:34.541 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:34 smithi153 systemd[1]: Started Ceph grafana.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: cluster 2024-04-06T03:22:32.767004+0000 osd.5 (osd.5) 33 : cluster 0 6.d scrub starts 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: cluster 2024-04-06T03:22:32.769335+0000 osd.5 (osd.5) 34 : cluster 0 6.d scrub ok 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: cluster 2024-04-06T03:22:33.375314+0000 osd.0 (osd.0) 37 : cluster 0 3.6 scrub starts 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: cluster 2024-04-06T03:22:33.379674+0000 osd.0 (osd.0) 38 : cluster 0 3.6 scrub ok 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: cluster 2024-04-06T03:22:33.742874+0000 osd.3 (osd.3) 31 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: cluster 2024-04-06T03:22:33.742985+0000 osd.3 (osd.3) 32 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: audit 2024-04-06T03:22:34.314942+0000 mon.a (mon.0) 903 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: audit 2024-04-06T03:22:34.326892+0000 mon.a (mon.0) 904 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: audit 2024-04-06T03:22:34.381501+0000 mon.a (mon.0) 905 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:34 smithi153 bash[23852]: audit 2024-04-06T03:22:34.389004+0000 mon.a (mon.0) 906 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: cluster 2024-04-06T03:22:32.767004+0000 osd.5 (osd.5) 33 : cluster 0 6.d scrub starts 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: cluster 2024-04-06T03:22:32.769335+0000 osd.5 (osd.5) 34 : cluster 0 6.d scrub ok 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: cluster 2024-04-06T03:22:33.375314+0000 osd.0 (osd.0) 37 : cluster 0 3.6 scrub starts 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: cluster 2024-04-06T03:22:33.379674+0000 osd.0 (osd.0) 38 : cluster 0 3.6 scrub ok 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: cluster 2024-04-06T03:22:33.742874+0000 osd.3 (osd.3) 31 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: cluster 2024-04-06T03:22:33.742985+0000 osd.3 (osd.3) 32 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: audit 2024-04-06T03:22:34.314942+0000 mon.a (mon.0) 903 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: audit 2024-04-06T03:22:34.326892+0000 mon.a (mon.0) 904 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: audit 2024-04-06T03:22:34.381501+0000 mon.a (mon.0) 905 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[29221]: audit 2024-04-06T03:22:34.389004+0000 mon.a (mon.0) 906 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: cluster 2024-04-06T03:22:32.767004+0000 osd.5 (osd.5) 33 : cluster 0 6.d scrub starts 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: cluster 2024-04-06T03:22:32.769335+0000 osd.5 (osd.5) 34 : cluster 0 6.d scrub ok 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: cluster 2024-04-06T03:22:33.375314+0000 osd.0 (osd.0) 37 : cluster 0 3.6 scrub starts 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: cluster 2024-04-06T03:22:33.379674+0000 osd.0 (osd.0) 38 : cluster 0 3.6 scrub ok 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: cluster 2024-04-06T03:22:33.742874+0000 osd.3 (osd.3) 31 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: cluster 2024-04-06T03:22:33.742985+0000 osd.3 (osd.3) 32 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: audit 2024-04-06T03:22:34.314942+0000 mon.a (mon.0) 903 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: audit 2024-04-06T03:22:34.326892+0000 mon.a (mon.0) 904 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: audit 2024-04-06T03:22:34.381501+0000 mon.a (mon.0) 905 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:34 smithi050 bash[21467]: audit 2024-04-06T03:22:34.389004+0000 mon.a (mon.0) 906 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:35.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: cluster 2024-04-06T03:22:33.962478+0000 mgr.y (mgr.24368) 53 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:35.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.372395+0000 osd.4 (osd.4) 31 : cluster 0 6.1a scrub starts 2024-04-06T03:22:35.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.374024+0000 osd.4 (osd.4) 32 : cluster 0 6.1a scrub ok 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.393196+0000 osd.0 (osd.0) 39 : cluster 0 6.10 scrub starts 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.395075+0000 osd.0 (osd.0) 40 : cluster 0 6.10 scrub ok 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: audit 2024-04-06T03:22:34.457268+0000 mon.a (mon.0) 907 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.463012+0000 mon.a (mon.0) 908 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: audit 2024-04-06T03:22:34.939056+0000 mon.a (mon.0) 909 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[21467]: audit 2024-04-06T03:22:34.945358+0000 mon.a (mon.0) 910 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: cluster 2024-04-06T03:22:33.962478+0000 mgr.y (mgr.24368) 53 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.372395+0000 osd.4 (osd.4) 31 : cluster 0 6.1a scrub starts 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.374024+0000 osd.4 (osd.4) 32 : cluster 0 6.1a scrub ok 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.393196+0000 osd.0 (osd.0) 39 : cluster 0 6.10 scrub starts 2024-04-06T03:22:35.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.395075+0000 osd.0 (osd.0) 40 : cluster 0 6.10 scrub ok 2024-04-06T03:22:35.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: audit 2024-04-06T03:22:34.457268+0000 mon.a (mon.0) 907 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:35.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.463012+0000 mon.a (mon.0) 908 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-04-06T03:22:35.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: audit 2024-04-06T03:22:34.939056+0000 mon.a (mon.0) 909 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:35.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:35 smithi050 bash[29221]: audit 2024-04-06T03:22:34.945358+0000 mon.a (mon.0) 910 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:35.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: cluster 2024-04-06T03:22:33.962478+0000 mgr.y (mgr.24368) 53 : cluster 0 pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.372395+0000 osd.4 (osd.4) 31 : cluster 0 6.1a scrub starts 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.374024+0000 osd.4 (osd.4) 32 : cluster 0 6.1a scrub ok 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.393196+0000 osd.0 (osd.0) 39 : cluster 0 6.10 scrub starts 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.395075+0000 osd.0 (osd.0) 40 : cluster 0 6.10 scrub ok 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: audit 2024-04-06T03:22:34.457268+0000 mon.a (mon.0) 907 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.463012+0000 mon.a (mon.0) 908 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: audit 2024-04-06T03:22:34.939056+0000 mon.a (mon.0) 909 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:35.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[23852]: audit 2024-04-06T03:22:34.945358+0000 mon.a (mon.0) 910 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:36.086 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837509036Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-06T03:22:36.086 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837709677Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-06T03:22:36.086 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837723839Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-06T03:22:36.086 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837727786Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-06T03:22:36.086 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837731103Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837734498Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837737583Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837741285Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837744485Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837749276Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837752426Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837755475Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837758551Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837761855Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837767411Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837772632Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837777939Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-06T03:22:36.087 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837783029Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=settings t=2024-04-06T03:22:35.837787185Z level=info msg="App mode production" 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=sqlstore t=2024-04-06T03:22:35.837845812Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=sqlstore t=2024-04-06T03:22:35.837874586Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:35.839237462Z level=info msg="Starting DB migrations" 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:35.839343643Z level=info msg="Executing migration" id="create migration_log table" 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:35 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:35.932337493Z level=info msg="Executing migration" id="create user table" 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.015815911Z level=info msg="Executing migration" id="add unique index user.login" 2024-04-06T03:22:36.088 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.091328758Z level=info msg="Executing migration" id="add unique index user.email" 2024-04-06T03:22:36.346 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.150229309Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-04-06T03:22:36.346 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.234428464Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-04-06T03:22:36.347 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.284961208Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-04-06T03:22:36.599 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.352526796Z level=info msg="Executing migration" id="create user table v2" 2024-04-06T03:22:36.599 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.436613686Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-04-06T03:22:36.599 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.514927889Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-04-06T03:22:36.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.779769+0000 osd.5 (osd.5) 35 : cluster 0 3.16 scrub starts 2024-04-06T03:22:36.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[23852]: cluster 2024-04-06T03:22:34.784866+0000 osd.5 (osd.5) 36 : cluster 0 3.16 scrub ok 2024-04-06T03:22:36.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[23852]: cluster 2024-04-06T03:22:35.747672+0000 osd.1 (osd.1) 39 : cluster 0 6.1d scrub starts 2024-04-06T03:22:36.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[23852]: cluster 2024-04-06T03:22:35.750062+0000 osd.1 (osd.1) 40 : cluster 0 6.1d scrub ok 2024-04-06T03:22:36.600 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[23852]: cluster 2024-04-06T03:22:36.026720+0000 osd.7 (osd.7) 35 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:36.600 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[23852]: cluster 2024-04-06T03:22:36.026866+0000 osd.7 (osd.7) 36 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.779769+0000 osd.5 (osd.5) 35 : cluster 0 3.16 scrub starts 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[21467]: cluster 2024-04-06T03:22:34.784866+0000 osd.5 (osd.5) 36 : cluster 0 3.16 scrub ok 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[21467]: cluster 2024-04-06T03:22:35.747672+0000 osd.1 (osd.1) 39 : cluster 0 6.1d scrub starts 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[21467]: cluster 2024-04-06T03:22:35.750062+0000 osd.1 (osd.1) 40 : cluster 0 6.1d scrub ok 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[21467]: cluster 2024-04-06T03:22:36.026720+0000 osd.7 (osd.7) 35 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[21467]: cluster 2024-04-06T03:22:36.026866+0000 osd.7 (osd.7) 36 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.779769+0000 osd.5 (osd.5) 35 : cluster 0 3.16 scrub starts 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[29221]: cluster 2024-04-06T03:22:34.784866+0000 osd.5 (osd.5) 36 : cluster 0 3.16 scrub ok 2024-04-06T03:22:36.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[29221]: cluster 2024-04-06T03:22:35.747672+0000 osd.1 (osd.1) 39 : cluster 0 6.1d scrub starts 2024-04-06T03:22:36.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[29221]: cluster 2024-04-06T03:22:35.750062+0000 osd.1 (osd.1) 40 : cluster 0 6.1d scrub ok 2024-04-06T03:22:36.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[29221]: cluster 2024-04-06T03:22:36.026720+0000 osd.7 (osd.7) 35 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:36.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:36 smithi050 bash[29221]: cluster 2024-04-06T03:22:36.026866+0000 osd.7 (osd.7) 36 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.605074898Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.647579521Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.672890275Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.698374503Z level=info msg="Executing migration" id="Update user table charset" 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.766007783Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.807388972Z level=info msg="Executing migration" id="Add missing user data" 2024-04-06T03:22:36.852 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.832752797Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-04-06T03:22:37.129 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.858015591Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-04-06T03:22:37.129 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.900111301Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-04-06T03:22:37.129 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.925362921Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-06T03:22:37.129 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.950685645Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-04-06T03:22:37.130 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:36 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:36.992758805Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-04-06T03:22:37.130 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.034780452Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-04-06T03:22:37.130 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.060051439Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-04-06T03:22:37.130 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.101992868Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-04-06T03:22:37.130 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.135623206Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-04-06T03:22:37.237 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:22:37.237 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":108,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:10:26.969128+0000","modified":"2024-04-06T03:22:36.460944+0000","last_up_change":"2024-04-06T03:20:09.072730+0000","last_in_change":"2024-04-06T03:19:42.545394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:15:48.096900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-06T03:20:50.011930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-06T03:20:51.687715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-06T03:20:53.301921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-06T03:20:53.810316+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-06T03:20:56.321502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e607f527-4592-4acd-a47f-76575615ae1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6803","nonce":3531754646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6805","nonce":3531754646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6809","nonce":3531754646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6807","nonce":3531754646}]},"public_addr":"172.21.15.50:6803/3531754646","cluster_addr":"172.21.15.50:6805/3531754646","heartbeat_back_addr":"172.21.15.50:6809/3531754646","heartbeat_front_addr":"172.21.15.50:6807/3531754646","state":["exists","up"]},{"osd":1,"uuid":"2ae8f784-d36a-4633-8630-2aed04e4ff10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6811","nonce":4128410069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6813","nonce":4128410069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6817","nonce":4128410069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6815","nonce":4128410069}]},"public_addr":"172.21.15.50:6811/4128410069","cluster_addr":"172.21.15.50:6813/4128410069","heartbeat_back_addr":"172.21.15.50:6817/4128410069","heartbeat_front_addr":"172.21.15.50:6815/4128410069","state":["exists","up"]},{"osd":2,"uuid":"43f31d09-5c16-4aea-8822-e0d877cc931b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6819","nonce":1307247904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6821","nonce":1307247904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6825","nonce":1307247904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6823","nonce":1307247904}]},"public_addr":"172.21.15.50:6819/1307247904","cluster_addr":"172.21.15.50:6821/1307247904","heartbeat_back_addr":"172.21.15.50:6825/1307247904","heartbeat_front_addr":"172.21.15.50:6823/1307247904","state":["exists","up"]},{"osd":3,"uuid":"4c913325-442e-4ee5-94f1-8b5bca0c3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6827","nonce":1397888508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6829","nonce":1397888508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6833","nonce":1397888508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6831","nonce":1397888508}]},"public_addr":"172.21.15.50:6827/1397888508","cluster_addr":"172.21.15.50:6829/1397888508","heartbeat_back_addr":"172.21.15.50:6833/1397888508","heartbeat_front_addr":"172.21.15.50:6831/1397888508","state":["exists","up"]},{"osd":4,"uuid":"50c02bfc-c2f2-4aae-b1d5-c38a86726b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6801","nonce":3377683473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6803","nonce":3377683473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6807","nonce":3377683473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6805","nonce":3377683473}]},"public_addr":"172.21.15.153:6801/3377683473","cluster_addr":"172.21.15.153:6803/3377683473","heartbeat_back_addr":"172.21.15.153:6807/3377683473","heartbeat_front_addr":"172.21.15.153:6805/3377683473","state":["exists","up"]},{"osd":5,"uuid":"c3e8194b-2c20-4ffb-859d-00c33902b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6809","nonce":357179117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6811","nonce":357179117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6815","nonce":357179117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6813","nonce":357179117}]},"public_addr":"172.21.15.153:6809/357179117","cluster_addr":"172.21.15.153:6811/357179117","heartbeat_back_addr":"172.21.15.153:6815/357179117","heartbeat_front_addr":"172.21.15.153:6813/357179117","state":["exists","up"]},{"osd":6,"uuid":"98027d53-4324-44d1-89ce-8da85be58efb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6817","nonce":1552538208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6819","nonce":1552538208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6823","nonce":1552538208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6821","nonce":1552538208}]},"public_addr":"172.21.15.153:6817/1552538208","cluster_addr":"172.21.15.153:6819/1552538208","heartbeat_back_addr":"172.21.15.153:6823/1552538208","heartbeat_front_addr":"172.21.15.153:6821/1552538208","state":["exists","up"]},{"osd":7,"uuid":"6b47d95a-1228-4fb1-a315-5371d9b42bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6825","nonce":2479193924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6827","nonce":2479193924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6831","nonce":2479193924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6829","nonce":2479193924}]},"public_addr":"172.21.15.153:6825/2479193924","cluster_addr":"172.21.15.153:6827/2479193924","heartbeat_back_addr":"172.21.15.153:6831/2479193924","heartbeat_front_addr":"172.21.15.153:6829/2479193924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:27.455468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:15.241309+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:21:55.773597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:33.742994+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:21:33.219570+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:11.557649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:21:34.605366+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:36.026877+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/283275195":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2738019433":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1268273764":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/567103210":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/2891059051":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2170219983":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/4082841962":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6800/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/570623774":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6801/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1379780741":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2880368026":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2044358308":"2024-04-07T03:21:47.915758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:22:37.390 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.169206628Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-04-06T03:22:37.390 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.202808136Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-04-06T03:22:37.390 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.236415788Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-04-06T03:22:37.390 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.27001356Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-04-06T03:22:37.390 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.303552929Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-06T03:22:37.390 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.354330937Z level=info msg="Executing migration" id="create temp_user v2" 2024-04-06T03:22:37.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[23852]: cluster 2024-04-06T03:22:35.963282+0000 mgr.y (mgr.24368) 54 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:37.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[23852]: cluster 2024-04-06T03:22:36.453765+0000 osd.7 (osd.7) 37 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:22:37.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[23852]: cluster 2024-04-06T03:22:36.461564+0000 osd.7 (osd.7) 38 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:22:37.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[23852]: cluster 2024-04-06T03:22:36.479179+0000 mon.a (mon.0) 911 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-04-06T03:22:37.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[23852]: audit 2024-04-06T03:22:37.243744+0000 mon.a (mon.0) 912 : audit 0 from='client.? 172.21.15.50:0/629413671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.396116344Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.421324617Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.446619774Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.48883776Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.514237569Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.547625936Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.581433459Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.631735467Z level=info msg="Executing migration" id="create star table" 2024-04-06T03:22:37.677 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.682299673Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[29221]: cluster 2024-04-06T03:22:35.963282+0000 mgr.y (mgr.24368) 54 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[29221]: cluster 2024-04-06T03:22:36.453765+0000 osd.7 (osd.7) 37 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[29221]: cluster 2024-04-06T03:22:36.461564+0000 osd.7 (osd.7) 38 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[29221]: cluster 2024-04-06T03:22:36.479179+0000 mon.a (mon.0) 911 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[29221]: audit 2024-04-06T03:22:37.243744+0000 mon.a (mon.0) 912 : audit 0 from='client.? 172.21.15.50:0/629413671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[21467]: cluster 2024-04-06T03:22:35.963282+0000 mgr.y (mgr.24368) 54 : cluster 0 pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[21467]: cluster 2024-04-06T03:22:36.453765+0000 osd.7 (osd.7) 37 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[21467]: cluster 2024-04-06T03:22:36.461564+0000 osd.7 (osd.7) 38 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:22:37.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[21467]: cluster 2024-04-06T03:22:36.479179+0000 mon.a (mon.0) 911 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2024-04-06T03:22:37.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:37 smithi050 bash[21467]: audit 2024-04-06T03:22:37.243744+0000 mon.a (mon.0) 912 : audit 0 from='client.? 172.21.15.50:0/629413671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:22:37.928 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.715955734Z level=info msg="Executing migration" id="create org table v1" 2024-04-06T03:22:37.928 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.766307495Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-04-06T03:22:37.928 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.816591624Z level=info msg="Executing migration" id="create org_user table v1" 2024-04-06T03:22:37.928 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.850278787Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-04-06T03:22:37.928 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.883758679Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-04-06T03:22:37.928 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.9340853Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-04-06T03:22:38.037 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-06T03:22:38.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:37 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:37.967457225Z level=info msg="Executing migration" id="Update org table charset" 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.017913937Z level=info msg="Executing migration" id="Update org_user table charset" 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.051411679Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.10205877Z level=info msg="Executing migration" id="create dashboard table" 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.135645182Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.152675302Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-04-06T03:22:38.197 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.178062468Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-04-06T03:22:38.198 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.203332487Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.228626191Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.25389235Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.279221987Z level=info msg="Executing migration" id="create dashboard v2" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.304441492Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.346488407Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.388657317Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-04-06T03:22:38.458 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.413885959Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-04-06T03:22:38.459 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.439098428Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-04-06T03:22:38.459 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.464381449Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-04-06T03:22:38.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.506372298Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-04-06T03:22:38.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.531969498Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-04-06T03:22:38.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.573921834Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-04-06T03:22:38.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.599267908Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-04-06T03:22:38.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.624522257Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-04-06T03:22:38.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.666562495Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-04-06T03:22:38.728 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.691802207Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-04-06T03:22:38.990 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.733799665Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-04-06T03:22:38.990 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.77586689Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-04-06T03:22:38.991 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.817915574Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-04-06T03:22:38.991 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.843140593Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-04-06T03:22:38.991 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.868415691Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-04-06T03:22:38.991 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.893729138Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-04-06T03:22:38.991 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.927305044Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-04-06T03:22:38.992 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:38.97767684Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-04-06T03:22:39.241 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:38 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:22:39.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[23852]: cluster 2024-04-06T03:22:37.736624+0000 osd.5 (osd.5) 37 : cluster 0 3.c scrub starts 2024-04-06T03:22:39.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[23852]: cluster 2024-04-06T03:22:37.740762+0000 osd.5 (osd.5) 38 : cluster 0 3.c scrub ok 2024-04-06T03:22:39.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[23852]: cluster 2024-04-06T03:22:37.964016+0000 mgr.y (mgr.24368) 55 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:39.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[23852]: audit 2024-04-06T03:22:38.065887+0000 mon.a (mon.0) 913 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:39.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[23852]: cluster 2024-04-06T03:22:38.319376+0000 osd.4 (osd.4) 33 : cluster 0 6.1e scrub starts 2024-04-06T03:22:39.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[23852]: cluster 2024-04-06T03:22:38.321804+0000 osd.4 (osd.4) 34 : cluster 0 6.1e scrub ok 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.028322841Z level=info msg="Executing migration" id="Update dashboard title length" 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.078555293Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.104098932Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.12937132Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.154619529Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.196576959Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-06T03:22:39.242 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.221849057Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-06T03:22:39.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:22:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21467]: cluster 2024-04-06T03:22:37.736624+0000 osd.5 (osd.5) 37 : cluster 0 3.c scrub starts 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21467]: cluster 2024-04-06T03:22:37.740762+0000 osd.5 (osd.5) 38 : cluster 0 3.c scrub ok 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21467]: cluster 2024-04-06T03:22:37.964016+0000 mgr.y (mgr.24368) 55 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21467]: audit 2024-04-06T03:22:38.065887+0000 mon.a (mon.0) 913 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21467]: cluster 2024-04-06T03:22:38.319376+0000 osd.4 (osd.4) 33 : cluster 0 6.1e scrub starts 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[21467]: cluster 2024-04-06T03:22:38.321804+0000 osd.4 (osd.4) 34 : cluster 0 6.1e scrub ok 2024-04-06T03:22:39.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[29221]: cluster 2024-04-06T03:22:37.736624+0000 osd.5 (osd.5) 37 : cluster 0 3.c scrub starts 2024-04-06T03:22:39.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[29221]: cluster 2024-04-06T03:22:37.740762+0000 osd.5 (osd.5) 38 : cluster 0 3.c scrub ok 2024-04-06T03:22:39.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[29221]: cluster 2024-04-06T03:22:37.964016+0000 mgr.y (mgr.24368) 55 : cluster 0 pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:39.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[29221]: audit 2024-04-06T03:22:38.065887+0000 mon.a (mon.0) 913 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:39.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[29221]: cluster 2024-04-06T03:22:38.319376+0000 osd.4 (osd.4) 33 : cluster 0 6.1e scrub starts 2024-04-06T03:22:39.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:39 smithi050 bash[29221]: cluster 2024-04-06T03:22:38.321804+0000 osd.4 (osd.4) 34 : cluster 0 6.1e scrub ok 2024-04-06T03:22:39.509 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.247144289Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.297318151Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.372726729Z level=info msg="Executing migration" id="Add check_sum column" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.397912036Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.43146439Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.465232277Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.490787184Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-04-06T03:22:39.510 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.516086936Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-06T03:22:39.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.558013713Z level=info msg="Executing migration" id="create data_source table" 2024-04-06T03:22:39.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.583284235Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-04-06T03:22:39.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.6253643Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-04-06T03:22:39.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.667403138Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-04-06T03:22:39.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.700937962Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-04-06T03:22:39.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.751320449Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-04-06T03:22:40.055 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.801820379Z level=info msg="Executing migration" id="create data_source table v2" 2024-04-06T03:22:40.055 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.885565235Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-04-06T03:22:40.055 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.919057331Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-04-06T03:22:40.056 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:39 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:39.971068661Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-04-06T03:22:40.056 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.019835494Z level=info msg="Executing migration" id="Add column with_credentials" 2024-04-06T03:22:40.056 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.036866527Z level=info msg="Executing migration" id="Add secure json data column" 2024-04-06T03:22:40.056 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.062157442Z level=info msg="Executing migration" id="Update data_source table charset" 2024-04-06T03:22:40.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[23852]: cluster 2024-04-06T03:22:38.673836+0000 osd.3 (osd.3) 33 : cluster 0 5.13 scrub starts 2024-04-06T03:22:40.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[23852]: cluster 2024-04-06T03:22:38.676324+0000 osd.3 (osd.3) 34 : cluster 0 5.13 scrub ok 2024-04-06T03:22:40.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[23852]: audit 2024-04-06T03:22:38.998980+0000 mgr.y (mgr.24368) 56 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:40.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[23852]: cluster 2024-04-06T03:22:39.287863+0000 osd.4 (osd.4) 35 : cluster 0 5.18 scrub starts 2024-04-06T03:22:40.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[23852]: cluster 2024-04-06T03:22:39.290197+0000 osd.4 (osd.4) 36 : cluster 0 5.18 scrub ok 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.104227292Z level=info msg="Executing migration" id="Update initial version to 1" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.146282607Z level=info msg="Executing migration" id="Add read_only data column" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.171591354Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.196886633Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.222110152Z level=info msg="Executing migration" id="Add uid column" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.247385379Z level=info msg="Executing migration" id="Update uid value" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.28947201Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-04-06T03:22:40.309 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.314876966Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-04-06T03:22:40.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[29221]: cluster 2024-04-06T03:22:38.673836+0000 osd.3 (osd.3) 33 : cluster 0 5.13 scrub starts 2024-04-06T03:22:40.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[29221]: cluster 2024-04-06T03:22:38.676324+0000 osd.3 (osd.3) 34 : cluster 0 5.13 scrub ok 2024-04-06T03:22:40.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[29221]: audit 2024-04-06T03:22:38.998980+0000 mgr.y (mgr.24368) 56 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:40.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[29221]: cluster 2024-04-06T03:22:39.287863+0000 osd.4 (osd.4) 35 : cluster 0 5.18 scrub starts 2024-04-06T03:22:40.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[29221]: cluster 2024-04-06T03:22:39.290197+0000 osd.4 (osd.4) 36 : cluster 0 5.18 scrub ok 2024-04-06T03:22:40.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[21467]: cluster 2024-04-06T03:22:38.673836+0000 osd.3 (osd.3) 33 : cluster 0 5.13 scrub starts 2024-04-06T03:22:40.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[21467]: cluster 2024-04-06T03:22:38.676324+0000 osd.3 (osd.3) 34 : cluster 0 5.13 scrub ok 2024-04-06T03:22:40.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[21467]: audit 2024-04-06T03:22:38.998980+0000 mgr.y (mgr.24368) 56 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:40.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[21467]: cluster 2024-04-06T03:22:39.287863+0000 osd.4 (osd.4) 35 : cluster 0 5.18 scrub starts 2024-04-06T03:22:40.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:40 smithi050 bash[21467]: cluster 2024-04-06T03:22:39.290197+0000 osd.4 (osd.4) 36 : cluster 0 5.18 scrub ok 2024-04-06T03:22:40.569 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.38259423Z level=info msg="Executing migration" id="create api_key table" 2024-04-06T03:22:40.569 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.407704478Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-04-06T03:22:40.569 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.432979841Z level=info msg="Executing migration" id="add index api_key.key" 2024-04-06T03:22:40.569 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.491470815Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-04-06T03:22:40.569 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.533693484Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-04-06T03:22:40.569 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.575643071Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-04-06T03:22:40.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.626196874Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-04-06T03:22:40.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.659892869Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-04-06T03:22:40.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.701963461Z level=info msg="Executing migration" id="create api_key table v2" 2024-04-06T03:22:40.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.744157371Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-04-06T03:22:40.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.786333199Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-04-06T03:22:40.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.828217632Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.861986239Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.895621416Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.92917477Z level=info msg="Executing migration" id="Update api_key table charset" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:40 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:40.962831538Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.012998147Z level=info msg="Executing migration" id="Add service account foreign key" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.063558434Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-06T03:22:41.091 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.09724496Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-06T03:22:41.361 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.147571109Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-06T03:22:41.361 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.181292897Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-04-06T03:22:41.361 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.214880515Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-04-06T03:22:41.361 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.248479495Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-04-06T03:22:41.362 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.299069823Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-04-06T03:22:41.362 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.332814286Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-06T03:22:41.362 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.366364843Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: cluster 2024-04-06T03:22:39.709121+0000 osd.1 (osd.1) 41 : cluster 0 6.4 scrub starts 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: cluster 2024-04-06T03:22:39.710509+0000 osd.1 (osd.1) 42 : cluster 0 6.4 scrub ok 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: cluster 2024-04-06T03:22:39.964778+0000 mgr.y (mgr.24368) 57 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:40.103333+0000 mon.a (mon.0) 914 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:40.108019+0000 mon.a (mon.0) 915 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:40.986842+0000 mon.a (mon.0) 916 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:41.000486+0000 mon.a (mon.0) 917 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:41.003725+0000 mon.a (mon.0) 918 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:41.363 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:41.005856+0000 mon.a (mon.0) 919 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:41.363 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[23852]: audit 2024-04-06T03:22:41.017960+0000 mon.a (mon.0) 920 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: cluster 2024-04-06T03:22:39.709121+0000 osd.1 (osd.1) 41 : cluster 0 6.4 scrub starts 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: cluster 2024-04-06T03:22:39.710509+0000 osd.1 (osd.1) 42 : cluster 0 6.4 scrub ok 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: cluster 2024-04-06T03:22:39.964778+0000 mgr.y (mgr.24368) 57 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:40.103333+0000 mon.a (mon.0) 914 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:40.108019+0000 mon.a (mon.0) 915 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:40.986842+0000 mon.a (mon.0) 916 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:41.000486+0000 mon.a (mon.0) 917 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:41.003725+0000 mon.a (mon.0) 918 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:41.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:41.005856+0000 mon.a (mon.0) 919 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[21467]: audit 2024-04-06T03:22:41.017960+0000 mon.a (mon.0) 920 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: cluster 2024-04-06T03:22:39.709121+0000 osd.1 (osd.1) 41 : cluster 0 6.4 scrub starts 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: cluster 2024-04-06T03:22:39.710509+0000 osd.1 (osd.1) 42 : cluster 0 6.4 scrub ok 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: cluster 2024-04-06T03:22:39.964778+0000 mgr.y (mgr.24368) 57 : cluster 0 pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:40.103333+0000 mon.a (mon.0) 914 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:40.108019+0000 mon.a (mon.0) 915 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:40.986842+0000 mon.a (mon.0) 916 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:41.000486+0000 mon.a (mon.0) 917 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:41.003725+0000 mon.a (mon.0) 918 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:41.005856+0000 mon.a (mon.0) 919 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:22:41.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:41 smithi050 bash[29221]: audit 2024-04-06T03:22:41.017960+0000 mon.a (mon.0) 920 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:41.612 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.399938111Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-04-06T03:22:41.612 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.433470942Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-04-06T03:22:41.612 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.467096266Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-04-06T03:22:41.612 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.517509074Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-04-06T03:22:41.612 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.56776761Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-06T03:22:41.612 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.618157404Z level=info msg="Executing migration" id="create quota table v1" 2024-04-06T03:22:41.889 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.651802404Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.677071527Z level=info msg="Executing migration" id="Update quota table charset" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.719104317Z level=info msg="Executing migration" id="create plugin_setting table" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.744399979Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.769691166Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.811507425Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.853641231Z level=info msg="Executing migration" id="create session table" 2024-04-06T03:22:41.890 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.895761877Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-04-06T03:22:42.167 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.92097665Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-04-06T03:22:42.167 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.946302179Z level=info msg="Executing migration" id="create playlist table v2" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:41 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:41.988288614Z level=info msg="Executing migration" id="create playlist item table v2" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.013625039Z level=info msg="Executing migration" id="Update playlist table charset" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.03885278Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.064082943Z level=info msg="Executing migration" id="drop preferences table v2" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.089382666Z level=info msg="Executing migration" id="drop preferences table v3" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.131542118Z level=info msg="Executing migration" id="create preferences table v3" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.173635432Z level=info msg="Executing migration" id="Update preferences table charset" 2024-04-06T03:22:42.168 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[23852]: cluster 2024-04-06T03:22:40.707353+0000 osd.5 (osd.5) 39 : cluster 0 3.5 scrub starts 2024-04-06T03:22:42.168 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[23852]: cluster 2024-04-06T03:22:40.710322+0000 osd.5 (osd.5) 40 : cluster 0 3.5 scrub ok 2024-04-06T03:22:42.168 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[23852]: cluster 2024-04-06T03:22:40.752860+0000 osd.1 (osd.1) 43 : cluster 0 5.c scrub starts 2024-04-06T03:22:42.168 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[23852]: cluster 2024-04-06T03:22:40.755067+0000 osd.1 (osd.1) 44 : cluster 0 5.c scrub ok 2024-04-06T03:22:42.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[23852]: cephadm 2024-04-06T03:22:41.097665+0000 mgr.y (mgr.24368) 58 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T03:22:42.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[23852]: cephadm 2024-04-06T03:22:41.108604+0000 mgr.y (mgr.24368) 59 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi050 2024-04-06T03:22:42.437 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.224304253Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-04-06T03:22:42.437 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.266356898Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-04-06T03:22:42.437 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.291553745Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-04-06T03:22:42.437 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.316869024Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-06T03:22:42.437 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.342189917Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-06T03:22:42.438 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.36742807Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-06T03:22:42.438 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.392754543Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-06T03:22:42.438 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.418017326Z level=info msg="Executing migration" id="create alert table v1" 2024-04-06T03:22:42.438 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.443285993Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-04-06T03:22:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[29221]: cluster 2024-04-06T03:22:40.707353+0000 osd.5 (osd.5) 39 : cluster 0 3.5 scrub starts 2024-04-06T03:22:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[29221]: cluster 2024-04-06T03:22:40.710322+0000 osd.5 (osd.5) 40 : cluster 0 3.5 scrub ok 2024-04-06T03:22:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[29221]: cluster 2024-04-06T03:22:40.752860+0000 osd.1 (osd.1) 43 : cluster 0 5.c scrub starts 2024-04-06T03:22:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[29221]: cluster 2024-04-06T03:22:40.755067+0000 osd.1 (osd.1) 44 : cluster 0 5.c scrub ok 2024-04-06T03:22:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[29221]: cephadm 2024-04-06T03:22:41.097665+0000 mgr.y (mgr.24368) 58 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T03:22:42.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[29221]: cephadm 2024-04-06T03:22:41.108604+0000 mgr.y (mgr.24368) 59 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi050 2024-04-06T03:22:42.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[21467]: cluster 2024-04-06T03:22:40.707353+0000 osd.5 (osd.5) 39 : cluster 0 3.5 scrub starts 2024-04-06T03:22:42.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[21467]: cluster 2024-04-06T03:22:40.710322+0000 osd.5 (osd.5) 40 : cluster 0 3.5 scrub ok 2024-04-06T03:22:42.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[21467]: cluster 2024-04-06T03:22:40.752860+0000 osd.1 (osd.1) 43 : cluster 0 5.c scrub starts 2024-04-06T03:22:42.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[21467]: cluster 2024-04-06T03:22:40.755067+0000 osd.1 (osd.1) 44 : cluster 0 5.c scrub ok 2024-04-06T03:22:42.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[21467]: cephadm 2024-04-06T03:22:41.097665+0000 mgr.y (mgr.24368) 58 : cephadm 1 Reconfiguring alertmanager.a (dependencies changed)... 2024-04-06T03:22:42.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:42 smithi050 bash[21467]: cephadm 2024-04-06T03:22:41.108604+0000 mgr.y (mgr.24368) 59 : cephadm 1 Reconfiguring daemon alertmanager.a on smithi050 2024-04-06T03:22:42.538 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.468614092Z level=info msg="Executing migration" id="add index alert state" 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.510576587Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.552661025Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.577876061Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.619801571Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.661815766Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-06T03:22:42.689 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.69541091Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-04-06T03:22:42.983 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.737362849Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-06T03:22:42.983 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.76274369Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-04-06T03:22:42.983 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.804750972Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-04-06T03:22:42.983 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.855342451Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-04-06T03:22:42.983 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.888795177Z level=info msg="Executing migration" id="Add column is_default" 2024-04-06T03:22:42.983 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.939149543Z level=info msg="Executing migration" id="Add column frequency" 2024-04-06T03:22:43.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[23852]: cluster 2024-04-06T03:22:41.689825+0000 osd.5 (osd.5) 41 : cluster 0 5.1b scrub starts 2024-04-06T03:22:43.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[23852]: cluster 2024-04-06T03:22:41.692151+0000 osd.5 (osd.5) 42 : cluster 0 5.1b scrub ok 2024-04-06T03:22:43.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[23852]: cluster 2024-04-06T03:22:41.965389+0000 mgr.y (mgr.24368) 60 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:42 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:42.989402403Z level=info msg="Executing migration" id="Add column send_reminder" 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.039799919Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.07346716Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.107193957Z level=info msg="Executing migration" id="Update alert table charset" 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.166120638Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.208302274Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-04-06T03:22:43.253 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.25869498Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-06T03:22:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:43 smithi050 bash[29221]: cluster 2024-04-06T03:22:41.689825+0000 osd.5 (osd.5) 41 : cluster 0 5.1b scrub starts 2024-04-06T03:22:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:43 smithi050 bash[29221]: cluster 2024-04-06T03:22:41.692151+0000 osd.5 (osd.5) 42 : cluster 0 5.1b scrub ok 2024-04-06T03:22:43.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:43 smithi050 bash[29221]: cluster 2024-04-06T03:22:41.965389+0000 mgr.y (mgr.24368) 60 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:43 smithi050 bash[21467]: cluster 2024-04-06T03:22:41.689825+0000 osd.5 (osd.5) 41 : cluster 0 5.1b scrub starts 2024-04-06T03:22:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:43 smithi050 bash[21467]: cluster 2024-04-06T03:22:41.692151+0000 osd.5 (osd.5) 42 : cluster 0 5.1b scrub ok 2024-04-06T03:22:43.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:43 smithi050 bash[21467]: cluster 2024-04-06T03:22:41.965389+0000 mgr.y (mgr.24368) 60 : cluster 0 pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:43.521 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.309153145Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.342604489Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.376260623Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.401460004Z level=info msg="Executing migration" id="Add for to alert table" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.426745831Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.451942202Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.477237183Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.502593362Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-04-06T03:22:43.522 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.527726188Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-04-06T03:22:43.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.553058961Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-04-06T03:22:43.782 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.595088262Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-04-06T03:22:43.783 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.620367892Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-04-06T03:22:43.783 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.662351362Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-04-06T03:22:43.783 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.704431032Z level=info msg="Executing migration" id="create annotation table v5" 2024-04-06T03:22:43.783 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.746405582Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-04-06T03:22:43.783 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.78843686Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-04-06T03:22:44.041 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.822169703Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.839093779Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.872820905Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.889753398Z level=info msg="Executing migration" id="Update annotation table charset" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.906617328Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.923538247Z level=info msg="Executing migration" id="Drop category_id index" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.940427577Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:43 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:43.974511319Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-04-06T03:22:44.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.008229632Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-06T03:22:44.305 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:42.726971+0000 osd.3 (osd.3) 35 : cluster 0 2.1b scrub starts 2024-04-06T03:22:44.305 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:42.729631+0000 osd.3 (osd.3) 36 : cluster 0 2.1b scrub ok 2024-04-06T03:22:44.306 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:42.788304+0000 osd.1 (osd.1) 45 : cluster 0 5.19 scrub starts 2024-04-06T03:22:44.306 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:42.790024+0000 osd.1 (osd.1) 46 : cluster 0 5.19 scrub ok 2024-04-06T03:22:44.306 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:42.984392+0000 osd.4 (osd.4) 37 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:44.306 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:42.984504+0000 osd.4 (osd.4) 38 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:44.306 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.121599+0000 mon.a (mon.0) 921 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-04-06T03:22:44.306 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.623722+0000 osd.7 (osd.7) 39 : cluster 0 6.3 scrub starts 2024-04-06T03:22:44.307 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.626219+0000 osd.7 (osd.7) 40 : cluster 0 6.3 scrub ok 2024-04-06T03:22:44.307 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.050134876Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-06T03:22:44.307 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.07546441Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-06T03:22:44.307 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.109082759Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-04-06T03:22:44.307 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.151235826Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-06T03:22:44.308 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.193364793Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-04-06T03:22:44.308 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.218615941Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-04-06T03:22:44.308 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.243849216Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-04-06T03:22:44.308 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.269139113Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-04-06T03:22:44.308 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.311253875Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-04-06T03:22:44.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:42.726971+0000 osd.3 (osd.3) 35 : cluster 0 2.1b scrub starts 2024-04-06T03:22:44.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:42.729631+0000 osd.3 (osd.3) 36 : cluster 0 2.1b scrub ok 2024-04-06T03:22:44.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:42.788304+0000 osd.1 (osd.1) 45 : cluster 0 5.19 scrub starts 2024-04-06T03:22:44.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:42.790024+0000 osd.1 (osd.1) 46 : cluster 0 5.19 scrub ok 2024-04-06T03:22:44.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:42.984392+0000 osd.4 (osd.4) 37 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:44.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:42.984504+0000 osd.4 (osd.4) 38 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:44.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.121599+0000 mon.a (mon.0) 921 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-04-06T03:22:44.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.623722+0000 osd.7 (osd.7) 39 : cluster 0 6.3 scrub starts 2024-04-06T03:22:44.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.626219+0000 osd.7 (osd.7) 40 : cluster 0 6.3 scrub ok 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:42.726971+0000 osd.3 (osd.3) 35 : cluster 0 2.1b scrub starts 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:42.729631+0000 osd.3 (osd.3) 36 : cluster 0 2.1b scrub ok 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:42.788304+0000 osd.1 (osd.1) 45 : cluster 0 5.19 scrub starts 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:42.790024+0000 osd.1 (osd.1) 46 : cluster 0 5.19 scrub ok 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:42.984392+0000 osd.4 (osd.4) 37 : cluster 0 purged_snaps scrub starts 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:42.984504+0000 osd.4 (osd.4) 38 : cluster 0 purged_snaps scrub ok 2024-04-06T03:22:44.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.121599+0000 mon.a (mon.0) 921 : cluster 0 osdmap e109: 8 total, 8 up, 8 in 2024-04-06T03:22:44.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.623722+0000 osd.7 (osd.7) 39 : cluster 0 6.3 scrub starts 2024-04-06T03:22:44.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:44 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.626219+0000 osd.7 (osd.7) 40 : cluster 0 6.3 scrub ok 2024-04-06T03:22:44.574 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.33658771Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-04-06T03:22:44.574 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.361835169Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.387102429Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.412286358Z level=info msg="Executing migration" id="Add epoch_end column" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.437598851Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.462919138Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.488083601Z level=info msg="Executing migration" id="Move region to single row" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.513401447Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.555397953Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-06T03:22:44.575 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.5806772Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-06T03:22:44.860 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.622656321Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-06T03:22:44.861 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.664591887Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-06T03:22:44.861 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.706660939Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-04-06T03:22:44.861 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.731930413Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-06T03:22:44.861 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.773974196Z level=info msg="Executing migration" id="create test_data table" 2024-04-06T03:22:44.861 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.816385343Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-04-06T03:22:44.861 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.866687225Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-04-06T03:22:45.121 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.900292127Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-06T03:22:45.121 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:44 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:44.925567431Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-04-06T03:22:45.121 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.009198952Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-04-06T03:22:45.121 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.034536216Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-04-06T03:22:45.121 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.076490234Z level=info msg="Executing migration" id="create team table" 2024-04-06T03:22:45.121 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.126905731Z level=info msg="Executing migration" id="add index team.org_id" 2024-04-06T03:22:45.372 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.177519365Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-04-06T03:22:45.372 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.261155438Z level=info msg="Executing migration" id="create team member table" 2024-04-06T03:22:45.372 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.29477793Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-04-06T03:22:45.372 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.328334514Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.716736+0000 osd.5 (osd.5) 43 : cluster 0 6.19 scrub starts 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.718939+0000 osd.5 (osd.5) 44 : cluster 0 6.19 scrub ok 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.758246+0000 osd.3 (osd.3) 37 : cluster 0 6.1b scrub starts 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.760292+0000 osd.3 (osd.3) 38 : cluster 0 6.1b scrub ok 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:43.966834+0000 mgr.y (mgr.24368) 61 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:44.383435+0000 osd.0 (osd.0) 41 : cluster 0 6.0 deep-scrub starts 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:44.385987+0000 osd.0 (osd.0) 42 : cluster 0 6.0 deep-scrub ok 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:44.640636+0000 osd.7 (osd.7) 41 : cluster 0 2.8 scrub starts 2024-04-06T03:22:45.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[23852]: cluster 2024-04-06T03:22:44.643170+0000 osd.7 (osd.7) 42 : cluster 0 2.8 scrub ok 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.716736+0000 osd.5 (osd.5) 43 : cluster 0 6.19 scrub starts 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.718939+0000 osd.5 (osd.5) 44 : cluster 0 6.19 scrub ok 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.758246+0000 osd.3 (osd.3) 37 : cluster 0 6.1b scrub starts 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.760292+0000 osd.3 (osd.3) 38 : cluster 0 6.1b scrub ok 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:43.966834+0000 mgr.y (mgr.24368) 61 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:44.383435+0000 osd.0 (osd.0) 41 : cluster 0 6.0 deep-scrub starts 2024-04-06T03:22:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:44.385987+0000 osd.0 (osd.0) 42 : cluster 0 6.0 deep-scrub ok 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:44.640636+0000 osd.7 (osd.7) 41 : cluster 0 2.8 scrub starts 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[21467]: cluster 2024-04-06T03:22:44.643170+0000 osd.7 (osd.7) 42 : cluster 0 2.8 scrub ok 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.716736+0000 osd.5 (osd.5) 43 : cluster 0 6.19 scrub starts 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.718939+0000 osd.5 (osd.5) 44 : cluster 0 6.19 scrub ok 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.758246+0000 osd.3 (osd.3) 37 : cluster 0 6.1b scrub starts 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.760292+0000 osd.3 (osd.3) 38 : cluster 0 6.1b scrub ok 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:43.966834+0000 mgr.y (mgr.24368) 61 : cluster 0 pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:44.383435+0000 osd.0 (osd.0) 41 : cluster 0 6.0 deep-scrub starts 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:44.385987+0000 osd.0 (osd.0) 42 : cluster 0 6.0 deep-scrub ok 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:44.640636+0000 osd.7 (osd.7) 41 : cluster 0 2.8 scrub starts 2024-04-06T03:22:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:45 smithi050 bash[29221]: cluster 2024-04-06T03:22:44.643170+0000 osd.7 (osd.7) 42 : cluster 0 2.8 scrub ok 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.378787191Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.412462535Z level=info msg="Executing migration" id="Add column email to team table" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.446008752Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.479587945Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.513229828Z level=info msg="Executing migration" id="create dashboard acl table" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.546640503Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.572117194Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-06T03:22:45.633 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.59737987Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-06T03:22:45.634 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.639350493Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-04-06T03:22:45.926 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.664781549Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-04-06T03:22:45.926 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.690018318Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-04-06T03:22:45.927 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.790343392Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-04-06T03:22:45.927 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.840534302Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-04-06T03:22:46.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:45 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:45.932931204Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-04-06T03:22:46.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.033246432Z level=info msg="Executing migration" id="create tag table" 2024-04-06T03:22:46.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.100383887Z level=info msg="Executing migration" id="add index tag.key_value" 2024-04-06T03:22:46.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.125724941Z level=info msg="Executing migration" id="create login attempt table" 2024-04-06T03:22:46.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.167618887Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.201344105Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.218257665Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.243594346Z level=info msg="Executing migration" id="create login_attempt v2" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.260538435Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.277496618Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.294379672Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-04-06T03:22:46.449 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.328106741Z level=info msg="Executing migration" id="create user auth table" 2024-04-06T03:22:46.450 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.34508704Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-06T03:22:46.450 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.378848081Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-04-06T03:22:46.450 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.40405037Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-04-06T03:22:46.450 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.429399951Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-04-06T03:22:46.450 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.454642173Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-04-06T03:22:46.726 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.480057432Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-04-06T03:22:46.726 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.505318901Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.530564979Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.555754685Z level=info msg="Executing migration" id="create server_lock table" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.597846751Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.623256941Z level=info msg="Executing migration" id="create user auth token table" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.665186514Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.690506847Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-04-06T03:22:46.727 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.732554897Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-04-06T03:22:47.012 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.783297018Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-04-06T03:22:47.012 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.82522537Z level=info msg="Executing migration" id="create cache_data table" 2024-04-06T03:22:47.012 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.867335482Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-04-06T03:22:47.012 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.892636151Z level=info msg="Executing migration" id="create short_url table v1" 2024-04-06T03:22:47.013 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.934604069Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-04-06T03:22:47.013 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:46 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:46.976621759Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-06T03:22:47.013 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.01861124Z level=info msg="Executing migration" id="delete alert_definition table" 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[29221]: cluster 2024-04-06T03:22:45.967838+0000 mgr.y (mgr.24368) 62 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.183598+0000 osd.4 (osd.4) 39 : cluster 0 5.1c scrub starts 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.186313+0000 osd.4 (osd.4) 40 : cluster 0 5.1c scrub ok 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.581837+0000 osd.7 (osd.7) 43 : cluster 0 3.15 scrub starts 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[21467]: cluster 2024-04-06T03:22:45.967838+0000 mgr.y (mgr.24368) 62 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.183598+0000 osd.4 (osd.4) 39 : cluster 0 5.1c scrub starts 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.186313+0000 osd.4 (osd.4) 40 : cluster 0 5.1c scrub ok 2024-04-06T03:22:47.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.581837+0000 osd.7 (osd.7) 43 : cluster 0 3.15 scrub starts 2024-04-06T03:22:47.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.586079+0000 osd.7 (osd.7) 44 : cluster 0 3.15 scrub ok 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.043878249Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.069244579Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.111188435Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.153253153Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.186907149Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.212177948Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-04-06T03:22:47.274 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.229137108Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-04-06T03:22:47.275 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.246089991Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-04-06T03:22:47.275 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.263080824Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-04-06T03:22:47.275 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.279956391Z level=info msg="Executing migration" id="drop alert_definition table" 2024-04-06T03:22:47.275 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[23852]: cluster 2024-04-06T03:22:45.967838+0000 mgr.y (mgr.24368) 62 : cluster 0 pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:22:47.275 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.183598+0000 osd.4 (osd.4) 39 : cluster 0 5.1c scrub starts 2024-04-06T03:22:47.275 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.186313+0000 osd.4 (osd.4) 40 : cluster 0 5.1c scrub ok 2024-04-06T03:22:47.275 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.581837+0000 osd.7 (osd.7) 43 : cluster 0 3.15 scrub starts 2024-04-06T03:22:47.275 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.586079+0000 osd.7 (osd.7) 44 : cluster 0 3.15 scrub ok 2024-04-06T03:22:47.481 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:47 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.586079+0000 osd.7 (osd.7) 44 : cluster 0 3.15 scrub ok 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.305306382Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.330596814Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.355694122Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.381033506Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.414650903Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.448176772Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.481881144Z level=info msg="Executing migration" id="create alert_instance table" 2024-04-06T03:22:47.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.515447096Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-06T03:22:47.828 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.565808985Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-06T03:22:47.828 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.616505834Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-04-06T03:22:47.828 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.666834203Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-06T03:22:47.829 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.717113124Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-04-06T03:22:47.829 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.767605022Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-04-06T03:22:48.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.834672375Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-04-06T03:22:48.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.88491945Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-06T03:22:48.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.935243594Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-04-06T03:22:48.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:47 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:47.968871967Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-06T03:22:48.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.040826805Z level=info msg="Executing migration" id="create alert_rule table" 2024-04-06T03:22:48.357 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.718261+0000 osd.5 (osd.5) 45 : cluster 0 6.a deep-scrub starts 2024-04-06T03:22:48.357 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.720600+0000 osd.5 (osd.5) 46 : cluster 0 6.a deep-scrub ok 2024-04-06T03:22:48.357 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.802475+0000 osd.3 (osd.3) 39 : cluster 0 3.7 scrub starts 2024-04-06T03:22:48.357 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[23852]: cluster 2024-04-06T03:22:46.807058+0000 osd.3 (osd.3) 40 : cluster 0 3.7 scrub ok 2024-04-06T03:22:48.357 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.102912886Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.153169372Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.203547339Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.237292525Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.270886319Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.296112022Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.338093185Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-04-06T03:22:48.358 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.36341775Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-04-06T03:22:48.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.718261+0000 osd.5 (osd.5) 45 : cluster 0 6.a deep-scrub starts 2024-04-06T03:22:48.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.720600+0000 osd.5 (osd.5) 46 : cluster 0 6.a deep-scrub ok 2024-04-06T03:22:48.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.802475+0000 osd.3 (osd.3) 39 : cluster 0 3.7 scrub starts 2024-04-06T03:22:48.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[29221]: cluster 2024-04-06T03:22:46.807058+0000 osd.3 (osd.3) 40 : cluster 0 3.7 scrub ok 2024-04-06T03:22:48.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.718261+0000 osd.5 (osd.5) 45 : cluster 0 6.a deep-scrub starts 2024-04-06T03:22:48.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.720600+0000 osd.5 (osd.5) 46 : cluster 0 6.a deep-scrub ok 2024-04-06T03:22:48.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.802475+0000 osd.3 (osd.3) 39 : cluster 0 3.7 scrub starts 2024-04-06T03:22:48.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[21467]: cluster 2024-04-06T03:22:46.807058+0000 osd.3 (osd.3) 40 : cluster 0 3.7 scrub ok 2024-04-06T03:22:48.461 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:48 smithi050 systemd[1]: Stopping Ceph alertmanager.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:22:48.461 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[58229]: ts=2024-04-06T03:22:48.220Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-06T03:22:48.536 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:22:48.536 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":109,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:10:26.969128+0000","modified":"2024-04-06T03:22:43.112492+0000","last_up_change":"2024-04-06T03:20:09.072730+0000","last_in_change":"2024-04-06T03:19:42.545394+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:15:48.096900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-06T03:20:50.011930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-06T03:20:51.687715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-06T03:20:53.301921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-06T03:20:53.810316+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-06T03:20:56.321502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e607f527-4592-4acd-a47f-76575615ae1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6803","nonce":3531754646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6805","nonce":3531754646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6809","nonce":3531754646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6807","nonce":3531754646}]},"public_addr":"172.21.15.50:6803/3531754646","cluster_addr":"172.21.15.50:6805/3531754646","heartbeat_back_addr":"172.21.15.50:6809/3531754646","heartbeat_front_addr":"172.21.15.50:6807/3531754646","state":["exists","up"]},{"osd":1,"uuid":"2ae8f784-d36a-4633-8630-2aed04e4ff10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6811","nonce":4128410069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6813","nonce":4128410069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6817","nonce":4128410069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6815","nonce":4128410069}]},"public_addr":"172.21.15.50:6811/4128410069","cluster_addr":"172.21.15.50:6813/4128410069","heartbeat_back_addr":"172.21.15.50:6817/4128410069","heartbeat_front_addr":"172.21.15.50:6815/4128410069","state":["exists","up"]},{"osd":2,"uuid":"43f31d09-5c16-4aea-8822-e0d877cc931b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6819","nonce":1307247904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6821","nonce":1307247904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6825","nonce":1307247904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6823","nonce":1307247904}]},"public_addr":"172.21.15.50:6819/1307247904","cluster_addr":"172.21.15.50:6821/1307247904","heartbeat_back_addr":"172.21.15.50:6825/1307247904","heartbeat_front_addr":"172.21.15.50:6823/1307247904","state":["exists","up"]},{"osd":3,"uuid":"4c913325-442e-4ee5-94f1-8b5bca0c3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6827","nonce":1397888508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6829","nonce":1397888508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6833","nonce":1397888508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6831","nonce":1397888508}]},"public_addr":"172.21.15.50:6827/1397888508","cluster_addr":"172.21.15.50:6829/1397888508","heartbeat_back_addr":"172.21.15.50:6833/1397888508","heartbeat_front_addr":"172.21.15.50:6831/1397888508","state":["exists","up"]},{"osd":4,"uuid":"50c02bfc-c2f2-4aae-b1d5-c38a86726b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6801","nonce":3377683473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6803","nonce":3377683473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6807","nonce":3377683473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6805","nonce":3377683473}]},"public_addr":"172.21.15.153:6801/3377683473","cluster_addr":"172.21.15.153:6803/3377683473","heartbeat_back_addr":"172.21.15.153:6807/3377683473","heartbeat_front_addr":"172.21.15.153:6805/3377683473","state":["exists","up"]},{"osd":5,"uuid":"c3e8194b-2c20-4ffb-859d-00c33902b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6809","nonce":357179117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6811","nonce":357179117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6815","nonce":357179117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6813","nonce":357179117}]},"public_addr":"172.21.15.153:6809/357179117","cluster_addr":"172.21.15.153:6811/357179117","heartbeat_back_addr":"172.21.15.153:6815/357179117","heartbeat_front_addr":"172.21.15.153:6813/357179117","state":["exists","up"]},{"osd":6,"uuid":"98027d53-4324-44d1-89ce-8da85be58efb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6817","nonce":1552538208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6819","nonce":1552538208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6823","nonce":1552538208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6821","nonce":1552538208}]},"public_addr":"172.21.15.153:6817/1552538208","cluster_addr":"172.21.15.153:6819/1552538208","heartbeat_back_addr":"172.21.15.153:6823/1552538208","heartbeat_front_addr":"172.21.15.153:6821/1552538208","state":["exists","up"]},{"osd":7,"uuid":"6b47d95a-1228-4fb1-a315-5371d9b42bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6825","nonce":2479193924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6827","nonce":2479193924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6831","nonce":2479193924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6829","nonce":2479193924}]},"public_addr":"172.21.15.153:6825/2479193924","cluster_addr":"172.21.15.153:6827/2479193924","heartbeat_back_addr":"172.21.15.153:6831/2479193924","heartbeat_front_addr":"172.21.15.153:6829/2479193924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:27.455468+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:15.241309+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:21:55.773597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:33.742994+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:42.984510+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:11.557649+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:21:34.605366+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:22:36.026877+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/283275195":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2738019433":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1268273764":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/567103210":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/2891059051":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2170219983":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/4082841962":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6800/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/570623774":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6801/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1379780741":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2880368026":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2044358308":"2024-04-07T03:21:47.915758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:22:48.626 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.388693116Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.413922175Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.439155644Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.46445869Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.489687246Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.515014131Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.540190118Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.56555458Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.607753683Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-06T03:22:48.627 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.632797997Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-06T03:22:48.906 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.676945785Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-06T03:22:48.906 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.710570052Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-04-06T03:22:48.906 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.76088018Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-04-06T03:22:48.906 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.811200468Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-04-06T03:22:48.906 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.861571168Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-06T03:22:48.906 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.911901499Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-06T03:22:49.128 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:48 smithi050 bash[59656]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T03:22:49.157 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:22:49.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[23852]: cluster 2024-04-06T03:22:47.785579+0000 osd.3 (osd.3) 41 : cluster 0 6.13 deep-scrub starts 2024-04-06T03:22:49.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[23852]: cluster 2024-04-06T03:22:47.788070+0000 osd.3 (osd.3) 42 : cluster 0 6.13 deep-scrub ok 2024-04-06T03:22:49.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[23852]: cluster 2024-04-06T03:22:47.968582+0000 mgr.y (mgr.24368) 63 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:49.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[23852]: audit 2024-04-06T03:22:48.201663+0000 mon.a (mon.0) 922 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:49.158 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[23852]: audit 2024-04-06T03:22:48.543571+0000 mon.a (mon.0) 923 : audit 0 from='client.? 172.21.15.50:0/4086499757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:22:49.158 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.945486697Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-06T03:22:49.159 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:48 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:48.995923397Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-04-06T03:22:49.159 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.046458732Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-04-06T03:22:49.159 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.079957629Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-06T03:22:49.159 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.113517928Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.16402566Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.206124262Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.231396095Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.256652804Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.281896677Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.307143641Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.332401204Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.35764187Z level=info msg="Executing migration" id="create alert_image table" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.38290248Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.408113394Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-06T03:22:49.436 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.441695476Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-06T03:22:49.482 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[59775]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T03:22:49.482 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:22:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[21467]: cluster 2024-04-06T03:22:47.785579+0000 osd.3 (osd.3) 41 : cluster 0 6.13 deep-scrub starts 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[21467]: cluster 2024-04-06T03:22:47.788070+0000 osd.3 (osd.3) 42 : cluster 0 6.13 deep-scrub ok 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[21467]: cluster 2024-04-06T03:22:47.968582+0000 mgr.y (mgr.24368) 63 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[21467]: audit 2024-04-06T03:22:48.201663+0000 mon.a (mon.0) 922 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[21467]: audit 2024-04-06T03:22:48.543571+0000 mon.a (mon.0) 923 : audit 0 from='client.? 172.21.15.50:0/4086499757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[29221]: cluster 2024-04-06T03:22:47.785579+0000 osd.3 (osd.3) 41 : cluster 0 6.13 deep-scrub starts 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[29221]: cluster 2024-04-06T03:22:47.788070+0000 osd.3 (osd.3) 42 : cluster 0 6.13 deep-scrub ok 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[29221]: cluster 2024-04-06T03:22:47.968582+0000 mgr.y (mgr.24368) 63 : cluster 0 pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[29221]: audit 2024-04-06T03:22:48.201663+0000 mon.a (mon.0) 922 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:22:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:49 smithi050 bash[29221]: audit 2024-04-06T03:22:48.543571+0000 mon.a (mon.0) 923 : audit 0 from='client.? 172.21.15.50:0/4086499757' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:22:49.696 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.475412799Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.5005686Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.500856199Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.525871772Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.567792676Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.60977998Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.610313052Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.610709183Z level=info msg="alerts found to migrate" alerts=0 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.635368188Z level=info msg="Executing migration" id="create library_element table v1" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.660663227Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-04-06T03:22:49.697 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.702602518Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-04-06T03:22:49.948 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.744627077Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-04-06T03:22:49.949 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.769841638Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-04-06T03:22:49.949 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.811781201Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-06T03:22:49.949 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.853720205Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-04-06T03:22:49.949 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.879186318Z level=info msg="Executing migration" id="create data_keys table" 2024-04-06T03:22:49.949 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.912759467Z level=info msg="Executing migration" id="create secrets table" 2024-04-06T03:22:49.981 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:49 smithi050 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-06T03:22:49.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:49 smithi050 systemd[1]: Stopped Ceph alertmanager.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:49.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:49 smithi050 systemd[1]: Started Ceph alertmanager.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:50.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:49 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:49.954855129Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-06T03:22:50.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.0217113Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-06T03:22:50.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.072018354Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-06T03:22:50.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.105611756Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-06T03:22:50.234 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.172737748Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-06T03:22:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[23852]: audit 2024-04-06T03:22:49.006781+0000 mgr.y (mgr.24368) 64 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[23852]: audit 2024-04-06T03:22:49.585049+0000 mon.a (mon.0) 924 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:50.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[23852]: audit 2024-04-06T03:22:49.591032+0000 mon.a (mon.0) 925 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:50.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-06T03:22:50.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-06T03:22:50.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-06T03:22:50.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-06T03:22:50.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-06T03:22:50.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-06T03:22:50.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-06T03:22:50.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-06T03:22:50.481 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[21467]: audit 2024-04-06T03:22:49.006781+0000 mgr.y (mgr.24368) 64 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:50.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[21467]: audit 2024-04-06T03:22:49.585049+0000 mon.a (mon.0) 924 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:50.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[21467]: audit 2024-04-06T03:22:49.591032+0000 mon.a (mon.0) 925 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:50.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[29221]: audit 2024-04-06T03:22:49.006781+0000 mgr.y (mgr.24368) 64 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:22:50.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[29221]: audit 2024-04-06T03:22:49.585049+0000 mon.a (mon.0) 924 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[29221]: audit 2024-04-06T03:22:49.591032+0000 mon.a (mon.0) 925 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:50.535 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.239786381Z level=info msg="Executing migration" id="create kv_store table v1" 2024-04-06T03:22:50.535 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.27339949Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-04-06T03:22:50.536 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.36870695Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-04-06T03:22:50.791 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.541792343Z level=info msg="Executing migration" id="create permission table" 2024-04-06T03:22:50.791 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:50 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:50.670196265Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.749Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.750Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.751Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.50 port=9094 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.752Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.781Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.782Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-06T03:22:50.982 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.785Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-06T03:22:50.983 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:50 smithi050 bash[59802]: ts=2024-04-06T03:22:50.785Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-06T03:22:51.291 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.043099993Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-04-06T03:22:51.292 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.22155954Z level=info msg="Executing migration" id="create role table" 2024-04-06T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[23852]: cephadm 2024-04-06T03:22:49.607486+0000 mgr.y (mgr.24368) 65 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[23852]: cluster 2024-04-06T03:22:49.777487+0000 osd.3 (osd.3) 43 : cluster 0 3.18 deep-scrub starts 2024-04-06T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[23852]: cluster 2024-04-06T03:22:49.786466+0000 osd.3 (osd.3) 44 : cluster 0 3.18 deep-scrub ok 2024-04-06T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[23852]: cephadm 2024-04-06T03:22:49.867458+0000 mgr.y (mgr.24368) 66 : cephadm 1 Reconfiguring daemon prometheus.a on smithi153 2024-04-06T03:22:51.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[23852]: cluster 2024-04-06T03:22:49.970125+0000 mgr.y (mgr.24368) 67 : cluster 0 pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[21467]: cephadm 2024-04-06T03:22:49.607486+0000 mgr.y (mgr.24368) 65 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T03:22:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[21467]: cluster 2024-04-06T03:22:49.777487+0000 osd.3 (osd.3) 43 : cluster 0 3.18 deep-scrub starts 2024-04-06T03:22:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[21467]: cluster 2024-04-06T03:22:49.786466+0000 osd.3 (osd.3) 44 : cluster 0 3.18 deep-scrub ok 2024-04-06T03:22:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[21467]: cephadm 2024-04-06T03:22:49.867458+0000 mgr.y (mgr.24368) 66 : cephadm 1 Reconfiguring daemon prometheus.a on smithi153 2024-04-06T03:22:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[21467]: cluster 2024-04-06T03:22:49.970125+0000 mgr.y (mgr.24368) 67 : cluster 0 pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[29221]: cephadm 2024-04-06T03:22:49.607486+0000 mgr.y (mgr.24368) 65 : cephadm 1 Reconfiguring prometheus.a (dependencies changed)... 2024-04-06T03:22:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[29221]: cluster 2024-04-06T03:22:49.777487+0000 osd.3 (osd.3) 43 : cluster 0 3.18 deep-scrub starts 2024-04-06T03:22:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[29221]: cluster 2024-04-06T03:22:49.786466+0000 osd.3 (osd.3) 44 : cluster 0 3.18 deep-scrub ok 2024-04-06T03:22:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[29221]: cephadm 2024-04-06T03:22:49.867458+0000 mgr.y (mgr.24368) 66 : cephadm 1 Reconfiguring daemon prometheus.a on smithi153 2024-04-06T03:22:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:51 smithi050 bash[29221]: cluster 2024-04-06T03:22:49.970125+0000 mgr.y (mgr.24368) 67 : cluster 0 pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:51.776 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.453776866Z level=info msg="Executing migration" id="add column display_name" 2024-04-06T03:22:51.776 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.562679409Z level=info msg="Executing migration" id="add column group_name" 2024-04-06T03:22:51.776 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.665638082Z level=info msg="Executing migration" id="add index role.org_id" 2024-04-06T03:22:52.041 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.782307169Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-04-06T03:22:52.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.897839419Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-04-06T03:22:52.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:51 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:51.9566143Z level=info msg="Executing migration" id="create team role table" 2024-04-06T03:22:52.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.015367041Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-04-06T03:22:52.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[23852]: cluster 2024-04-06T03:22:50.727771+0000 osd.3 (osd.3) 45 : cluster 0 5.17 scrub starts 2024-04-06T03:22:52.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[23852]: cluster 2024-04-06T03:22:50.729288+0000 osd.3 (osd.3) 46 : cluster 0 5.17 scrub ok 2024-04-06T03:22:52.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[23852]: cluster 2024-04-06T03:22:51.218413+0000 osd.0 (osd.0) 43 : cluster 0 3.12 scrub starts 2024-04-06T03:22:52.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[23852]: cluster 2024-04-06T03:22:51.222572+0000 osd.0 (osd.0) 44 : cluster 0 3.12 scrub ok 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.119156852Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.174590153Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.200001006Z level=info msg="Executing migration" id="create user role table" 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.242246268Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.301257447Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.318212538Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-04-06T03:22:52.400 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.405326977Z level=info msg="Executing migration" id="create builtin role table" 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[29221]: cluster 2024-04-06T03:22:50.727771+0000 osd.3 (osd.3) 45 : cluster 0 5.17 scrub starts 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[29221]: cluster 2024-04-06T03:22:50.729288+0000 osd.3 (osd.3) 46 : cluster 0 5.17 scrub ok 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[29221]: cluster 2024-04-06T03:22:51.218413+0000 osd.0 (osd.0) 43 : cluster 0 3.12 scrub starts 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[29221]: cluster 2024-04-06T03:22:51.222572+0000 osd.0 (osd.0) 44 : cluster 0 3.12 scrub ok 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[21467]: cluster 2024-04-06T03:22:50.727771+0000 osd.3 (osd.3) 45 : cluster 0 5.17 scrub starts 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[21467]: cluster 2024-04-06T03:22:50.729288+0000 osd.3 (osd.3) 46 : cluster 0 5.17 scrub ok 2024-04-06T03:22:52.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[21467]: cluster 2024-04-06T03:22:51.218413+0000 osd.0 (osd.0) 43 : cluster 0 3.12 scrub starts 2024-04-06T03:22:52.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[21467]: cluster 2024-04-06T03:22:51.222572+0000 osd.0 (osd.0) 44 : cluster 0 3.12 scrub ok 2024-04-06T03:22:52.791 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.501747799Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-04-06T03:22:52.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.610033749Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-04-06T03:22:53.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:52 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:52.833411122Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-04-06T03:22:53.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:53 smithi050 bash[21467]: cluster 2024-04-06T03:22:51.970812+0000 mgr.y (mgr.24368) 68 : cluster 0 pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:53.154 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:22:52 smithi050 bash[59802]: ts=2024-04-06T03:22:52.753Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.0005676s 2024-04-06T03:22:53.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[23852]: cluster 2024-04-06T03:22:51.970812+0000 mgr.y (mgr.24368) 68 : cluster 0 pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:53.410 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:53.105241809Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-04-06T03:22:53.410 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:53.239034953Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-04-06T03:22:53.410 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:53.322926159Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-04-06T03:22:53.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:53 smithi050 bash[29221]: cluster 2024-04-06T03:22:51.970812+0000 mgr.y (mgr.24368) 68 : cluster 0 pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:22:53.791 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:53.415919405Z level=info msg="Executing migration" id="add unique index role.uid" 2024-04-06T03:22:53.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:53.566574594Z level=info msg="Executing migration" id="create seed assignment table" 2024-04-06T03:22:54.291 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:53 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:53.926229695Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-04-06T03:22:54.292 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.144489486Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-06T03:22:54.594 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.303651504Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-06T03:22:54.594 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.39874027Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-06T03:22:54.594 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.498631271Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-06T03:22:54.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[23852]: cluster 2024-04-06T03:22:53.515081+0000 osd.7 (osd.7) 45 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:22:54.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[23852]: cluster 2024-04-06T03:22:53.517720+0000 osd.7 (osd.7) 46 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:22:54.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:54 smithi050 bash[21467]: cluster 2024-04-06T03:22:53.515081+0000 osd.7 (osd.7) 45 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:22:54.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:54 smithi050 bash[21467]: cluster 2024-04-06T03:22:53.517720+0000 osd.7 (osd.7) 46 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:22:54.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:54 smithi050 bash[29221]: cluster 2024-04-06T03:22:53.515081+0000 osd.7 (osd.7) 45 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:22:54.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:54 smithi050 bash[29221]: cluster 2024-04-06T03:22:53.517720+0000 osd.7 (osd.7) 46 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:22:54.850 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.599564182Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-06T03:22:54.850 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.655592229Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-06T03:22:54.850 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.735592221Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-06T03:22:54.850 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.806677585Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-06T03:22:54.850 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.848666147Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-06T03:22:55.171 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.172 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.172 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.173 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.173 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.173 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.175 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.177 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:22:55.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.891023467Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-06T03:22:55.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:54 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:54.933111351Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-06T03:22:55.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.000381189Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-06T03:22:55.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.054529597Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-06T03:22:55.195 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.1096472Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-06T03:22:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:55 smithi050 bash[29221]: cluster 2024-04-06T03:22:53.972177+0000 mgr.y (mgr.24368) 69 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:22:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:55 smithi050 bash[29221]: cluster 2024-04-06T03:22:54.513259+0000 osd.7 (osd.7) 47 : cluster 0 2.0 scrub starts 2024-04-06T03:22:55.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:55 smithi050 bash[29221]: cluster 2024-04-06T03:22:54.515728+0000 osd.7 (osd.7) 48 : cluster 0 2.0 scrub ok 2024-04-06T03:22:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:55 smithi050 bash[21467]: cluster 2024-04-06T03:22:53.972177+0000 mgr.y (mgr.24368) 69 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:22:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:55 smithi050 bash[21467]: cluster 2024-04-06T03:22:54.513259+0000 osd.7 (osd.7) 47 : cluster 0 2.0 scrub starts 2024-04-06T03:22:55.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:55 smithi050 bash[21467]: cluster 2024-04-06T03:22:54.515728+0000 osd.7 (osd.7) 48 : cluster 0 2.0 scrub ok 2024-04-06T03:22:55.518 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.201257109Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-06T03:22:55.519 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.329353189Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-06T03:22:55.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[23852]: cluster 2024-04-06T03:22:53.972177+0000 mgr.y (mgr.24368) 69 : cluster 0 pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:22:55.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[23852]: cluster 2024-04-06T03:22:54.513259+0000 osd.7 (osd.7) 47 : cluster 0 2.0 scrub starts 2024-04-06T03:22:55.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[23852]: cluster 2024-04-06T03:22:54.515728+0000 osd.7 (osd.7) 48 : cluster 0 2.0 scrub ok 2024-04-06T03:22:55.791 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.522409831Z level=info msg="Executing migration" id="add correlation config column" 2024-04-06T03:22:55.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.732556884Z level=info msg="Executing migration" id="create entity_events table" 2024-04-06T03:22:56.169 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 systemd[1]: Stopping Ceph prometheus.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:22:56.169 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.86322627Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-06T03:22:56.169 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.955704326Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-06T03:22:56.169 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:55 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:55.956111294Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-06T03:22:56.169 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.006701544Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-06T03:22:56.169 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.007104418Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-06T03:22:56.169 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.048704989Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-06T03:22:56.170 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.099199745Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-06T03:22:56.170 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.174883798Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-06T03:22:56.429 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.21709682Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-06T03:22:56.429 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.268459847Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-06T03:22:56.429 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.348191806Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-06T03:22:56.429 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-06T03:22:56.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[49872]: ts=2024-04-06T03:22:56.205Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-06T03:22:56.430 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[23852]: cluster 2024-04-06T03:22:55.506420+0000 osd.7 (osd.7) 49 : cluster 0 6.8 scrub starts 2024-04-06T03:22:56.431 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[23852]: cluster 2024-04-06T03:22:55.508937+0000 osd.7 (osd.7) 50 : cluster 0 6.8 scrub ok 2024-04-06T03:22:56.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:56 smithi050 bash[29221]: cluster 2024-04-06T03:22:55.506420+0000 osd.7 (osd.7) 49 : cluster 0 6.8 scrub starts 2024-04-06T03:22:56.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:56 smithi050 bash[29221]: cluster 2024-04-06T03:22:55.508937+0000 osd.7 (osd.7) 50 : cluster 0 6.8 scrub ok 2024-04-06T03:22:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:56 smithi050 bash[21467]: cluster 2024-04-06T03:22:55.506420+0000 osd.7 (osd.7) 49 : cluster 0 6.8 scrub starts 2024-04-06T03:22:56.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:56 smithi050 bash[21467]: cluster 2024-04-06T03:22:55.508937+0000 osd.7 (osd.7) 50 : cluster 0 6.8 scrub ok 2024-04-06T03:22:56.791 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.435272654Z level=info msg="Executing migration" id="Drop public config table" 2024-04-06T03:22:56.792 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.630862656Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-06T03:22:57.123 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[52445]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-06T03:22:57.124 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:56 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:56.814120374Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-06T03:22:57.124 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:57.062465483Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-06T03:22:57.378 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:57.130001407Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-06T03:22:57.378 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:57.205711964Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-06T03:22:57.378 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:57.264652548Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-06T03:22:57.378 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[52507]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-prometheus-a 2024-04-06T03:22:57.379 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:57 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@prometheus.a.service: Deactivated successfully. 2024-04-06T03:22:57.379 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:57 smithi153 systemd[1]: Stopped Ceph prometheus.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:57.379 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:57 smithi153 systemd[1]: Started Ceph prometheus.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: cluster 2024-04-06T03:22:55.743286+0000 osd.1 (osd.1) 47 : cluster 0 5.12 scrub starts 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: cluster 2024-04-06T03:22:55.745775+0000 osd.1 (osd.1) 48 : cluster 0 5.12 scrub ok 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: cluster 2024-04-06T03:22:55.972900+0000 mgr.y (mgr.24368) 70 : cluster 0 pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: cluster 2024-04-06T03:22:56.160136+0000 osd.0 (osd.0) 45 : cluster 0 6.18 scrub starts 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: cluster 2024-04-06T03:22:56.162201+0000 osd.0 (osd.0) 46 : cluster 0 6.18 scrub ok 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.280384+0000 mon.a (mon.0) 926 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.288051+0000 mon.a (mon.0) 927 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.323751+0000 mon.a (mon.0) 928 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.327013+0000 mon.a (mon.0) 929 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:22:57.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.334749+0000 mon.a (mon.0) 930 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.360352+0000 mon.a (mon.0) 931 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21467]: audit 2024-04-06T03:22:57.369568+0000 mon.a (mon.0) 932 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: cluster 2024-04-06T03:22:55.743286+0000 osd.1 (osd.1) 47 : cluster 0 5.12 scrub starts 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: cluster 2024-04-06T03:22:55.745775+0000 osd.1 (osd.1) 48 : cluster 0 5.12 scrub ok 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: cluster 2024-04-06T03:22:55.972900+0000 mgr.y (mgr.24368) 70 : cluster 0 pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: cluster 2024-04-06T03:22:56.160136+0000 osd.0 (osd.0) 45 : cluster 0 6.18 scrub starts 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: cluster 2024-04-06T03:22:56.162201+0000 osd.0 (osd.0) 46 : cluster 0 6.18 scrub ok 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.280384+0000 mon.a (mon.0) 926 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.288051+0000 mon.a (mon.0) 927 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.323751+0000 mon.a (mon.0) 928 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.327013+0000 mon.a (mon.0) 929 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.334749+0000 mon.a (mon.0) 930 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.360352+0000 mon.a (mon.0) 931 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[29221]: audit 2024-04-06T03:22:57.369568+0000 mon.a (mon.0) 932 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STOPPING 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STOPPED 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STARTING 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Serving on http://:::9283 2024-04-06T03:22:57.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STARTED 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STOPPING 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STOPPED 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STARTING 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Serving on http://:::9283 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STARTED 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STOPPING 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STOPPED 2024-04-06T03:22:57.596 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STARTING 2024-04-06T03:22:57.739 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:57.432850416Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: cluster 2024-04-06T03:22:55.743286+0000 osd.1 (osd.1) 47 : cluster 0 5.12 scrub starts 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: cluster 2024-04-06T03:22:55.745775+0000 osd.1 (osd.1) 48 : cluster 0 5.12 scrub ok 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: cluster 2024-04-06T03:22:55.972900+0000 mgr.y (mgr.24368) 70 : cluster 0 pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: cluster 2024-04-06T03:22:56.160136+0000 osd.0 (osd.0) 45 : cluster 0 6.18 scrub starts 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: cluster 2024-04-06T03:22:56.162201+0000 osd.0 (osd.0) 46 : cluster 0 6.18 scrub ok 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.280384+0000 mon.a (mon.0) 926 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.288051+0000 mon.a (mon.0) 927 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.323751+0000 mon.a (mon.0) 928 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:22:57.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.327013+0000 mon.a (mon.0) 929 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:22:57.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.334749+0000 mon.a (mon.0) 930 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:57.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.360352+0000 mon.a (mon.0) 931 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:57.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[23852]: audit 2024-04-06T03:22:57.369568+0000 mon.a (mon.0) 932 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:22:57.865 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Serving on http://:::9283 2024-04-06T03:22:57.865 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:22:57 smithi050 bash[21725]: [06/Apr/2024:03:22:57] ENGINE Bus STARTED 2024-04-06T03:22:58.037 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:57 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:57.745935472Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-06T03:22:58.292 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.043964658Z level=info msg="Executing migration" id="add share column" 2024-04-06T03:22:58.292 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.203132674Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-06T03:22:58.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.324713+0000 mgr.y (mgr.24368) 71 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:22:58.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.327794+0000 mgr.y (mgr.24368) 72 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:22:58.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.361488+0000 mgr.y (mgr.24368) 73 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:58.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.370509+0000 mgr.y (mgr.24368) 74 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.380546+0000 mon.a (mon.0) 933 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.399460+0000 mon.a (mon.0) 934 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.400026+0000 mgr.y (mgr.24368) 75 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.401663+0000 mon.a (mon.0) 935 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.402028+0000 mgr.y (mgr.24368) 76 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.406598+0000 mon.a (mon.0) 936 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: audit 2024-04-06T03:22:57.425110+0000 mon.a (mon.0) 937 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: cluster 2024-04-06T03:22:57.483234+0000 osd.7 (osd.7) 51 : cluster 0 3.d scrub starts 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: cluster 2024-04-06T03:22:57.488161+0000 osd.7 (osd.7) 52 : cluster 0 3.d scrub ok 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: cluster 2024-04-06T03:22:57.658115+0000 osd.3 (osd.3) 47 : cluster 0 6.11 scrub starts 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: cluster 2024-04-06T03:22:57.659276+0000 osd.3 (osd.3) 48 : cluster 0 6.11 scrub ok 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[29221]: cluster 2024-04-06T03:22:58.129862+0000 osd.0 (osd.0) 47 : cluster 0 5.3 scrub starts 2024-04-06T03:22:58.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.324713+0000 mgr.y (mgr.24368) 71 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.327794+0000 mgr.y (mgr.24368) 72 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.361488+0000 mgr.y (mgr.24368) 73 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.370509+0000 mgr.y (mgr.24368) 74 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.380546+0000 mon.a (mon.0) 933 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.399460+0000 mon.a (mon.0) 934 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.400026+0000 mgr.y (mgr.24368) 75 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.401663+0000 mon.a (mon.0) 935 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.402028+0000 mgr.y (mgr.24368) 76 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.406598+0000 mon.a (mon.0) 936 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: audit 2024-04-06T03:22:57.425110+0000 mon.a (mon.0) 937 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: cluster 2024-04-06T03:22:57.483234+0000 osd.7 (osd.7) 51 : cluster 0 3.d scrub starts 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: cluster 2024-04-06T03:22:57.488161+0000 osd.7 (osd.7) 52 : cluster 0 3.d scrub ok 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: cluster 2024-04-06T03:22:57.658115+0000 osd.3 (osd.3) 47 : cluster 0 6.11 scrub starts 2024-04-06T03:22:58.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: cluster 2024-04-06T03:22:57.659276+0000 osd.3 (osd.3) 48 : cluster 0 6.11 scrub ok 2024-04-06T03:22:58.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:58 smithi050 bash[21467]: cluster 2024-04-06T03:22:58.129862+0000 osd.0 (osd.0) 47 : cluster 0 5.3 scrub starts 2024-04-06T03:22:58.567 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.304399321Z level=info msg="Executing migration" id="create file table" 2024-04-06T03:22:58.567 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.412830193Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-06T03:22:58.567 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.489537697Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-06T03:22:58.567 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.324713+0000 mgr.y (mgr.24368) 71 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-06T03:22:58.567 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.327794+0000 mgr.y (mgr.24368) 72 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-06T03:22:58.567 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.361488+0000 mgr.y (mgr.24368) 73 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.370509+0000 mgr.y (mgr.24368) 74 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.380546+0000 mon.a (mon.0) 933 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.399460+0000 mon.a (mon.0) 934 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.400026+0000 mgr.y (mgr.24368) 75 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.401663+0000 mon.a (mon.0) 935 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.402028+0000 mgr.y (mgr.24368) 76 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.406598+0000 mon.a (mon.0) 936 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: audit 2024-04-06T03:22:57.425110+0000 mon.a (mon.0) 937 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: cluster 2024-04-06T03:22:57.483234+0000 osd.7 (osd.7) 51 : cluster 0 3.d scrub starts 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: cluster 2024-04-06T03:22:57.488161+0000 osd.7 (osd.7) 52 : cluster 0 3.d scrub ok 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: cluster 2024-04-06T03:22:57.658115+0000 osd.3 (osd.3) 47 : cluster 0 6.11 scrub starts 2024-04-06T03:22:58.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: cluster 2024-04-06T03:22:57.659276+0000 osd.3 (osd.3) 48 : cluster 0 6.11 scrub ok 2024-04-06T03:22:58.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[23852]: cluster 2024-04-06T03:22:58.129862+0000 osd.0 (osd.0) 47 : cluster 0 5.3 scrub starts 2024-04-06T03:22:58.838 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.572024645Z level=info msg="Executing migration" id="create file_meta table" 2024-04-06T03:22:58.838 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.659661295Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-06T03:22:58.838 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.731374371Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-06T03:22:58.838 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.790103329Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-06T03:22:59.095 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:22:59.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.843626278Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-06T03:22:59.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.916410189Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:58.966880747Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.017358581Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.050848026Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.10137339Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.852Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.852Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.852Z caller=main.go:570 level=info host_details="(Linux 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 smithi153 (none))" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.852Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.852Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.854Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.854Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.856Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-06T03:22:59.096 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.856Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.858Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.858Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.214µs 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.858Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.858Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.859Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.859Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.388µs wal_replay_duration=634.812µs wbl_replay_duration=101ns total_replay_duration=683.184µs 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.860Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.860Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.860Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.880Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.220335ms db_storage=866ns remote_storage=1.271µs web_handler=379ns query_engine=665ns scrape=248.414µs scrape_sd=92.461µs notify=11.066µs notify_sd=19.802µs rules=19.579759ms tracing=6.147µs 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.880Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-06T03:22:59.097 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:22:58 smithi153 bash[52536]: ts=2024-04-06T03:22:58.880Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.151826055Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.202313893Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.269292746Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.286266219Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.311495254Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.328427216Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-06T03:22:59.381 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.387047924Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[21467]: cluster 2024-04-06T03:22:57.973308+0000 mgr.y (mgr.24368) 77 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[21467]: cluster 2024-04-06T03:22:58.133353+0000 osd.0 (osd.0) 48 : cluster 0 5.3 scrub ok 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[21467]: cluster 2024-04-06T03:22:58.494711+0000 osd.7 (osd.7) 53 : cluster 0 6.1c scrub starts 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[21467]: cluster 2024-04-06T03:22:58.497517+0000 osd.7 (osd.7) 54 : cluster 0 6.1c scrub ok 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[21467]: cluster 2024-04-06T03:22:58.651814+0000 osd.3 (osd.3) 49 : cluster 0 5.0 deep-scrub starts 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[21467]: cluster 2024-04-06T03:22:58.657234+0000 osd.3 (osd.3) 50 : cluster 0 5.0 deep-scrub ok 2024-04-06T03:22:59.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[29221]: cluster 2024-04-06T03:22:57.973308+0000 mgr.y (mgr.24368) 77 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:59.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[29221]: cluster 2024-04-06T03:22:58.133353+0000 osd.0 (osd.0) 48 : cluster 0 5.3 scrub ok 2024-04-06T03:22:59.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[29221]: cluster 2024-04-06T03:22:58.494711+0000 osd.7 (osd.7) 53 : cluster 0 6.1c scrub starts 2024-04-06T03:22:59.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[29221]: cluster 2024-04-06T03:22:58.497517+0000 osd.7 (osd.7) 54 : cluster 0 6.1c scrub ok 2024-04-06T03:22:59.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[29221]: cluster 2024-04-06T03:22:58.651814+0000 osd.3 (osd.3) 49 : cluster 0 5.0 deep-scrub starts 2024-04-06T03:22:59.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:22:59 smithi050 bash[29221]: cluster 2024-04-06T03:22:58.657234+0000 osd.3 (osd.3) 50 : cluster 0 5.0 deep-scrub ok 2024-04-06T03:22:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[23852]: cluster 2024-04-06T03:22:57.973308+0000 mgr.y (mgr.24368) 77 : cluster 0 pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:22:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[23852]: cluster 2024-04-06T03:22:58.133353+0000 osd.0 (osd.0) 48 : cluster 0 5.3 scrub ok 2024-04-06T03:22:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[23852]: cluster 2024-04-06T03:22:58.494711+0000 osd.7 (osd.7) 53 : cluster 0 6.1c scrub starts 2024-04-06T03:22:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[23852]: cluster 2024-04-06T03:22:58.497517+0000 osd.7 (osd.7) 54 : cluster 0 6.1c scrub ok 2024-04-06T03:22:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[23852]: cluster 2024-04-06T03:22:58.651814+0000 osd.3 (osd.3) 49 : cluster 0 5.0 deep-scrub starts 2024-04-06T03:22:59.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[23852]: cluster 2024-04-06T03:22:58.657234+0000 osd.3 (osd.3) 50 : cluster 0 5.0 deep-scrub ok 2024-04-06T03:22:59.667 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.420798882Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-06T03:22:59.667 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.446097495Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-06T03:22:59.667 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.488009286Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-06T03:22:59.668 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.52157231Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-06T03:22:59.668 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.55525832Z level=info msg="Executing migration" id="create folder table" 2024-04-06T03:22:59.668 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.580500823Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-06T03:22:59.668 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.60578874Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-06T03:22:59.668 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.631009894Z level=info msg="Executing migration" id="Update folder title length" 2024-04-06T03:22:59.668 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.672968371Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-06T03:23:00.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=migrator t=2024-04-06T03:22:59.698181279Z level=info msg="migrations completed" performed=485 skipped=0 duration=23.858862849s 2024-04-06T03:23:00.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=sqlstore t=2024-04-06T03:22:59.699576667Z level=info msg="Created default organization" 2024-04-06T03:23:00.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=plugin.finder t=2024-04-06T03:22:59.809239247Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-06T03:23:00.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=secrets t=2024-04-06T03:22:59.80941221Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-06T03:23:00.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=query_data t=2024-04-06T03:22:59.813287404Z level=info msg="Query Service initialization" 2024-04-06T03:23:00.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:22:59 smithi153 bash[51781]: logger=live.push_http t=2024-04-06T03:22:59.818376305Z level=info msg="Live Push Gateway initialization" 2024-04-06T03:23:00.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[21467]: audit 2024-04-06T03:22:59.014919+0000 mgr.y (mgr.24368) 78 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:00.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[21467]: cluster 2024-04-06T03:22:59.653895+0000 osd.3 (osd.3) 51 : cluster 0 5.14 deep-scrub starts 2024-04-06T03:23:00.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[21467]: cluster 2024-04-06T03:22:59.658755+0000 osd.3 (osd.3) 52 : cluster 0 5.14 deep-scrub ok 2024-04-06T03:23:00.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[29221]: audit 2024-04-06T03:22:59.014919+0000 mgr.y (mgr.24368) 78 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:00.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[29221]: cluster 2024-04-06T03:22:59.653895+0000 osd.3 (osd.3) 51 : cluster 0 5.14 deep-scrub starts 2024-04-06T03:23:00.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[29221]: cluster 2024-04-06T03:22:59.658755+0000 osd.3 (osd.3) 52 : cluster 0 5.14 deep-scrub ok 2024-04-06T03:23:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:00 smithi153 bash[23852]: audit 2024-04-06T03:22:59.014919+0000 mgr.y (mgr.24368) 78 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:00 smithi153 bash[23852]: cluster 2024-04-06T03:22:59.653895+0000 osd.3 (osd.3) 51 : cluster 0 5.14 deep-scrub starts 2024-04-06T03:23:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:00 smithi153 bash[23852]: cluster 2024-04-06T03:22:59.658755+0000 osd.3 (osd.3) 52 : cluster 0 5.14 deep-scrub ok 2024-04-06T03:23:00.998 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:23:00 smithi050 bash[59802]: ts=2024-04-06T03:23:00.756Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003588508s 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=infra.usagestats.collector t=2024-04-06T03:23:01.017951762Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=provisioning.datasources t=2024-04-06T03:23:01.020071396Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=provisioning.datasources t=2024-04-06T03:23:01.108019441Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=provisioning.alerting t=2024-04-06T03:23:01.158103637Z level=info msg="starting to provision alerting" 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=provisioning.alerting t=2024-04-06T03:23:01.15813025Z level=info msg="finished to provision alerting" 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=ngalert.state.manager t=2024-04-06T03:23:01.158412399Z level=info msg="Warming state cache for startup" 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=ngalert.state.manager t=2024-04-06T03:23:01.158807812Z level=info msg="State cache has been initialized" states=0 duration=394.674µs 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=ticker t=2024-04-06T03:23:01.158918677Z level=info msg=starting first_tick=2024-04-06T03:23:10Z 2024-04-06T03:23:01.283 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=ngalert.multiorg.alertmanager t=2024-04-06T03:23:01.158986481Z level=info msg="starting MultiOrg Alertmanager" 2024-04-06T03:23:01.284 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=grafanaStorageLogger t=2024-04-06T03:23:01.15907898Z level=info msg="storage starting" 2024-04-06T03:23:01.284 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=http.server t=2024-04-06T03:23:01.161790302Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-06T03:23:01.284 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:23:01.22327707Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-06T03:23:01.284 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:23:01.288803503Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-06T03:23:01.541 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:23:01.294996683Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-06T03:23:01.542 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:23:01.300235217Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-04-06T03:23:01.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[23852]: cluster 2024-04-06T03:22:59.974739+0000 mgr.y (mgr.24368) 79 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:01.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[23852]: cluster 2024-04-06T03:23:00.608284+0000 osd.3 (osd.3) 53 : cluster 0 6.6 scrub starts 2024-04-06T03:23:01.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[23852]: cluster 2024-04-06T03:23:00.610109+0000 osd.3 (osd.3) 54 : cluster 0 6.6 scrub ok 2024-04-06T03:23:01.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:01 smithi050 bash[21467]: cluster 2024-04-06T03:22:59.974739+0000 mgr.y (mgr.24368) 79 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:01.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:01 smithi050 bash[21467]: cluster 2024-04-06T03:23:00.608284+0000 osd.3 (osd.3) 53 : cluster 0 6.6 scrub starts 2024-04-06T03:23:01.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:01 smithi050 bash[21467]: cluster 2024-04-06T03:23:00.610109+0000 osd.3 (osd.3) 54 : cluster 0 6.6 scrub ok 2024-04-06T03:23:01.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:01 smithi050 bash[29221]: cluster 2024-04-06T03:22:59.974739+0000 mgr.y (mgr.24368) 79 : cluster 0 pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:01.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:01 smithi050 bash[29221]: cluster 2024-04-06T03:23:00.608284+0000 osd.3 (osd.3) 53 : cluster 0 6.6 scrub starts 2024-04-06T03:23:01.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:01 smithi050 bash[29221]: cluster 2024-04-06T03:23:00.610109+0000 osd.3 (osd.3) 54 : cluster 0 6.6 scrub ok 2024-04-06T03:23:02.041 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:23:01.639934653Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-04-06T03:23:02.042 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:23:01 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:23:01.729447389Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-04-06T03:23:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[21467]: cluster 2024-04-06T03:23:01.381164+0000 osd.2 (osd.2) 35 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[21467]: cluster 2024-04-06T03:23:01.381279+0000 osd.2 (osd.2) 36 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[21467]: cluster 2024-04-06T03:23:01.428008+0000 mon.a (mon.0) 938 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-04-06T03:23:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[21467]: cluster 2024-04-06T03:23:01.684190+0000 osd.1 (osd.1) 49 : cluster 0 5.1d deep-scrub starts 2024-04-06T03:23:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[21467]: cluster 2024-04-06T03:23:01.686786+0000 osd.1 (osd.1) 50 : cluster 0 5.1d deep-scrub ok 2024-04-06T03:23:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[29221]: cluster 2024-04-06T03:23:01.381164+0000 osd.2 (osd.2) 35 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[29221]: cluster 2024-04-06T03:23:01.381279+0000 osd.2 (osd.2) 36 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[29221]: cluster 2024-04-06T03:23:01.428008+0000 mon.a (mon.0) 938 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-04-06T03:23:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[29221]: cluster 2024-04-06T03:23:01.684190+0000 osd.1 (osd.1) 49 : cluster 0 5.1d deep-scrub starts 2024-04-06T03:23:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:02 smithi050 bash[29221]: cluster 2024-04-06T03:23:01.686786+0000 osd.1 (osd.1) 50 : cluster 0 5.1d deep-scrub ok 2024-04-06T03:23:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:02 smithi153 bash[23852]: cluster 2024-04-06T03:23:01.381164+0000 osd.2 (osd.2) 35 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:02 smithi153 bash[23852]: cluster 2024-04-06T03:23:01.381279+0000 osd.2 (osd.2) 36 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:02 smithi153 bash[23852]: cluster 2024-04-06T03:23:01.428008+0000 mon.a (mon.0) 938 : cluster 0 osdmap e110: 8 total, 8 up, 8 in 2024-04-06T03:23:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:02 smithi153 bash[23852]: cluster 2024-04-06T03:23:01.684190+0000 osd.1 (osd.1) 49 : cluster 0 5.1d deep-scrub starts 2024-04-06T03:23:02.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:02 smithi153 bash[23852]: cluster 2024-04-06T03:23:01.686786+0000 osd.1 (osd.1) 50 : cluster 0 5.1d deep-scrub ok 2024-04-06T03:23:03.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:03 smithi050 bash[21467]: cluster 2024-04-06T03:23:01.975441+0000 mgr.y (mgr.24368) 80 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:03.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:03 smithi050 bash[21467]: audit 2024-04-06T03:23:03.217288+0000 mon.a (mon.0) 939 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:03 smithi050 bash[29221]: cluster 2024-04-06T03:23:01.975441+0000 mgr.y (mgr.24368) 80 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:03 smithi050 bash[29221]: audit 2024-04-06T03:23:03.217288+0000 mon.a (mon.0) 939 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:03.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:03 smithi153 bash[23852]: cluster 2024-04-06T03:23:01.975441+0000 mgr.y (mgr.24368) 80 : cluster 0 pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:03.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:03 smithi153 bash[23852]: audit 2024-04-06T03:23:03.217288+0000 mon.a (mon.0) 939 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:05.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: cluster 2024-04-06T03:23:03.977012+0000 mgr.y (mgr.24368) 81 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:05.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: cluster 2024-04-06T03:23:04.090302+0000 osd.6 (osd.6) 29 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:05.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: cluster 2024-04-06T03:23:04.090381+0000 osd.6 (osd.6) 30 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: cluster 2024-04-06T03:23:04.453958+0000 mon.a (mon.0) 940 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: audit 2024-04-06T03:23:04.521309+0000 mon.a (mon.0) 941 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: audit 2024-04-06T03:23:04.534079+0000 mon.a (mon.0) 942 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: cluster 2024-04-06T03:23:04.810531+0000 osd.1 (osd.1) 51 : cluster 0 3.19 deep-scrub starts 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[29221]: cluster 2024-04-06T03:23:04.830536+0000 osd.1 (osd.1) 52 : cluster 0 3.19 deep-scrub ok 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: cluster 2024-04-06T03:23:03.977012+0000 mgr.y (mgr.24368) 81 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: cluster 2024-04-06T03:23:04.090302+0000 osd.6 (osd.6) 29 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: cluster 2024-04-06T03:23:04.090381+0000 osd.6 (osd.6) 30 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: cluster 2024-04-06T03:23:04.453958+0000 mon.a (mon.0) 940 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: audit 2024-04-06T03:23:04.521309+0000 mon.a (mon.0) 941 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:05.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: audit 2024-04-06T03:23:04.534079+0000 mon.a (mon.0) 942 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:05.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: cluster 2024-04-06T03:23:04.810531+0000 osd.1 (osd.1) 51 : cluster 0 3.19 deep-scrub starts 2024-04-06T03:23:05.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:05 smithi050 bash[21467]: cluster 2024-04-06T03:23:04.830536+0000 osd.1 (osd.1) 52 : cluster 0 3.19 deep-scrub ok 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: cluster 2024-04-06T03:23:03.977012+0000 mgr.y (mgr.24368) 81 : cluster 0 pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: cluster 2024-04-06T03:23:04.090302+0000 osd.6 (osd.6) 29 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: cluster 2024-04-06T03:23:04.090381+0000 osd.6 (osd.6) 30 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: cluster 2024-04-06T03:23:04.453958+0000 mon.a (mon.0) 940 : cluster 0 osdmap e111: 8 total, 8 up, 8 in 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: audit 2024-04-06T03:23:04.521309+0000 mon.a (mon.0) 941 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: audit 2024-04-06T03:23:04.534079+0000 mon.a (mon.0) 942 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: cluster 2024-04-06T03:23:04.810531+0000 osd.1 (osd.1) 51 : cluster 0 3.19 deep-scrub starts 2024-04-06T03:23:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:05 smithi153 bash[23852]: cluster 2024-04-06T03:23:04.830536+0000 osd.1 (osd.1) 52 : cluster 0 3.19 deep-scrub ok 2024-04-06T03:23:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:07 smithi050 bash[21467]: cluster 2024-04-06T03:23:05.977738+0000 mgr.y (mgr.24368) 82 : cluster 0 pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:07 smithi050 bash[21467]: cluster 2024-04-06T03:23:06.628002+0000 osd.3 (osd.3) 55 : cluster 0 4.0 scrub starts 2024-04-06T03:23:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:07 smithi050 bash[21467]: cluster 2024-04-06T03:23:06.631524+0000 osd.3 (osd.3) 56 : cluster 0 4.0 scrub ok 2024-04-06T03:23:07.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:07 smithi050 bash[29221]: cluster 2024-04-06T03:23:05.977738+0000 mgr.y (mgr.24368) 82 : cluster 0 pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:07.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:07 smithi050 bash[29221]: cluster 2024-04-06T03:23:06.628002+0000 osd.3 (osd.3) 55 : cluster 0 4.0 scrub starts 2024-04-06T03:23:07.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:07 smithi050 bash[29221]: cluster 2024-04-06T03:23:06.631524+0000 osd.3 (osd.3) 56 : cluster 0 4.0 scrub ok 2024-04-06T03:23:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:07 smithi153 bash[23852]: cluster 2024-04-06T03:23:05.977738+0000 mgr.y (mgr.24368) 82 : cluster 0 pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:07 smithi153 bash[23852]: cluster 2024-04-06T03:23:06.628002+0000 osd.3 (osd.3) 55 : cluster 0 4.0 scrub starts 2024-04-06T03:23:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:07 smithi153 bash[23852]: cluster 2024-04-06T03:23:06.631524+0000 osd.3 (osd.3) 56 : cluster 0 4.0 scrub ok 2024-04-06T03:23:08.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:08 smithi153 bash[23852]: cluster 2024-04-06T03:23:07.795681+0000 osd.1 (osd.1) 53 : cluster 0 2.a scrub starts 2024-04-06T03:23:08.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:08 smithi153 bash[23852]: cluster 2024-04-06T03:23:07.798081+0000 osd.1 (osd.1) 54 : cluster 0 2.a scrub ok 2024-04-06T03:23:08.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:08 smithi050 bash[29221]: cluster 2024-04-06T03:23:07.795681+0000 osd.1 (osd.1) 53 : cluster 0 2.a scrub starts 2024-04-06T03:23:08.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:08 smithi050 bash[29221]: cluster 2024-04-06T03:23:07.798081+0000 osd.1 (osd.1) 54 : cluster 0 2.a scrub ok 2024-04-06T03:23:08.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:08 smithi050 bash[21467]: cluster 2024-04-06T03:23:07.795681+0000 osd.1 (osd.1) 53 : cluster 0 2.a scrub starts 2024-04-06T03:23:08.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:08 smithi050 bash[21467]: cluster 2024-04-06T03:23:07.798081+0000 osd.1 (osd.1) 54 : cluster 0 2.a scrub ok 2024-04-06T03:23:09.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:23:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:23:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[21467]: cluster 2024-04-06T03:23:07.600731+0000 osd.2 (osd.2) 37 : cluster 0 2.e scrub starts 2024-04-06T03:23:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[21467]: cluster 2024-04-06T03:23:07.604079+0000 osd.2 (osd.2) 38 : cluster 0 2.e scrub ok 2024-04-06T03:23:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[21467]: cluster 2024-04-06T03:23:07.978640+0000 mgr.y (mgr.24368) 83 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:09.732 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:23:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:23:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[29221]: cluster 2024-04-06T03:23:07.600731+0000 osd.2 (osd.2) 37 : cluster 0 2.e scrub starts 2024-04-06T03:23:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[29221]: cluster 2024-04-06T03:23:07.604079+0000 osd.2 (osd.2) 38 : cluster 0 2.e scrub ok 2024-04-06T03:23:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:09 smithi050 bash[29221]: cluster 2024-04-06T03:23:07.978640+0000 mgr.y (mgr.24368) 83 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:09 smithi153 bash[23852]: cluster 2024-04-06T03:23:07.600731+0000 osd.2 (osd.2) 37 : cluster 0 2.e scrub starts 2024-04-06T03:23:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:09 smithi153 bash[23852]: cluster 2024-04-06T03:23:07.604079+0000 osd.2 (osd.2) 38 : cluster 0 2.e scrub ok 2024-04-06T03:23:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:09 smithi153 bash[23852]: cluster 2024-04-06T03:23:07.978640+0000 mgr.y (mgr.24368) 83 : cluster 0 pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[21467]: audit 2024-04-06T03:23:09.026292+0000 mgr.y (mgr.24368) 84 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[21467]: cluster 2024-04-06T03:23:09.159644+0000 osd.4 (osd.4) 41 : cluster 0 2.f scrub starts 2024-04-06T03:23:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[21467]: cluster 2024-04-06T03:23:09.163243+0000 osd.4 (osd.4) 42 : cluster 0 2.f scrub ok 2024-04-06T03:23:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[21467]: cluster 2024-04-06T03:23:09.788785+0000 osd.1 (osd.1) 55 : cluster 0 3.4 scrub starts 2024-04-06T03:23:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[21467]: cluster 2024-04-06T03:23:09.792365+0000 osd.1 (osd.1) 56 : cluster 0 3.4 scrub ok 2024-04-06T03:23:10.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[29221]: audit 2024-04-06T03:23:09.026292+0000 mgr.y (mgr.24368) 84 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[29221]: cluster 2024-04-06T03:23:09.159644+0000 osd.4 (osd.4) 41 : cluster 0 2.f scrub starts 2024-04-06T03:23:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[29221]: cluster 2024-04-06T03:23:09.163243+0000 osd.4 (osd.4) 42 : cluster 0 2.f scrub ok 2024-04-06T03:23:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[29221]: cluster 2024-04-06T03:23:09.788785+0000 osd.1 (osd.1) 55 : cluster 0 3.4 scrub starts 2024-04-06T03:23:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:10 smithi050 bash[29221]: cluster 2024-04-06T03:23:09.792365+0000 osd.1 (osd.1) 56 : cluster 0 3.4 scrub ok 2024-04-06T03:23:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:10 smithi153 bash[23852]: audit 2024-04-06T03:23:09.026292+0000 mgr.y (mgr.24368) 84 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:10 smithi153 bash[23852]: cluster 2024-04-06T03:23:09.159644+0000 osd.4 (osd.4) 41 : cluster 0 2.f scrub starts 2024-04-06T03:23:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:10 smithi153 bash[23852]: cluster 2024-04-06T03:23:09.163243+0000 osd.4 (osd.4) 42 : cluster 0 2.f scrub ok 2024-04-06T03:23:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:10 smithi153 bash[23852]: cluster 2024-04-06T03:23:09.788785+0000 osd.1 (osd.1) 55 : cluster 0 3.4 scrub starts 2024-04-06T03:23:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:10 smithi153 bash[23852]: cluster 2024-04-06T03:23:09.792365+0000 osd.1 (osd.1) 56 : cluster 0 3.4 scrub ok 2024-04-06T03:23:10.940 INFO:teuthology.orchestra.run.smithi050.stdout:197568495675 2024-04-06T03:23:10.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-06T03:23:11.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:11 smithi050 bash[21467]: cluster 2024-04-06T03:23:09.980073+0000 mgr.y (mgr.24368) 85 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:11.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:11 smithi050 bash[29221]: cluster 2024-04-06T03:23:09.980073+0000 mgr.y (mgr.24368) 85 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:11 smithi153 bash[23852]: cluster 2024-04-06T03:23:09.980073+0000 mgr.y (mgr.24368) 85 : cluster 0 pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:12 smithi050 bash[21467]: cluster 2024-04-06T03:23:10.625369+0000 osd.2 (osd.2) 39 : cluster 0 6.14 deep-scrub starts 2024-04-06T03:23:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:12 smithi050 bash[21467]: cluster 2024-04-06T03:23:10.631045+0000 osd.2 (osd.2) 40 : cluster 0 6.14 deep-scrub ok 2024-04-06T03:23:12.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:12 smithi050 bash[29221]: cluster 2024-04-06T03:23:10.625369+0000 osd.2 (osd.2) 39 : cluster 0 6.14 deep-scrub starts 2024-04-06T03:23:12.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:12 smithi050 bash[29221]: cluster 2024-04-06T03:23:10.631045+0000 osd.2 (osd.2) 40 : cluster 0 6.14 deep-scrub ok 2024-04-06T03:23:12.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:12 smithi153 bash[23852]: cluster 2024-04-06T03:23:10.625369+0000 osd.2 (osd.2) 39 : cluster 0 6.14 deep-scrub starts 2024-04-06T03:23:12.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:12 smithi153 bash[23852]: cluster 2024-04-06T03:23:10.631045+0000 osd.2 (osd.2) 40 : cluster 0 6.14 deep-scrub ok 2024-04-06T03:23:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:13 smithi153 bash[23852]: cluster 2024-04-06T03:23:11.687617+0000 osd.5 (osd.5) 47 : cluster 0 5.f scrub starts 2024-04-06T03:23:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:13 smithi153 bash[23852]: cluster 2024-04-06T03:23:11.690160+0000 osd.5 (osd.5) 48 : cluster 0 5.f scrub ok 2024-04-06T03:23:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:13 smithi153 bash[23852]: cluster 2024-04-06T03:23:11.980737+0000 mgr.y (mgr.24368) 86 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:13 smithi153 bash[23852]: cluster 2024-04-06T03:23:12.822799+0000 osd.1 (osd.1) 57 : cluster 0 2.15 deep-scrub starts 2024-04-06T03:23:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:13 smithi153 bash[23852]: cluster 2024-04-06T03:23:12.824471+0000 osd.1 (osd.1) 58 : cluster 0 2.15 deep-scrub ok 2024-04-06T03:23:13.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[21467]: cluster 2024-04-06T03:23:11.687617+0000 osd.5 (osd.5) 47 : cluster 0 5.f scrub starts 2024-04-06T03:23:13.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[21467]: cluster 2024-04-06T03:23:11.690160+0000 osd.5 (osd.5) 48 : cluster 0 5.f scrub ok 2024-04-06T03:23:13.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[21467]: cluster 2024-04-06T03:23:11.980737+0000 mgr.y (mgr.24368) 86 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:13.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[21467]: cluster 2024-04-06T03:23:12.822799+0000 osd.1 (osd.1) 57 : cluster 0 2.15 deep-scrub starts 2024-04-06T03:23:13.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[21467]: cluster 2024-04-06T03:23:12.824471+0000 osd.1 (osd.1) 58 : cluster 0 2.15 deep-scrub ok 2024-04-06T03:23:13.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[29221]: cluster 2024-04-06T03:23:11.687617+0000 osd.5 (osd.5) 47 : cluster 0 5.f scrub starts 2024-04-06T03:23:13.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[29221]: cluster 2024-04-06T03:23:11.690160+0000 osd.5 (osd.5) 48 : cluster 0 5.f scrub ok 2024-04-06T03:23:13.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[29221]: cluster 2024-04-06T03:23:11.980737+0000 mgr.y (mgr.24368) 86 : cluster 0 pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:13.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[29221]: cluster 2024-04-06T03:23:12.822799+0000 osd.1 (osd.1) 57 : cluster 0 2.15 deep-scrub starts 2024-04-06T03:23:13.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:13 smithi050 bash[29221]: cluster 2024-04-06T03:23:12.824471+0000 osd.1 (osd.1) 58 : cluster 0 2.15 deep-scrub ok 2024-04-06T03:23:14.515 INFO:teuthology.orchestra.run.smithi050.stdout:244813135920 2024-04-06T03:23:14.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-06T03:23:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[21467]: cluster 2024-04-06T03:23:12.713308+0000 osd.5 (osd.5) 49 : cluster 0 2.16 scrub starts 2024-04-06T03:23:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[21467]: cluster 2024-04-06T03:23:12.715599+0000 osd.5 (osd.5) 50 : cluster 0 2.16 scrub ok 2024-04-06T03:23:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[21467]: cluster 2024-04-06T03:23:14.037945+0000 osd.0 (osd.0) 49 : cluster 0 3.1b scrub starts 2024-04-06T03:23:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[21467]: cluster 2024-04-06T03:23:14.042037+0000 osd.0 (osd.0) 50 : cluster 0 3.1b scrub ok 2024-04-06T03:23:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[29221]: cluster 2024-04-06T03:23:12.713308+0000 osd.5 (osd.5) 49 : cluster 0 2.16 scrub starts 2024-04-06T03:23:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[29221]: cluster 2024-04-06T03:23:12.715599+0000 osd.5 (osd.5) 50 : cluster 0 2.16 scrub ok 2024-04-06T03:23:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[29221]: cluster 2024-04-06T03:23:14.037945+0000 osd.0 (osd.0) 49 : cluster 0 3.1b scrub starts 2024-04-06T03:23:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:14 smithi050 bash[29221]: cluster 2024-04-06T03:23:14.042037+0000 osd.0 (osd.0) 50 : cluster 0 3.1b scrub ok 2024-04-06T03:23:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:14 smithi153 bash[23852]: cluster 2024-04-06T03:23:12.713308+0000 osd.5 (osd.5) 49 : cluster 0 2.16 scrub starts 2024-04-06T03:23:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:14 smithi153 bash[23852]: cluster 2024-04-06T03:23:12.715599+0000 osd.5 (osd.5) 50 : cluster 0 2.16 scrub ok 2024-04-06T03:23:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:14 smithi153 bash[23852]: cluster 2024-04-06T03:23:14.037945+0000 osd.0 (osd.0) 49 : cluster 0 3.1b scrub starts 2024-04-06T03:23:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:14 smithi153 bash[23852]: cluster 2024-04-06T03:23:14.042037+0000 osd.0 (osd.0) 50 : cluster 0 3.1b scrub ok 2024-04-06T03:23:15.216 INFO:teuthology.orchestra.run.smithi050.stdout:34359738479 2024-04-06T03:23:15.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-06T03:23:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:81604378715 2024-04-06T03:23:15.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-06T03:23:15.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:15 smithi050 bash[21467]: cluster 2024-04-06T03:23:13.982069+0000 mgr.y (mgr.24368) 87 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:15.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:15 smithi050 bash[29221]: cluster 2024-04-06T03:23:13.982069+0000 mgr.y (mgr.24368) 87 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:15.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:15 smithi153 bash[23852]: cluster 2024-04-06T03:23:13.982069+0000 mgr.y (mgr.24368) 87 : cluster 0 pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:16.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:16 smithi050 bash[21467]: cluster 2024-04-06T03:23:15.707430+0000 osd.7 (osd.7) 55 : cluster 0 2.b scrub starts 2024-04-06T03:23:16.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:16 smithi050 bash[21467]: cluster 2024-04-06T03:23:15.710027+0000 osd.7 (osd.7) 56 : cluster 0 2.b scrub ok 2024-04-06T03:23:16.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:16 smithi050 bash[29221]: cluster 2024-04-06T03:23:15.707430+0000 osd.7 (osd.7) 55 : cluster 0 2.b scrub starts 2024-04-06T03:23:16.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:16 smithi050 bash[29221]: cluster 2024-04-06T03:23:15.710027+0000 osd.7 (osd.7) 56 : cluster 0 2.b scrub ok 2024-04-06T03:23:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:16 smithi153 bash[23852]: cluster 2024-04-06T03:23:15.707430+0000 osd.7 (osd.7) 55 : cluster 0 2.b scrub starts 2024-04-06T03:23:16.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:16 smithi153 bash[23852]: cluster 2024-04-06T03:23:15.710027+0000 osd.7 (osd.7) 56 : cluster 0 2.b scrub ok 2024-04-06T03:23:16.946 INFO:teuthology.orchestra.run.smithi050.stdout:296352743462 2024-04-06T03:23:16.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-06T03:23:16.950 INFO:teuthology.orchestra.run.smithi050.stdout:150323855431 2024-04-06T03:23:16.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-06T03:23:16.952 INFO:teuthology.orchestra.run.smithi050.stdout:120259084367 2024-04-06T03:23:16.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-06T03:23:17.377 INFO:teuthology.orchestra.run.smithi050.stdout:55834574949 2024-04-06T03:23:17.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-06T03:23:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[21467]: cluster 2024-04-06T03:23:15.982829+0000 mgr.y (mgr.24368) 88 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-04-06T03:23:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[21467]: cluster 2024-04-06T03:23:16.812229+0000 osd.1 (osd.1) 59 : cluster 0 2.d scrub starts 2024-04-06T03:23:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[21467]: cluster 2024-04-06T03:23:16.814724+0000 osd.1 (osd.1) 60 : cluster 0 2.d scrub ok 2024-04-06T03:23:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[21467]: cluster 2024-04-06T03:23:16.848771+0000 osd.6 (osd.6) 31 : cluster 0 2.14 scrub starts 2024-04-06T03:23:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[21467]: cluster 2024-04-06T03:23:16.851009+0000 osd.6 (osd.6) 32 : cluster 0 2.14 scrub ok 2024-04-06T03:23:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[29221]: cluster 2024-04-06T03:23:15.982829+0000 mgr.y (mgr.24368) 88 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-04-06T03:23:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[29221]: cluster 2024-04-06T03:23:16.812229+0000 osd.1 (osd.1) 59 : cluster 0 2.d scrub starts 2024-04-06T03:23:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[29221]: cluster 2024-04-06T03:23:16.814724+0000 osd.1 (osd.1) 60 : cluster 0 2.d scrub ok 2024-04-06T03:23:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[29221]: cluster 2024-04-06T03:23:16.848771+0000 osd.6 (osd.6) 31 : cluster 0 2.14 scrub starts 2024-04-06T03:23:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:17 smithi050 bash[29221]: cluster 2024-04-06T03:23:16.851009+0000 osd.6 (osd.6) 32 : cluster 0 2.14 scrub ok 2024-04-06T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:17 smithi153 bash[23852]: cluster 2024-04-06T03:23:15.982829+0000 mgr.y (mgr.24368) 88 : cluster 0 pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 887 B/s rd, 0 op/s 2024-04-06T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:17 smithi153 bash[23852]: cluster 2024-04-06T03:23:16.812229+0000 osd.1 (osd.1) 59 : cluster 0 2.d scrub starts 2024-04-06T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:17 smithi153 bash[23852]: cluster 2024-04-06T03:23:16.814724+0000 osd.1 (osd.1) 60 : cluster 0 2.d scrub ok 2024-04-06T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:17 smithi153 bash[23852]: cluster 2024-04-06T03:23:16.848771+0000 osd.6 (osd.6) 31 : cluster 0 2.14 scrub starts 2024-04-06T03:23:17.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:17 smithi153 bash[23852]: cluster 2024-04-06T03:23:16.851009+0000 osd.6 (osd.6) 32 : cluster 0 2.14 scrub ok 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: cluster 2024-04-06T03:23:17.693886+0000 osd.3 (osd.3) 57 : cluster 0 2.1e scrub starts 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: cluster 2024-04-06T03:23:17.696463+0000 osd.3 (osd.3) 58 : cluster 0 2.1e scrub ok 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: cluster 2024-04-06T03:23:17.861763+0000 osd.6 (osd.6) 33 : cluster 0 2.7 deep-scrub starts 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: cluster 2024-04-06T03:23:17.864170+0000 osd.6 (osd.6) 34 : cluster 0 2.7 deep-scrub ok 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: cluster 2024-04-06T03:23:18.053910+0000 osd.0 (osd.0) 51 : cluster 0 3.17 scrub starts 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: cluster 2024-04-06T03:23:18.056020+0000 osd.0 (osd.0) 52 : cluster 0 3.17 scrub ok 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: audit 2024-04-06T03:23:18.233851+0000 mon.a (mon.0) 943 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:18.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: audit 2024-04-06T03:23:18.452579+0000 mon.a (mon.0) 944 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:18.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:18 smithi153 bash[23852]: audit 2024-04-06T03:23:18.458802+0000 mon.a (mon.0) 945 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: cluster 2024-04-06T03:23:17.693886+0000 osd.3 (osd.3) 57 : cluster 0 2.1e scrub starts 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: cluster 2024-04-06T03:23:17.696463+0000 osd.3 (osd.3) 58 : cluster 0 2.1e scrub ok 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: cluster 2024-04-06T03:23:17.861763+0000 osd.6 (osd.6) 33 : cluster 0 2.7 deep-scrub starts 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: cluster 2024-04-06T03:23:17.864170+0000 osd.6 (osd.6) 34 : cluster 0 2.7 deep-scrub ok 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: cluster 2024-04-06T03:23:18.053910+0000 osd.0 (osd.0) 51 : cluster 0 3.17 scrub starts 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: cluster 2024-04-06T03:23:18.056020+0000 osd.0 (osd.0) 52 : cluster 0 3.17 scrub ok 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: audit 2024-04-06T03:23:18.233851+0000 mon.a (mon.0) 943 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: audit 2024-04-06T03:23:18.452579+0000 mon.a (mon.0) 944 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[21467]: audit 2024-04-06T03:23:18.458802+0000 mon.a (mon.0) 945 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:18.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: cluster 2024-04-06T03:23:17.693886+0000 osd.3 (osd.3) 57 : cluster 0 2.1e scrub starts 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: cluster 2024-04-06T03:23:17.696463+0000 osd.3 (osd.3) 58 : cluster 0 2.1e scrub ok 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: cluster 2024-04-06T03:23:17.861763+0000 osd.6 (osd.6) 33 : cluster 0 2.7 deep-scrub starts 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: cluster 2024-04-06T03:23:17.864170+0000 osd.6 (osd.6) 34 : cluster 0 2.7 deep-scrub ok 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: cluster 2024-04-06T03:23:18.053910+0000 osd.0 (osd.0) 51 : cluster 0 3.17 scrub starts 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: cluster 2024-04-06T03:23:18.056020+0000 osd.0 (osd.0) 52 : cluster 0 3.17 scrub ok 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: audit 2024-04-06T03:23:18.233851+0000 mon.a (mon.0) 943 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: audit 2024-04-06T03:23:18.452579+0000 mon.a (mon.0) 944 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:18 smithi050 bash[29221]: audit 2024-04-06T03:23:18.458802+0000 mon.a (mon.0) 945 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:23:19.732 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:23:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:23:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: cluster 2024-04-06T03:23:17.983246+0000 mgr.y (mgr.24368) 89 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: cluster 2024-04-06T03:23:18.278001+0000 osd.4 (osd.4) 43 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: cluster 2024-04-06T03:23:18.287113+0000 osd.4 (osd.4) 44 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.035483+0000 mon.a (mon.0) 946 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.036713+0000 mon.a (mon.0) 947 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.042399+0000 mon.a (mon.0) 948 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.125171+0000 mon.a (mon.0) 949 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.128576+0000 mon.a (mon.0) 950 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.130221+0000 mon.a (mon.0) 951 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[21467]: audit 2024-04-06T03:23:19.139676+0000 mon.a (mon.0) 952 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: cluster 2024-04-06T03:23:17.983246+0000 mgr.y (mgr.24368) 89 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: cluster 2024-04-06T03:23:18.278001+0000 osd.4 (osd.4) 43 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: cluster 2024-04-06T03:23:18.287113+0000 osd.4 (osd.4) 44 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:23:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.035483+0000 mon.a (mon.0) 946 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.036713+0000 mon.a (mon.0) 947 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.042399+0000 mon.a (mon.0) 948 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.125171+0000 mon.a (mon.0) 949 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.128576+0000 mon.a (mon.0) 950 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.130221+0000 mon.a (mon.0) 951 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:19.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:19 smithi050 bash[29221]: audit 2024-04-06T03:23:19.139676+0000 mon.a (mon.0) 952 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: cluster 2024-04-06T03:23:17.983246+0000 mgr.y (mgr.24368) 89 : cluster 0 pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: cluster 2024-04-06T03:23:18.278001+0000 osd.4 (osd.4) 43 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: cluster 2024-04-06T03:23:18.287113+0000 osd.4 (osd.4) 44 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.035483+0000 mon.a (mon.0) 946 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.036713+0000 mon.a (mon.0) 947 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:19.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.042399+0000 mon.a (mon.0) 948 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.125171+0000 mon.a (mon.0) 949 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:23:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.128576+0000 mon.a (mon.0) 950 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:23:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.130221+0000 mon.a (mon.0) 951 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:23:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:19 smithi153 bash[23852]: audit 2024-04-06T03:23:19.139676+0000 mon.a (mon.0) 952 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:23:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:19.975 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:19.977 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:19.979 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:20 smithi153 bash[23852]: audit 2024-04-06T03:23:19.037403+0000 mgr.y (mgr.24368) 90 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:20 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.296137+0000 osd.4 (osd.4) 45 : cluster 0 6.17 scrub starts 2024-04-06T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:20 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.298477+0000 osd.4 (osd.4) 46 : cluster 0 6.17 scrub ok 2024-04-06T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:20 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.817974+0000 osd.7 (osd.7) 57 : cluster 0 3.e scrub starts 2024-04-06T03:23:20.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:20 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.822121+0000 osd.7 (osd.7) 58 : cluster 0 3.e scrub ok 2024-04-06T03:23:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[21467]: audit 2024-04-06T03:23:19.037403+0000 mgr.y (mgr.24368) 90 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.296137+0000 osd.4 (osd.4) 45 : cluster 0 6.17 scrub starts 2024-04-06T03:23:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.298477+0000 osd.4 (osd.4) 46 : cluster 0 6.17 scrub ok 2024-04-06T03:23:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.817974+0000 osd.7 (osd.7) 57 : cluster 0 3.e scrub starts 2024-04-06T03:23:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.822121+0000 osd.7 (osd.7) 58 : cluster 0 3.e scrub ok 2024-04-06T03:23:20.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[29221]: audit 2024-04-06T03:23:19.037403+0000 mgr.y (mgr.24368) 90 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.296137+0000 osd.4 (osd.4) 45 : cluster 0 6.17 scrub starts 2024-04-06T03:23:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.298477+0000 osd.4 (osd.4) 46 : cluster 0 6.17 scrub ok 2024-04-06T03:23:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.817974+0000 osd.7 (osd.7) 57 : cluster 0 3.e scrub starts 2024-04-06T03:23:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:20 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.822121+0000 osd.7 (osd.7) 58 : cluster 0 3.e scrub ok 2024-04-06T03:23:20.992 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:20.992 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:20.993 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.759706+0000 osd.5 (osd.5) 51 : cluster 0 2.18 scrub starts 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.763169+0000 osd.5 (osd.5) 52 : cluster 0 2.18 scrub ok 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:19.984156+0000 mgr.y (mgr.24368) 91 : cluster 0 pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:20.742780+0000 osd.3 (osd.3) 59 : cluster 0 3.1e scrub starts 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:20.747047+0000 osd.3 (osd.3) 60 : cluster 0 3.1e scrub ok 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:20.823677+0000 osd.7 (osd.7) 59 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:23:21.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:21 smithi153 bash[23852]: cluster 2024-04-06T03:23:20.835502+0000 osd.7 (osd.7) 60 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:23:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.759706+0000 osd.5 (osd.5) 51 : cluster 0 2.18 scrub starts 2024-04-06T03:23:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.763169+0000 osd.5 (osd.5) 52 : cluster 0 2.18 scrub ok 2024-04-06T03:23:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:19.984156+0000 mgr.y (mgr.24368) 91 : cluster 0 pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:20.742780+0000 osd.3 (osd.3) 59 : cluster 0 3.1e scrub starts 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:20.747047+0000 osd.3 (osd.3) 60 : cluster 0 3.1e scrub ok 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:20.823677+0000 osd.7 (osd.7) 59 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[21467]: cluster 2024-04-06T03:23:20.835502+0000 osd.7 (osd.7) 60 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.759706+0000 osd.5 (osd.5) 51 : cluster 0 2.18 scrub starts 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.763169+0000 osd.5 (osd.5) 52 : cluster 0 2.18 scrub ok 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:19.984156+0000 mgr.y (mgr.24368) 91 : cluster 0 pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:20.742780+0000 osd.3 (osd.3) 59 : cluster 0 3.1e scrub starts 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:20.747047+0000 osd.3 (osd.3) 60 : cluster 0 3.1e scrub ok 2024-04-06T03:23:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:20.823677+0000 osd.7 (osd.7) 59 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:23:21.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:21 smithi050 bash[29221]: cluster 2024-04-06T03:23:20.835502+0000 osd.7 (osd.7) 60 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:23:22.027 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[21467]: cluster 2024-04-06T03:23:20.740412+0000 osd.5 (osd.5) 53 : cluster 0 2.3 scrub starts 2024-04-06T03:23:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[21467]: cluster 2024-04-06T03:23:20.742757+0000 osd.5 (osd.5) 54 : cluster 0 2.3 scrub ok 2024-04-06T03:23:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.798292+0000 osd.7 (osd.7) 61 : cluster 0 2.1d scrub starts 2024-04-06T03:23:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.801327+0000 osd.7 (osd.7) 62 : cluster 0 2.1d scrub ok 2024-04-06T03:23:22.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.869280+0000 osd.6 (osd.6) 35 : cluster 0 3.10 deep-scrub starts 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.875223+0000 osd.6 (osd.6) 36 : cluster 0 3.10 deep-scrub ok 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[29221]: cluster 2024-04-06T03:23:20.740412+0000 osd.5 (osd.5) 53 : cluster 0 2.3 scrub starts 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[29221]: cluster 2024-04-06T03:23:20.742757+0000 osd.5 (osd.5) 54 : cluster 0 2.3 scrub ok 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.798292+0000 osd.7 (osd.7) 61 : cluster 0 2.1d scrub starts 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.801327+0000 osd.7 (osd.7) 62 : cluster 0 2.1d scrub ok 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.869280+0000 osd.6 (osd.6) 35 : cluster 0 3.10 deep-scrub starts 2024-04-06T03:23:22.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:22 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.875223+0000 osd.6 (osd.6) 36 : cluster 0 3.10 deep-scrub ok 2024-04-06T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:22 smithi153 bash[23852]: cluster 2024-04-06T03:23:20.740412+0000 osd.5 (osd.5) 53 : cluster 0 2.3 scrub starts 2024-04-06T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:22 smithi153 bash[23852]: cluster 2024-04-06T03:23:20.742757+0000 osd.5 (osd.5) 54 : cluster 0 2.3 scrub ok 2024-04-06T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:22 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.798292+0000 osd.7 (osd.7) 61 : cluster 0 2.1d scrub starts 2024-04-06T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:22 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.801327+0000 osd.7 (osd.7) 62 : cluster 0 2.1d scrub ok 2024-04-06T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:22 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.869280+0000 osd.6 (osd.6) 35 : cluster 0 3.10 deep-scrub starts 2024-04-06T03:23:22.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:22 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.875223+0000 osd.6 (osd.6) 36 : cluster 0 3.10 deep-scrub ok 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.737273+0000 osd.5 (osd.5) 55 : cluster 0 2.12 scrub starts 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.739661+0000 osd.5 (osd.5) 56 : cluster 0 2.12 scrub ok 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:21.984980+0000 mgr.y (mgr.24368) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:22.865673+0000 osd.6 (osd.6) 37 : cluster 0 3.a scrub starts 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:22.871137+0000 osd.6 (osd.6) 38 : cluster 0 3.a scrub ok 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.072747+0000 osd.0 (osd.0) 53 : cluster 0 3.1 scrub starts 2024-04-06T03:23:23.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:23 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.076263+0000 osd.0 (osd.0) 54 : cluster 0 3.1 scrub ok 2024-04-06T03:23:23.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.737273+0000 osd.5 (osd.5) 55 : cluster 0 2.12 scrub starts 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.739661+0000 osd.5 (osd.5) 56 : cluster 0 2.12 scrub ok 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:21.984980+0000 mgr.y (mgr.24368) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:22.865673+0000 osd.6 (osd.6) 37 : cluster 0 3.a scrub starts 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:22.871137+0000 osd.6 (osd.6) 38 : cluster 0 3.a scrub ok 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.072747+0000 osd.0 (osd.0) 53 : cluster 0 3.1 scrub starts 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.076263+0000 osd.0 (osd.0) 54 : cluster 0 3.1 scrub ok 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.737273+0000 osd.5 (osd.5) 55 : cluster 0 2.12 scrub starts 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.739661+0000 osd.5 (osd.5) 56 : cluster 0 2.12 scrub ok 2024-04-06T03:23:23.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:21.984980+0000 mgr.y (mgr.24368) 92 : cluster 0 pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:23.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:22.865673+0000 osd.6 (osd.6) 37 : cluster 0 3.a scrub starts 2024-04-06T03:23:23.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:22.871137+0000 osd.6 (osd.6) 38 : cluster 0 3.a scrub ok 2024-04-06T03:23:23.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.072747+0000 osd.0 (osd.0) 53 : cluster 0 3.1 scrub starts 2024-04-06T03:23:23.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:23 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.076263+0000 osd.0 (osd.0) 54 : cluster 0 3.1 scrub ok 2024-04-06T03:23:24.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:24 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.292066+0000 osd.4 (osd.4) 47 : cluster 0 5.e scrub starts 2024-04-06T03:23:24.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:24 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.295109+0000 osd.4 (osd.4) 48 : cluster 0 5.e scrub ok 2024-04-06T03:23:24.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:24 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.587187+0000 osd.2 (osd.2) 41 : cluster 0 2.10 scrub starts 2024-04-06T03:23:24.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:24 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.589462+0000 osd.2 (osd.2) 42 : cluster 0 2.10 scrub ok 2024-04-06T03:23:24.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.292066+0000 osd.4 (osd.4) 47 : cluster 0 5.e scrub starts 2024-04-06T03:23:24.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.295109+0000 osd.4 (osd.4) 48 : cluster 0 5.e scrub ok 2024-04-06T03:23:24.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.587187+0000 osd.2 (osd.2) 41 : cluster 0 2.10 scrub starts 2024-04-06T03:23:24.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.589462+0000 osd.2 (osd.2) 42 : cluster 0 2.10 scrub ok 2024-04-06T03:23:24.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.292066+0000 osd.4 (osd.4) 47 : cluster 0 5.e scrub starts 2024-04-06T03:23:24.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.295109+0000 osd.4 (osd.4) 48 : cluster 0 5.e scrub ok 2024-04-06T03:23:24.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.587187+0000 osd.2 (osd.2) 41 : cluster 0 2.10 scrub starts 2024-04-06T03:23:24.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:24 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.589462+0000 osd.2 (osd.2) 42 : cluster 0 2.10 scrub ok 2024-04-06T03:23:25.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:25 smithi050 bash[21467]: cluster 2024-04-06T03:23:23.985775+0000 mgr.y (mgr.24368) 93 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:25.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:25 smithi050 bash[21467]: cluster 2024-04-06T03:23:24.817204+0000 osd.3 (osd.3) 61 : cluster 0 3.b scrub starts 2024-04-06T03:23:25.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:25 smithi050 bash[21467]: cluster 2024-04-06T03:23:24.821154+0000 osd.3 (osd.3) 62 : cluster 0 3.b scrub ok 2024-04-06T03:23:25.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:25 smithi050 bash[29221]: cluster 2024-04-06T03:23:23.985775+0000 mgr.y (mgr.24368) 93 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:25.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:25 smithi050 bash[29221]: cluster 2024-04-06T03:23:24.817204+0000 osd.3 (osd.3) 61 : cluster 0 3.b scrub starts 2024-04-06T03:23:25.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:25 smithi050 bash[29221]: cluster 2024-04-06T03:23:24.821154+0000 osd.3 (osd.3) 62 : cluster 0 3.b scrub ok 2024-04-06T03:23:25.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:25 smithi153 bash[23852]: cluster 2024-04-06T03:23:23.985775+0000 mgr.y (mgr.24368) 93 : cluster 0 pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:25.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:25 smithi153 bash[23852]: cluster 2024-04-06T03:23:24.817204+0000 osd.3 (osd.3) 61 : cluster 0 3.b scrub starts 2024-04-06T03:23:25.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:25 smithi153 bash[23852]: cluster 2024-04-06T03:23:24.821154+0000 osd.3 (osd.3) 62 : cluster 0 3.b scrub ok 2024-04-06T03:23:26.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:26 smithi050 bash[21467]: cluster 2024-04-06T03:23:25.884709+0000 osd.6 (osd.6) 39 : cluster 0 5.11 scrub starts 2024-04-06T03:23:26.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:26 smithi050 bash[21467]: cluster 2024-04-06T03:23:25.887143+0000 osd.6 (osd.6) 40 : cluster 0 5.11 scrub ok 2024-04-06T03:23:26.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:26 smithi050 bash[29221]: cluster 2024-04-06T03:23:25.884709+0000 osd.6 (osd.6) 39 : cluster 0 5.11 scrub starts 2024-04-06T03:23:26.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:26 smithi050 bash[29221]: cluster 2024-04-06T03:23:25.887143+0000 osd.6 (osd.6) 40 : cluster 0 5.11 scrub ok 2024-04-06T03:23:27.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:26 smithi153 bash[23852]: cluster 2024-04-06T03:23:25.884709+0000 osd.6 (osd.6) 39 : cluster 0 5.11 scrub starts 2024-04-06T03:23:27.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:26 smithi153 bash[23852]: cluster 2024-04-06T03:23:25.887143+0000 osd.6 (osd.6) 40 : cluster 0 5.11 scrub ok 2024-04-06T03:23:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:27 smithi050 bash[21467]: cluster 2024-04-06T03:23:25.986476+0000 mgr.y (mgr.24368) 94 : cluster 0 pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:27 smithi050 bash[21467]: cluster 2024-04-06T03:23:26.861501+0000 osd.6 (osd.6) 41 : cluster 0 2.17 deep-scrub starts 2024-04-06T03:23:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:27 smithi050 bash[21467]: cluster 2024-04-06T03:23:26.863680+0000 osd.6 (osd.6) 42 : cluster 0 2.17 deep-scrub ok 2024-04-06T03:23:27.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:27 smithi050 bash[29221]: cluster 2024-04-06T03:23:25.986476+0000 mgr.y (mgr.24368) 94 : cluster 0 pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:27.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:27 smithi050 bash[29221]: cluster 2024-04-06T03:23:26.861501+0000 osd.6 (osd.6) 41 : cluster 0 2.17 deep-scrub starts 2024-04-06T03:23:27.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:27 smithi050 bash[29221]: cluster 2024-04-06T03:23:26.863680+0000 osd.6 (osd.6) 42 : cluster 0 2.17 deep-scrub ok 2024-04-06T03:23:28.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:27 smithi153 bash[23852]: cluster 2024-04-06T03:23:25.986476+0000 mgr.y (mgr.24368) 94 : cluster 0 pgmap v65: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:28.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:27 smithi153 bash[23852]: cluster 2024-04-06T03:23:26.861501+0000 osd.6 (osd.6) 41 : cluster 0 2.17 deep-scrub starts 2024-04-06T03:23:28.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:27 smithi153 bash[23852]: cluster 2024-04-06T03:23:26.863680+0000 osd.6 (osd.6) 42 : cluster 0 2.17 deep-scrub ok 2024-04-06T03:23:28.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[29221]: cluster 2024-04-06T03:23:27.843280+0000 osd.5 (osd.5) 57 : cluster 0 2.2 scrub starts 2024-04-06T03:23:28.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[29221]: cluster 2024-04-06T03:23:27.845700+0000 osd.5 (osd.5) 58 : cluster 0 2.2 scrub ok 2024-04-06T03:23:28.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[29221]: cluster 2024-04-06T03:23:27.987096+0000 mgr.y (mgr.24368) 95 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:23:28.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[29221]: cluster 2024-04-06T03:23:28.065729+0000 osd.0 (osd.0) 55 : cluster 0 6.16 deep-scrub starts 2024-04-06T03:23:28.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[29221]: cluster 2024-04-06T03:23:28.067960+0000 osd.0 (osd.0) 56 : cluster 0 6.16 deep-scrub ok 2024-04-06T03:23:28.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[21467]: cluster 2024-04-06T03:23:27.843280+0000 osd.5 (osd.5) 57 : cluster 0 2.2 scrub starts 2024-04-06T03:23:28.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[21467]: cluster 2024-04-06T03:23:27.845700+0000 osd.5 (osd.5) 58 : cluster 0 2.2 scrub ok 2024-04-06T03:23:28.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[21467]: cluster 2024-04-06T03:23:27.987096+0000 mgr.y (mgr.24368) 95 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:23:28.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[21467]: cluster 2024-04-06T03:23:28.065729+0000 osd.0 (osd.0) 55 : cluster 0 6.16 deep-scrub starts 2024-04-06T03:23:28.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:28 smithi050 bash[21467]: cluster 2024-04-06T03:23:28.067960+0000 osd.0 (osd.0) 56 : cluster 0 6.16 deep-scrub ok 2024-04-06T03:23:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:28 smithi153 bash[23852]: cluster 2024-04-06T03:23:27.843280+0000 osd.5 (osd.5) 57 : cluster 0 2.2 scrub starts 2024-04-06T03:23:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:28 smithi153 bash[23852]: cluster 2024-04-06T03:23:27.845700+0000 osd.5 (osd.5) 58 : cluster 0 2.2 scrub ok 2024-04-06T03:23:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:28 smithi153 bash[23852]: cluster 2024-04-06T03:23:27.987096+0000 mgr.y (mgr.24368) 95 : cluster 0 pgmap v66: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:23:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:28 smithi153 bash[23852]: cluster 2024-04-06T03:23:28.065729+0000 osd.0 (osd.0) 55 : cluster 0 6.16 deep-scrub starts 2024-04-06T03:23:29.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:28 smithi153 bash[23852]: cluster 2024-04-06T03:23:28.067960+0000 osd.0 (osd.0) 56 : cluster 0 6.16 deep-scrub ok 2024-04-06T03:23:29.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:23:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:23:29.677 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:23:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:23:29.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[29221]: cluster 2024-04-06T03:23:28.867036+0000 osd.5 (osd.5) 59 : cluster 0 5.7 scrub starts 2024-04-06T03:23:29.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[29221]: cluster 2024-04-06T03:23:28.869058+0000 osd.5 (osd.5) 60 : cluster 0 5.7 scrub ok 2024-04-06T03:23:29.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[29221]: audit 2024-04-06T03:23:29.048652+0000 mgr.y (mgr.24368) 96 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[21467]: cluster 2024-04-06T03:23:28.867036+0000 osd.5 (osd.5) 59 : cluster 0 5.7 scrub starts 2024-04-06T03:23:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[21467]: cluster 2024-04-06T03:23:28.869058+0000 osd.5 (osd.5) 60 : cluster 0 5.7 scrub ok 2024-04-06T03:23:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:29 smithi050 bash[21467]: audit 2024-04-06T03:23:29.048652+0000 mgr.y (mgr.24368) 96 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:30.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:29 smithi153 bash[23852]: cluster 2024-04-06T03:23:28.867036+0000 osd.5 (osd.5) 59 : cluster 0 5.7 scrub starts 2024-04-06T03:23:30.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:29 smithi153 bash[23852]: cluster 2024-04-06T03:23:28.869058+0000 osd.5 (osd.5) 60 : cluster 0 5.7 scrub ok 2024-04-06T03:23:30.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:29 smithi153 bash[23852]: audit 2024-04-06T03:23:29.048652+0000 mgr.y (mgr.24368) 96 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[21467]: cluster 2024-04-06T03:23:29.334336+0000 osd.4 (osd.4) 49 : cluster 0 3.14 scrub starts 2024-04-06T03:23:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[21467]: cluster 2024-04-06T03:23:29.339042+0000 osd.4 (osd.4) 50 : cluster 0 3.14 scrub ok 2024-04-06T03:23:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[21467]: cluster 2024-04-06T03:23:29.524159+0000 osd.2 (osd.2) 43 : cluster 0 5.d deep-scrub starts 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[21467]: cluster 2024-04-06T03:23:29.527994+0000 osd.2 (osd.2) 44 : cluster 0 5.d deep-scrub ok 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[21467]: cluster 2024-04-06T03:23:29.988559+0000 mgr.y (mgr.24368) 97 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[29221]: cluster 2024-04-06T03:23:29.334336+0000 osd.4 (osd.4) 49 : cluster 0 3.14 scrub starts 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[29221]: cluster 2024-04-06T03:23:29.339042+0000 osd.4 (osd.4) 50 : cluster 0 3.14 scrub ok 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[29221]: cluster 2024-04-06T03:23:29.524159+0000 osd.2 (osd.2) 43 : cluster 0 5.d deep-scrub starts 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[29221]: cluster 2024-04-06T03:23:29.527994+0000 osd.2 (osd.2) 44 : cluster 0 5.d deep-scrub ok 2024-04-06T03:23:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:30 smithi050 bash[29221]: cluster 2024-04-06T03:23:29.988559+0000 mgr.y (mgr.24368) 97 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:31.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:30 smithi153 bash[23852]: cluster 2024-04-06T03:23:29.334336+0000 osd.4 (osd.4) 49 : cluster 0 3.14 scrub starts 2024-04-06T03:23:31.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:30 smithi153 bash[23852]: cluster 2024-04-06T03:23:29.339042+0000 osd.4 (osd.4) 50 : cluster 0 3.14 scrub ok 2024-04-06T03:23:31.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:30 smithi153 bash[23852]: cluster 2024-04-06T03:23:29.524159+0000 osd.2 (osd.2) 43 : cluster 0 5.d deep-scrub starts 2024-04-06T03:23:31.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:30 smithi153 bash[23852]: cluster 2024-04-06T03:23:29.527994+0000 osd.2 (osd.2) 44 : cluster 0 5.d deep-scrub ok 2024-04-06T03:23:31.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:30 smithi153 bash[23852]: cluster 2024-04-06T03:23:29.988559+0000 mgr.y (mgr.24368) 97 : cluster 0 pgmap v67: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:31.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:31 smithi050 bash[21467]: cluster 2024-04-06T03:23:30.860022+0000 osd.3 (osd.3) 63 : cluster 0 6.1f scrub starts 2024-04-06T03:23:31.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:31 smithi050 bash[21467]: cluster 2024-04-06T03:23:30.862636+0000 osd.3 (osd.3) 64 : cluster 0 6.1f scrub ok 2024-04-06T03:23:31.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:31 smithi050 bash[29221]: cluster 2024-04-06T03:23:30.860022+0000 osd.3 (osd.3) 63 : cluster 0 6.1f scrub starts 2024-04-06T03:23:31.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:31 smithi050 bash[29221]: cluster 2024-04-06T03:23:30.862636+0000 osd.3 (osd.3) 64 : cluster 0 6.1f scrub ok 2024-04-06T03:23:32.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:31 smithi153 bash[23852]: cluster 2024-04-06T03:23:30.860022+0000 osd.3 (osd.3) 63 : cluster 0 6.1f scrub starts 2024-04-06T03:23:32.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:31 smithi153 bash[23852]: cluster 2024-04-06T03:23:30.862636+0000 osd.3 (osd.3) 64 : cluster 0 6.1f scrub ok 2024-04-06T03:23:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:32 smithi050 bash[21467]: cluster 2024-04-06T03:23:31.953060+0000 osd.5 (osd.5) 61 : cluster 0 5.16 scrub starts 2024-04-06T03:23:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:32 smithi050 bash[21467]: cluster 2024-04-06T03:23:31.955356+0000 osd.5 (osd.5) 62 : cluster 0 5.16 scrub ok 2024-04-06T03:23:32.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:32 smithi050 bash[21467]: cluster 2024-04-06T03:23:31.989426+0000 mgr.y (mgr.24368) 98 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:32.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:32 smithi050 bash[29221]: cluster 2024-04-06T03:23:31.953060+0000 osd.5 (osd.5) 61 : cluster 0 5.16 scrub starts 2024-04-06T03:23:32.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:32 smithi050 bash[29221]: cluster 2024-04-06T03:23:31.955356+0000 osd.5 (osd.5) 62 : cluster 0 5.16 scrub ok 2024-04-06T03:23:32.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:32 smithi050 bash[29221]: cluster 2024-04-06T03:23:31.989426+0000 mgr.y (mgr.24368) 98 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:33.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:32 smithi153 bash[23852]: cluster 2024-04-06T03:23:31.953060+0000 osd.5 (osd.5) 61 : cluster 0 5.16 scrub starts 2024-04-06T03:23:33.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:32 smithi153 bash[23852]: cluster 2024-04-06T03:23:31.955356+0000 osd.5 (osd.5) 62 : cluster 0 5.16 scrub ok 2024-04-06T03:23:33.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:32 smithi153 bash[23852]: cluster 2024-04-06T03:23:31.989426+0000 mgr.y (mgr.24368) 98 : cluster 0 pgmap v68: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:23:33.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:33 smithi050 bash[21467]: audit 2024-04-06T03:23:33.247784+0000 mon.a (mon.0) 953 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:33.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:33 smithi050 bash[29221]: audit 2024-04-06T03:23:33.247784+0000 mon.a (mon.0) 953 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:34.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:33 smithi153 bash[23852]: audit 2024-04-06T03:23:33.247784+0000 mon.a (mon.0) 953 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.817465+0000 osd.7 (osd.7) 63 : cluster 0 5.9 scrub starts 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.820679+0000 osd.7 (osd.7) 64 : cluster 0 5.9 scrub ok 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.866579+0000 osd.6 (osd.6) 43 : cluster 0 5.1f scrub starts 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.869518+0000 osd.6 (osd.6) 44 : cluster 0 5.1f scrub ok 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.950677+0000 osd.5 (osd.5) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.950772+0000 osd.5 (osd.5) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:33.990950+0000 mgr.y (mgr.24368) 99 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:34.007571+0000 osd.0 (osd.0) 57 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[21467]: cluster 2024-04-06T03:23:34.009804+0000 osd.0 (osd.0) 58 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:23:34.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.817465+0000 osd.7 (osd.7) 63 : cluster 0 5.9 scrub starts 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.820679+0000 osd.7 (osd.7) 64 : cluster 0 5.9 scrub ok 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.866579+0000 osd.6 (osd.6) 43 : cluster 0 5.1f scrub starts 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.869518+0000 osd.6 (osd.6) 44 : cluster 0 5.1f scrub ok 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.950677+0000 osd.5 (osd.5) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.950772+0000 osd.5 (osd.5) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:33.990950+0000 mgr.y (mgr.24368) 99 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:34.007571+0000 osd.0 (osd.0) 57 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:23:34.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:34 smithi050 bash[29221]: cluster 2024-04-06T03:23:34.009804+0000 osd.0 (osd.0) 58 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.817465+0000 osd.7 (osd.7) 63 : cluster 0 5.9 scrub starts 2024-04-06T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.820679+0000 osd.7 (osd.7) 64 : cluster 0 5.9 scrub ok 2024-04-06T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.866579+0000 osd.6 (osd.6) 43 : cluster 0 5.1f scrub starts 2024-04-06T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.869518+0000 osd.6 (osd.6) 44 : cluster 0 5.1f scrub ok 2024-04-06T03:23:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.950677+0000 osd.5 (osd.5) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.950772+0000 osd.5 (osd.5) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:33.990950+0000 mgr.y (mgr.24368) 99 : cluster 0 pgmap v69: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:34.007571+0000 osd.0 (osd.0) 57 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:23:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:34 smithi153 bash[23852]: cluster 2024-04-06T03:23:34.009804+0000 osd.0 (osd.0) 58 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:23:35.417 INFO:teuthology.orchestra.run.smithi050.stdout:34359738484 2024-04-06T03:23:35.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:35 smithi050 bash[21467]: cluster 2024-04-06T03:23:34.609593+0000 mon.a (mon.0) 954 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-04-06T03:23:35.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:35 smithi050 bash[21467]: audit 2024-04-06T03:23:35.426948+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.50:0/4045056504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:23:35.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:35 smithi050 bash[29221]: cluster 2024-04-06T03:23:34.609593+0000 mon.a (mon.0) 954 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-04-06T03:23:35.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:35 smithi050 bash[29221]: audit 2024-04-06T03:23:35.426948+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.50:0/4045056504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:23:36.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:35 smithi153 bash[23852]: cluster 2024-04-06T03:23:34.609593+0000 mon.a (mon.0) 954 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2024-04-06T03:23:36.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:35 smithi153 bash[23852]: audit 2024-04-06T03:23:35.426948+0000 mon.c (mon.2) 24 : audit 0 from='client.? 172.21.15.50:0/4045056504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:23:36.570 INFO:teuthology.orchestra.run.smithi050.stdout:197568495681 2024-04-06T03:23:36.934 INFO:teuthology.orchestra.run.smithi050.stdout:120259084372 2024-04-06T03:23:36.948 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.362021+0000 osd.4 (osd.4) 51 : cluster 0 6.2 scrub starts 2024-04-06T03:23:36.948 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.364346+0000 osd.4 (osd.4) 52 : cluster 0 6.2 scrub ok 2024-04-06T03:23:36.948 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.676191+0000 osd.1 (osd.1) 61 : cluster 0 3.0 scrub starts 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.681895+0000 osd.1 (osd.1) 62 : cluster 0 3.0 scrub ok 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.832510+0000 osd.6 (osd.6) 45 : cluster 0 2.11 scrub starts 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.834833+0000 osd.6 (osd.6) 46 : cluster 0 2.11 scrub ok 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: cluster 2024-04-06T03:23:35.991690+0000 mgr.y (mgr.24368) 100 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[21467]: audit 2024-04-06T03:23:36.580022+0000 mon.a (mon.0) 955 : audit 0 from='client.? 172.21.15.50:0/1576282067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.362021+0000 osd.4 (osd.4) 51 : cluster 0 6.2 scrub starts 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.364346+0000 osd.4 (osd.4) 52 : cluster 0 6.2 scrub ok 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.676191+0000 osd.1 (osd.1) 61 : cluster 0 3.0 scrub starts 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.681895+0000 osd.1 (osd.1) 62 : cluster 0 3.0 scrub ok 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.832510+0000 osd.6 (osd.6) 45 : cluster 0 2.11 scrub starts 2024-04-06T03:23:36.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.834833+0000 osd.6 (osd.6) 46 : cluster 0 2.11 scrub ok 2024-04-06T03:23:36.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: cluster 2024-04-06T03:23:35.991690+0000 mgr.y (mgr.24368) 100 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:36.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:36 smithi050 bash[29221]: audit 2024-04-06T03:23:36.580022+0000 mon.a (mon.0) 955 : audit 0 from='client.? 172.21.15.50:0/1576282067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:23:37.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.362021+0000 osd.4 (osd.4) 51 : cluster 0 6.2 scrub starts 2024-04-06T03:23:37.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.364346+0000 osd.4 (osd.4) 52 : cluster 0 6.2 scrub ok 2024-04-06T03:23:37.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.676191+0000 osd.1 (osd.1) 61 : cluster 0 3.0 scrub starts 2024-04-06T03:23:37.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.681895+0000 osd.1 (osd.1) 62 : cluster 0 3.0 scrub ok 2024-04-06T03:23:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.832510+0000 osd.6 (osd.6) 45 : cluster 0 2.11 scrub starts 2024-04-06T03:23:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.834833+0000 osd.6 (osd.6) 46 : cluster 0 2.11 scrub ok 2024-04-06T03:23:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: cluster 2024-04-06T03:23:35.991690+0000 mgr.y (mgr.24368) 100 : cluster 0 pgmap v71: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:23:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:36 smithi153 bash[23852]: audit 2024-04-06T03:23:36.580022+0000 mon.a (mon.0) 955 : audit 0 from='client.? 172.21.15.50:0/1576282067' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:23:37.112 INFO:teuthology.orchestra.run.smithi050.stdout:244813135925 2024-04-06T03:23:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:36.388746+0000 osd.4 (osd.4) 53 : cluster 0 6.e scrub starts 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:36.390792+0000 osd.4 (osd.4) 54 : cluster 0 6.e scrub ok 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:36.845906+0000 osd.6 (osd.6) 47 : cluster 0 2.1a scrub starts 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:36.848353+0000 osd.6 (osd.6) 48 : cluster 0 2.1a scrub ok 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:36.893270+0000 osd.5 (osd.5) 65 : cluster 0 3.1c deep-scrub starts 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:36.905488+0000 osd.5 (osd.5) 66 : cluster 0 3.1c deep-scrub ok 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: audit 2024-04-06T03:23:36.944005+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.50:0/2649818204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: audit 2024-04-06T03:23:37.121178+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.50:0/2211857135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.255168+0000 osd.1 (osd.1) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.255271+0000 osd.1 (osd.1) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:36.388746+0000 osd.4 (osd.4) 53 : cluster 0 6.e scrub starts 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:36.390792+0000 osd.4 (osd.4) 54 : cluster 0 6.e scrub ok 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:36.845906+0000 osd.6 (osd.6) 47 : cluster 0 2.1a scrub starts 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:36.848353+0000 osd.6 (osd.6) 48 : cluster 0 2.1a scrub ok 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:36.893270+0000 osd.5 (osd.5) 65 : cluster 0 3.1c deep-scrub starts 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:36.905488+0000 osd.5 (osd.5) 66 : cluster 0 3.1c deep-scrub ok 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: audit 2024-04-06T03:23:36.944005+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.50:0/2649818204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: audit 2024-04-06T03:23:37.121178+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.50:0/2211857135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.255168+0000 osd.1 (osd.1) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:37 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.255271+0000 osd.1 (osd.1) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:38.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:36.388746+0000 osd.4 (osd.4) 53 : cluster 0 6.e scrub starts 2024-04-06T03:23:38.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:36.390792+0000 osd.4 (osd.4) 54 : cluster 0 6.e scrub ok 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:36.845906+0000 osd.6 (osd.6) 47 : cluster 0 2.1a scrub starts 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:36.848353+0000 osd.6 (osd.6) 48 : cluster 0 2.1a scrub ok 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:36.893270+0000 osd.5 (osd.5) 65 : cluster 0 3.1c deep-scrub starts 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:36.905488+0000 osd.5 (osd.5) 66 : cluster 0 3.1c deep-scrub ok 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: audit 2024-04-06T03:23:36.944005+0000 mon.c (mon.2) 25 : audit 0 from='client.? 172.21.15.50:0/2649818204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: audit 2024-04-06T03:23:37.121178+0000 mon.c (mon.2) 26 : audit 0 from='client.? 172.21.15.50:0/2211857135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.255168+0000 osd.1 (osd.1) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:37 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.255271+0000 osd.1 (osd.1) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:38.132 INFO:teuthology.orchestra.run.smithi050.stdout:81604378720 2024-04-06T03:23:38.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738479 got 34359738484 for osd.0 2024-04-06T03:23:38.445 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:38.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.627042+0000 mon.a (mon.0) 956 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-04-06T03:23:38.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.690595+0000 osd.1 (osd.1) 65 : cluster 0 2.6 scrub starts 2024-04-06T03:23:38.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.757410+0000 osd.1 (osd.1) 66 : cluster 0 2.6 scrub ok 2024-04-06T03:23:38.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.851940+0000 osd.5 (osd.5) 67 : cluster 0 6.7 scrub starts 2024-04-06T03:23:38.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.853999+0000 osd.5 (osd.5) 68 : cluster 0 6.7 scrub ok 2024-04-06T03:23:38.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: cluster 2024-04-06T03:23:37.992179+0000 mgr.y (mgr.24368) 101 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[21467]: audit 2024-04-06T03:23:38.141671+0000 mon.a (mon.0) 957 : audit 0 from='client.? 172.21.15.50:0/1141516941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.627042+0000 mon.a (mon.0) 956 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.690595+0000 osd.1 (osd.1) 65 : cluster 0 2.6 scrub starts 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.757410+0000 osd.1 (osd.1) 66 : cluster 0 2.6 scrub ok 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.851940+0000 osd.5 (osd.5) 67 : cluster 0 6.7 scrub starts 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.853999+0000 osd.5 (osd.5) 68 : cluster 0 6.7 scrub ok 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: cluster 2024-04-06T03:23:37.992179+0000 mgr.y (mgr.24368) 101 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:38.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:38 smithi050 bash[29221]: audit 2024-04-06T03:23:38.141671+0000 mon.a (mon.0) 957 : audit 0 from='client.? 172.21.15.50:0/1141516941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:23:38.984 INFO:teuthology.orchestra.run.smithi050.stdout:55834574954 2024-04-06T03:23:39.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.627042+0000 mon.a (mon.0) 956 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2024-04-06T03:23:39.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.690595+0000 osd.1 (osd.1) 65 : cluster 0 2.6 scrub starts 2024-04-06T03:23:39.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.757410+0000 osd.1 (osd.1) 66 : cluster 0 2.6 scrub ok 2024-04-06T03:23:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.851940+0000 osd.5 (osd.5) 67 : cluster 0 6.7 scrub starts 2024-04-06T03:23:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.853999+0000 osd.5 (osd.5) 68 : cluster 0 6.7 scrub ok 2024-04-06T03:23:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: cluster 2024-04-06T03:23:37.992179+0000 mgr.y (mgr.24368) 101 : cluster 0 pgmap v73: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:38 smithi153 bash[23852]: audit 2024-04-06T03:23:38.141671+0000 mon.a (mon.0) 957 : audit 0 from='client.? 172.21.15.50:0/1141516941' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:23:39.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:23:39.621 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:23:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:23:39.689 INFO:teuthology.orchestra.run.smithi050.stdout:150323855435 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: cluster 2024-04-06T03:23:38.880203+0000 osd.3 (osd.3) 65 : cluster 0 6.c deep-scrub starts 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: cluster 2024-04-06T03:23:38.881648+0000 osd.5 (osd.5) 69 : cluster 0 6.d scrub starts 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: cluster 2024-04-06T03:23:38.882262+0000 osd.3 (osd.3) 66 : cluster 0 6.c deep-scrub ok 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: cluster 2024-04-06T03:23:38.883725+0000 osd.5 (osd.5) 70 : cluster 0 6.d scrub ok 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: audit 2024-04-06T03:23:38.993620+0000 mon.c (mon.2) 27 : audit 0 from='client.? 172.21.15.50:0/533771983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.029214+0000 osd.0 (osd.0) 59 : cluster 0 3.6 scrub starts 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.032614+0000 osd.0 (osd.0) 60 : cluster 0 3.6 scrub ok 2024-04-06T03:23:39.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[21467]: audit 2024-04-06T03:23:39.059028+0000 mgr.y (mgr.24368) 102 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: cluster 2024-04-06T03:23:38.880203+0000 osd.3 (osd.3) 65 : cluster 0 6.c deep-scrub starts 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: cluster 2024-04-06T03:23:38.881648+0000 osd.5 (osd.5) 69 : cluster 0 6.d scrub starts 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: cluster 2024-04-06T03:23:38.882262+0000 osd.3 (osd.3) 66 : cluster 0 6.c deep-scrub ok 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: cluster 2024-04-06T03:23:38.883725+0000 osd.5 (osd.5) 70 : cluster 0 6.d scrub ok 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: audit 2024-04-06T03:23:38.993620+0000 mon.c (mon.2) 27 : audit 0 from='client.? 172.21.15.50:0/533771983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.029214+0000 osd.0 (osd.0) 59 : cluster 0 3.6 scrub starts 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.032614+0000 osd.0 (osd.0) 60 : cluster 0 3.6 scrub ok 2024-04-06T03:23:39.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:39 smithi050 bash[29221]: audit 2024-04-06T03:23:39.059028+0000 mgr.y (mgr.24368) 102 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:40.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: cluster 2024-04-06T03:23:38.880203+0000 osd.3 (osd.3) 65 : cluster 0 6.c deep-scrub starts 2024-04-06T03:23:40.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: cluster 2024-04-06T03:23:38.881648+0000 osd.5 (osd.5) 69 : cluster 0 6.d scrub starts 2024-04-06T03:23:40.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: cluster 2024-04-06T03:23:38.882262+0000 osd.3 (osd.3) 66 : cluster 0 6.c deep-scrub ok 2024-04-06T03:23:40.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: cluster 2024-04-06T03:23:38.883725+0000 osd.5 (osd.5) 70 : cluster 0 6.d scrub ok 2024-04-06T03:23:40.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: audit 2024-04-06T03:23:38.993620+0000 mon.c (mon.2) 27 : audit 0 from='client.? 172.21.15.50:0/533771983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:23:40.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.029214+0000 osd.0 (osd.0) 59 : cluster 0 3.6 scrub starts 2024-04-06T03:23:40.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.032614+0000 osd.0 (osd.0) 60 : cluster 0 3.6 scrub ok 2024-04-06T03:23:40.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:39 smithi153 bash[23852]: audit 2024-04-06T03:23:39.059028+0000 mgr.y (mgr.24368) 102 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:40.062 INFO:teuthology.orchestra.run.smithi050.stdout:296352743467 2024-04-06T03:23:40.612 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495675 got 197568495681 for osd.5 2024-04-06T03:23:40.612 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:40.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: audit 2024-04-06T03:23:39.698426+0000 mon.c (mon.2) 28 : audit 0 from='client.? 172.21.15.50:0/1921535998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:23:40.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.774632+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.776923+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.988230+0000 osd.0 (osd.0) 61 : cluster 0 6.9 scrub starts 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.989993+0000 osd.0 (osd.0) 62 : cluster 0 6.9 scrub ok 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: cluster 2024-04-06T03:23:39.992830+0000 mgr.y (mgr.24368) 103 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[21467]: audit 2024-04-06T03:23:40.071892+0000 mon.c (mon.2) 29 : audit 0 from='client.? 172.21.15.50:0/2824013332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: audit 2024-04-06T03:23:39.698426+0000 mon.c (mon.2) 28 : audit 0 from='client.? 172.21.15.50:0/1921535998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.774632+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.776923+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.988230+0000 osd.0 (osd.0) 61 : cluster 0 6.9 scrub starts 2024-04-06T03:23:40.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.989993+0000 osd.0 (osd.0) 62 : cluster 0 6.9 scrub ok 2024-04-06T03:23:40.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: cluster 2024-04-06T03:23:39.992830+0000 mgr.y (mgr.24368) 103 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:40.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:40 smithi050 bash[29221]: audit 2024-04-06T03:23:40.071892+0000 mon.c (mon.2) 29 : audit 0 from='client.? 172.21.15.50:0/2824013332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:23:41.034 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135920 got 244813135925 for osd.6 2024-04-06T03:23:41.034 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: audit 2024-04-06T03:23:39.698426+0000 mon.c (mon.2) 28 : audit 0 from='client.? 172.21.15.50:0/1921535998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.774632+0000 osd.6 (osd.6) 49 : cluster 0 5.2 scrub starts 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.776923+0000 osd.6 (osd.6) 50 : cluster 0 5.2 scrub ok 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.988230+0000 osd.0 (osd.0) 61 : cluster 0 6.9 scrub starts 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.989993+0000 osd.0 (osd.0) 62 : cluster 0 6.9 scrub ok 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: cluster 2024-04-06T03:23:39.992830+0000 mgr.y (mgr.24368) 103 : cluster 0 pgmap v74: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:40 smithi153 bash[23852]: audit 2024-04-06T03:23:40.071892+0000 mon.c (mon.2) 29 : audit 0 from='client.? 172.21.15.50:0/2824013332' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:41 smithi050 bash[29221]: cluster 2024-04-06T03:23:40.953706+0000 osd.0 (osd.0) 63 : cluster 0 2.13 scrub starts 2024-04-06T03:23:41.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:41 smithi050 bash[29221]: cluster 2024-04-06T03:23:40.956154+0000 osd.0 (osd.0) 64 : cluster 0 2.13 scrub ok 2024-04-06T03:23:41.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:41 smithi050 bash[21467]: cluster 2024-04-06T03:23:40.953706+0000 osd.0 (osd.0) 63 : cluster 0 2.13 scrub starts 2024-04-06T03:23:41.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:41 smithi050 bash[21467]: cluster 2024-04-06T03:23:40.956154+0000 osd.0 (osd.0) 64 : cluster 0 2.13 scrub ok 2024-04-06T03:23:41.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378715 got 81604378720 for osd.2 2024-04-06T03:23:41.962 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:42.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:41 smithi153 bash[23852]: cluster 2024-04-06T03:23:40.953706+0000 osd.0 (osd.0) 63 : cluster 0 2.13 scrub starts 2024-04-06T03:23:42.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:41 smithi153 bash[23852]: cluster 2024-04-06T03:23:40.956154+0000 osd.0 (osd.0) 64 : cluster 0 2.13 scrub ok 2024-04-06T03:23:42.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084367 got 120259084372 for osd.3 2024-04-06T03:23:42.059 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:42.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:42 smithi050 bash[21467]: cluster 2024-04-06T03:23:41.698451+0000 osd.7 (osd.7) 65 : cluster 0 3.f scrub starts 2024-04-06T03:23:42.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:42 smithi050 bash[21467]: cluster 2024-04-06T03:23:41.706561+0000 osd.7 (osd.7) 66 : cluster 0 3.f scrub ok 2024-04-06T03:23:42.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:42 smithi050 bash[21467]: cluster 2024-04-06T03:23:41.993343+0000 mgr.y (mgr.24368) 104 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:42.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:42 smithi050 bash[29221]: cluster 2024-04-06T03:23:41.698451+0000 osd.7 (osd.7) 65 : cluster 0 3.f scrub starts 2024-04-06T03:23:42.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:42 smithi050 bash[29221]: cluster 2024-04-06T03:23:41.706561+0000 osd.7 (osd.7) 66 : cluster 0 3.f scrub ok 2024-04-06T03:23:42.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:42 smithi050 bash[29221]: cluster 2024-04-06T03:23:41.993343+0000 mgr.y (mgr.24368) 104 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:42.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743462 got 296352743467 for osd.7 2024-04-06T03:23:42.913 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:42.981 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574949 got 55834574954 for osd.1 2024-04-06T03:23:42.982 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:42.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855431 got 150323855435 for osd.4 2024-04-06T03:23:42.989 DEBUG:teuthology.parallel:result is None 2024-04-06T03:23:42.990 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-06T03:23:42.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:23:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:42 smithi153 bash[23852]: cluster 2024-04-06T03:23:41.698451+0000 osd.7 (osd.7) 65 : cluster 0 3.f scrub starts 2024-04-06T03:23:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:42 smithi153 bash[23852]: cluster 2024-04-06T03:23:41.706561+0000 osd.7 (osd.7) 66 : cluster 0 3.f scrub ok 2024-04-06T03:23:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:42 smithi153 bash[23852]: cluster 2024-04-06T03:23:41.993343+0000 mgr.y (mgr.24368) 104 : cluster 0 pgmap v75: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:44.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:44 smithi050 bash[21467]: cluster 2024-04-06T03:23:43.920701+0000 osd.3 (osd.3) 67 : cluster 0 2.19 scrub starts 2024-04-06T03:23:44.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:44 smithi050 bash[21467]: cluster 2024-04-06T03:23:43.923780+0000 osd.3 (osd.3) 68 : cluster 0 2.19 scrub ok 2024-04-06T03:23:44.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:44 smithi050 bash[29221]: cluster 2024-04-06T03:23:43.920701+0000 osd.3 (osd.3) 67 : cluster 0 2.19 scrub starts 2024-04-06T03:23:44.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:44 smithi050 bash[29221]: cluster 2024-04-06T03:23:43.923780+0000 osd.3 (osd.3) 68 : cluster 0 2.19 scrub ok 2024-04-06T03:23:44.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:44 smithi153 bash[23852]: cluster 2024-04-06T03:23:43.920701+0000 osd.3 (osd.3) 67 : cluster 0 2.19 scrub starts 2024-04-06T03:23:44.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:44 smithi153 bash[23852]: cluster 2024-04-06T03:23:43.923780+0000 osd.3 (osd.3) 68 : cluster 0 2.19 scrub ok 2024-04-06T03:23:45.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:43.563496+0000 osd.2 (osd.2) 45 : cluster 0 5.b deep-scrub starts 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:43.565919+0000 osd.2 (osd.2) 46 : cluster 0 5.b deep-scrub ok 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:43.994718+0000 mgr.y (mgr.24368) 105 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.158644+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.158750+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.385011+0000 mon.a (mon.0) 958 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.973550+0000 osd.3 (osd.3) 69 : cluster 0 6.b scrub starts 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.976005+0000 osd.3 (osd.3) 70 : cluster 0 6.b scrub ok 2024-04-06T03:23:45.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:43.563496+0000 osd.2 (osd.2) 45 : cluster 0 5.b deep-scrub starts 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:43.565919+0000 osd.2 (osd.2) 46 : cluster 0 5.b deep-scrub ok 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:43.994718+0000 mgr.y (mgr.24368) 105 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.158644+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.158750+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.385011+0000 mon.a (mon.0) 958 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.973550+0000 osd.3 (osd.3) 69 : cluster 0 6.b scrub starts 2024-04-06T03:23:45.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:45 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.976005+0000 osd.3 (osd.3) 70 : cluster 0 6.b scrub ok 2024-04-06T03:23:45.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:43.563496+0000 osd.2 (osd.2) 45 : cluster 0 5.b deep-scrub starts 2024-04-06T03:23:45.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:43.565919+0000 osd.2 (osd.2) 46 : cluster 0 5.b deep-scrub ok 2024-04-06T03:23:45.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:43.994718+0000 mgr.y (mgr.24368) 105 : cluster 0 pgmap v76: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:23:45.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.158644+0000 osd.7 (osd.7) 67 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:45.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.158750+0000 osd.7 (osd.7) 68 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:45.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.385011+0000 mon.a (mon.0) 958 : cluster 0 osdmap e114: 8 total, 8 up, 8 in 2024-04-06T03:23:45.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.973550+0000 osd.3 (osd.3) 69 : cluster 0 6.b scrub starts 2024-04-06T03:23:45.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:45 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.976005+0000 osd.3 (osd.3) 70 : cluster 0 6.b scrub ok 2024-04-06T03:23:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.523034+0000 osd.2 (osd.2) 47 : cluster 0 5.a deep-scrub starts 2024-04-06T03:23:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[21467]: cluster 2024-04-06T03:23:44.525605+0000 osd.2 (osd.2) 48 : cluster 0 5.a deep-scrub ok 2024-04-06T03:23:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[21467]: cluster 2024-04-06T03:23:45.584018+0000 osd.3 (osd.3) 71 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[21467]: cluster 2024-04-06T03:23:45.584136+0000 osd.3 (osd.3) 72 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.523034+0000 osd.2 (osd.2) 47 : cluster 0 5.a deep-scrub starts 2024-04-06T03:23:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[29221]: cluster 2024-04-06T03:23:44.525605+0000 osd.2 (osd.2) 48 : cluster 0 5.a deep-scrub ok 2024-04-06T03:23:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[29221]: cluster 2024-04-06T03:23:45.584018+0000 osd.3 (osd.3) 71 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:46 smithi050 bash[29221]: cluster 2024-04-06T03:23:45.584136+0000 osd.3 (osd.3) 72 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:46.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:46 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.523034+0000 osd.2 (osd.2) 47 : cluster 0 5.a deep-scrub starts 2024-04-06T03:23:46.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:46 smithi153 bash[23852]: cluster 2024-04-06T03:23:44.525605+0000 osd.2 (osd.2) 48 : cluster 0 5.a deep-scrub ok 2024-04-06T03:23:46.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:46 smithi153 bash[23852]: cluster 2024-04-06T03:23:45.584018+0000 osd.3 (osd.3) 71 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:46.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:46 smithi153 bash[23852]: cluster 2024-04-06T03:23:45.584136+0000 osd.3 (osd.3) 72 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:47.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[21467]: cluster 2024-04-06T03:23:45.887740+0000 osd.5 (osd.5) 71 : cluster 0 5.15 scrub starts 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[21467]: cluster 2024-04-06T03:23:45.890790+0000 osd.5 (osd.5) 72 : cluster 0 5.15 scrub ok 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[21467]: cluster 2024-04-06T03:23:45.995424+0000 mgr.y (mgr.24368) 106 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[21467]: cluster 2024-04-06T03:23:46.372109+0000 osd.4 (osd.4) 55 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[21467]: cluster 2024-04-06T03:23:46.374345+0000 osd.4 (osd.4) 56 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[21467]: cluster 2024-04-06T03:23:46.397192+0000 mon.a (mon.0) 959 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[29221]: cluster 2024-04-06T03:23:45.887740+0000 osd.5 (osd.5) 71 : cluster 0 5.15 scrub starts 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[29221]: cluster 2024-04-06T03:23:45.890790+0000 osd.5 (osd.5) 72 : cluster 0 5.15 scrub ok 2024-04-06T03:23:47.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[29221]: cluster 2024-04-06T03:23:45.995424+0000 mgr.y (mgr.24368) 106 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:47.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[29221]: cluster 2024-04-06T03:23:46.372109+0000 osd.4 (osd.4) 55 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:23:47.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[29221]: cluster 2024-04-06T03:23:46.374345+0000 osd.4 (osd.4) 56 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:23:47.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:47 smithi050 bash[29221]: cluster 2024-04-06T03:23:46.397192+0000 mon.a (mon.0) 959 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-04-06T03:23:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:47 smithi153 bash[23852]: cluster 2024-04-06T03:23:45.887740+0000 osd.5 (osd.5) 71 : cluster 0 5.15 scrub starts 2024-04-06T03:23:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:47 smithi153 bash[23852]: cluster 2024-04-06T03:23:45.890790+0000 osd.5 (osd.5) 72 : cluster 0 5.15 scrub ok 2024-04-06T03:23:47.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:47 smithi153 bash[23852]: cluster 2024-04-06T03:23:45.995424+0000 mgr.y (mgr.24368) 106 : cluster 0 pgmap v78: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:23:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:47 smithi153 bash[23852]: cluster 2024-04-06T03:23:46.372109+0000 osd.4 (osd.4) 55 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:23:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:47 smithi153 bash[23852]: cluster 2024-04-06T03:23:46.374345+0000 osd.4 (osd.4) 56 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:23:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:47 smithi153 bash[23852]: cluster 2024-04-06T03:23:46.397192+0000 mon.a (mon.0) 959 : cluster 0 osdmap e115: 8 total, 8 up, 8 in 2024-04-06T03:23:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:48.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:48 smithi050 bash[21467]: cluster 2024-04-06T03:23:46.859009+0000 osd.5 (osd.5) 73 : cluster 0 3.1d scrub starts 2024-04-06T03:23:48.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:48 smithi050 bash[21467]: cluster 2024-04-06T03:23:46.865065+0000 osd.5 (osd.5) 74 : cluster 0 3.1d scrub ok 2024-04-06T03:23:48.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:48 smithi050 bash[21467]: audit 2024-04-06T03:23:48.279738+0000 mon.a (mon.0) 960 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:48.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:48 smithi050 bash[29221]: cluster 2024-04-06T03:23:46.859009+0000 osd.5 (osd.5) 73 : cluster 0 3.1d scrub starts 2024-04-06T03:23:48.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:48 smithi050 bash[29221]: cluster 2024-04-06T03:23:46.865065+0000 osd.5 (osd.5) 74 : cluster 0 3.1d scrub ok 2024-04-06T03:23:48.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:48 smithi050 bash[29221]: audit 2024-04-06T03:23:48.279738+0000 mon.a (mon.0) 960 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:48.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:48 smithi153 bash[23852]: cluster 2024-04-06T03:23:46.859009+0000 osd.5 (osd.5) 73 : cluster 0 3.1d scrub starts 2024-04-06T03:23:48.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:48 smithi153 bash[23852]: cluster 2024-04-06T03:23:46.865065+0000 osd.5 (osd.5) 74 : cluster 0 3.1d scrub ok 2024-04-06T03:23:48.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:48 smithi153 bash[23852]: audit 2024-04-06T03:23:48.279738+0000 mon.a (mon.0) 960 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:23:49.396 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:23:49.482 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:23:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:47.596230+0000 osd.2 (osd.2) 49 : cluster 0 5.6 scrub starts 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:47.598671+0000 osd.2 (osd.2) 50 : cluster 0 5.6 scrub ok 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:47.904127+0000 osd.5 (osd.5) 75 : cluster 0 3.16 scrub starts 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:47.908276+0000 osd.5 (osd.5) 76 : cluster 0 3.16 scrub ok 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:47.996234+0000 mgr.y (mgr.24368) 107 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:48.687529+0000 osd.1 (osd.1) 67 : cluster 0 6.1 scrub starts 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[21467]: cluster 2024-04-06T03:23:48.690049+0000 osd.1 (osd.1) 68 : cluster 0 6.1 scrub ok 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:47.596230+0000 osd.2 (osd.2) 49 : cluster 0 5.6 scrub starts 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:47.598671+0000 osd.2 (osd.2) 50 : cluster 0 5.6 scrub ok 2024-04-06T03:23:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:47.904127+0000 osd.5 (osd.5) 75 : cluster 0 3.16 scrub starts 2024-04-06T03:23:49.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:47.908276+0000 osd.5 (osd.5) 76 : cluster 0 3.16 scrub ok 2024-04-06T03:23:49.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:47.996234+0000 mgr.y (mgr.24368) 107 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:49.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:48.687529+0000 osd.1 (osd.1) 67 : cluster 0 6.1 scrub starts 2024-04-06T03:23:49.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:49 smithi050 bash[29221]: cluster 2024-04-06T03:23:48.690049+0000 osd.1 (osd.1) 68 : cluster 0 6.1 scrub ok 2024-04-06T03:23:49.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:47.596230+0000 osd.2 (osd.2) 49 : cluster 0 5.6 scrub starts 2024-04-06T03:23:49.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:47.598671+0000 osd.2 (osd.2) 50 : cluster 0 5.6 scrub ok 2024-04-06T03:23:49.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:47.904127+0000 osd.5 (osd.5) 75 : cluster 0 3.16 scrub starts 2024-04-06T03:23:49.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:47.908276+0000 osd.5 (osd.5) 76 : cluster 0 3.16 scrub ok 2024-04-06T03:23:49.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:47.996234+0000 mgr.y (mgr.24368) 107 : cluster 0 pgmap v80: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:23:49.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:48.687529+0000 osd.1 (osd.1) 67 : cluster 0 6.1 scrub starts 2024-04-06T03:23:49.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:49 smithi153 bash[23852]: cluster 2024-04-06T03:23:48.690049+0000 osd.1 (osd.1) 68 : cluster 0 6.1 scrub ok 2024-04-06T03:23:50.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:50 smithi050 bash[21467]: cluster 2024-04-06T03:23:48.920450+0000 osd.5 (osd.5) 77 : cluster 0 3.c scrub starts 2024-04-06T03:23:50.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:50 smithi050 bash[21467]: cluster 2024-04-06T03:23:48.924664+0000 osd.5 (osd.5) 78 : cluster 0 3.c scrub ok 2024-04-06T03:23:50.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:50 smithi050 bash[21467]: audit 2024-04-06T03:23:49.066035+0000 mgr.y (mgr.24368) 108 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:50.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:50 smithi050 bash[29221]: cluster 2024-04-06T03:23:48.920450+0000 osd.5 (osd.5) 77 : cluster 0 3.c scrub starts 2024-04-06T03:23:50.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:50 smithi050 bash[29221]: cluster 2024-04-06T03:23:48.924664+0000 osd.5 (osd.5) 78 : cluster 0 3.c scrub ok 2024-04-06T03:23:50.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:50 smithi050 bash[29221]: audit 2024-04-06T03:23:49.066035+0000 mgr.y (mgr.24368) 108 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:50.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:50 smithi153 bash[23852]: cluster 2024-04-06T03:23:48.920450+0000 osd.5 (osd.5) 77 : cluster 0 3.c scrub starts 2024-04-06T03:23:50.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:50 smithi153 bash[23852]: cluster 2024-04-06T03:23:48.924664+0000 osd.5 (osd.5) 78 : cluster 0 3.c scrub ok 2024-04-06T03:23:50.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:50 smithi153 bash[23852]: audit 2024-04-06T03:23:49.066035+0000 mgr.y (mgr.24368) 108 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:23:51.186 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:23:51.189 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:23:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:51 smithi050 bash[21467]: cluster 2024-04-06T03:23:49.997722+0000 mgr.y (mgr.24368) 109 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:23:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:51 smithi050 bash[21467]: cluster 2024-04-06T03:23:50.714211+0000 osd.1 (osd.1) 69 : cluster 0 2.4 scrub starts 2024-04-06T03:23:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:51 smithi050 bash[21467]: cluster 2024-04-06T03:23:50.717344+0000 osd.1 (osd.1) 70 : cluster 0 2.4 scrub ok 2024-04-06T03:23:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:51 smithi050 bash[29221]: cluster 2024-04-06T03:23:49.997722+0000 mgr.y (mgr.24368) 109 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:23:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:51 smithi050 bash[29221]: cluster 2024-04-06T03:23:50.714211+0000 osd.1 (osd.1) 69 : cluster 0 2.4 scrub starts 2024-04-06T03:23:51.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:51 smithi050 bash[29221]: cluster 2024-04-06T03:23:50.717344+0000 osd.1 (osd.1) 70 : cluster 0 2.4 scrub ok 2024-04-06T03:23:51.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:51 smithi153 bash[23852]: cluster 2024-04-06T03:23:49.997722+0000 mgr.y (mgr.24368) 109 : cluster 0 pgmap v81: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:23:51.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:51 smithi153 bash[23852]: cluster 2024-04-06T03:23:50.714211+0000 osd.1 (osd.1) 69 : cluster 0 2.4 scrub starts 2024-04-06T03:23:51.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:51 smithi153 bash[23852]: cluster 2024-04-06T03:23:50.717344+0000 osd.1 (osd.1) 70 : cluster 0 2.4 scrub ok 2024-04-06T03:23:51.931 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-04-06T03:23:49.996563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1394,"num_read_kb":1208,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":881,"ondisk_log_size":881,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249312,"kb_used_data":8188,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711904,"statfs":{"total":767960285184,"available":767704989696,"internally_reserved":0,"allocated":8384512,"data_stored":5101858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003535"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.400704+0000","last_change":"2024-04-06T03:22:43.760443+0000","last_active":"2024-04-06T03:23:46.400704+0000","last_peered":"2024-04-06T03:23:46.400704+0000","last_clean":"2024-04-06T03:23:46.400704+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:23:46.400704+0000","last_undegraded":"2024-04-06T03:23:46.400704+0000","last_fullsized":"2024-04-06T03:23:46.400704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:43.760298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:43.760298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388222+0000","last_change":"2024-04-06T03:22:39.290421+0000","last_active":"2024-04-06T03:23:46.388222+0000","last_peered":"2024-04-06T03:23:46.388222+0000","last_clean":"2024-04-06T03:23:46.388222+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:23:46.388222+0000","last_undegraded":"2024-04-06T03:23:46.388222+0000","last_fullsized":"2024-04-06T03:23:46.388222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:39.290214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:39.290214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:58.769754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":98,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.400146+0000","last_change":"2024-04-06T03:22:31.407417+0000","last_active":"2024-04-06T03:23:46.400146+0000","last_peered":"2024-04-06T03:23:46.400146+0000","last_clean":"2024-04-06T03:23:46.400146+0000","last_became_active":"2024-04-06T03:21:35.160225+0000","last_became_peered":"2024-04-06T03:21:35.160225+0000","last_unstale":"2024-04-06T03:23:46.400146+0000","last_undegraded":"2024-04-06T03:23:46.400146+0000","last_fullsized":"2024-04-06T03:23:46.400146+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:31.407192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.407192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:52.957016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":125,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401652+0000","last_change":"2024-04-06T03:23:20.747373+0000","last_active":"2024-04-06T03:23:46.401652+0000","last_peered":"2024-04-06T03:23:46.401652+0000","last_clean":"2024-04-06T03:23:46.401652+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:23:46.401652+0000","last_undegraded":"2024-04-06T03:23:46.401652+0000","last_fullsized":"2024-04-06T03:23:46.401652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:23:20.747059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.747059+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.765610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388662+0000","last_change":"2024-04-06T03:22:34.374145+0000","last_active":"2024-04-06T03:23:46.388662+0000","last_peered":"2024-04-06T03:23:46.388662+0000","last_clean":"2024-04-06T03:23:46.388662+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:23:46.388662+0000","last_undegraded":"2024-04-06T03:23:46.388662+0000","last_fullsized":"2024-04-06T03:23:46.388662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:34.374028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:34.374028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:04.316032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.876778+0000","last_change":"2024-04-06T03:22:42.790165+0000","last_active":"2024-04-06T03:23:46.876778+0000","last_peered":"2024-04-06T03:23:46.876778+0000","last_clean":"2024-04-06T03:23:46.876778+0000","last_became_active":"2024-04-06T03:20:55.772175+0000","last_became_peered":"2024-04-06T03:20:55.772175+0000","last_unstale":"2024-04-06T03:23:46.876778+0000","last_undegraded":"2024-04-06T03:23:46.876778+0000","last_fullsized":"2024-04-06T03:23:46.876778+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:42.790029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:42.790029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:52.067624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401987+0000","last_change":"2024-04-06T03:23:17.696688+0000","last_active":"2024-04-06T03:23:46.401987+0000","last_peered":"2024-04-06T03:23:46.401987+0000","last_clean":"2024-04-06T03:23:46.401987+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:23:46.401987+0000","last_undegraded":"2024-04-06T03:23:46.401987+0000","last_fullsized":"2024-04-06T03:23:46.401987+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.696473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.696473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.518460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":119,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399723+0000","last_change":"2024-04-06T03:22:23.375990+0000","last_active":"2024-04-06T03:23:46.399723+0000","last_peered":"2024-04-06T03:23:46.399723+0000","last_clean":"2024-04-06T03:23:46.399723+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:23:46.399723+0000","last_undegraded":"2024-04-06T03:23:46.399723+0000","last_fullsized":"2024-04-06T03:23:46.399723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:22:23.375802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:23.375802+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:49.742377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.859758+0000","last_change":"2024-04-06T03:22:43.719297+0000","last_active":"2024-04-06T03:23:46.859758+0000","last_peered":"2024-04-06T03:23:46.859758+0000","last_clean":"2024-04-06T03:23:46.859758+0000","last_became_active":"2024-04-06T03:20:57.825746+0000","last_became_peered":"2024-04-06T03:20:57.825746+0000","last_unstale":"2024-04-06T03:23:46.859758+0000","last_undegraded":"2024-04-06T03:23:46.859758+0000","last_fullsized":"2024-04-06T03:23:46.859758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:43.718948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:43.718948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:09.217552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873736+0000","last_change":"2024-04-06T03:22:31.422734+0000","last_active":"2024-04-06T03:23:46.873736+0000","last_peered":"2024-04-06T03:23:46.873736+0000","last_clean":"2024-04-06T03:23:46.873736+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:23:46.873736+0000","last_undegraded":"2024-04-06T03:23:46.873736+0000","last_fullsized":"2024-04-06T03:23:46.873736+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.422613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:59.334545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":108,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875535+0000","last_change":"2024-04-06T03:23:21.801571+0000","last_active":"2024-04-06T03:23:46.875535+0000","last_peered":"2024-04-06T03:23:46.875535+0000","last_clean":"2024-04-06T03:23:46.875535+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:23:46.875535+0000","last_undegraded":"2024-04-06T03:23:46.875535+0000","last_fullsized":"2024-04-06T03:23:46.875535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:21.801341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.801341+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.356673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"112'22","reported_seq":129,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.861793+0000","last_change":"2024-04-06T03:23:36.905749+0000","last_active":"2024-04-06T03:23:46.861793+0000","last_peered":"2024-04-06T03:23:46.861793+0000","last_clean":"2024-04-06T03:23:46.861793+0000","last_became_active":"2024-04-06T03:20:53.352525+0000","last_became_peered":"2024-04-06T03:20:53.352525+0000","last_unstale":"2024-04-06T03:23:46.861793+0000","last_undegraded":"2024-04-06T03:23:46.861793+0000","last_fullsized":"2024-04-06T03:23:46.861793+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'22","last_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.905499+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.467751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.392610+0000","last_change":"2024-04-06T03:22:56.162313+0000","last_active":"2024-04-06T03:23:46.392610+0000","last_peered":"2024-04-06T03:23:46.392610+0000","last_clean":"2024-04-06T03:23:46.392610+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:23:46.392610+0000","last_undegraded":"2024-04-06T03:23:46.392610+0000","last_fullsized":"2024-04-06T03:23:46.392610+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:56.162207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:56.162207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:57.161121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.860384+0000","last_change":"2024-04-06T03:22:41.692374+0000","last_active":"2024-04-06T03:23:46.860384+0000","last_peered":"2024-04-06T03:23:46.860384+0000","last_clean":"2024-04-06T03:23:46.860384+0000","last_became_active":"2024-04-06T03:20:55.755058+0000","last_became_peered":"2024-04-06T03:20:55.755058+0000","last_unstale":"2024-04-06T03:23:46.860384+0000","last_undegraded":"2024-04-06T03:23:46.860384+0000","last_fullsized":"2024-04-06T03:23:46.860384+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:41.692162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:41.692162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:48.619689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873731+0000","last_change":"2024-04-06T03:22:17.421828+0000","last_active":"2024-04-06T03:23:46.873731+0000","last_peered":"2024-04-06T03:23:46.873731+0000","last_clean":"2024-04-06T03:23:46.873731+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:23:46.873731+0000","last_undegraded":"2024-04-06T03:23:46.873731+0000","last_fullsized":"2024-04-06T03:23:46.873731+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:17.421734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:17.421734+0000","last_clean_scrub_stamp":"2024-04-06T03:22:17.421734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"105'24","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.865342+0000","last_change":"2024-04-06T03:23:46.865483+0000","last_active":"2024-04-06T03:23:46.865342+0000","last_peered":"2024-04-06T03:23:46.865342+0000","last_clean":"2024-04-06T03:23:46.865342+0000","last_became_active":"2024-04-06T03:20:53.346381+0000","last_became_peered":"2024-04-06T03:20:53.346381+0000","last_unstale":"2024-04-06T03:23:46.865342+0000","last_undegraded":"2024-04-06T03:23:46.865342+0000","last_fullsized":"2024-04-06T03:23:46.865342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'24","last_scrub_stamp":"2024-04-06T03:23:46.865084+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.865084+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.280139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401948+0000","last_change":"2024-04-06T03:23:30.862864+0000","last_active":"2024-04-06T03:23:46.401948+0000","last_peered":"2024-04-06T03:23:46.401948+0000","last_clean":"2024-04-06T03:23:46.401948+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:23:46.401948+0000","last_undegraded":"2024-04-06T03:23:46.401948+0000","last_fullsized":"2024-04-06T03:23:46.401948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:30.862646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:30.862646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.636834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.389236+0000","last_change":"2024-04-06T03:22:46.186560+0000","last_active":"2024-04-06T03:23:46.389236+0000","last_peered":"2024-04-06T03:23:46.389236+0000","last_clean":"2024-04-06T03:23:46.389236+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:23:46.389236+0000","last_undegraded":"2024-04-06T03:23:46.389236+0000","last_fullsized":"2024-04-06T03:23:46.389236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:46.186324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.186324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:54.836821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.400937+0000","last_change":"2024-04-06T03:22:42.729912+0000","last_active":"2024-04-06T03:23:46.400937+0000","last_peered":"2024-04-06T03:23:46.400937+0000","last_clean":"2024-04-06T03:23:46.400937+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:23:46.400937+0000","last_undegraded":"2024-04-06T03:23:46.400937+0000","last_fullsized":"2024-04-06T03:23:46.400937+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:42.729641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:42.729641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:58.849864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"111'27","reported_seq":144,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.389117+0000","last_change":"2024-04-06T03:23:18.287395+0000","last_active":"2024-04-06T03:23:46.389117+0000","last_peered":"2024-04-06T03:23:46.389117+0000","last_clean":"2024-04-06T03:23:46.389117+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:23:46.389117+0000","last_undegraded":"2024-04-06T03:23:46.389117+0000","last_fullsized":"2024-04-06T03:23:46.389117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'27","last_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.287127+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.693232+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.390951+0000","last_change":"2024-04-06T03:22:38.322032+0000","last_active":"2024-04-06T03:23:46.390951+0000","last_peered":"2024-04-06T03:23:46.390951+0000","last_clean":"2024-04-06T03:23:46.390951+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:23:46.390951+0000","last_undegraded":"2024-04-06T03:23:46.390951+0000","last_fullsized":"2024-04-06T03:23:46.390951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:38.321815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:38.321815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:53.956206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.878218+0000","last_change":"2024-04-06T03:23:01.687019+0000","last_active":"2024-04-06T03:23:46.878218+0000","last_peered":"2024-04-06T03:23:46.878218+0000","last_clean":"2024-04-06T03:23:46.878218+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:23:46.878218+0000","last_undegraded":"2024-04-06T03:23:46.878218+0000","last_fullsized":"2024-04-06T03:23:46.878218+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_clean_scrub_stamp":"2024-04-06T03:23:01.686796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.412050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.865979+0000","last_change":"2024-04-06T03:23:36.848600+0000","last_active":"2024-04-06T03:23:46.865979+0000","last_peered":"2024-04-06T03:23:46.865979+0000","last_clean":"2024-04-06T03:23:46.865979+0000","last_became_active":"2024-04-06T03:20:51.143283+0000","last_became_peered":"2024-04-06T03:20:51.143283+0000","last_unstale":"2024-04-06T03:23:46.865979+0000","last_undegraded":"2024-04-06T03:23:46.865979+0000","last_fullsized":"2024-04-06T03:23:46.865979+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.848364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.848364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.392566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.393999+0000","last_change":"2024-04-06T03:23:14.042484+0000","last_active":"2024-04-06T03:23:46.393999+0000","last_peered":"2024-04-06T03:23:46.393999+0000","last_clean":"2024-04-06T03:23:46.393999+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:23:46.393999+0000","last_undegraded":"2024-04-06T03:23:46.393999+0000","last_fullsized":"2024-04-06T03:23:46.393999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:14.042058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:14.042058+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.768307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.878329+0000","last_change":"2024-04-06T03:22:35.750314+0000","last_active":"2024-04-06T03:23:46.878329+0000","last_peered":"2024-04-06T03:23:46.878329+0000","last_clean":"2024-04-06T03:23:46.878329+0000","last_became_active":"2024-04-06T03:20:57.845246+0000","last_became_peered":"2024-04-06T03:20:57.845246+0000","last_unstale":"2024-04-06T03:23:46.878329+0000","last_undegraded":"2024-04-06T03:23:46.878329+0000","last_fullsized":"2024-04-06T03:23:46.878329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:35.750071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:35.750071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.398783+0000","last_change":"2024-04-06T03:23:34.010043+0000","last_active":"2024-04-06T03:23:46.398783+0000","last_peered":"2024-04-06T03:23:46.398783+0000","last_clean":"2024-04-06T03:23:46.398783+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:23:46.398783+0000","last_undegraded":"2024-04-06T03:23:46.398783+0000","last_fullsized":"2024-04-06T03:23:46.398783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:23:34.009837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.282584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":103,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399609+0000","last_change":"2024-04-06T03:23:43.924012+0000","last_active":"2024-04-06T03:23:46.399609+0000","last_peered":"2024-04-06T03:23:46.399609+0000","last_clean":"2024-04-06T03:23:46.399609+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:23:46.399609+0000","last_undegraded":"2024-04-06T03:23:46.399609+0000","last_fullsized":"2024-04-06T03:23:46.399609+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:43.923791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.923791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:10.650300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"109'20","reported_seq":123,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.400274+0000","last_change":"2024-04-06T03:22:49.786714+0000","last_active":"2024-04-06T03:23:46.400274+0000","last_peered":"2024-04-06T03:23:46.400274+0000","last_clean":"2024-04-06T03:23:46.400274+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:23:46.400274+0000","last_undegraded":"2024-04-06T03:23:46.400274+0000","last_fullsized":"2024-04-06T03:23:46.400274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'20","last_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:22:49.786480+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:08.011745+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875552+0000","last_change":"2024-04-06T03:22:58.497756+0000","last_active":"2024-04-06T03:23:46.875552+0000","last_peered":"2024-04-06T03:23:46.875552+0000","last_clean":"2024-04-06T03:23:46.875552+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:23:46.875552+0000","last_undegraded":"2024-04-06T03:23:46.875552+0000","last_fullsized":"2024-04-06T03:23:46.875552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-06T03:22:58.497530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:58.497530+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:03.159362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"101'12","reported_seq":145,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.866571+0000","last_change":"2024-04-06T03:23:33.869748+0000","last_active":"2024-04-06T03:23:46.866571+0000","last_peered":"2024-04-06T03:23:46.866571+0000","last_clean":"2024-04-06T03:23:46.866571+0000","last_became_active":"2024-04-06T03:20:55.765174+0000","last_became_peered":"2024-04-06T03:20:55.765174+0000","last_unstale":"2024-04-06T03:23:46.866571+0000","last_undegraded":"2024-04-06T03:23:46.866571+0000","last_fullsized":"2024-04-06T03:23:46.866571+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-06T03:23:33.869531+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.869531+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.720001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.859537+0000","last_change":"2024-04-06T03:23:19.763448+0000","last_active":"2024-04-06T03:23:46.859537+0000","last_peered":"2024-04-06T03:23:46.859537+0000","last_clean":"2024-04-06T03:23:46.859537+0000","last_became_active":"2024-04-06T03:20:51.144514+0000","last_became_peered":"2024-04-06T03:20:51.144514+0000","last_unstale":"2024-04-06T03:23:46.859537+0000","last_undegraded":"2024-04-06T03:23:46.859537+0000","last_fullsized":"2024-04-06T03:23:46.859537+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.763180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:54.626052+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.763180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:46.950926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"111'50","reported_seq":173,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877162+0000","last_change":"2024-04-06T03:23:04.830805+0000","last_active":"2024-04-06T03:23:46.877162+0000","last_peered":"2024-04-06T03:23:46.877162+0000","last_clean":"2024-04-06T03:23:46.877162+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:23:46.877162+0000","last_undegraded":"2024-04-06T03:23:46.877162+0000","last_fullsized":"2024-04-06T03:23:46.877162+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'50","last_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_clean_scrub_stamp":"2024-04-06T03:23:04.830548+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:25.118064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401508+0000","last_change":"2024-04-06T03:22:47.788318+0000","last_active":"2024-04-06T03:23:46.401508+0000","last_peered":"2024-04-06T03:23:46.401508+0000","last_clean":"2024-04-06T03:23:46.401508+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:23:46.401508+0000","last_undegraded":"2024-04-06T03:23:46.401508+0000","last_fullsized":"2024-04-06T03:23:46.401508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:22:47.788083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.587930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.874026+0000","last_change":"2024-04-06T03:22:53.518032+0000","last_active":"2024-04-06T03:23:46.874026+0000","last_peered":"2024-04-06T03:23:46.874026+0000","last_clean":"2024-04-06T03:23:46.874026+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:23:46.874026+0000","last_undegraded":"2024-04-06T03:23:46.874026+0000","last_fullsized":"2024-04-06T03:23:46.874026+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_clean_scrub_stamp":"2024-04-06T03:22:53.517730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.828631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.867136+0000","last_change":"2024-04-06T03:23:26.863931+0000","last_active":"2024-04-06T03:23:46.867136+0000","last_peered":"2024-04-06T03:23:46.867136+0000","last_clean":"2024-04-06T03:23:46.867136+0000","last_became_active":"2024-04-06T03:20:51.142652+0000","last_became_peered":"2024-04-06T03:20:51.142652+0000","last_unstale":"2024-04-06T03:23:46.867136+0000","last_undegraded":"2024-04-06T03:23:46.867136+0000","last_fullsized":"2024-04-06T03:23:46.867136+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:23:26.863691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:41.711589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":137,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:47.908614+0000","last_change":"2024-04-06T03:23:47.908697+0000","last_active":"2024-04-06T03:23:47.908614+0000","last_peered":"2024-04-06T03:23:47.908614+0000","last_clean":"2024-04-06T03:23:47.908614+0000","last_became_active":"2024-04-06T03:20:53.350189+0000","last_became_peered":"2024-04-06T03:20:53.350189+0000","last_unstale":"2024-04-06T03:23:47.908614+0000","last_undegraded":"2024-04-06T03:23:47.908614+0000","last_fullsized":"2024-04-06T03:23:47.908614+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:47.908288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.908288+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.038863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873409+0000","last_change":"2024-04-06T03:22:29.395728+0000","last_active":"2024-04-06T03:23:46.873409+0000","last_peered":"2024-04-06T03:23:46.873409+0000","last_clean":"2024-04-06T03:23:46.873409+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:23:46.873409+0000","last_undegraded":"2024-04-06T03:23:46.873409+0000","last_fullsized":"2024-04-06T03:23:46.873409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_clean_scrub_stamp":"2024-04-06T03:22:29.395501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.867137+0000","last_change":"2024-04-06T03:23:25.887364+0000","last_active":"2024-04-06T03:23:46.867137+0000","last_peered":"2024-04-06T03:23:46.867137+0000","last_clean":"2024-04-06T03:23:46.867137+0000","last_became_active":"2024-04-06T03:20:55.769876+0000","last_became_peered":"2024-04-06T03:20:55.769876+0000","last_unstale":"2024-04-06T03:23:46.867137+0000","last_undegraded":"2024-04-06T03:23:46.867137+0000","last_fullsized":"2024-04-06T03:23:46.867137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:25.887152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:25.887152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.529607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.863376+0000","last_change":"2024-04-06T03:23:12.715806+0000","last_active":"2024-04-06T03:23:46.863376+0000","last_peered":"2024-04-06T03:23:46.863376+0000","last_clean":"2024-04-06T03:23:46.863376+0000","last_became_active":"2024-04-06T03:20:51.151269+0000","last_became_peered":"2024-04-06T03:20:51.151269+0000","last_unstale":"2024-04-06T03:23:46.863376+0000","last_undegraded":"2024-04-06T03:23:46.863376+0000","last_fullsized":"2024-04-06T03:23:46.863376+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:12.715608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:12.715608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:14.405678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399417+0000","last_change":"2024-04-06T03:23:18.056122+0000","last_active":"2024-04-06T03:23:46.399417+0000","last_peered":"2024-04-06T03:23:46.399417+0000","last_clean":"2024-04-06T03:23:46.399417+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:23:46.399417+0000","last_undegraded":"2024-04-06T03:23:46.399417+0000","last_fullsized":"2024-04-06T03:23:46.399417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:18.056024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.056024+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.251393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399480+0000","last_change":"2024-04-06T03:22:57.659396+0000","last_active":"2024-04-06T03:23:46.399480+0000","last_peered":"2024-04-06T03:23:46.399480+0000","last_clean":"2024-04-06T03:23:46.399480+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:23:46.399480+0000","last_undegraded":"2024-04-06T03:23:46.399480+0000","last_fullsized":"2024-04-06T03:23:46.399480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:57.659282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:57.659282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:09.284205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877431+0000","last_change":"2024-04-06T03:22:55.746092+0000","last_active":"2024-04-06T03:23:46.877431+0000","last_peered":"2024-04-06T03:23:46.877431+0000","last_clean":"2024-04-06T03:23:46.877431+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:23:46.877431+0000","last_undegraded":"2024-04-06T03:23:46.877431+0000","last_fullsized":"2024-04-06T03:23:46.877431+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:55.745785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:55.745785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:56.988854+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877543+0000","last_change":"2024-04-06T03:23:12.824588+0000","last_active":"2024-04-06T03:23:46.877543+0000","last_peered":"2024-04-06T03:23:46.877543+0000","last_clean":"2024-04-06T03:23:46.877543+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:23:46.877543+0000","last_undegraded":"2024-04-06T03:23:46.877543+0000","last_fullsized":"2024-04-06T03:23:46.877543+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_clean_scrub_stamp":"2024-04-06T03:23:12.824475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:24.655532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'23","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.389653+0000","last_change":"2024-04-06T03:23:29.339315+0000","last_active":"2024-04-06T03:23:46.389653+0000","last_peered":"2024-04-06T03:23:46.389653+0000","last_clean":"2024-04-06T03:23:46.389653+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:23:46.389653+0000","last_undegraded":"2024-04-06T03:23:46.389653+0000","last_fullsized":"2024-04-06T03:23:46.389653+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-06T03:23:29.339052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.339052+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.010581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.392243+0000","last_change":"2024-04-06T03:22:34.395252+0000","last_active":"2024-04-06T03:23:46.392243+0000","last_peered":"2024-04-06T03:23:46.392243+0000","last_clean":"2024-04-06T03:23:46.392243+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:23:46.392243+0000","last_undegraded":"2024-04-06T03:23:46.392243+0000","last_fullsized":"2024-04-06T03:23:46.392243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:34.395083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:34.395083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:51.274750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401227+0000","last_change":"2024-04-06T03:22:38.676629+0000","last_active":"2024-04-06T03:23:46.401227+0000","last_peered":"2024-04-06T03:23:46.401227+0000","last_clean":"2024-04-06T03:23:46.401227+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:23:46.401227+0000","last_undegraded":"2024-04-06T03:23:46.401227+0000","last_fullsized":"2024-04-06T03:23:46.401227+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:38.676335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:38.676335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.866144+0000","last_change":"2024-04-06T03:23:16.851233+0000","last_active":"2024-04-06T03:23:46.866144+0000","last_peered":"2024-04-06T03:23:46.866144+0000","last_clean":"2024-04-06T03:23:46.866144+0000","last_became_active":"2024-04-06T03:20:51.137376+0000","last_became_peered":"2024-04-06T03:20:51.137376+0000","last_unstale":"2024-04-06T03:23:46.866144+0000","last_undegraded":"2024-04-06T03:23:46.866144+0000","last_fullsized":"2024-04-06T03:23:46.866144+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:16.851024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:23:16.851024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:18.428925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":116,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875868+0000","last_change":"2024-04-06T03:22:46.586344+0000","last_active":"2024-04-06T03:23:46.875868+0000","last_peered":"2024-04-06T03:23:46.875868+0000","last_clean":"2024-04-06T03:23:46.875868+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:23:46.875868+0000","last_undegraded":"2024-04-06T03:23:46.875868+0000","last_fullsized":"2024-04-06T03:23:46.875868+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:22:46.586091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.586091+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:57.595934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.391212+0000","last_change":"2024-04-06T03:23:19.298817+0000","last_active":"2024-04-06T03:23:46.391212+0000","last_peered":"2024-04-06T03:23:46.391212+0000","last_clean":"2024-04-06T03:23:46.391212+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:23:46.391212+0000","last_undegraded":"2024-04-06T03:23:46.391212+0000","last_fullsized":"2024-04-06T03:23:46.391212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.298498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.298498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.565314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401174+0000","last_change":"2024-04-06T03:22:59.658875+0000","last_active":"2024-04-06T03:23:46.401174+0000","last_peered":"2024-04-06T03:23:46.401174+0000","last_clean":"2024-04-06T03:23:46.401174+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:23:46.401174+0000","last_undegraded":"2024-04-06T03:23:46.401174+0000","last_fullsized":"2024-04-06T03:23:46.401174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:22:59.658761+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:20.379794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.396052+0000","last_change":"2024-04-06T03:23:40.956392+0000","last_active":"2024-04-06T03:23:46.396052+0000","last_peered":"2024-04-06T03:23:46.396052+0000","last_clean":"2024-04-06T03:23:46.396052+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:23:46.396052+0000","last_undegraded":"2024-04-06T03:23:46.396052+0000","last_fullsized":"2024-04-06T03:23:46.396052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:40.956167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:40.956167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.850753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.393005+0000","last_change":"2024-04-06T03:22:51.222996+0000","last_active":"2024-04-06T03:23:46.393005+0000","last_peered":"2024-04-06T03:23:46.393005+0000","last_clean":"2024-04-06T03:23:46.393005+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:23:46.393005+0000","last_undegraded":"2024-04-06T03:23:46.393005+0000","last_fullsized":"2024-04-06T03:23:46.393005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-06T03:22:51.222590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:51.222590+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:57.067022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.398332+0000","last_change":"2024-04-06T03:23:28.068250+0000","last_active":"2024-04-06T03:23:46.398332+0000","last_peered":"2024-04-06T03:23:46.398332+0000","last_clean":"2024-04-06T03:23:46.398332+0000","last_became_active":"2024-04-06T03:20:57.843310+0000","last_became_peered":"2024-04-06T03:20:57.843310+0000","last_unstale":"2024-04-06T03:23:46.398332+0000","last_undegraded":"2024-04-06T03:23:46.398332+0000","last_fullsized":"2024-04-06T03:23:46.398332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.067970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.747904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"85'11","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.862399+0000","last_change":"2024-04-06T03:23:45.891079+0000","last_active":"2024-04-06T03:23:46.862399+0000","last_peered":"2024-04-06T03:23:46.862399+0000","last_clean":"2024-04-06T03:23:46.862399+0000","last_became_active":"2024-04-06T03:20:55.770173+0000","last_became_peered":"2024-04-06T03:20:55.770173+0000","last_unstale":"2024-04-06T03:23:46.862399+0000","last_undegraded":"2024-04-06T03:23:46.862399+0000","last_fullsized":"2024-04-06T03:23:46.862399+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:45.890804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:45.890804+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.597014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.860050+0000","last_change":"2024-04-06T03:23:21.739882+0000","last_active":"2024-04-06T03:23:46.860050+0000","last_peered":"2024-04-06T03:23:46.860050+0000","last_clean":"2024-04-06T03:23:46.860050+0000","last_became_active":"2024-04-06T03:20:51.157201+0000","last_became_peered":"2024-04-06T03:20:51.157201+0000","last_unstale":"2024-04-06T03:23:46.860050+0000","last_undegraded":"2024-04-06T03:23:46.860050+0000","last_fullsized":"2024-04-06T03:23:46.860050+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:21.739671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.739671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.621751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":107,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.874410+0000","last_change":"2024-04-06T03:22:15.465954+0000","last_active":"2024-04-06T03:23:46.874410+0000","last_peered":"2024-04-06T03:23:46.874410+0000","last_clean":"2024-04-06T03:23:46.874410+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:23:46.874410+0000","last_undegraded":"2024-04-06T03:23:46.874410+0000","last_fullsized":"2024-04-06T03:23:46.874410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:22:15.465569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:15.465569+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875132+0000","last_change":"2024-04-06T03:22:14.504308+0000","last_active":"2024-04-06T03:23:46.875132+0000","last_peered":"2024-04-06T03:23:46.875132+0000","last_clean":"2024-04-06T03:23:46.875132+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:23:46.875132+0000","last_undegraded":"2024-04-06T03:23:46.875132+0000","last_fullsized":"2024-04-06T03:23:46.875132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:14.504101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:14.504101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.863227+0000","last_change":"2024-04-06T03:23:31.955588+0000","last_active":"2024-04-06T03:23:46.863227+0000","last_peered":"2024-04-06T03:23:46.863227+0000","last_clean":"2024-04-06T03:23:46.863227+0000","last_became_active":"2024-04-06T03:20:55.756465+0000","last_became_peered":"2024-04-06T03:20:55.756465+0000","last_unstale":"2024-04-06T03:23:46.863227+0000","last_undegraded":"2024-04-06T03:23:46.863227+0000","last_fullsized":"2024-04-06T03:23:46.863227+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:31.955371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:31.955371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:56.516085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.865224+0000","last_change":"2024-04-06T03:23:35.835087+0000","last_active":"2024-04-06T03:23:46.865224+0000","last_peered":"2024-04-06T03:23:46.865224+0000","last_clean":"2024-04-06T03:23:46.865224+0000","last_became_active":"2024-04-06T03:20:51.145847+0000","last_became_peered":"2024-04-06T03:20:51.145847+0000","last_unstale":"2024-04-06T03:23:46.865224+0000","last_undegraded":"2024-04-06T03:23:46.865224+0000","last_fullsized":"2024-04-06T03:23:46.865224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.834842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.834842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.169606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"111'6","reported_seq":104,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.865050+0000","last_change":"2024-04-06T03:23:21.875463+0000","last_active":"2024-04-06T03:23:46.865050+0000","last_peered":"2024-04-06T03:23:46.865050+0000","last_clean":"2024-04-06T03:23:46.865050+0000","last_became_active":"2024-04-06T03:20:53.348930+0000","last_became_peered":"2024-04-06T03:20:53.348930+0000","last_unstale":"2024-04-06T03:23:46.865050+0000","last_undegraded":"2024-04-06T03:23:46.865050+0000","last_fullsized":"2024-04-06T03:23:46.865050+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6","last_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.875237+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.782953+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.394796+0000","last_change":"2024-04-06T03:23:10.631170+0000","last_active":"2024-04-06T03:23:46.394796+0000","last_peered":"2024-04-06T03:23:46.394796+0000","last_clean":"2024-04-06T03:23:46.394796+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:23:46.394796+0000","last_undegraded":"2024-04-06T03:23:46.394796+0000","last_fullsized":"2024-04-06T03:23:46.394796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:23:10.631051+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:10.891042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.402128+0000","last_change":"2024-04-06T03:22:50.729388+0000","last_active":"2024-04-06T03:23:46.402128+0000","last_peered":"2024-04-06T03:23:46.402128+0000","last_clean":"2024-04-06T03:23:46.402128+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:23:46.402128+0000","last_undegraded":"2024-04-06T03:23:46.402128+0000","last_fullsized":"2024-04-06T03:23:46.402128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:50.729294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:50.729294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:18.365897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.393124+0000","last_change":"2024-04-06T03:23:23.589775+0000","last_active":"2024-04-06T03:23:46.393124+0000","last_peered":"2024-04-06T03:23:46.393124+0000","last_clean":"2024-04-06T03:23:46.393124+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:23:46.393124+0000","last_undegraded":"2024-04-06T03:23:46.393124+0000","last_fullsized":"2024-04-06T03:23:46.393124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:23.589472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.589472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.531624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"111'23","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873469+0000","last_change":"2024-04-06T03:23:20.835769+0000","last_active":"2024-04-06T03:23:46.873469+0000","last_peered":"2024-04-06T03:23:46.873469+0000","last_clean":"2024-04-06T03:23:46.873469+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:23:46.873469+0000","last_undegraded":"2024-04-06T03:23:46.873469+0000","last_fullsized":"2024-04-06T03:23:46.873469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'23","last_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.835514+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:30.227431+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401331+0000","last_change":"2024-04-06T03:23:44.976222+0000","last_active":"2024-04-06T03:23:46.401331+0000","last_peered":"2024-04-06T03:23:46.401331+0000","last_clean":"2024-04-06T03:23:46.401331+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:23:46.401331+0000","last_undegraded":"2024-04-06T03:23:46.401331+0000","last_fullsized":"2024-04-06T03:23:46.401331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.976015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.976015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.423557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":118,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.390670+0000","last_change":"2024-04-06T03:23:09.163493+0000","last_active":"2024-04-06T03:23:46.390670+0000","last_peered":"2024-04-06T03:23:46.390670+0000","last_clean":"2024-04-06T03:23:46.390670+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:23:46.390670+0000","last_undegraded":"2024-04-06T03:23:46.390670+0000","last_fullsized":"2024-04-06T03:23:46.390670+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-06T03:23:09.163254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.163254+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.561644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":127,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875482+0000","last_change":"2024-04-06T03:23:19.822376+0000","last_active":"2024-04-06T03:23:46.875482+0000","last_peered":"2024-04-06T03:23:46.875482+0000","last_clean":"2024-04-06T03:23:46.875482+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:23:46.875482+0000","last_undegraded":"2024-04-06T03:23:46.875482+0000","last_fullsized":"2024-04-06T03:23:46.875482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:19.822131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.822131+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.155287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.392842+0000","last_change":"2024-04-06T03:22:16.421651+0000","last_active":"2024-04-06T03:23:46.392842+0000","last_peered":"2024-04-06T03:23:46.392842+0000","last_clean":"2024-04-06T03:23:46.392842+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:23:46.392842+0000","last_undegraded":"2024-04-06T03:23:46.392842+0000","last_fullsized":"2024-04-06T03:23:46.392842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:16.421443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:16.421443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.392285+0000","last_change":"2024-04-06T03:22:44.386258+0000","last_active":"2024-04-06T03:23:46.392285+0000","last_peered":"2024-04-06T03:23:46.392285+0000","last_clean":"2024-04-06T03:23:46.392285+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:23:46.392285+0000","last_undegraded":"2024-04-06T03:23:46.392285+0000","last_fullsized":"2024-04-06T03:23:46.392285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:22:44.386020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.128819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877717+0000","last_change":"2024-04-06T03:22:18.710566+0000","last_active":"2024-04-06T03:23:46.877717+0000","last_peered":"2024-04-06T03:23:46.877717+0000","last_clean":"2024-04-06T03:23:46.877717+0000","last_became_active":"2024-04-06T03:20:51.148632+0000","last_became_peered":"2024-04-06T03:20:51.148632+0000","last_unstale":"2024-04-06T03:23:46.877717+0000","last_undegraded":"2024-04-06T03:23:46.877717+0000","last_fullsized":"2024-04-06T03:23:46.877717+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:22:18.710336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:18.710336+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":147,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.861666+0000","last_change":"2024-04-06T03:22:40.710426+0000","last_active":"2024-04-06T03:23:46.861666+0000","last_peered":"2024-04-06T03:23:46.861666+0000","last_clean":"2024-04-06T03:23:46.861666+0000","last_became_active":"2024-04-06T03:20:53.351328+0000","last_became_peered":"2024-04-06T03:20:53.351328+0000","last_unstale":"2024-04-06T03:23:46.861666+0000","last_undegraded":"2024-04-06T03:23:46.861666+0000","last_fullsized":"2024-04-06T03:23:46.861666+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:22:40.710327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:40.710327+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:05.111206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":97,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877630+0000","last_change":"2024-04-06T03:22:20.722406+0000","last_active":"2024-04-06T03:23:46.877630+0000","last_peered":"2024-04-06T03:23:46.877630+0000","last_clean":"2024-04-06T03:23:46.877630+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:23:46.877630+0000","last_undegraded":"2024-04-06T03:23:46.877630+0000","last_fullsized":"2024-04-06T03:23:46.877630+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:22:20.722147+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.393844+0000","last_change":"2024-04-06T03:22:58.133626+0000","last_active":"2024-04-06T03:23:46.393844+0000","last_peered":"2024-04-06T03:23:46.393844+0000","last_clean":"2024-04-06T03:23:46.393844+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:23:46.393844+0000","last_undegraded":"2024-04-06T03:23:46.393844+0000","last_fullsized":"2024-04-06T03:23:46.393844+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:22:58.133367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:58.133367+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:20.887571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401823+0000","last_change":"2024-04-06T03:23:00.610221+0000","last_active":"2024-04-06T03:23:46.401823+0000","last_peered":"2024-04-06T03:23:46.401823+0000","last_clean":"2024-04-06T03:23:46.401823+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:23:46.401823+0000","last_undegraded":"2024-04-06T03:23:46.401823+0000","last_fullsized":"2024-04-06T03:23:46.401823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:00.610114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:00.610114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:06.812327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.860340+0000","last_change":"2024-04-06T03:23:27.845925+0000","last_active":"2024-04-06T03:23:46.860340+0000","last_peered":"2024-04-06T03:23:46.860340+0000","last_clean":"2024-04-06T03:23:46.860340+0000","last_became_active":"2024-04-06T03:20:51.148366+0000","last_became_peered":"2024-04-06T03:20:51.148366+0000","last_unstale":"2024-04-06T03:23:46.860340+0000","last_undegraded":"2024-04-06T03:23:46.860340+0000","last_fullsized":"2024-04-06T03:23:46.860340+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:27.845710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:27.845710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.306766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'34","reported_seq":147,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388676+0000","last_change":"2024-04-06T03:22:31.389374+0000","last_active":"2024-04-06T03:23:46.388676+0000","last_peered":"2024-04-06T03:23:46.388676+0000","last_clean":"2024-04-06T03:23:46.388676+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:23:46.388676+0000","last_undegraded":"2024-04-06T03:23:46.388676+0000","last_fullsized":"2024-04-06T03:23:46.388676+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'34","last_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.389226+0000","objects_scrubbed":9,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:54.731367+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.396924+0000","last_change":"2024-04-06T03:22:30.412875+0000","last_active":"2024-04-06T03:23:46.396924+0000","last_peered":"2024-04-06T03:23:46.396924+0000","last_clean":"2024-04-06T03:23:46.396924+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:23:46.396924+0000","last_undegraded":"2024-04-06T03:23:46.396924+0000","last_fullsized":"2024-04-06T03:23:46.396924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:30.412667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:30.412667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:00.364828+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875175+0000","last_change":"2024-04-06T03:22:30.391636+0000","last_active":"2024-04-06T03:23:46.875175+0000","last_peered":"2024-04-06T03:23:46.875175+0000","last_clean":"2024-04-06T03:23:46.875175+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:23:46.875175+0000","last_undegraded":"2024-04-06T03:23:46.875175+0000","last_fullsized":"2024-04-06T03:23:46.875175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:22:30.391413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":153,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877090+0000","last_change":"2024-04-06T03:23:35.682185+0000","last_active":"2024-04-06T03:23:46.877090+0000","last_peered":"2024-04-06T03:23:46.877090+0000","last_clean":"2024-04-06T03:23:46.877090+0000","last_became_active":"2024-04-06T03:20:53.358177+0000","last_became_peered":"2024-04-06T03:20:53.358177+0000","last_unstale":"2024-04-06T03:23:46.877090+0000","last_undegraded":"2024-04-06T03:23:46.877090+0000","last_fullsized":"2024-04-06T03:23:46.877090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-06T03:23:35.681909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.681909+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.508581+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.394062+0000","last_change":"2024-04-06T03:22:20.415485+0000","last_active":"2024-04-06T03:23:46.394062+0000","last_peered":"2024-04-06T03:23:46.394062+0000","last_clean":"2024-04-06T03:23:46.394062+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:23:46.394062+0000","last_undegraded":"2024-04-06T03:23:46.394062+0000","last_fullsized":"2024-04-06T03:23:46.394062+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:20.415265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:20.415265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:47.598849+0000","last_change":"2024-04-06T03:23:47.598906+0000","last_active":"2024-04-06T03:23:47.598849+0000","last_peered":"2024-04-06T03:23:47.598849+0000","last_clean":"2024-04-06T03:23:47.598849+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:23:47.598849+0000","last_undegraded":"2024-04-06T03:23:47.598849+0000","last_fullsized":"2024-04-06T03:23:47.598849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:47.598682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.598682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.431027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.863225+0000","last_change":"2024-04-06T03:23:37.854225+0000","last_active":"2024-04-06T03:23:46.863225+0000","last_peered":"2024-04-06T03:23:46.863225+0000","last_clean":"2024-04-06T03:23:46.863225+0000","last_became_active":"2024-04-06T03:20:57.822724+0000","last_became_peered":"2024-04-06T03:20:57.822724+0000","last_unstale":"2024-04-06T03:23:46.863225+0000","last_undegraded":"2024-04-06T03:23:46.863225+0000","last_fullsized":"2024-04-06T03:23:46.863225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:37.854012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.854012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.515577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":117,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.400012+0000","last_change":"2024-04-06T03:22:15.609664+0000","last_active":"2024-04-06T03:23:46.400012+0000","last_peered":"2024-04-06T03:23:46.400012+0000","last_clean":"2024-04-06T03:23:46.400012+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:23:46.400012+0000","last_undegraded":"2024-04-06T03:23:46.400012+0000","last_fullsized":"2024-04-06T03:23:46.400012+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:22:15.609395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:15.609395+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.860804+0000","last_change":"2024-04-06T03:23:20.743012+0000","last_active":"2024-04-06T03:23:46.860804+0000","last_peered":"2024-04-06T03:23:46.860804+0000","last_clean":"2024-04-06T03:23:46.860804+0000","last_became_active":"2024-04-06T03:20:51.151534+0000","last_became_peered":"2024-04-06T03:20:51.151534+0000","last_unstale":"2024-04-06T03:23:46.860804+0000","last_undegraded":"2024-04-06T03:23:46.860804+0000","last_fullsized":"2024-04-06T03:23:46.860804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:20.742768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.742768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:26.458288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":351,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873395+0000","last_change":"2024-04-06T03:22:36.461819+0000","last_active":"2024-04-06T03:23:46.873395+0000","last_peered":"2024-04-06T03:23:46.873395+0000","last_clean":"2024-04-06T03:23:46.873395+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:23:46.873395+0000","last_undegraded":"2024-04-06T03:23:46.873395+0000","last_fullsized":"2024-04-06T03:23:46.873395+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:22:36.461578+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:02.479275+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":84,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875224+0000","last_change":"2024-04-06T03:22:26.403696+0000","last_active":"2024-04-06T03:23:46.875224+0000","last_peered":"2024-04-06T03:23:46.875224+0000","last_clean":"2024-04-06T03:23:46.875224+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:23:46.875224+0000","last_undegraded":"2024-04-06T03:23:46.875224+0000","last_fullsized":"2024-04-06T03:23:46.875224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:26.403500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:26.403500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:55.369737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877110+0000","last_change":"2024-04-06T03:22:39.710602+0000","last_active":"2024-04-06T03:23:46.877110+0000","last_peered":"2024-04-06T03:23:46.877110+0000","last_clean":"2024-04-06T03:23:46.877110+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:23:46.877110+0000","last_undegraded":"2024-04-06T03:23:46.877110+0000","last_fullsized":"2024-04-06T03:23:46.877110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:39.710513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:39.710513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'21","reported_seq":130,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.393051+0000","last_change":"2024-04-06T03:23:23.076419+0000","last_active":"2024-04-06T03:23:46.393051+0000","last_peered":"2024-04-06T03:23:46.393051+0000","last_clean":"2024-04-06T03:23:46.393051+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:23:46.393051+0000","last_undegraded":"2024-04-06T03:23:46.393051+0000","last_fullsized":"2024-04-06T03:23:46.393051+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:23.076269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.076269+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.958707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873190+0000","last_change":"2024-04-06T03:22:54.515963+0000","last_active":"2024-04-06T03:23:46.873190+0000","last_peered":"2024-04-06T03:23:46.873190+0000","last_clean":"2024-04-06T03:23:46.873190+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:23:46.873190+0000","last_undegraded":"2024-04-06T03:23:46.873190+0000","last_fullsized":"2024-04-06T03:23:46.873190+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:54.515739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:54.515739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:15.695306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.862809+0000","last_change":"2024-04-06T03:23:28.869298+0000","last_active":"2024-04-06T03:23:46.862809+0000","last_peered":"2024-04-06T03:23:46.862809+0000","last_clean":"2024-04-06T03:23:46.862809+0000","last_became_active":"2024-04-06T03:20:55.754600+0000","last_became_peered":"2024-04-06T03:20:55.754600+0000","last_unstale":"2024-04-06T03:23:46.862809+0000","last_undegraded":"2024-04-06T03:23:46.862809+0000","last_fullsized":"2024-04-06T03:23:46.862809+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.869068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.869068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.304800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:48.690258+0000","last_change":"2024-04-06T03:23:48.690384+0000","last_active":"2024-04-06T03:23:48.690258+0000","last_peered":"2024-04-06T03:23:48.690258+0000","last_clean":"2024-04-06T03:23:48.690258+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:23:48.690258+0000","last_undegraded":"2024-04-06T03:23:48.690258+0000","last_fullsized":"2024-04-06T03:23:48.690258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:48.690059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.690059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.780733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875403+0000","last_change":"2024-04-06T03:22:10.497983+0000","last_active":"2024-04-06T03:23:46.875403+0000","last_peered":"2024-04-06T03:23:46.875403+0000","last_clean":"2024-04-06T03:23:46.875403+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:23:46.875403+0000","last_undegraded":"2024-04-06T03:23:46.875403+0000","last_fullsized":"2024-04-06T03:23:46.875403+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:10.497772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:10.497772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'37","reported_seq":196,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877894+0000","last_change":"2024-04-06T03:23:09.792573+0000","last_active":"2024-04-06T03:23:46.877894+0000","last_peered":"2024-04-06T03:23:46.877894+0000","last_clean":"2024-04-06T03:23:46.877894+0000","last_became_active":"2024-04-06T03:20:53.352453+0000","last_became_peered":"2024-04-06T03:20:53.352453+0000","last_unstale":"2024-04-06T03:23:46.877894+0000","last_undegraded":"2024-04-06T03:23:46.877894+0000","last_fullsized":"2024-04-06T03:23:46.877894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'37","last_scrub_stamp":"2024-04-06T03:23:09.792373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.792373+0000","objects_scrubbed":11,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.096219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.866136+0000","last_change":"2024-04-06T03:23:39.777144+0000","last_active":"2024-04-06T03:23:46.866136+0000","last_peered":"2024-04-06T03:23:46.866136+0000","last_clean":"2024-04-06T03:23:46.866136+0000","last_became_active":"2024-04-06T03:20:55.769795+0000","last_became_peered":"2024-04-06T03:20:55.769795+0000","last_unstale":"2024-04-06T03:23:46.866136+0000","last_undegraded":"2024-04-06T03:23:46.866136+0000","last_fullsized":"2024-04-06T03:23:46.866136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.776933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.776933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.208938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388675+0000","last_change":"2024-04-06T03:23:35.364623+0000","last_active":"2024-04-06T03:23:46.388675+0000","last_peered":"2024-04-06T03:23:46.388675+0000","last_clean":"2024-04-06T03:23:46.388675+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:23:46.388675+0000","last_undegraded":"2024-04-06T03:23:46.388675+0000","last_fullsized":"2024-04-06T03:23:46.388675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.364381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.332831+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.364381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.739996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.878478+0000","last_change":"2024-04-06T03:23:37.757642+0000","last_active":"2024-04-06T03:23:46.878478+0000","last_peered":"2024-04-06T03:23:46.878478+0000","last_clean":"2024-04-06T03:23:46.878478+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:23:46.878478+0000","last_undegraded":"2024-04-06T03:23:46.878478+0000","last_fullsized":"2024-04-06T03:23:46.878478+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:37.757420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.757420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.290369+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":129,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.402253+0000","last_change":"2024-04-06T03:22:46.807318+0000","last_active":"2024-04-06T03:23:46.402253+0000","last_peered":"2024-04-06T03:23:46.402253+0000","last_clean":"2024-04-06T03:23:46.402253+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:23:46.402253+0000","last_undegraded":"2024-04-06T03:23:46.402253+0000","last_fullsized":"2024-04-06T03:23:46.402253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-06T03:22:46.807072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.807072+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:03.325548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":262,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.971402+0000","last_change":"2024-04-06T03:23:06.631761+0000","last_active":"2024-04-06T03:23:46.971402+0000","last_peered":"2024-04-06T03:23:46.971402+0000","last_clean":"2024-04-06T03:23:46.971402+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:23:46.971402+0000","last_undegraded":"2024-04-06T03:23:46.971402+0000","last_fullsized":"2024-04-06T03:23:46.971402+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:23:06.631539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:23:06.631539+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:25.562835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":156,"num_read_kb":151,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":86,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388368+0000","last_change":"2024-04-06T03:22:29.384611+0000","last_active":"2024-04-06T03:23:46.388368+0000","last_peered":"2024-04-06T03:23:46.388368+0000","last_clean":"2024-04-06T03:23:46.388368+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:23:46.388368+0000","last_undegraded":"2024-04-06T03:23:46.388368+0000","last_fullsized":"2024-04-06T03:23:46.388368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:29.384319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:29.384319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:49.777527+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873968+0000","last_change":"2024-04-06T03:22:43.626547+0000","last_active":"2024-04-06T03:23:46.873968+0000","last_peered":"2024-04-06T03:23:46.873968+0000","last_clean":"2024-04-06T03:23:46.873968+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:23:46.873968+0000","last_undegraded":"2024-04-06T03:23:46.873968+0000","last_fullsized":"2024-04-06T03:23:46.873968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:43.626233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:43.626233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:05.538215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.866842+0000","last_change":"2024-04-06T03:23:17.864400+0000","last_active":"2024-04-06T03:23:46.866842+0000","last_peered":"2024-04-06T03:23:46.866842+0000","last_clean":"2024-04-06T03:23:46.866842+0000","last_became_active":"2024-04-06T03:20:51.134999+0000","last_became_peered":"2024-04-06T03:20:51.134999+0000","last_unstale":"2024-04-06T03:23:46.866842+0000","last_undegraded":"2024-04-06T03:23:46.866842+0000","last_fullsized":"2024-04-06T03:23:46.866842+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.864183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.536200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":114,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399005+0000","last_change":"2024-04-06T03:23:39.032748+0000","last_active":"2024-04-06T03:23:46.399005+0000","last_peered":"2024-04-06T03:23:46.399005+0000","last_clean":"2024-04-06T03:23:46.399005+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:23:46.399005+0000","last_undegraded":"2024-04-06T03:23:46.399005+0000","last_fullsized":"2024-04-06T03:23:46.399005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:39.032620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.032620+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:39.700020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":103,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388655+0000","last_change":"2024-04-06T03:23:46.374564+0000","last_active":"2024-04-06T03:23:46.388655+0000","last_peered":"2024-04-06T03:23:46.388655+0000","last_clean":"2024-04-06T03:23:46.388655+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:23:46.388655+0000","last_undegraded":"2024-04-06T03:23:46.388655+0000","last_fullsized":"2024-04-06T03:23:46.388655+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.374356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.995508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":133,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399297+0000","last_change":"2024-04-06T03:22:58.657483+0000","last_active":"2024-04-06T03:23:46.399297+0000","last_peered":"2024-04-06T03:23:46.399297+0000","last_clean":"2024-04-06T03:23:46.399297+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:23:46.399297+0000","last_undegraded":"2024-04-06T03:23:46.399297+0000","last_fullsized":"2024-04-06T03:23:46.399297+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:22:58.657247+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:12.252538+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401044+0000","last_change":"2024-04-06T03:23:38.882403+0000","last_active":"2024-04-06T03:23:46.401044+0000","last_peered":"2024-04-06T03:23:46.401044+0000","last_clean":"2024-04-06T03:23:46.401044+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:23:46.401044+0000","last_undegraded":"2024-04-06T03:23:46.401044+0000","last_fullsized":"2024-04-06T03:23:46.401044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.882269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.592970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.862952+0000","last_change":"2024-04-06T03:23:11.690418+0000","last_active":"2024-04-06T03:23:46.862952+0000","last_peered":"2024-04-06T03:23:46.862952+0000","last_clean":"2024-04-06T03:23:46.862952+0000","last_became_active":"2024-04-06T03:20:55.762055+0000","last_became_peered":"2024-04-06T03:20:55.762055+0000","last_unstale":"2024-04-06T03:23:46.862952+0000","last_undegraded":"2024-04-06T03:23:46.862952+0000","last_fullsized":"2024-04-06T03:23:46.862952+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:11.690170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:11.690170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:35.579295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":92,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873462+0000","last_change":"2024-04-06T03:22:44.643404+0000","last_active":"2024-04-06T03:23:46.873462+0000","last_peered":"2024-04-06T03:23:46.873462+0000","last_clean":"2024-04-06T03:23:46.873462+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:23:46.873462+0000","last_undegraded":"2024-04-06T03:23:46.873462+0000","last_fullsized":"2024-04-06T03:23:46.873462+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:44.643181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:44.643181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:10.998024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":137,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.388373+0000","last_change":"2024-04-06T03:22:25.366358+0000","last_active":"2024-04-06T03:23:46.388373+0000","last_peered":"2024-04-06T03:23:46.388373+0000","last_clean":"2024-04-06T03:23:46.388373+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:23:46.388373+0000","last_undegraded":"2024-04-06T03:23:46.388373+0000","last_fullsized":"2024-04-06T03:23:46.388373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-06T03:22:25.366108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:25.366108+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:50.494711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":89,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.861388+0000","last_change":"2024-04-06T03:23:38.883987+0000","last_active":"2024-04-06T03:23:46.861388+0000","last_peered":"2024-04-06T03:23:46.861388+0000","last_clean":"2024-04-06T03:23:46.861388+0000","last_became_active":"2024-04-06T03:20:57.845535+0000","last_became_peered":"2024-04-06T03:20:57.845535+0000","last_unstale":"2024-04-06T03:23:46.861388+0000","last_undegraded":"2024-04-06T03:23:46.861388+0000","last_fullsized":"2024-04-06T03:23:46.861388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.883738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.883738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.269641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"85'11","reported_seq":140,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.390548+0000","last_change":"2024-04-06T03:23:23.295365+0000","last_active":"2024-04-06T03:23:46.390548+0000","last_peered":"2024-04-06T03:23:46.390548+0000","last_clean":"2024-04-06T03:23:46.390548+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:23:46.390548+0000","last_undegraded":"2024-04-06T03:23:46.390548+0000","last_fullsized":"2024-04-06T03:23:46.390548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:23.295124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.295124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.215136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":93,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.876880+0000","last_change":"2024-04-06T03:22:31.784166+0000","last_active":"2024-04-06T03:23:46.876880+0000","last_peered":"2024-04-06T03:23:46.876880+0000","last_clean":"2024-04-06T03:23:46.876880+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:23:46.876880+0000","last_undegraded":"2024-04-06T03:23:46.876880+0000","last_fullsized":"2024-04-06T03:23:46.876880+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:31.784009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.784009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"85'27","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.399798+0000","last_change":"2024-04-06T03:22:25.585530+0000","last_active":"2024-04-06T03:23:46.399798+0000","last_peered":"2024-04-06T03:23:46.399798+0000","last_clean":"2024-04-06T03:23:46.399798+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:23:46.399798+0000","last_undegraded":"2024-04-06T03:23:46.399798+0000","last_fullsized":"2024-04-06T03:23:46.399798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'27","last_scrub_stamp":"2024-04-06T03:22:25.585294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:25.585294+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:52.602972+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":90,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.390761+0000","last_change":"2024-04-06T03:23:36.391113+0000","last_active":"2024-04-06T03:23:46.390761+0000","last_peered":"2024-04-06T03:23:46.390761+0000","last_clean":"2024-04-06T03:23:46.390761+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:23:46.390761+0000","last_undegraded":"2024-04-06T03:23:46.390761+0000","last_fullsized":"2024-04-06T03:23:46.390761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.390806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.390806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:00.809816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":141,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.395814+0000","last_change":"2024-04-06T03:23:29.528196+0000","last_active":"2024-04-06T03:23:46.395814+0000","last_peered":"2024-04-06T03:23:46.395814+0000","last_clean":"2024-04-06T03:23:46.395814+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:23:46.395814+0000","last_undegraded":"2024-04-06T03:23:46.395814+0000","last_fullsized":"2024-04-06T03:23:46.395814+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.528005+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:29.753304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877076+0000","last_change":"2024-04-06T03:23:07.798328+0000","last_active":"2024-04-06T03:23:46.877076+0000","last_peered":"2024-04-06T03:23:46.877076+0000","last_clean":"2024-04-06T03:23:46.877076+0000","last_became_active":"2024-04-06T03:20:51.167121+0000","last_became_peered":"2024-04-06T03:20:51.167121+0000","last_unstale":"2024-04-06T03:23:46.877076+0000","last_undegraded":"2024-04-06T03:23:46.877076+0000","last_fullsized":"2024-04-06T03:23:46.877076+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:07.798091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:07.798091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:28.780888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"85'15","reported_seq":126,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.401018+0000","last_change":"2024-04-06T03:23:24.821416+0000","last_active":"2024-04-06T03:23:46.401018+0000","last_peered":"2024-04-06T03:23:46.401018+0000","last_clean":"2024-04-06T03:23:46.401018+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:23:46.401018+0000","last_undegraded":"2024-04-06T03:23:46.401018+0000","last_fullsized":"2024-04-06T03:23:46.401018+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:24.821164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:24.821164+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.815641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":82,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.398800+0000","last_change":"2024-04-06T03:22:28.514833+0000","last_active":"2024-04-06T03:23:46.398800+0000","last_peered":"2024-04-06T03:23:46.398800+0000","last_clean":"2024-04-06T03:23:46.398800+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:23:46.398800+0000","last_undegraded":"2024-04-06T03:23:46.398800+0000","last_fullsized":"2024-04-06T03:23:46.398800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:28.514624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:28.514624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:51.914339+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":85,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877291+0000","last_change":"2024-04-06T03:22:40.755288+0000","last_active":"2024-04-06T03:23:46.877291+0000","last_peered":"2024-04-06T03:23:46.877291+0000","last_clean":"2024-04-06T03:23:46.877291+0000","last_became_active":"2024-04-06T03:20:55.758951+0000","last_became_peered":"2024-04-06T03:20:55.758951+0000","last_unstale":"2024-04-06T03:23:46.877291+0000","last_undegraded":"2024-04-06T03:23:46.877291+0000","last_fullsized":"2024-04-06T03:23:46.877291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:40.755077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:40.755077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":100,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.874280+0000","last_change":"2024-04-06T03:23:15.710252+0000","last_active":"2024-04-06T03:23:46.874280+0000","last_peered":"2024-04-06T03:23:46.874280+0000","last_clean":"2024-04-06T03:23:46.874280+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:23:46.874280+0000","last_undegraded":"2024-04-06T03:23:46.874280+0000","last_fullsized":"2024-04-06T03:23:46.874280+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:15.710038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:15.710038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.258607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'25","reported_seq":139,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.865653+0000","last_change":"2024-04-06T03:23:22.871567+0000","last_active":"2024-04-06T03:23:46.865653+0000","last_peered":"2024-04-06T03:23:46.865653+0000","last_clean":"2024-04-06T03:23:46.865653+0000","last_became_active":"2024-04-06T03:20:53.350784+0000","last_became_peered":"2024-04-06T03:20:53.350784+0000","last_unstale":"2024-04-06T03:23:46.865653+0000","last_undegraded":"2024-04-06T03:23:46.865653+0000","last_fullsized":"2024-04-06T03:23:46.865653+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-06T03:23:22.871158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:22.871158+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:30.808433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":80,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.873068+0000","last_change":"2024-04-06T03:22:55.509249+0000","last_active":"2024-04-06T03:23:46.873068+0000","last_peered":"2024-04-06T03:23:46.873068+0000","last_clean":"2024-04-06T03:23:46.873068+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:23:46.873068+0000","last_undegraded":"2024-04-06T03:23:46.873068+0000","last_fullsized":"2024-04-06T03:23:46.873068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:55.508950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:55.508950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:15.654754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.394663+0000","last_change":"2024-04-06T03:23:43.566143+0000","last_active":"2024-04-06T03:23:46.394663+0000","last_peered":"2024-04-06T03:23:46.394663+0000","last_clean":"2024-04-06T03:23:46.394663+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:23:46.394663+0000","last_undegraded":"2024-04-06T03:23:46.394663+0000","last_fullsized":"2024-04-06T03:23:46.394663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.565930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.924455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.394356+0000","last_change":"2024-04-06T03:22:25.481597+0000","last_active":"2024-04-06T03:23:46.394356+0000","last_peered":"2024-04-06T03:23:46.394356+0000","last_clean":"2024-04-06T03:23:46.394356+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:23:46.394356+0000","last_undegraded":"2024-04-06T03:23:46.394356+0000","last_fullsized":"2024-04-06T03:23:46.394356+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:25.481353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:25.481353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:53.575199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":116,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875788+0000","last_change":"2024-04-06T03:22:57.488426+0000","last_active":"2024-04-06T03:23:46.875788+0000","last_peered":"2024-04-06T03:23:46.875788+0000","last_clean":"2024-04-06T03:23:46.875788+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:23:46.875788+0000","last_undegraded":"2024-04-06T03:23:46.875788+0000","last_fullsized":"2024-04-06T03:23:46.875788+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:22:57.488176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:57.488176+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.958962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":88,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.392539+0000","last_change":"2024-04-06T03:23:39.990151+0000","last_active":"2024-04-06T03:23:46.392539+0000","last_peered":"2024-04-06T03:23:46.392539+0000","last_clean":"2024-04-06T03:23:46.392539+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:23:46.392539+0000","last_undegraded":"2024-04-06T03:23:46.392539+0000","last_fullsized":"2024-04-06T03:23:46.392539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.990000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.990000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.631759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.393744+0000","last_change":"2024-04-06T03:23:44.525846+0000","last_active":"2024-04-06T03:23:46.393744+0000","last_peered":"2024-04-06T03:23:46.393744+0000","last_clean":"2024-04-06T03:23:46.393744+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:23:46.393744+0000","last_undegraded":"2024-04-06T03:23:46.393744+0000","last_fullsized":"2024-04-06T03:23:46.393744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.525616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.599062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":101,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.877836+0000","last_change":"2024-04-06T03:23:16.814953+0000","last_active":"2024-04-06T03:23:46.877836+0000","last_peered":"2024-04-06T03:23:46.877836+0000","last_clean":"2024-04-06T03:23:46.877836+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:23:46.877836+0000","last_undegraded":"2024-04-06T03:23:46.877836+0000","last_fullsized":"2024-04-06T03:23:46.877836+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:16.814738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.771476+0000","last_clean_scrub_stamp":"2024-04-06T03:23:16.814738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:27.108754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"85'10","reported_seq":112,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:48.924824+0000","last_change":"2024-04-06T03:23:48.924901+0000","last_active":"2024-04-06T03:23:48.924824+0000","last_peered":"2024-04-06T03:23:48.924824+0000","last_clean":"2024-04-06T03:23:48.924824+0000","last_became_active":"2024-04-06T03:20:53.345059+0000","last_became_peered":"2024-04-06T03:20:53.345059+0000","last_unstale":"2024-04-06T03:23:48.924824+0000","last_undegraded":"2024-04-06T03:23:48.924824+0000","last_fullsized":"2024-04-06T03:23:48.924824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'10","last_scrub_stamp":"2024-04-06T03:23:48.924677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.924677+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.542114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":81,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.859969+0000","last_change":"2024-04-06T03:22:46.720865+0000","last_active":"2024-04-06T03:23:46.859969+0000","last_peered":"2024-04-06T03:23:46.859969+0000","last_clean":"2024-04-06T03:23:46.859969+0000","last_became_active":"2024-04-06T03:20:58.128530+0000","last_became_peered":"2024-04-06T03:20:58.128530+0000","last_unstale":"2024-04-06T03:23:46.859969+0000","last_undegraded":"2024-04-06T03:23:46.859969+0000","last_fullsized":"2024-04-06T03:23:46.859969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.720611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:55.882241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":102,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.397820+0000","last_change":"2024-04-06T03:23:07.604317+0000","last_active":"2024-04-06T03:23:46.397820+0000","last_peered":"2024-04-06T03:23:46.397820+0000","last_clean":"2024-04-06T03:23:46.397820+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:23:46.397820+0000","last_undegraded":"2024-04-06T03:23:46.397820+0000","last_fullsized":"2024-04-06T03:23:46.397820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:07.604089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:23:07.604089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:21.624479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":121,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.875917+0000","last_change":"2024-04-06T03:23:41.707141+0000","last_active":"2024-04-06T03:23:46.875917+0000","last_peered":"2024-04-06T03:23:46.875917+0000","last_clean":"2024-04-06T03:23:46.875917+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:23:46.875917+0000","last_undegraded":"2024-04-06T03:23:46.875917+0000","last_fullsized":"2024-04-06T03:23:46.875917+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:41.706589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:41.706589+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.161763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"85'11","reported_seq":138,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-04-06T03:23:46.874076+0000","last_change":"2024-04-06T03:23:33.820933+0000","last_active":"2024-04-06T03:23:46.874076+0000","last_peered":"2024-04-06T03:23:46.874076+0000","last_clean":"2024-04-06T03:23:46.874076+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:23:46.874076+0000","last_undegraded":"2024-04-06T03:23:46.874076+0000","last_fullsized":"2024-04-06T03:23:46.874076+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:33.820691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.820691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.287595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":156,"num_read_kb":151,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":642,"ondisk_log_size":642,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743470,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32588,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712564,"statfs":{"total":95995035648,"available":95961665536,"internally_reserved":0,"allocated":1441792,"data_stored":1008268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135928,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1404928,"data_stored":1006548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":827392,"data_stored":416034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36732,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708420,"statfs":{"total":95995035648,"available":95957422080,"internally_reserved":0,"allocated":1425408,"data_stored":1007601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378723,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":815104,"data_stored":415778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084375,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":806912,"data_stored":415692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855438,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":851968,"data_stored":416263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495684,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":415674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:23:51.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:23:52.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:52 smithi050 bash[21467]: audit 2024-04-06T03:23:51.156355+0000 mgr.y (mgr.24368) 110 : audit 0 from='client.24533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:23:52.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:52 smithi050 bash[21467]: cluster 2024-04-06T03:23:51.857357+0000 osd.3 (osd.3) 73 : cluster 0 3.2 scrub starts 2024-04-06T03:23:52.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:52 smithi050 bash[21467]: cluster 2024-04-06T03:23:51.861981+0000 osd.3 (osd.3) 74 : cluster 0 3.2 scrub ok 2024-04-06T03:23:52.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:52 smithi050 bash[29221]: audit 2024-04-06T03:23:51.156355+0000 mgr.y (mgr.24368) 110 : audit 0 from='client.24533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:23:52.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:52 smithi050 bash[29221]: cluster 2024-04-06T03:23:51.857357+0000 osd.3 (osd.3) 73 : cluster 0 3.2 scrub starts 2024-04-06T03:23:52.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:52 smithi050 bash[29221]: cluster 2024-04-06T03:23:51.861981+0000 osd.3 (osd.3) 74 : cluster 0 3.2 scrub ok 2024-04-06T03:23:52.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:52 smithi153 bash[23852]: audit 2024-04-06T03:23:51.156355+0000 mgr.y (mgr.24368) 110 : audit 0 from='client.24533 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:23:52.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:52 smithi153 bash[23852]: cluster 2024-04-06T03:23:51.857357+0000 osd.3 (osd.3) 73 : cluster 0 3.2 scrub starts 2024-04-06T03:23:52.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:52 smithi153 bash[23852]: cluster 2024-04-06T03:23:51.861981+0000 osd.3 (osd.3) 74 : cluster 0 3.2 scrub ok 2024-04-06T03:23:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:51.977872+0000 osd.5 (osd.5) 79 : cluster 0 5.1b scrub starts 2024-04-06T03:23:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:51.980097+0000 osd.5 (osd.5) 80 : cluster 0 5.1b scrub ok 2024-04-06T03:23:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:51.998155+0000 mgr.y (mgr.24368) 111 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:52.831503+0000 osd.0 (osd.0) 65 : cluster 0 3.1f scrub starts 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:52.836496+0000 osd.0 (osd.0) 66 : cluster 0 3.1f scrub ok 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:52.838995+0000 osd.3 (osd.3) 75 : cluster 0 5.13 scrub starts 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[21467]: cluster 2024-04-06T03:23:52.841280+0000 osd.3 (osd.3) 76 : cluster 0 5.13 scrub ok 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:51.977872+0000 osd.5 (osd.5) 79 : cluster 0 5.1b scrub starts 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:51.980097+0000 osd.5 (osd.5) 80 : cluster 0 5.1b scrub ok 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:51.998155+0000 mgr.y (mgr.24368) 111 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:52.831503+0000 osd.0 (osd.0) 65 : cluster 0 3.1f scrub starts 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:52.836496+0000 osd.0 (osd.0) 66 : cluster 0 3.1f scrub ok 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:52.838995+0000 osd.3 (osd.3) 75 : cluster 0 5.13 scrub starts 2024-04-06T03:23:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:53 smithi050 bash[29221]: cluster 2024-04-06T03:23:52.841280+0000 osd.3 (osd.3) 76 : cluster 0 5.13 scrub ok 2024-04-06T03:23:53.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:51.977872+0000 osd.5 (osd.5) 79 : cluster 0 5.1b scrub starts 2024-04-06T03:23:53.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:51.980097+0000 osd.5 (osd.5) 80 : cluster 0 5.1b scrub ok 2024-04-06T03:23:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:51.998155+0000 mgr.y (mgr.24368) 111 : cluster 0 pgmap v82: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:23:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:52.831503+0000 osd.0 (osd.0) 65 : cluster 0 3.1f scrub starts 2024-04-06T03:23:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:52.836496+0000 osd.0 (osd.0) 66 : cluster 0 3.1f scrub ok 2024-04-06T03:23:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:52.838995+0000 osd.3 (osd.3) 75 : cluster 0 5.13 scrub starts 2024-04-06T03:23:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:53 smithi153 bash[23852]: cluster 2024-04-06T03:23:52.841280+0000 osd.3 (osd.3) 76 : cluster 0 5.13 scrub ok 2024-04-06T03:23:54.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[21467]: cluster 2024-04-06T03:23:52.626649+0000 osd.2 (osd.2) 51 : cluster 0 5.8 scrub starts 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[21467]: cluster 2024-04-06T03:23:52.628992+0000 osd.2 (osd.2) 52 : cluster 0 5.8 scrub ok 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[21467]: cluster 2024-04-06T03:23:53.412007+0000 osd.4 (osd.4) 57 : cluster 0 5.1 scrub starts 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[21467]: cluster 2024-04-06T03:23:53.414206+0000 osd.4 (osd.4) 58 : cluster 0 5.1 scrub ok 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[29221]: cluster 2024-04-06T03:23:52.626649+0000 osd.2 (osd.2) 51 : cluster 0 5.8 scrub starts 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[29221]: cluster 2024-04-06T03:23:52.628992+0000 osd.2 (osd.2) 52 : cluster 0 5.8 scrub ok 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[29221]: cluster 2024-04-06T03:23:53.412007+0000 osd.4 (osd.4) 57 : cluster 0 5.1 scrub starts 2024-04-06T03:23:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:54 smithi050 bash[29221]: cluster 2024-04-06T03:23:53.414206+0000 osd.4 (osd.4) 58 : cluster 0 5.1 scrub ok 2024-04-06T03:23:54.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:54 smithi153 bash[23852]: cluster 2024-04-06T03:23:52.626649+0000 osd.2 (osd.2) 51 : cluster 0 5.8 scrub starts 2024-04-06T03:23:54.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:54 smithi153 bash[23852]: cluster 2024-04-06T03:23:52.628992+0000 osd.2 (osd.2) 52 : cluster 0 5.8 scrub ok 2024-04-06T03:23:54.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:54 smithi153 bash[23852]: cluster 2024-04-06T03:23:53.412007+0000 osd.4 (osd.4) 57 : cluster 0 5.1 scrub starts 2024-04-06T03:23:54.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:54 smithi153 bash[23852]: cluster 2024-04-06T03:23:53.414206+0000 osd.4 (osd.4) 58 : cluster 0 5.1 scrub ok 2024-04-06T03:23:55.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:55 smithi050 bash[21467]: cluster 2024-04-06T03:23:53.591866+0000 osd.2 (osd.2) 53 : cluster 0 2.1 scrub starts 2024-04-06T03:23:55.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:55 smithi050 bash[21467]: cluster 2024-04-06T03:23:53.594478+0000 osd.2 (osd.2) 54 : cluster 0 2.1 scrub ok 2024-04-06T03:23:55.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:55 smithi050 bash[21467]: cluster 2024-04-06T03:23:53.999483+0000 mgr.y (mgr.24368) 112 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-04-06T03:23:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:55 smithi050 bash[29221]: cluster 2024-04-06T03:23:53.591866+0000 osd.2 (osd.2) 53 : cluster 0 2.1 scrub starts 2024-04-06T03:23:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:55 smithi050 bash[29221]: cluster 2024-04-06T03:23:53.594478+0000 osd.2 (osd.2) 54 : cluster 0 2.1 scrub ok 2024-04-06T03:23:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:55 smithi050 bash[29221]: cluster 2024-04-06T03:23:53.999483+0000 mgr.y (mgr.24368) 112 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-04-06T03:23:55.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:55 smithi153 bash[23852]: cluster 2024-04-06T03:23:53.591866+0000 osd.2 (osd.2) 53 : cluster 0 2.1 scrub starts 2024-04-06T03:23:55.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:55 smithi153 bash[23852]: cluster 2024-04-06T03:23:53.594478+0000 osd.2 (osd.2) 54 : cluster 0 2.1 scrub ok 2024-04-06T03:23:55.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:55 smithi153 bash[23852]: cluster 2024-04-06T03:23:53.999483+0000 mgr.y (mgr.24368) 112 : cluster 0 pgmap v83: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-04-06T03:23:56.730 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:23:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.778019+0000 osd.0 (osd.0) 67 : cluster 0 6.10 scrub starts 2024-04-06T03:23:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.780386+0000 osd.0 (osd.0) 68 : cluster 0 6.10 scrub ok 2024-04-06T03:23:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.804211+0000 osd.7 (osd.7) 69 : cluster 0 6.15 scrub starts 2024-04-06T03:23:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.806602+0000 osd.7 (osd.7) 70 : cluster 0 6.15 scrub ok 2024-04-06T03:23:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.826184+0000 osd.3 (osd.3) 77 : cluster 0 6.1b scrub starts 2024-04-06T03:23:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.828633+0000 osd.3 (osd.3) 78 : cluster 0 6.1b scrub ok 2024-04-06T03:23:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.778019+0000 osd.0 (osd.0) 67 : cluster 0 6.10 scrub starts 2024-04-06T03:23:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.780386+0000 osd.0 (osd.0) 68 : cluster 0 6.10 scrub ok 2024-04-06T03:23:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.804211+0000 osd.7 (osd.7) 69 : cluster 0 6.15 scrub starts 2024-04-06T03:23:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.806602+0000 osd.7 (osd.7) 70 : cluster 0 6.15 scrub ok 2024-04-06T03:23:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.826184+0000 osd.3 (osd.3) 77 : cluster 0 6.1b scrub starts 2024-04-06T03:23:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:56 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.828633+0000 osd.3 (osd.3) 78 : cluster 0 6.1b scrub ok 2024-04-06T03:23:56.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:56 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.778019+0000 osd.0 (osd.0) 67 : cluster 0 6.10 scrub starts 2024-04-06T03:23:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:56 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.780386+0000 osd.0 (osd.0) 68 : cluster 0 6.10 scrub ok 2024-04-06T03:23:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:56 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.804211+0000 osd.7 (osd.7) 69 : cluster 0 6.15 scrub starts 2024-04-06T03:23:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:56 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.806602+0000 osd.7 (osd.7) 70 : cluster 0 6.15 scrub ok 2024-04-06T03:23:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:56 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.826184+0000 osd.3 (osd.3) 77 : cluster 0 6.1b scrub starts 2024-04-06T03:23:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:56 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.828633+0000 osd.3 (osd.3) 78 : cluster 0 6.1b scrub ok 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.614137+0000 osd.2 (osd.2) 55 : cluster 0 6.f deep-scrub starts 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:55.616529+0000 osd.2 (osd.2) 56 : cluster 0 6.f deep-scrub ok 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.000158+0000 mgr.y (mgr.24368) 113 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.437692+0000 osd.4 (osd.4) 59 : cluster 0 3.9 scrub starts 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.442647+0000 osd.4 (osd.4) 60 : cluster 0 3.9 scrub ok 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.655170+0000 osd.4 (osd.4) 61 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.655276+0000 osd.4 (osd.4) 62 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:57.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.662665+0000 osd.1 (osd.1) 71 : cluster 0 4.2 scrub starts 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.666423+0000 osd.1 (osd.1) 72 : cluster 0 4.2 scrub ok 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.614137+0000 osd.2 (osd.2) 55 : cluster 0 6.f deep-scrub starts 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:55.616529+0000 osd.2 (osd.2) 56 : cluster 0 6.f deep-scrub ok 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.000158+0000 mgr.y (mgr.24368) 113 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.437692+0000 osd.4 (osd.4) 59 : cluster 0 3.9 scrub starts 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.442647+0000 osd.4 (osd.4) 60 : cluster 0 3.9 scrub ok 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.655170+0000 osd.4 (osd.4) 61 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.655276+0000 osd.4 (osd.4) 62 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.662665+0000 osd.1 (osd.1) 71 : cluster 0 4.2 scrub starts 2024-04-06T03:23:57.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:57 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.666423+0000 osd.1 (osd.1) 72 : cluster 0 4.2 scrub ok 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.614137+0000 osd.2 (osd.2) 55 : cluster 0 6.f deep-scrub starts 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:55.616529+0000 osd.2 (osd.2) 56 : cluster 0 6.f deep-scrub ok 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.000158+0000 mgr.y (mgr.24368) 113 : cluster 0 pgmap v84: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.437692+0000 osd.4 (osd.4) 59 : cluster 0 3.9 scrub starts 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.442647+0000 osd.4 (osd.4) 60 : cluster 0 3.9 scrub ok 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.655170+0000 osd.4 (osd.4) 61 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.655276+0000 osd.4 (osd.4) 62 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.662665+0000 osd.1 (osd.1) 71 : cluster 0 4.2 scrub starts 2024-04-06T03:23:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:57 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.666423+0000 osd.1 (osd.1) 72 : cluster 0 4.2 scrub ok 2024-04-06T03:23:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.572686+0000 osd.2 (osd.2) 57 : cluster 0 2.c deep-scrub starts 2024-04-06T03:23:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:56.575080+0000 osd.2 (osd.2) 58 : cluster 0 2.c deep-scrub ok 2024-04-06T03:23:58.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.454930+0000 mon.a (mon.0) 961 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.471580+0000 osd.0 (osd.0) 69 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.471694+0000 osd.0 (osd.0) 70 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.472582+0000 osd.4 (osd.4) 63 : cluster 0 6.1e scrub starts 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.474763+0000 osd.4 (osd.4) 64 : cluster 0 6.1e scrub ok 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.834757+0000 osd.3 (osd.3) 79 : cluster 0 3.8 scrub starts 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[21467]: cluster 2024-04-06T03:23:57.837991+0000 osd.3 (osd.3) 80 : cluster 0 3.8 scrub ok 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.572686+0000 osd.2 (osd.2) 57 : cluster 0 2.c deep-scrub starts 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:56.575080+0000 osd.2 (osd.2) 58 : cluster 0 2.c deep-scrub ok 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.454930+0000 mon.a (mon.0) 961 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-04-06T03:23:58.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.471580+0000 osd.0 (osd.0) 69 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.471694+0000 osd.0 (osd.0) 70 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.472582+0000 osd.4 (osd.4) 63 : cluster 0 6.1e scrub starts 2024-04-06T03:23:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.474763+0000 osd.4 (osd.4) 64 : cluster 0 6.1e scrub ok 2024-04-06T03:23:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.834757+0000 osd.3 (osd.3) 79 : cluster 0 3.8 scrub starts 2024-04-06T03:23:58.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:58 smithi050 bash[29221]: cluster 2024-04-06T03:23:57.837991+0000 osd.3 (osd.3) 80 : cluster 0 3.8 scrub ok 2024-04-06T03:23:58.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.572686+0000 osd.2 (osd.2) 57 : cluster 0 2.c deep-scrub starts 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:56.575080+0000 osd.2 (osd.2) 58 : cluster 0 2.c deep-scrub ok 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.454930+0000 mon.a (mon.0) 961 : cluster 0 osdmap e116: 8 total, 8 up, 8 in 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.471580+0000 osd.0 (osd.0) 69 : cluster 0 purged_snaps scrub starts 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.471694+0000 osd.0 (osd.0) 70 : cluster 0 purged_snaps scrub ok 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.472582+0000 osd.4 (osd.4) 63 : cluster 0 6.1e scrub starts 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.474763+0000 osd.4 (osd.4) 64 : cluster 0 6.1e scrub ok 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.834757+0000 osd.3 (osd.3) 79 : cluster 0 3.8 scrub starts 2024-04-06T03:23:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:58 smithi153 bash[23852]: cluster 2024-04-06T03:23:57.837991+0000 osd.3 (osd.3) 80 : cluster 0 3.8 scrub ok 2024-04-06T03:23:59.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:23:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:23:59.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:59 smithi050 bash[21467]: cluster 2024-04-06T03:23:58.000965+0000 mgr.y (mgr.24368) 114 : cluster 0 pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:23:59.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:23:59 smithi050 bash[21467]: cluster 2024-04-06T03:23:58.479009+0000 mon.a (mon.0) 962 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-04-06T03:23:59.664 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:23:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:23:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:23:59.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:59 smithi050 bash[29221]: cluster 2024-04-06T03:23:58.000965+0000 mgr.y (mgr.24368) 114 : cluster 0 pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:23:59.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:23:59 smithi050 bash[29221]: cluster 2024-04-06T03:23:58.479009+0000 mon.a (mon.0) 962 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-04-06T03:23:59.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:59 smithi153 bash[23852]: cluster 2024-04-06T03:23:58.000965+0000 mgr.y (mgr.24368) 114 : cluster 0 pgmap v86: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:23:59.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:23:59 smithi153 bash[23852]: cluster 2024-04-06T03:23:58.479009+0000 mon.a (mon.0) 962 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2024-04-06T03:24:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:24:00.081 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:24:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: audit 2024-04-06T03:23:59.077519+0000 mgr.y (mgr.24368) 115 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: cluster 2024-04-06T03:23:59.437569+0000 osd.4 (osd.4) 65 : cluster 0 3.3 scrub starts 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: cluster 2024-04-06T03:23:59.442503+0000 osd.4 (osd.4) 66 : cluster 0 3.3 scrub ok 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: cluster 2024-04-06T03:23:59.662421+0000 osd.1 (osd.1) 73 : cluster 0 2.9 scrub starts 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: cluster 2024-04-06T03:23:59.665044+0000 osd.1 (osd.1) 74 : cluster 0 2.9 scrub ok 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: cluster 2024-04-06T03:23:59.741101+0000 osd.0 (osd.0) 71 : cluster 0 2.1f scrub starts 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[21467]: cluster 2024-04-06T03:23:59.742634+0000 osd.0 (osd.0) 72 : cluster 0 2.1f scrub ok 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: audit 2024-04-06T03:23:59.077519+0000 mgr.y (mgr.24368) 115 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: cluster 2024-04-06T03:23:59.437569+0000 osd.4 (osd.4) 65 : cluster 0 3.3 scrub starts 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: cluster 2024-04-06T03:23:59.442503+0000 osd.4 (osd.4) 66 : cluster 0 3.3 scrub ok 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: cluster 2024-04-06T03:23:59.662421+0000 osd.1 (osd.1) 73 : cluster 0 2.9 scrub starts 2024-04-06T03:24:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: cluster 2024-04-06T03:23:59.665044+0000 osd.1 (osd.1) 74 : cluster 0 2.9 scrub ok 2024-04-06T03:24:00.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: cluster 2024-04-06T03:23:59.741101+0000 osd.0 (osd.0) 71 : cluster 0 2.1f scrub starts 2024-04-06T03:24:00.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:00 smithi050 bash[29221]: cluster 2024-04-06T03:23:59.742634+0000 osd.0 (osd.0) 72 : cluster 0 2.1f scrub ok 2024-04-06T03:24:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: audit 2024-04-06T03:23:59.077519+0000 mgr.y (mgr.24368) 115 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: cluster 2024-04-06T03:23:59.437569+0000 osd.4 (osd.4) 65 : cluster 0 3.3 scrub starts 2024-04-06T03:24:00.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: cluster 2024-04-06T03:23:59.442503+0000 osd.4 (osd.4) 66 : cluster 0 3.3 scrub ok 2024-04-06T03:24:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: cluster 2024-04-06T03:23:59.662421+0000 osd.1 (osd.1) 73 : cluster 0 2.9 scrub starts 2024-04-06T03:24:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: cluster 2024-04-06T03:23:59.665044+0000 osd.1 (osd.1) 74 : cluster 0 2.9 scrub ok 2024-04-06T03:24:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: cluster 2024-04-06T03:23:59.741101+0000 osd.0 (osd.0) 71 : cluster 0 2.1f scrub starts 2024-04-06T03:24:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:00 smithi153 bash[23852]: cluster 2024-04-06T03:23:59.742634+0000 osd.0 (osd.0) 72 : cluster 0 2.1f scrub ok 2024-04-06T03:24:00.818 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-04-06T03:24:00.001238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1404,"num_read_kb":1218,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":881,"ondisk_log_size":881,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":249436,"kb_used_data":7348,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749711780,"statfs":{"total":767960285184,"available":767704862720,"internally_reserved":0,"allocated":7524352,"data_stored":4758178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003309"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836553+0000","last_change":"2024-04-06T03:23:55.828897+0000","last_active":"2024-04-06T03:23:58.836553+0000","last_peered":"2024-04-06T03:23:58.836553+0000","last_clean":"2024-04-06T03:23:58.836553+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:23:58.836553+0000","last_undegraded":"2024-04-06T03:23:58.836553+0000","last_fullsized":"2024-04-06T03:23:58.836553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.828643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.828643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.076125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.471682+0000","last_change":"2024-04-06T03:22:39.290421+0000","last_active":"2024-04-06T03:23:58.471682+0000","last_peered":"2024-04-06T03:23:58.471682+0000","last_clean":"2024-04-06T03:23:58.471682+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:23:58.471682+0000","last_undegraded":"2024-04-06T03:23:58.471682+0000","last_fullsized":"2024-04-06T03:23:58.471682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:39.290214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:39.290214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:58.769754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478950+0000","last_change":"2024-04-06T03:22:31.407417+0000","last_active":"2024-04-06T03:23:58.478950+0000","last_peered":"2024-04-06T03:23:58.478950+0000","last_clean":"2024-04-06T03:23:58.478950+0000","last_became_active":"2024-04-06T03:21:35.160225+0000","last_became_peered":"2024-04-06T03:21:35.160225+0000","last_unstale":"2024-04-06T03:23:58.478950+0000","last_undegraded":"2024-04-06T03:23:58.478950+0000","last_fullsized":"2024-04-06T03:23:58.478950+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:31.407192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.407192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837336+0000","last_change":"2024-04-06T03:23:20.747373+0000","last_active":"2024-04-06T03:23:58.837336+0000","last_peered":"2024-04-06T03:23:58.837336+0000","last_clean":"2024-04-06T03:23:58.837336+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:23:58.837336+0000","last_undegraded":"2024-04-06T03:23:58.837336+0000","last_fullsized":"2024-04-06T03:23:58.837336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:23:20.747059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.747059+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.765610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472080+0000","last_change":"2024-04-06T03:22:34.374145+0000","last_active":"2024-04-06T03:23:58.472080+0000","last_peered":"2024-04-06T03:23:58.472080+0000","last_clean":"2024-04-06T03:23:58.472080+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:23:58.472080+0000","last_undegraded":"2024-04-06T03:23:58.472080+0000","last_fullsized":"2024-04-06T03:23:58.472080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:34.374028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:34.374028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:04.316032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955141+0000","last_change":"2024-04-06T03:22:42.790165+0000","last_active":"2024-04-06T03:23:57.955141+0000","last_peered":"2024-04-06T03:23:57.955141+0000","last_clean":"2024-04-06T03:23:57.955141+0000","last_became_active":"2024-04-06T03:20:55.772175+0000","last_became_peered":"2024-04-06T03:20:55.772175+0000","last_unstale":"2024-04-06T03:23:57.955141+0000","last_undegraded":"2024-04-06T03:23:57.955141+0000","last_fullsized":"2024-04-06T03:23:57.955141+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:42.790029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:42.790029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837892+0000","last_change":"2024-04-06T03:23:17.696688+0000","last_active":"2024-04-06T03:23:58.837892+0000","last_peered":"2024-04-06T03:23:58.837892+0000","last_clean":"2024-04-06T03:23:58.837892+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:23:58.837892+0000","last_undegraded":"2024-04-06T03:23:58.837892+0000","last_fullsized":"2024-04-06T03:23:58.837892+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.696473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.696473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.518460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":131,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478758+0000","last_change":"2024-04-06T03:23:52.836932+0000","last_active":"2024-04-06T03:23:58.478758+0000","last_peered":"2024-04-06T03:23:58.478758+0000","last_clean":"2024-04-06T03:23:58.478758+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:23:58.478758+0000","last_undegraded":"2024-04-06T03:23:58.478758+0000","last_fullsized":"2024-04-06T03:23:58.478758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:52.836543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.836543+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:09.578286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.465686+0000","last_change":"2024-04-06T03:22:43.719297+0000","last_active":"2024-04-06T03:23:57.465686+0000","last_peered":"2024-04-06T03:23:57.465686+0000","last_clean":"2024-04-06T03:23:57.465686+0000","last_became_active":"2024-04-06T03:20:57.825746+0000","last_became_peered":"2024-04-06T03:20:57.825746+0000","last_unstale":"2024-04-06T03:23:57.465686+0000","last_undegraded":"2024-04-06T03:23:57.465686+0000","last_fullsized":"2024-04-06T03:23:57.465686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:43.718948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:43.718948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:09.217552+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952726+0000","last_change":"2024-04-06T03:22:31.422734+0000","last_active":"2024-04-06T03:23:57.952726+0000","last_peered":"2024-04-06T03:23:57.952726+0000","last_clean":"2024-04-06T03:23:57.952726+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:23:57.952726+0000","last_undegraded":"2024-04-06T03:23:57.952726+0000","last_fullsized":"2024-04-06T03:23:57.952726+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.422613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:59.334545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.954358+0000","last_change":"2024-04-06T03:23:21.801571+0000","last_active":"2024-04-06T03:23:57.954358+0000","last_peered":"2024-04-06T03:23:57.954358+0000","last_clean":"2024-04-06T03:23:57.954358+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:23:57.954358+0000","last_undegraded":"2024-04-06T03:23:57.954358+0000","last_fullsized":"2024-04-06T03:23:57.954358+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:21.801341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.801341+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.356673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"112'22","reported_seq":131,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.467514+0000","last_change":"2024-04-06T03:23:36.905749+0000","last_active":"2024-04-06T03:23:57.467514+0000","last_peered":"2024-04-06T03:23:57.467514+0000","last_clean":"2024-04-06T03:23:57.467514+0000","last_became_active":"2024-04-06T03:20:53.352525+0000","last_became_peered":"2024-04-06T03:20:53.352525+0000","last_unstale":"2024-04-06T03:23:57.467514+0000","last_undegraded":"2024-04-06T03:23:57.467514+0000","last_fullsized":"2024-04-06T03:23:57.467514+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'22","last_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.905499+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.467751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474950+0000","last_change":"2024-04-06T03:22:56.162313+0000","last_active":"2024-04-06T03:23:58.474950+0000","last_peered":"2024-04-06T03:23:58.474950+0000","last_clean":"2024-04-06T03:23:58.474950+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:23:58.474950+0000","last_undegraded":"2024-04-06T03:23:58.474950+0000","last_fullsized":"2024-04-06T03:23:58.474950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:56.162207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:56.162207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466204+0000","last_change":"2024-04-06T03:23:51.980318+0000","last_active":"2024-04-06T03:23:57.466204+0000","last_peered":"2024-04-06T03:23:57.466204+0000","last_clean":"2024-04-06T03:23:57.466204+0000","last_became_active":"2024-04-06T03:20:55.755058+0000","last_became_peered":"2024-04-06T03:20:55.755058+0000","last_unstale":"2024-04-06T03:23:57.466204+0000","last_undegraded":"2024-04-06T03:23:57.466204+0000","last_fullsized":"2024-04-06T03:23:57.466204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:51.980107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:51.980107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:15.105749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952901+0000","last_change":"2024-04-06T03:22:17.421828+0000","last_active":"2024-04-06T03:23:57.952901+0000","last_peered":"2024-04-06T03:23:57.952901+0000","last_clean":"2024-04-06T03:23:57.952901+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:23:57.952901+0000","last_undegraded":"2024-04-06T03:23:57.952901+0000","last_fullsized":"2024-04-06T03:23:57.952901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:17.421734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:17.421734+0000","last_clean_scrub_stamp":"2024-04-06T03:22:17.421734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"105'24","reported_seq":140,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.467078+0000","last_change":"2024-04-06T03:23:46.865483+0000","last_active":"2024-04-06T03:23:57.467078+0000","last_peered":"2024-04-06T03:23:57.467078+0000","last_clean":"2024-04-06T03:23:57.467078+0000","last_became_active":"2024-04-06T03:20:53.346381+0000","last_became_peered":"2024-04-06T03:20:53.346381+0000","last_unstale":"2024-04-06T03:23:57.467078+0000","last_undegraded":"2024-04-06T03:23:57.467078+0000","last_fullsized":"2024-04-06T03:23:57.467078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'24","last_scrub_stamp":"2024-04-06T03:23:46.865084+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.865084+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.280139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.838032+0000","last_change":"2024-04-06T03:23:30.862864+0000","last_active":"2024-04-06T03:23:58.838032+0000","last_peered":"2024-04-06T03:23:58.838032+0000","last_clean":"2024-04-06T03:23:58.838032+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:23:58.838032+0000","last_undegraded":"2024-04-06T03:23:58.838032+0000","last_fullsized":"2024-04-06T03:23:58.838032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:30.862646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:30.862646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.636834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472046+0000","last_change":"2024-04-06T03:22:46.186560+0000","last_active":"2024-04-06T03:23:58.472046+0000","last_peered":"2024-04-06T03:23:58.472046+0000","last_clean":"2024-04-06T03:23:58.472046+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:23:58.472046+0000","last_undegraded":"2024-04-06T03:23:58.472046+0000","last_fullsized":"2024-04-06T03:23:58.472046+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:46.186324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.186324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836953+0000","last_change":"2024-04-06T03:22:42.729912+0000","last_active":"2024-04-06T03:23:58.836953+0000","last_peered":"2024-04-06T03:23:58.836953+0000","last_clean":"2024-04-06T03:23:58.836953+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:23:58.836953+0000","last_undegraded":"2024-04-06T03:23:58.836953+0000","last_fullsized":"2024-04-06T03:23:58.836953+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:42.729641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:42.729641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:23:58.849864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"111'27","reported_seq":148,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472490+0000","last_change":"2024-04-06T03:23:18.287395+0000","last_active":"2024-04-06T03:23:58.472490+0000","last_peered":"2024-04-06T03:23:58.472490+0000","last_clean":"2024-04-06T03:23:58.472490+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:23:58.472490+0000","last_undegraded":"2024-04-06T03:23:58.472490+0000","last_fullsized":"2024-04-06T03:23:58.472490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'27","last_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.287127+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.693232+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474078+0000","last_change":"2024-04-06T03:23:57.475036+0000","last_active":"2024-04-06T03:23:58.474078+0000","last_peered":"2024-04-06T03:23:58.474078+0000","last_clean":"2024-04-06T03:23:58.474078+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:23:58.474078+0000","last_undegraded":"2024-04-06T03:23:58.474078+0000","last_fullsized":"2024-04-06T03:23:58.474078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:57.474774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:57.474774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.260532+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.956164+0000","last_change":"2024-04-06T03:23:01.687019+0000","last_active":"2024-04-06T03:23:57.956164+0000","last_peered":"2024-04-06T03:23:57.956164+0000","last_clean":"2024-04-06T03:23:57.956164+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:23:57.956164+0000","last_undegraded":"2024-04-06T03:23:57.956164+0000","last_fullsized":"2024-04-06T03:23:57.956164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_clean_scrub_stamp":"2024-04-06T03:23:01.686796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.412050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769491+0000","last_change":"2024-04-06T03:23:36.848600+0000","last_active":"2024-04-06T03:23:58.769491+0000","last_peered":"2024-04-06T03:23:58.769491+0000","last_clean":"2024-04-06T03:23:58.769491+0000","last_became_active":"2024-04-06T03:20:51.143283+0000","last_became_peered":"2024-04-06T03:20:51.143283+0000","last_unstale":"2024-04-06T03:23:58.769491+0000","last_undegraded":"2024-04-06T03:23:58.769491+0000","last_fullsized":"2024-04-06T03:23:58.769491+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.848364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.848364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.392566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477025+0000","last_change":"2024-04-06T03:23:14.042484+0000","last_active":"2024-04-06T03:23:58.477025+0000","last_peered":"2024-04-06T03:23:58.477025+0000","last_clean":"2024-04-06T03:23:58.477025+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:23:58.477025+0000","last_undegraded":"2024-04-06T03:23:58.477025+0000","last_fullsized":"2024-04-06T03:23:58.477025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:14.042058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:14.042058+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.768307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.956315+0000","last_change":"2024-04-06T03:22:35.750314+0000","last_active":"2024-04-06T03:23:57.956315+0000","last_peered":"2024-04-06T03:23:57.956315+0000","last_clean":"2024-04-06T03:23:57.956315+0000","last_became_active":"2024-04-06T03:20:57.845246+0000","last_became_peered":"2024-04-06T03:20:57.845246+0000","last_unstale":"2024-04-06T03:23:57.956315+0000","last_undegraded":"2024-04-06T03:23:57.956315+0000","last_fullsized":"2024-04-06T03:23:57.956315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:35.750071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:35.750071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477684+0000","last_change":"2024-04-06T03:23:34.010043+0000","last_active":"2024-04-06T03:23:58.477684+0000","last_peered":"2024-04-06T03:23:58.477684+0000","last_clean":"2024-04-06T03:23:58.477684+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:23:58.477684+0000","last_undegraded":"2024-04-06T03:23:58.477684+0000","last_fullsized":"2024-04-06T03:23:58.477684+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:23:34.009837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.282584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836179+0000","last_change":"2024-04-06T03:23:43.924012+0000","last_active":"2024-04-06T03:23:58.836179+0000","last_peered":"2024-04-06T03:23:58.836179+0000","last_clean":"2024-04-06T03:23:58.836179+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:23:58.836179+0000","last_undegraded":"2024-04-06T03:23:58.836179+0000","last_fullsized":"2024-04-06T03:23:58.836179+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:43.923791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.923791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:10.650300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"109'20","reported_seq":127,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836393+0000","last_change":"2024-04-06T03:22:49.786714+0000","last_active":"2024-04-06T03:23:58.836393+0000","last_peered":"2024-04-06T03:23:58.836393+0000","last_clean":"2024-04-06T03:23:58.836393+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:23:58.836393+0000","last_undegraded":"2024-04-06T03:23:58.836393+0000","last_fullsized":"2024-04-06T03:23:58.836393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'20","last_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:22:49.786480+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:08.011745+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953882+0000","last_change":"2024-04-06T03:22:58.497756+0000","last_active":"2024-04-06T03:23:57.953882+0000","last_peered":"2024-04-06T03:23:57.953882+0000","last_clean":"2024-04-06T03:23:57.953882+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:23:57.953882+0000","last_undegraded":"2024-04-06T03:23:57.953882+0000","last_fullsized":"2024-04-06T03:23:57.953882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-06T03:22:58.497530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:58.497530+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:03.159362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"101'12","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770105+0000","last_change":"2024-04-06T03:23:33.869748+0000","last_active":"2024-04-06T03:23:58.770105+0000","last_peered":"2024-04-06T03:23:58.770105+0000","last_clean":"2024-04-06T03:23:58.770105+0000","last_became_active":"2024-04-06T03:20:55.765174+0000","last_became_peered":"2024-04-06T03:20:55.765174+0000","last_unstale":"2024-04-06T03:23:58.770105+0000","last_undegraded":"2024-04-06T03:23:58.770105+0000","last_fullsized":"2024-04-06T03:23:58.770105+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-06T03:23:33.869531+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.869531+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.720001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.465512+0000","last_change":"2024-04-06T03:23:19.763448+0000","last_active":"2024-04-06T03:23:57.465512+0000","last_peered":"2024-04-06T03:23:57.465512+0000","last_clean":"2024-04-06T03:23:57.465512+0000","last_became_active":"2024-04-06T03:20:51.144514+0000","last_became_peered":"2024-04-06T03:20:51.144514+0000","last_unstale":"2024-04-06T03:23:57.465512+0000","last_undegraded":"2024-04-06T03:23:57.465512+0000","last_fullsized":"2024-04-06T03:23:57.465512+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.763180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:54.626052+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.763180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:46.950926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"111'50","reported_seq":175,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955541+0000","last_change":"2024-04-06T03:23:04.830805+0000","last_active":"2024-04-06T03:23:57.955541+0000","last_peered":"2024-04-06T03:23:57.955541+0000","last_clean":"2024-04-06T03:23:57.955541+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:23:57.955541+0000","last_undegraded":"2024-04-06T03:23:57.955541+0000","last_fullsized":"2024-04-06T03:23:57.955541+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'50","last_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_clean_scrub_stamp":"2024-04-06T03:23:04.830548+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:25.118064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837665+0000","last_change":"2024-04-06T03:22:47.788318+0000","last_active":"2024-04-06T03:23:58.837665+0000","last_peered":"2024-04-06T03:23:58.837665+0000","last_clean":"2024-04-06T03:23:58.837665+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:23:58.837665+0000","last_undegraded":"2024-04-06T03:23:58.837665+0000","last_fullsized":"2024-04-06T03:23:58.837665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:22:47.788083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.587930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953476+0000","last_change":"2024-04-06T03:22:53.518032+0000","last_active":"2024-04-06T03:23:57.953476+0000","last_peered":"2024-04-06T03:23:57.953476+0000","last_clean":"2024-04-06T03:23:57.953476+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:23:57.953476+0000","last_undegraded":"2024-04-06T03:23:57.953476+0000","last_fullsized":"2024-04-06T03:23:57.953476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_clean_scrub_stamp":"2024-04-06T03:22:53.517730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.828631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770383+0000","last_change":"2024-04-06T03:23:26.863931+0000","last_active":"2024-04-06T03:23:58.770383+0000","last_peered":"2024-04-06T03:23:58.770383+0000","last_clean":"2024-04-06T03:23:58.770383+0000","last_became_active":"2024-04-06T03:20:51.142652+0000","last_became_peered":"2024-04-06T03:20:51.142652+0000","last_unstale":"2024-04-06T03:23:58.770383+0000","last_undegraded":"2024-04-06T03:23:58.770383+0000","last_fullsized":"2024-04-06T03:23:58.770383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:23:26.863691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:41.711589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":139,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466748+0000","last_change":"2024-04-06T03:23:47.908697+0000","last_active":"2024-04-06T03:23:57.466748+0000","last_peered":"2024-04-06T03:23:57.466748+0000","last_clean":"2024-04-06T03:23:57.466748+0000","last_became_active":"2024-04-06T03:20:53.350189+0000","last_became_peered":"2024-04-06T03:20:53.350189+0000","last_unstale":"2024-04-06T03:23:57.466748+0000","last_undegraded":"2024-04-06T03:23:57.466748+0000","last_fullsized":"2024-04-06T03:23:57.466748+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:47.908288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.908288+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.038863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952399+0000","last_change":"2024-04-06T03:22:29.395728+0000","last_active":"2024-04-06T03:23:57.952399+0000","last_peered":"2024-04-06T03:23:57.952399+0000","last_clean":"2024-04-06T03:23:57.952399+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:23:57.952399+0000","last_undegraded":"2024-04-06T03:23:57.952399+0000","last_fullsized":"2024-04-06T03:23:57.952399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_clean_scrub_stamp":"2024-04-06T03:22:29.395501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769129+0000","last_change":"2024-04-06T03:23:25.887364+0000","last_active":"2024-04-06T03:23:58.769129+0000","last_peered":"2024-04-06T03:23:58.769129+0000","last_clean":"2024-04-06T03:23:58.769129+0000","last_became_active":"2024-04-06T03:20:55.769876+0000","last_became_peered":"2024-04-06T03:20:55.769876+0000","last_unstale":"2024-04-06T03:23:58.769129+0000","last_undegraded":"2024-04-06T03:23:58.769129+0000","last_fullsized":"2024-04-06T03:23:58.769129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:25.887152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:25.887152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.529607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.467542+0000","last_change":"2024-04-06T03:23:12.715806+0000","last_active":"2024-04-06T03:23:57.467542+0000","last_peered":"2024-04-06T03:23:57.467542+0000","last_clean":"2024-04-06T03:23:57.467542+0000","last_became_active":"2024-04-06T03:20:51.151269+0000","last_became_peered":"2024-04-06T03:20:51.151269+0000","last_unstale":"2024-04-06T03:23:57.467542+0000","last_undegraded":"2024-04-06T03:23:57.467542+0000","last_fullsized":"2024-04-06T03:23:57.467542+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:12.715608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:12.715608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:14.405678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":121,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478669+0000","last_change":"2024-04-06T03:23:18.056122+0000","last_active":"2024-04-06T03:23:58.478669+0000","last_peered":"2024-04-06T03:23:58.478669+0000","last_clean":"2024-04-06T03:23:58.478669+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:23:58.478669+0000","last_undegraded":"2024-04-06T03:23:58.478669+0000","last_fullsized":"2024-04-06T03:23:58.478669+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:18.056024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.056024+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.251393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.835973+0000","last_change":"2024-04-06T03:22:57.659396+0000","last_active":"2024-04-06T03:23:58.835973+0000","last_peered":"2024-04-06T03:23:58.835973+0000","last_clean":"2024-04-06T03:23:58.835973+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:23:58.835973+0000","last_undegraded":"2024-04-06T03:23:58.835973+0000","last_fullsized":"2024-04-06T03:23:58.835973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:57.659282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:57.659282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:09.284205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955717+0000","last_change":"2024-04-06T03:22:55.746092+0000","last_active":"2024-04-06T03:23:57.955717+0000","last_peered":"2024-04-06T03:23:57.955717+0000","last_clean":"2024-04-06T03:23:57.955717+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:23:57.955717+0000","last_undegraded":"2024-04-06T03:23:57.955717+0000","last_fullsized":"2024-04-06T03:23:57.955717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:55.745785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:55.745785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955850+0000","last_change":"2024-04-06T03:23:12.824588+0000","last_active":"2024-04-06T03:23:57.955850+0000","last_peered":"2024-04-06T03:23:57.955850+0000","last_clean":"2024-04-06T03:23:57.955850+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:23:57.955850+0000","last_undegraded":"2024-04-06T03:23:57.955850+0000","last_fullsized":"2024-04-06T03:23:57.955850+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_clean_scrub_stamp":"2024-04-06T03:23:12.824475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:24.655532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'23","reported_seq":142,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472549+0000","last_change":"2024-04-06T03:23:29.339315+0000","last_active":"2024-04-06T03:23:58.472549+0000","last_peered":"2024-04-06T03:23:58.472549+0000","last_clean":"2024-04-06T03:23:58.472549+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:23:58.472549+0000","last_undegraded":"2024-04-06T03:23:58.472549+0000","last_fullsized":"2024-04-06T03:23:58.472549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-06T03:23:29.339052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.339052+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.010581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474688+0000","last_change":"2024-04-06T03:23:55.780644+0000","last_active":"2024-04-06T03:23:58.474688+0000","last_peered":"2024-04-06T03:23:58.474688+0000","last_clean":"2024-04-06T03:23:58.474688+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:23:58.474688+0000","last_undegraded":"2024-04-06T03:23:58.474688+0000","last_fullsized":"2024-04-06T03:23:58.474688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.780397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.780397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.956035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837343+0000","last_change":"2024-04-06T03:23:52.841531+0000","last_active":"2024-04-06T03:23:58.837343+0000","last_peered":"2024-04-06T03:23:58.837343+0000","last_clean":"2024-04-06T03:23:58.837343+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:23:58.837343+0000","last_undegraded":"2024-04-06T03:23:58.837343+0000","last_fullsized":"2024-04-06T03:23:58.837343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:52.841291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.841291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.389698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769616+0000","last_change":"2024-04-06T03:23:16.851233+0000","last_active":"2024-04-06T03:23:58.769616+0000","last_peered":"2024-04-06T03:23:58.769616+0000","last_clean":"2024-04-06T03:23:58.769616+0000","last_became_active":"2024-04-06T03:20:51.137376+0000","last_became_peered":"2024-04-06T03:20:51.137376+0000","last_unstale":"2024-04-06T03:23:58.769616+0000","last_undegraded":"2024-04-06T03:23:58.769616+0000","last_fullsized":"2024-04-06T03:23:58.769616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:16.851024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:23:16.851024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:18.428925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.954528+0000","last_change":"2024-04-06T03:22:46.586344+0000","last_active":"2024-04-06T03:23:57.954528+0000","last_peered":"2024-04-06T03:23:57.954528+0000","last_clean":"2024-04-06T03:23:57.954528+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:23:57.954528+0000","last_undegraded":"2024-04-06T03:23:57.954528+0000","last_fullsized":"2024-04-06T03:23:57.954528+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:22:46.586091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.586091+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473782+0000","last_change":"2024-04-06T03:23:19.298817+0000","last_active":"2024-04-06T03:23:58.473782+0000","last_peered":"2024-04-06T03:23:58.473782+0000","last_clean":"2024-04-06T03:23:58.473782+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:23:58.473782+0000","last_undegraded":"2024-04-06T03:23:58.473782+0000","last_fullsized":"2024-04-06T03:23:58.473782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.298498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.298498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.565314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":139,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837334+0000","last_change":"2024-04-06T03:22:59.658875+0000","last_active":"2024-04-06T03:23:58.837334+0000","last_peered":"2024-04-06T03:23:58.837334+0000","last_clean":"2024-04-06T03:23:58.837334+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:23:58.837334+0000","last_undegraded":"2024-04-06T03:23:58.837334+0000","last_fullsized":"2024-04-06T03:23:58.837334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:22:59.658761+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:20.379794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477187+0000","last_change":"2024-04-06T03:23:40.956392+0000","last_active":"2024-04-06T03:23:58.477187+0000","last_peered":"2024-04-06T03:23:58.477187+0000","last_clean":"2024-04-06T03:23:58.477187+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:23:58.477187+0000","last_undegraded":"2024-04-06T03:23:58.477187+0000","last_fullsized":"2024-04-06T03:23:58.477187+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:40.956167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:40.956167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.850753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.476161+0000","last_change":"2024-04-06T03:22:51.222996+0000","last_active":"2024-04-06T03:23:58.476161+0000","last_peered":"2024-04-06T03:23:58.476161+0000","last_clean":"2024-04-06T03:23:58.476161+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:23:58.476161+0000","last_undegraded":"2024-04-06T03:23:58.476161+0000","last_fullsized":"2024-04-06T03:23:58.476161+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-06T03:22:51.222590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:51.222590+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477224+0000","last_change":"2024-04-06T03:23:28.068250+0000","last_active":"2024-04-06T03:23:58.477224+0000","last_peered":"2024-04-06T03:23:58.477224+0000","last_clean":"2024-04-06T03:23:58.477224+0000","last_became_active":"2024-04-06T03:20:57.843310+0000","last_became_peered":"2024-04-06T03:20:57.843310+0000","last_unstale":"2024-04-06T03:23:58.477224+0000","last_undegraded":"2024-04-06T03:23:58.477224+0000","last_fullsized":"2024-04-06T03:23:58.477224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.067970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.747904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"85'11","reported_seq":143,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.467148+0000","last_change":"2024-04-06T03:23:45.891079+0000","last_active":"2024-04-06T03:23:57.467148+0000","last_peered":"2024-04-06T03:23:57.467148+0000","last_clean":"2024-04-06T03:23:57.467148+0000","last_became_active":"2024-04-06T03:20:55.770173+0000","last_became_peered":"2024-04-06T03:20:55.770173+0000","last_unstale":"2024-04-06T03:23:57.467148+0000","last_undegraded":"2024-04-06T03:23:57.467148+0000","last_fullsized":"2024-04-06T03:23:57.467148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:45.890804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:45.890804+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.597014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466038+0000","last_change":"2024-04-06T03:23:21.739882+0000","last_active":"2024-04-06T03:23:57.466038+0000","last_peered":"2024-04-06T03:23:57.466038+0000","last_clean":"2024-04-06T03:23:57.466038+0000","last_became_active":"2024-04-06T03:20:51.157201+0000","last_became_peered":"2024-04-06T03:20:51.157201+0000","last_unstale":"2024-04-06T03:23:57.466038+0000","last_undegraded":"2024-04-06T03:23:57.466038+0000","last_fullsized":"2024-04-06T03:23:57.466038+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:21.739671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.739671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.621751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":109,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953720+0000","last_change":"2024-04-06T03:22:15.465954+0000","last_active":"2024-04-06T03:23:57.953720+0000","last_peered":"2024-04-06T03:23:57.953720+0000","last_clean":"2024-04-06T03:23:57.953720+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:23:57.953720+0000","last_undegraded":"2024-04-06T03:23:57.953720+0000","last_fullsized":"2024-04-06T03:23:57.953720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:22:15.465569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:15.465569+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953402+0000","last_change":"2024-04-06T03:23:55.806848+0000","last_active":"2024-04-06T03:23:57.953402+0000","last_peered":"2024-04-06T03:23:57.953402+0000","last_clean":"2024-04-06T03:23:57.953402+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:23:57.953402+0000","last_undegraded":"2024-04-06T03:23:57.953402+0000","last_fullsized":"2024-04-06T03:23:57.953402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.806612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.806612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.253217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466926+0000","last_change":"2024-04-06T03:23:31.955588+0000","last_active":"2024-04-06T03:23:57.466926+0000","last_peered":"2024-04-06T03:23:57.466926+0000","last_clean":"2024-04-06T03:23:57.466926+0000","last_became_active":"2024-04-06T03:20:55.756465+0000","last_became_peered":"2024-04-06T03:20:55.756465+0000","last_unstale":"2024-04-06T03:23:57.466926+0000","last_undegraded":"2024-04-06T03:23:57.466926+0000","last_fullsized":"2024-04-06T03:23:57.466926+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:31.955371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:31.955371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:56.516085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769295+0000","last_change":"2024-04-06T03:23:35.835087+0000","last_active":"2024-04-06T03:23:58.769295+0000","last_peered":"2024-04-06T03:23:58.769295+0000","last_clean":"2024-04-06T03:23:58.769295+0000","last_became_active":"2024-04-06T03:20:51.145847+0000","last_became_peered":"2024-04-06T03:20:51.145847+0000","last_unstale":"2024-04-06T03:23:58.769295+0000","last_undegraded":"2024-04-06T03:23:58.769295+0000","last_fullsized":"2024-04-06T03:23:58.769295+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.834842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.834842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.169606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"111'6","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769153+0000","last_change":"2024-04-06T03:23:21.875463+0000","last_active":"2024-04-06T03:23:58.769153+0000","last_peered":"2024-04-06T03:23:58.769153+0000","last_clean":"2024-04-06T03:23:58.769153+0000","last_became_active":"2024-04-06T03:20:53.348930+0000","last_became_peered":"2024-04-06T03:20:53.348930+0000","last_unstale":"2024-04-06T03:23:58.769153+0000","last_undegraded":"2024-04-06T03:23:58.769153+0000","last_fullsized":"2024-04-06T03:23:58.769153+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6","last_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.875237+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.782953+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.475400+0000","last_change":"2024-04-06T03:23:10.631170+0000","last_active":"2024-04-06T03:23:58.475400+0000","last_peered":"2024-04-06T03:23:58.475400+0000","last_clean":"2024-04-06T03:23:58.475400+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:23:58.475400+0000","last_undegraded":"2024-04-06T03:23:58.475400+0000","last_fullsized":"2024-04-06T03:23:58.475400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:23:10.631051+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:10.891042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.838113+0000","last_change":"2024-04-06T03:22:50.729388+0000","last_active":"2024-04-06T03:23:58.838113+0000","last_peered":"2024-04-06T03:23:58.838113+0000","last_clean":"2024-04-06T03:23:58.838113+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:23:58.838113+0000","last_undegraded":"2024-04-06T03:23:58.838113+0000","last_fullsized":"2024-04-06T03:23:58.838113+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:50.729294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:50.729294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:18.365897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473357+0000","last_change":"2024-04-06T03:23:23.589775+0000","last_active":"2024-04-06T03:23:58.473357+0000","last_peered":"2024-04-06T03:23:58.473357+0000","last_clean":"2024-04-06T03:23:58.473357+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:23:58.473357+0000","last_undegraded":"2024-04-06T03:23:58.473357+0000","last_fullsized":"2024-04-06T03:23:58.473357+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:23.589472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.589472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.531624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"111'23","reported_seq":135,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952489+0000","last_change":"2024-04-06T03:23:20.835769+0000","last_active":"2024-04-06T03:23:57.952489+0000","last_peered":"2024-04-06T03:23:57.952489+0000","last_clean":"2024-04-06T03:23:57.952489+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:23:57.952489+0000","last_undegraded":"2024-04-06T03:23:57.952489+0000","last_fullsized":"2024-04-06T03:23:57.952489+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'23","last_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.835514+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:30.227431+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837493+0000","last_change":"2024-04-06T03:23:44.976222+0000","last_active":"2024-04-06T03:23:58.837493+0000","last_peered":"2024-04-06T03:23:58.837493+0000","last_clean":"2024-04-06T03:23:58.837493+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:23:58.837493+0000","last_undegraded":"2024-04-06T03:23:58.837493+0000","last_fullsized":"2024-04-06T03:23:58.837493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.976015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.976015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.423557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473045+0000","last_change":"2024-04-06T03:23:09.163493+0000","last_active":"2024-04-06T03:23:58.473045+0000","last_peered":"2024-04-06T03:23:58.473045+0000","last_clean":"2024-04-06T03:23:58.473045+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:23:58.473045+0000","last_undegraded":"2024-04-06T03:23:58.473045+0000","last_fullsized":"2024-04-06T03:23:58.473045+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-06T03:23:09.163254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.163254+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.561644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":129,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953916+0000","last_change":"2024-04-06T03:23:19.822376+0000","last_active":"2024-04-06T03:23:57.953916+0000","last_peered":"2024-04-06T03:23:57.953916+0000","last_clean":"2024-04-06T03:23:57.953916+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:23:57.953916+0000","last_undegraded":"2024-04-06T03:23:57.953916+0000","last_fullsized":"2024-04-06T03:23:57.953916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:19.822131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.822131+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.155287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473044+0000","last_change":"2024-04-06T03:23:52.629246+0000","last_active":"2024-04-06T03:23:58.473044+0000","last_peered":"2024-04-06T03:23:58.473044+0000","last_clean":"2024-04-06T03:23:58.473044+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:23:58.473044+0000","last_undegraded":"2024-04-06T03:23:58.473044+0000","last_fullsized":"2024-04-06T03:23:58.473044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:52.629002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.629002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.822619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474932+0000","last_change":"2024-04-06T03:22:44.386258+0000","last_active":"2024-04-06T03:23:58.474932+0000","last_peered":"2024-04-06T03:23:58.474932+0000","last_clean":"2024-04-06T03:23:58.474932+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:23:58.474932+0000","last_undegraded":"2024-04-06T03:23:58.474932+0000","last_fullsized":"2024-04-06T03:23:58.474932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:22:44.386020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.128819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":111,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.956217+0000","last_change":"2024-04-06T03:23:50.717593+0000","last_active":"2024-04-06T03:23:57.956217+0000","last_peered":"2024-04-06T03:23:57.956217+0000","last_clean":"2024-04-06T03:23:57.956217+0000","last_became_active":"2024-04-06T03:20:51.148632+0000","last_became_peered":"2024-04-06T03:20:51.148632+0000","last_unstale":"2024-04-06T03:23:57.956217+0000","last_undegraded":"2024-04-06T03:23:57.956217+0000","last_fullsized":"2024-04-06T03:23:57.956217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:50.717354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:50.717354+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:52.790825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":151,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466808+0000","last_change":"2024-04-06T03:22:40.710426+0000","last_active":"2024-04-06T03:23:57.466808+0000","last_peered":"2024-04-06T03:23:57.466808+0000","last_clean":"2024-04-06T03:23:57.466808+0000","last_became_active":"2024-04-06T03:20:53.351328+0000","last_became_peered":"2024-04-06T03:20:53.351328+0000","last_unstale":"2024-04-06T03:23:57.466808+0000","last_undegraded":"2024-04-06T03:23:57.466808+0000","last_fullsized":"2024-04-06T03:23:57.466808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:22:40.710327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:40.710327+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:05.111206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955928+0000","last_change":"2024-04-06T03:23:56.666850+0000","last_active":"2024-04-06T03:23:57.955928+0000","last_peered":"2024-04-06T03:23:57.955928+0000","last_clean":"2024-04-06T03:23:57.955928+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:23:57.955928+0000","last_undegraded":"2024-04-06T03:23:57.955928+0000","last_fullsized":"2024-04-06T03:23:57.955928+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:23:56.666442+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.666442+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:17.097001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":136,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.476840+0000","last_change":"2024-04-06T03:22:58.133626+0000","last_active":"2024-04-06T03:23:58.476840+0000","last_peered":"2024-04-06T03:23:58.476840+0000","last_clean":"2024-04-06T03:23:58.476840+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:23:58.476840+0000","last_undegraded":"2024-04-06T03:23:58.476840+0000","last_fullsized":"2024-04-06T03:23:58.476840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:22:58.133367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:58.133367+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:20.887571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837492+0000","last_change":"2024-04-06T03:23:00.610221+0000","last_active":"2024-04-06T03:23:58.837492+0000","last_peered":"2024-04-06T03:23:58.837492+0000","last_clean":"2024-04-06T03:23:58.837492+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:23:58.837492+0000","last_undegraded":"2024-04-06T03:23:58.837492+0000","last_fullsized":"2024-04-06T03:23:58.837492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:00.610114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:00.610114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:06.812327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466179+0000","last_change":"2024-04-06T03:23:27.845925+0000","last_active":"2024-04-06T03:23:57.466179+0000","last_peered":"2024-04-06T03:23:57.466179+0000","last_clean":"2024-04-06T03:23:57.466179+0000","last_became_active":"2024-04-06T03:20:51.148366+0000","last_became_peered":"2024-04-06T03:20:51.148366+0000","last_unstale":"2024-04-06T03:23:57.466179+0000","last_undegraded":"2024-04-06T03:23:57.466179+0000","last_fullsized":"2024-04-06T03:23:57.466179+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:27.845710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:27.845710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.306766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'34","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472076+0000","last_change":"2024-04-06T03:22:31.389374+0000","last_active":"2024-04-06T03:23:58.472076+0000","last_peered":"2024-04-06T03:23:58.472076+0000","last_clean":"2024-04-06T03:23:58.472076+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:23:58.472076+0000","last_undegraded":"2024-04-06T03:23:58.472076+0000","last_fullsized":"2024-04-06T03:23:58.472076+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'34","last_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.389226+0000","objects_scrubbed":9,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.476167+0000","last_change":"2024-04-06T03:22:30.412875+0000","last_active":"2024-04-06T03:23:58.476167+0000","last_peered":"2024-04-06T03:23:58.476167+0000","last_clean":"2024-04-06T03:23:58.476167+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:23:58.476167+0000","last_undegraded":"2024-04-06T03:23:58.476167+0000","last_fullsized":"2024-04-06T03:23:58.476167+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:30.412667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:30.412667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:00.364828+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953720+0000","last_change":"2024-04-06T03:22:30.391636+0000","last_active":"2024-04-06T03:23:57.953720+0000","last_peered":"2024-04-06T03:23:57.953720+0000","last_clean":"2024-04-06T03:23:57.953720+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:23:57.953720+0000","last_undegraded":"2024-04-06T03:23:57.953720+0000","last_fullsized":"2024-04-06T03:23:57.953720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:22:30.391413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":155,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955266+0000","last_change":"2024-04-06T03:23:35.682185+0000","last_active":"2024-04-06T03:23:57.955266+0000","last_peered":"2024-04-06T03:23:57.955266+0000","last_clean":"2024-04-06T03:23:57.955266+0000","last_became_active":"2024-04-06T03:20:53.358177+0000","last_became_peered":"2024-04-06T03:20:53.358177+0000","last_unstale":"2024-04-06T03:23:57.955266+0000","last_undegraded":"2024-04-06T03:23:57.955266+0000","last_fullsized":"2024-04-06T03:23:57.955266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-06T03:23:35.681909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.681909+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.508581+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473242+0000","last_change":"2024-04-06T03:23:53.594743+0000","last_active":"2024-04-06T03:23:58.473242+0000","last_peered":"2024-04-06T03:23:58.473242+0000","last_clean":"2024-04-06T03:23:58.473242+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:23:58.473242+0000","last_undegraded":"2024-04-06T03:23:58.473242+0000","last_fullsized":"2024-04-06T03:23:58.473242+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:53.594492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:53.594492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:18.217623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473560+0000","last_change":"2024-04-06T03:23:47.598906+0000","last_active":"2024-04-06T03:23:58.473560+0000","last_peered":"2024-04-06T03:23:58.473560+0000","last_clean":"2024-04-06T03:23:58.473560+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:23:58.473560+0000","last_undegraded":"2024-04-06T03:23:58.473560+0000","last_fullsized":"2024-04-06T03:23:58.473560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:47.598682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.598682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.431027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.468336+0000","last_change":"2024-04-06T03:23:37.854225+0000","last_active":"2024-04-06T03:23:57.468336+0000","last_peered":"2024-04-06T03:23:57.468336+0000","last_clean":"2024-04-06T03:23:57.468336+0000","last_became_active":"2024-04-06T03:20:57.822724+0000","last_became_peered":"2024-04-06T03:20:57.822724+0000","last_unstale":"2024-04-06T03:23:57.468336+0000","last_undegraded":"2024-04-06T03:23:57.468336+0000","last_fullsized":"2024-04-06T03:23:57.468336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:37.854012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.854012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.515577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836547+0000","last_change":"2024-04-06T03:23:51.862357+0000","last_active":"2024-04-06T03:23:58.836547+0000","last_peered":"2024-04-06T03:23:58.836547+0000","last_clean":"2024-04-06T03:23:58.836547+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:23:58.836547+0000","last_undegraded":"2024-04-06T03:23:58.836547+0000","last_fullsized":"2024-04-06T03:23:58.836547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:23:51.861992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:51.861992+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.736112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466195+0000","last_change":"2024-04-06T03:23:20.743012+0000","last_active":"2024-04-06T03:23:57.466195+0000","last_peered":"2024-04-06T03:23:57.466195+0000","last_clean":"2024-04-06T03:23:57.466195+0000","last_became_active":"2024-04-06T03:20:51.151534+0000","last_became_peered":"2024-04-06T03:20:51.151534+0000","last_unstale":"2024-04-06T03:23:57.466195+0000","last_undegraded":"2024-04-06T03:23:57.466195+0000","last_fullsized":"2024-04-06T03:23:57.466195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:20.742768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.742768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:26.458288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":353,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952503+0000","last_change":"2024-04-06T03:22:36.461819+0000","last_active":"2024-04-06T03:23:57.952503+0000","last_peered":"2024-04-06T03:23:57.952503+0000","last_clean":"2024-04-06T03:23:57.952503+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:23:57.952503+0000","last_undegraded":"2024-04-06T03:23:57.952503+0000","last_fullsized":"2024-04-06T03:23:57.952503+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:22:36.461578+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:02.479275+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":86,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953564+0000","last_change":"2024-04-06T03:22:26.403696+0000","last_active":"2024-04-06T03:23:57.953564+0000","last_peered":"2024-04-06T03:23:57.953564+0000","last_clean":"2024-04-06T03:23:57.953564+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:23:57.953564+0000","last_undegraded":"2024-04-06T03:23:57.953564+0000","last_fullsized":"2024-04-06T03:23:57.953564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:26.403500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:26.403500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955699+0000","last_change":"2024-04-06T03:22:39.710602+0000","last_active":"2024-04-06T03:23:57.955699+0000","last_peered":"2024-04-06T03:23:57.955699+0000","last_clean":"2024-04-06T03:23:57.955699+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:23:57.955699+0000","last_undegraded":"2024-04-06T03:23:57.955699+0000","last_fullsized":"2024-04-06T03:23:57.955699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:39.710513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:39.710513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'21","reported_seq":134,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478175+0000","last_change":"2024-04-06T03:23:23.076419+0000","last_active":"2024-04-06T03:23:58.478175+0000","last_peered":"2024-04-06T03:23:58.478175+0000","last_clean":"2024-04-06T03:23:58.478175+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:23:58.478175+0000","last_undegraded":"2024-04-06T03:23:58.478175+0000","last_fullsized":"2024-04-06T03:23:58.478175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:23.076269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.076269+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.958707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952354+0000","last_change":"2024-04-06T03:22:54.515963+0000","last_active":"2024-04-06T03:23:57.952354+0000","last_peered":"2024-04-06T03:23:57.952354+0000","last_clean":"2024-04-06T03:23:57.952354+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:23:57.952354+0000","last_undegraded":"2024-04-06T03:23:57.952354+0000","last_fullsized":"2024-04-06T03:23:57.952354+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:54.515739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:54.515739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:15.695306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.468080+0000","last_change":"2024-04-06T03:23:28.869298+0000","last_active":"2024-04-06T03:23:57.468080+0000","last_peered":"2024-04-06T03:23:57.468080+0000","last_clean":"2024-04-06T03:23:57.468080+0000","last_became_active":"2024-04-06T03:20:55.754600+0000","last_became_peered":"2024-04-06T03:20:55.754600+0000","last_unstale":"2024-04-06T03:23:57.468080+0000","last_undegraded":"2024-04-06T03:23:57.468080+0000","last_fullsized":"2024-04-06T03:23:57.468080+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.869068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.869068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.304800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955126+0000","last_change":"2024-04-06T03:23:48.690384+0000","last_active":"2024-04-06T03:23:57.955126+0000","last_peered":"2024-04-06T03:23:57.955126+0000","last_clean":"2024-04-06T03:23:57.955126+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:23:57.955126+0000","last_undegraded":"2024-04-06T03:23:57.955126+0000","last_fullsized":"2024-04-06T03:23:57.955126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:48.690059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.690059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.780733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.954022+0000","last_change":"2024-04-06T03:22:10.497983+0000","last_active":"2024-04-06T03:23:57.954022+0000","last_peered":"2024-04-06T03:23:57.954022+0000","last_clean":"2024-04-06T03:23:57.954022+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:23:57.954022+0000","last_undegraded":"2024-04-06T03:23:57.954022+0000","last_fullsized":"2024-04-06T03:23:57.954022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:10.497772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:10.497772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'37","reported_seq":200,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.956249+0000","last_change":"2024-04-06T03:23:09.792573+0000","last_active":"2024-04-06T03:23:57.956249+0000","last_peered":"2024-04-06T03:23:57.956249+0000","last_clean":"2024-04-06T03:23:57.956249+0000","last_became_active":"2024-04-06T03:20:53.352453+0000","last_became_peered":"2024-04-06T03:20:53.352453+0000","last_unstale":"2024-04-06T03:23:57.956249+0000","last_undegraded":"2024-04-06T03:23:57.956249+0000","last_fullsized":"2024-04-06T03:23:57.956249+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'37","last_scrub_stamp":"2024-04-06T03:23:09.792373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.792373+0000","objects_scrubbed":11,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.096219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770224+0000","last_change":"2024-04-06T03:23:39.777144+0000","last_active":"2024-04-06T03:23:58.770224+0000","last_peered":"2024-04-06T03:23:58.770224+0000","last_clean":"2024-04-06T03:23:58.770224+0000","last_became_active":"2024-04-06T03:20:55.769795+0000","last_became_peered":"2024-04-06T03:20:55.769795+0000","last_unstale":"2024-04-06T03:23:58.770224+0000","last_undegraded":"2024-04-06T03:23:58.770224+0000","last_fullsized":"2024-04-06T03:23:58.770224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.776933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.776933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.208938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474197+0000","last_change":"2024-04-06T03:23:35.364623+0000","last_active":"2024-04-06T03:23:58.474197+0000","last_peered":"2024-04-06T03:23:58.474197+0000","last_clean":"2024-04-06T03:23:58.474197+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:23:58.474197+0000","last_undegraded":"2024-04-06T03:23:58.474197+0000","last_fullsized":"2024-04-06T03:23:58.474197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.364381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.332831+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.364381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.739996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.956538+0000","last_change":"2024-04-06T03:23:37.757642+0000","last_active":"2024-04-06T03:23:57.956538+0000","last_peered":"2024-04-06T03:23:57.956538+0000","last_clean":"2024-04-06T03:23:57.956538+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:23:57.956538+0000","last_undegraded":"2024-04-06T03:23:57.956538+0000","last_fullsized":"2024-04-06T03:23:57.956538+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:37.757420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.757420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.290369+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.838233+0000","last_change":"2024-04-06T03:22:46.807318+0000","last_active":"2024-04-06T03:23:58.838233+0000","last_peered":"2024-04-06T03:23:58.838233+0000","last_clean":"2024-04-06T03:23:58.838233+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:23:58.838233+0000","last_undegraded":"2024-04-06T03:23:58.838233+0000","last_fullsized":"2024-04-06T03:23:58.838233+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-06T03:22:46.807072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.807072+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:03.325548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":276,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.835643+0000","last_change":"2024-04-06T03:23:06.631761+0000","last_active":"2024-04-06T03:23:58.835643+0000","last_peered":"2024-04-06T03:23:58.835643+0000","last_clean":"2024-04-06T03:23:58.835643+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:23:58.835643+0000","last_undegraded":"2024-04-06T03:23:58.835643+0000","last_fullsized":"2024-04-06T03:23:58.835643+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:23:06.631539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:23:06.631539+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:25.562835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":166,"num_read_kb":161,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472783+0000","last_change":"2024-04-06T03:23:53.414430+0000","last_active":"2024-04-06T03:23:58.472783+0000","last_peered":"2024-04-06T03:23:58.472783+0000","last_clean":"2024-04-06T03:23:58.472783+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:23:58.472783+0000","last_undegraded":"2024-04-06T03:23:58.472783+0000","last_fullsized":"2024-04-06T03:23:58.472783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:53.414220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:53.414220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:07.437199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952793+0000","last_change":"2024-04-06T03:22:43.626547+0000","last_active":"2024-04-06T03:23:57.952793+0000","last_peered":"2024-04-06T03:23:57.952793+0000","last_clean":"2024-04-06T03:23:57.952793+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:23:57.952793+0000","last_undegraded":"2024-04-06T03:23:57.952793+0000","last_fullsized":"2024-04-06T03:23:57.952793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:43.626233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:43.626233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:05.538215+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770152+0000","last_change":"2024-04-06T03:23:17.864400+0000","last_active":"2024-04-06T03:23:58.770152+0000","last_peered":"2024-04-06T03:23:58.770152+0000","last_clean":"2024-04-06T03:23:58.770152+0000","last_became_active":"2024-04-06T03:20:51.134999+0000","last_became_peered":"2024-04-06T03:20:51.134999+0000","last_unstale":"2024-04-06T03:23:58.770152+0000","last_undegraded":"2024-04-06T03:23:58.770152+0000","last_fullsized":"2024-04-06T03:23:58.770152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.864183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.536200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478081+0000","last_change":"2024-04-06T03:23:39.032748+0000","last_active":"2024-04-06T03:23:58.478081+0000","last_peered":"2024-04-06T03:23:58.478081+0000","last_clean":"2024-04-06T03:23:58.478081+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:23:58.478081+0000","last_undegraded":"2024-04-06T03:23:58.478081+0000","last_fullsized":"2024-04-06T03:23:58.478081+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:39.032620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.032620+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:39.700020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472144+0000","last_change":"2024-04-06T03:23:46.374564+0000","last_active":"2024-04-06T03:23:58.472144+0000","last_peered":"2024-04-06T03:23:58.472144+0000","last_clean":"2024-04-06T03:23:58.472144+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:23:58.472144+0000","last_undegraded":"2024-04-06T03:23:58.472144+0000","last_fullsized":"2024-04-06T03:23:58.472144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.374356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.995508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":139,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.835923+0000","last_change":"2024-04-06T03:22:58.657483+0000","last_active":"2024-04-06T03:23:58.835923+0000","last_peered":"2024-04-06T03:23:58.835923+0000","last_clean":"2024-04-06T03:23:58.835923+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:23:58.835923+0000","last_undegraded":"2024-04-06T03:23:58.835923+0000","last_fullsized":"2024-04-06T03:23:58.835923+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:22:58.657247+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:12.252538+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837001+0000","last_change":"2024-04-06T03:23:38.882403+0000","last_active":"2024-04-06T03:23:58.837001+0000","last_peered":"2024-04-06T03:23:58.837001+0000","last_clean":"2024-04-06T03:23:58.837001+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:23:58.837001+0000","last_undegraded":"2024-04-06T03:23:58.837001+0000","last_fullsized":"2024-04-06T03:23:58.837001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.882269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.592970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.468176+0000","last_change":"2024-04-06T03:23:11.690418+0000","last_active":"2024-04-06T03:23:57.468176+0000","last_peered":"2024-04-06T03:23:57.468176+0000","last_clean":"2024-04-06T03:23:57.468176+0000","last_became_active":"2024-04-06T03:20:55.762055+0000","last_became_peered":"2024-04-06T03:20:55.762055+0000","last_unstale":"2024-04-06T03:23:57.468176+0000","last_undegraded":"2024-04-06T03:23:57.468176+0000","last_fullsized":"2024-04-06T03:23:57.468176+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:11.690170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:11.690170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:35.579295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952833+0000","last_change":"2024-04-06T03:22:44.643404+0000","last_active":"2024-04-06T03:23:57.952833+0000","last_peered":"2024-04-06T03:23:57.952833+0000","last_clean":"2024-04-06T03:23:57.952833+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:23:57.952833+0000","last_undegraded":"2024-04-06T03:23:57.952833+0000","last_fullsized":"2024-04-06T03:23:57.952833+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:44.643181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:44.643181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:10.998024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":149,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.471804+0000","last_change":"2024-04-06T03:23:56.442918+0000","last_active":"2024-04-06T03:23:58.471804+0000","last_peered":"2024-04-06T03:23:58.471804+0000","last_clean":"2024-04-06T03:23:58.471804+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:23:58.471804+0000","last_undegraded":"2024-04-06T03:23:58.471804+0000","last_fullsized":"2024-04-06T03:23:58.471804+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-06T03:23:56.442661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.442661+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.735553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.466571+0000","last_change":"2024-04-06T03:23:38.883987+0000","last_active":"2024-04-06T03:23:57.466571+0000","last_peered":"2024-04-06T03:23:57.466571+0000","last_clean":"2024-04-06T03:23:57.466571+0000","last_became_active":"2024-04-06T03:20:57.845535+0000","last_became_peered":"2024-04-06T03:20:57.845535+0000","last_unstale":"2024-04-06T03:23:57.466571+0000","last_undegraded":"2024-04-06T03:23:57.466571+0000","last_fullsized":"2024-04-06T03:23:57.466571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.883738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.883738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.269641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"85'11","reported_seq":146,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473700+0000","last_change":"2024-04-06T03:23:23.295365+0000","last_active":"2024-04-06T03:23:58.473700+0000","last_peered":"2024-04-06T03:23:58.473700+0000","last_clean":"2024-04-06T03:23:58.473700+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:23:58.473700+0000","last_undegraded":"2024-04-06T03:23:58.473700+0000","last_fullsized":"2024-04-06T03:23:58.473700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:23.295124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.295124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.215136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955412+0000","last_change":"2024-04-06T03:22:31.784166+0000","last_active":"2024-04-06T03:23:57.955412+0000","last_peered":"2024-04-06T03:23:57.955412+0000","last_clean":"2024-04-06T03:23:57.955412+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:23:57.955412+0000","last_undegraded":"2024-04-06T03:23:57.955412+0000","last_fullsized":"2024-04-06T03:23:57.955412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:31.784009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:31.784009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"85'27","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836070+0000","last_change":"2024-04-06T03:23:57.838208+0000","last_active":"2024-04-06T03:23:58.836070+0000","last_peered":"2024-04-06T03:23:58.836070+0000","last_clean":"2024-04-06T03:23:58.836070+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:23:58.836070+0000","last_undegraded":"2024-04-06T03:23:58.836070+0000","last_fullsized":"2024-04-06T03:23:58.836070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'27","last_scrub_stamp":"2024-04-06T03:23:57.837997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:57.837997+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.275078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473984+0000","last_change":"2024-04-06T03:23:36.391113+0000","last_active":"2024-04-06T03:23:58.473984+0000","last_peered":"2024-04-06T03:23:58.473984+0000","last_clean":"2024-04-06T03:23:58.473984+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:23:58.473984+0000","last_undegraded":"2024-04-06T03:23:58.473984+0000","last_fullsized":"2024-04-06T03:23:58.473984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.390806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.390806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:00.809816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":147,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474218+0000","last_change":"2024-04-06T03:23:29.528196+0000","last_active":"2024-04-06T03:23:58.474218+0000","last_peered":"2024-04-06T03:23:58.474218+0000","last_clean":"2024-04-06T03:23:58.474218+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:23:58.474218+0000","last_undegraded":"2024-04-06T03:23:58.474218+0000","last_fullsized":"2024-04-06T03:23:58.474218+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.528005+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:29.753304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955575+0000","last_change":"2024-04-06T03:23:07.798328+0000","last_active":"2024-04-06T03:23:57.955575+0000","last_peered":"2024-04-06T03:23:57.955575+0000","last_clean":"2024-04-06T03:23:57.955575+0000","last_became_active":"2024-04-06T03:20:51.167121+0000","last_became_peered":"2024-04-06T03:20:51.167121+0000","last_unstale":"2024-04-06T03:23:57.955575+0000","last_undegraded":"2024-04-06T03:23:57.955575+0000","last_fullsized":"2024-04-06T03:23:57.955575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:07.798091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:07.798091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:28.780888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"85'15","reported_seq":130,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837140+0000","last_change":"2024-04-06T03:23:24.821416+0000","last_active":"2024-04-06T03:23:58.837140+0000","last_peered":"2024-04-06T03:23:58.837140+0000","last_clean":"2024-04-06T03:23:58.837140+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:23:58.837140+0000","last_undegraded":"2024-04-06T03:23:58.837140+0000","last_fullsized":"2024-04-06T03:23:58.837140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:24.821164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:24.821164+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.815641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.479797+0000","last_change":"2024-04-06T03:23:55.616868+0000","last_active":"2024-04-06T03:23:58.479797+0000","last_peered":"2024-04-06T03:23:58.479797+0000","last_clean":"2024-04-06T03:23:58.479797+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:23:58.479797+0000","last_undegraded":"2024-04-06T03:23:58.479797+0000","last_fullsized":"2024-04-06T03:23:58.479797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.616541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:04.260063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.955911+0000","last_change":"2024-04-06T03:22:40.755288+0000","last_active":"2024-04-06T03:23:57.955911+0000","last_peered":"2024-04-06T03:23:57.955911+0000","last_clean":"2024-04-06T03:23:57.955911+0000","last_became_active":"2024-04-06T03:20:55.758951+0000","last_became_peered":"2024-04-06T03:20:55.758951+0000","last_unstale":"2024-04-06T03:23:57.955911+0000","last_undegraded":"2024-04-06T03:23:57.955911+0000","last_fullsized":"2024-04-06T03:23:57.955911+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:40.755077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:22:40.755077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953098+0000","last_change":"2024-04-06T03:23:15.710252+0000","last_active":"2024-04-06T03:23:57.953098+0000","last_peered":"2024-04-06T03:23:57.953098+0000","last_clean":"2024-04-06T03:23:57.953098+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:23:57.953098+0000","last_undegraded":"2024-04-06T03:23:57.953098+0000","last_fullsized":"2024-04-06T03:23:57.953098+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:15.710038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:15.710038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.258607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'25","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769247+0000","last_change":"2024-04-06T03:23:22.871567+0000","last_active":"2024-04-06T03:23:58.769247+0000","last_peered":"2024-04-06T03:23:58.769247+0000","last_clean":"2024-04-06T03:23:58.769247+0000","last_became_active":"2024-04-06T03:20:53.350784+0000","last_became_peered":"2024-04-06T03:20:53.350784+0000","last_unstale":"2024-04-06T03:23:58.769247+0000","last_undegraded":"2024-04-06T03:23:58.769247+0000","last_fullsized":"2024-04-06T03:23:58.769247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-06T03:23:22.871158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:22.871158+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:30.808433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.952149+0000","last_change":"2024-04-06T03:22:55.509249+0000","last_active":"2024-04-06T03:23:57.952149+0000","last_peered":"2024-04-06T03:23:57.952149+0000","last_clean":"2024-04-06T03:23:57.952149+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:23:57.952149+0000","last_undegraded":"2024-04-06T03:23:57.952149+0000","last_fullsized":"2024-04-06T03:23:57.952149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:55.508950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:55.508950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:15.654754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473124+0000","last_change":"2024-04-06T03:23:43.566143+0000","last_active":"2024-04-06T03:23:58.473124+0000","last_peered":"2024-04-06T03:23:58.473124+0000","last_clean":"2024-04-06T03:23:58.473124+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:23:58.473124+0000","last_undegraded":"2024-04-06T03:23:58.473124+0000","last_fullsized":"2024-04-06T03:23:58.473124+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.565930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.924455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474970+0000","last_change":"2024-04-06T03:23:56.575312+0000","last_active":"2024-04-06T03:23:58.474970+0000","last_peered":"2024-04-06T03:23:58.474970+0000","last_clean":"2024-04-06T03:23:58.474970+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:23:58.474970+0000","last_undegraded":"2024-04-06T03:23:58.474970+0000","last_fullsized":"2024-04-06T03:23:58.474970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.575091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.850617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":118,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.954382+0000","last_change":"2024-04-06T03:22:57.488426+0000","last_active":"2024-04-06T03:23:57.954382+0000","last_peered":"2024-04-06T03:23:57.954382+0000","last_clean":"2024-04-06T03:23:57.954382+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:23:57.954382+0000","last_undegraded":"2024-04-06T03:23:57.954382+0000","last_fullsized":"2024-04-06T03:23:57.954382+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:22:57.488176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:57.488176+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.958962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474919+0000","last_change":"2024-04-06T03:23:39.990151+0000","last_active":"2024-04-06T03:23:58.474919+0000","last_peered":"2024-04-06T03:23:58.474919+0000","last_clean":"2024-04-06T03:23:58.474919+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:23:58.474919+0000","last_undegraded":"2024-04-06T03:23:58.474919+0000","last_fullsized":"2024-04-06T03:23:58.474919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.990000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.990000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.631759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474039+0000","last_change":"2024-04-06T03:23:44.525846+0000","last_active":"2024-04-06T03:23:58.474039+0000","last_peered":"2024-04-06T03:23:58.474039+0000","last_clean":"2024-04-06T03:23:58.474039+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:23:58.474039+0000","last_undegraded":"2024-04-06T03:23:58.474039+0000","last_fullsized":"2024-04-06T03:23:58.474039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.525616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.599062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.956115+0000","last_change":"2024-04-06T03:23:16.814953+0000","last_active":"2024-04-06T03:23:57.956115+0000","last_peered":"2024-04-06T03:23:57.956115+0000","last_clean":"2024-04-06T03:23:57.956115+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:23:57.956115+0000","last_undegraded":"2024-04-06T03:23:57.956115+0000","last_fullsized":"2024-04-06T03:23:57.956115+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:16.814738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.771476+0000","last_clean_scrub_stamp":"2024-04-06T03:23:16.814738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:27.108754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"85'10","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.467118+0000","last_change":"2024-04-06T03:23:48.924901+0000","last_active":"2024-04-06T03:23:57.467118+0000","last_peered":"2024-04-06T03:23:57.467118+0000","last_clean":"2024-04-06T03:23:57.467118+0000","last_became_active":"2024-04-06T03:20:53.345059+0000","last_became_peered":"2024-04-06T03:20:53.345059+0000","last_unstale":"2024-04-06T03:23:57.467118+0000","last_undegraded":"2024-04-06T03:23:57.467118+0000","last_fullsized":"2024-04-06T03:23:57.467118+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'10","last_scrub_stamp":"2024-04-06T03:23:48.924677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.924677+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.542114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":83,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.465891+0000","last_change":"2024-04-06T03:22:46.720865+0000","last_active":"2024-04-06T03:23:57.465891+0000","last_peered":"2024-04-06T03:23:57.465891+0000","last_clean":"2024-04-06T03:23:57.465891+0000","last_became_active":"2024-04-06T03:20:58.128530+0000","last_became_peered":"2024-04-06T03:20:58.128530+0000","last_unstale":"2024-04-06T03:23:57.465891+0000","last_undegraded":"2024-04-06T03:23:57.465891+0000","last_fullsized":"2024-04-06T03:23:57.465891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:22:46.720611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474172+0000","last_change":"2024-04-06T03:23:07.604317+0000","last_active":"2024-04-06T03:23:58.474172+0000","last_peered":"2024-04-06T03:23:58.474172+0000","last_clean":"2024-04-06T03:23:58.474172+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:23:58.474172+0000","last_undegraded":"2024-04-06T03:23:58.474172+0000","last_fullsized":"2024-04-06T03:23:58.474172+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:07.604089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:23:07.604089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:21.624479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":123,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953701+0000","last_change":"2024-04-06T03:23:41.707141+0000","last_active":"2024-04-06T03:23:57.953701+0000","last_peered":"2024-04-06T03:23:57.953701+0000","last_clean":"2024-04-06T03:23:57.953701+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:23:57.953701+0000","last_undegraded":"2024-04-06T03:23:57.953701+0000","last_fullsized":"2024-04-06T03:23:57.953701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:41.706589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:41.706589+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.161763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"85'11","reported_seq":142,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-06T03:23:57.953540+0000","last_change":"2024-04-06T03:23:33.820933+0000","last_active":"2024-04-06T03:23:57.953540+0000","last_peered":"2024-04-06T03:23:57.953540+0000","last_clean":"2024-04-06T03:23:57.953540+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:23:57.953540+0000","last_undegraded":"2024-04-06T03:23:57.953540+0000","last_fullsized":"2024-04-06T03:23:57.953540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:33.820691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.820691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.287595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":161,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":642,"ondisk_log_size":642,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743472,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1331200,"data_stored":963383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135930,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1294336,"data_stored":961663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":716800,"data_stored":371149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36760,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708392,"statfs":{"total":95995035648,"available":95957393408,"internally_reserved":0,"allocated":1327104,"data_stored":970416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378725,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":716800,"data_stored":378593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084377,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":696320,"data_stored":370807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32024,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713128,"statfs":{"total":95995035648,"available":95962243072,"internally_reserved":0,"allocated":741376,"data_stored":371378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495686,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":700416,"data_stored":370789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:24:00.824 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-06T03:24:00.824 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-06T03:24:00.824 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-06T03:24:00.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.002400+0000 mgr.y (mgr.24368) 116 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: audit 2024-04-06T03:24:00.052153+0000 mgr.y (mgr.24368) 117 : audit 0 from='client.24539 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.076036+0000 osd.5 (osd.5) 81 : cluster 0 6.a scrub starts 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.078186+0000 osd.5 (osd.5) 82 : cluster 0 6.a scrub ok 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.466114+0000 osd.4 (osd.4) 67 : cluster 0 5.1c scrub starts 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.468650+0000 osd.4 (osd.4) 68 : cluster 0 5.1c scrub ok 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.749892+0000 osd.0 (osd.0) 73 : cluster 0 3.12 scrub starts 2024-04-06T03:24:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.753894+0000 osd.0 (osd.0) 74 : cluster 0 3.12 scrub ok 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.785858+0000 osd.7 (osd.7) 71 : cluster 0 3.13 scrub starts 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.790212+0000 osd.7 (osd.7) 72 : cluster 0 3.13 scrub ok 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.822984+0000 osd.3 (osd.3) 81 : cluster 0 2.1b scrub starts 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[21467]: cluster 2024-04-06T03:24:00.825174+0000 osd.3 (osd.3) 82 : cluster 0 2.1b scrub ok 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.002400+0000 mgr.y (mgr.24368) 116 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: audit 2024-04-06T03:24:00.052153+0000 mgr.y (mgr.24368) 117 : audit 0 from='client.24539 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.076036+0000 osd.5 (osd.5) 81 : cluster 0 6.a scrub starts 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.078186+0000 osd.5 (osd.5) 82 : cluster 0 6.a scrub ok 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.466114+0000 osd.4 (osd.4) 67 : cluster 0 5.1c scrub starts 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.468650+0000 osd.4 (osd.4) 68 : cluster 0 5.1c scrub ok 2024-04-06T03:24:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.749892+0000 osd.0 (osd.0) 73 : cluster 0 3.12 scrub starts 2024-04-06T03:24:01.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.753894+0000 osd.0 (osd.0) 74 : cluster 0 3.12 scrub ok 2024-04-06T03:24:01.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.785858+0000 osd.7 (osd.7) 71 : cluster 0 3.13 scrub starts 2024-04-06T03:24:01.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.790212+0000 osd.7 (osd.7) 72 : cluster 0 3.13 scrub ok 2024-04-06T03:24:01.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.822984+0000 osd.3 (osd.3) 81 : cluster 0 2.1b scrub starts 2024-04-06T03:24:01.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:01 smithi050 bash[29221]: cluster 2024-04-06T03:24:00.825174+0000 osd.3 (osd.3) 82 : cluster 0 2.1b scrub ok 2024-04-06T03:24:01.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.002400+0000 mgr.y (mgr.24368) 116 : cluster 0 pgmap v88: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: audit 2024-04-06T03:24:00.052153+0000 mgr.y (mgr.24368) 117 : audit 0 from='client.24539 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.076036+0000 osd.5 (osd.5) 81 : cluster 0 6.a scrub starts 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.078186+0000 osd.5 (osd.5) 82 : cluster 0 6.a scrub ok 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.466114+0000 osd.4 (osd.4) 67 : cluster 0 5.1c scrub starts 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.468650+0000 osd.4 (osd.4) 68 : cluster 0 5.1c scrub ok 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.749892+0000 osd.0 (osd.0) 73 : cluster 0 3.12 scrub starts 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.753894+0000 osd.0 (osd.0) 74 : cluster 0 3.12 scrub ok 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.785858+0000 osd.7 (osd.7) 71 : cluster 0 3.13 scrub starts 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.790212+0000 osd.7 (osd.7) 72 : cluster 0 3.13 scrub ok 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.822984+0000 osd.3 (osd.3) 81 : cluster 0 2.1b scrub starts 2024-04-06T03:24:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:01 smithi153 bash[23852]: cluster 2024-04-06T03:24:00.825174+0000 osd.3 (osd.3) 82 : cluster 0 2.1b scrub ok 2024-04-06T03:24:03.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[21467]: cluster 2024-04-06T03:24:02.003037+0000 mgr.y (mgr.24368) 118 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[21467]: cluster 2024-04-06T03:24:02.701213+0000 osd.7 (osd.7) 73 : cluster 0 6.5 scrub starts 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[21467]: cluster 2024-04-06T03:24:02.704508+0000 osd.7 (osd.7) 74 : cluster 0 6.5 scrub ok 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[21467]: audit 2024-04-06T03:24:03.311211+0000 mon.a (mon.0) 963 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[29221]: cluster 2024-04-06T03:24:02.003037+0000 mgr.y (mgr.24368) 118 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[29221]: cluster 2024-04-06T03:24:02.701213+0000 osd.7 (osd.7) 73 : cluster 0 6.5 scrub starts 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[29221]: cluster 2024-04-06T03:24:02.704508+0000 osd.7 (osd.7) 74 : cluster 0 6.5 scrub ok 2024-04-06T03:24:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:03 smithi050 bash[29221]: audit 2024-04-06T03:24:03.311211+0000 mon.a (mon.0) 963 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:03 smithi153 bash[23852]: cluster 2024-04-06T03:24:02.003037+0000 mgr.y (mgr.24368) 118 : cluster 0 pgmap v89: 132 pgs: 132 active+clean; 582 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:03.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:03 smithi153 bash[23852]: cluster 2024-04-06T03:24:02.701213+0000 osd.7 (osd.7) 73 : cluster 0 6.5 scrub starts 2024-04-06T03:24:03.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:03 smithi153 bash[23852]: cluster 2024-04-06T03:24:02.704508+0000 osd.7 (osd.7) 74 : cluster 0 6.5 scrub ok 2024-04-06T03:24:03.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:03 smithi153 bash[23852]: audit 2024-04-06T03:24:03.311211+0000 mon.a (mon.0) 963 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:04.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:04 smithi153 bash[23852]: cluster 2024-04-06T03:24:03.506108+0000 osd.4 (osd.4) 69 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:24:04.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:04 smithi153 bash[23852]: cluster 2024-04-06T03:24:03.509170+0000 osd.4 (osd.4) 70 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:24:04.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:04 smithi153 bash[23852]: cluster 2024-04-06T03:24:03.670538+0000 osd.1 (osd.1) 75 : cluster 0 6.1d scrub starts 2024-04-06T03:24:04.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:04 smithi153 bash[23852]: cluster 2024-04-06T03:24:03.673139+0000 osd.1 (osd.1) 76 : cluster 0 6.1d scrub ok 2024-04-06T03:24:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[21467]: cluster 2024-04-06T03:24:03.506108+0000 osd.4 (osd.4) 69 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:24:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[21467]: cluster 2024-04-06T03:24:03.509170+0000 osd.4 (osd.4) 70 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:24:04.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[21467]: cluster 2024-04-06T03:24:03.670538+0000 osd.1 (osd.1) 75 : cluster 0 6.1d scrub starts 2024-04-06T03:24:04.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[21467]: cluster 2024-04-06T03:24:03.673139+0000 osd.1 (osd.1) 76 : cluster 0 6.1d scrub ok 2024-04-06T03:24:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[29221]: cluster 2024-04-06T03:24:03.506108+0000 osd.4 (osd.4) 69 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:24:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[29221]: cluster 2024-04-06T03:24:03.509170+0000 osd.4 (osd.4) 70 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:24:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[29221]: cluster 2024-04-06T03:24:03.670538+0000 osd.1 (osd.1) 75 : cluster 0 6.1d scrub starts 2024-04-06T03:24:04.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:04 smithi050 bash[29221]: cluster 2024-04-06T03:24:03.673139+0000 osd.1 (osd.1) 76 : cluster 0 6.1d scrub ok 2024-04-06T03:24:05.615 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:05.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:05 smithi153 bash[23852]: cluster 2024-04-06T03:24:04.004556+0000 mgr.y (mgr.24368) 119 : cluster 0 pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:05.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:05 smithi153 bash[23852]: cluster 2024-04-06T03:24:04.678837+0000 osd.7 (osd.7) 75 : cluster 0 6.12 scrub starts 2024-04-06T03:24:05.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:05 smithi153 bash[23852]: cluster 2024-04-06T03:24:04.681302+0000 osd.7 (osd.7) 76 : cluster 0 6.12 scrub ok 2024-04-06T03:24:05.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:05 smithi153 bash[23852]: cluster 2024-04-06T03:24:04.716511+0000 osd.0 (osd.0) 75 : cluster 0 6.18 scrub starts 2024-04-06T03:24:05.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:05 smithi153 bash[23852]: cluster 2024-04-06T03:24:04.719061+0000 osd.0 (osd.0) 76 : cluster 0 6.18 scrub ok 2024-04-06T03:24:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[21467]: cluster 2024-04-06T03:24:04.004556+0000 mgr.y (mgr.24368) 119 : cluster 0 pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:05.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[21467]: cluster 2024-04-06T03:24:04.678837+0000 osd.7 (osd.7) 75 : cluster 0 6.12 scrub starts 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[21467]: cluster 2024-04-06T03:24:04.681302+0000 osd.7 (osd.7) 76 : cluster 0 6.12 scrub ok 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[21467]: cluster 2024-04-06T03:24:04.716511+0000 osd.0 (osd.0) 75 : cluster 0 6.18 scrub starts 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[21467]: cluster 2024-04-06T03:24:04.719061+0000 osd.0 (osd.0) 76 : cluster 0 6.18 scrub ok 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[29221]: cluster 2024-04-06T03:24:04.004556+0000 mgr.y (mgr.24368) 119 : cluster 0 pgmap v90: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[29221]: cluster 2024-04-06T03:24:04.678837+0000 osd.7 (osd.7) 75 : cluster 0 6.12 scrub starts 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[29221]: cluster 2024-04-06T03:24:04.681302+0000 osd.7 (osd.7) 76 : cluster 0 6.12 scrub ok 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[29221]: cluster 2024-04-06T03:24:04.716511+0000 osd.0 (osd.0) 75 : cluster 0 6.18 scrub starts 2024-04-06T03:24:05.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:05 smithi050 bash[29221]: cluster 2024-04-06T03:24:04.719061+0000 osd.0 (osd.0) 76 : cluster 0 6.18 scrub ok 2024-04-06T03:24:06.782 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:06 smithi050 bash[21467]: cluster 2024-04-06T03:24:05.553472+0000 osd.4 (osd.4) 71 : cluster 0 6.1a scrub starts 2024-04-06T03:24:06.782 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:06 smithi050 bash[21467]: cluster 2024-04-06T03:24:05.555871+0000 osd.4 (osd.4) 72 : cluster 0 6.1a scrub ok 2024-04-06T03:24:06.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:06 smithi050 bash[29221]: cluster 2024-04-06T03:24:05.553472+0000 osd.4 (osd.4) 71 : cluster 0 6.1a scrub starts 2024-04-06T03:24:06.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:06 smithi050 bash[29221]: cluster 2024-04-06T03:24:05.555871+0000 osd.4 (osd.4) 72 : cluster 0 6.1a scrub ok 2024-04-06T03:24:06.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:06 smithi153 bash[23852]: cluster 2024-04-06T03:24:05.553472+0000 osd.4 (osd.4) 71 : cluster 0 6.1a scrub starts 2024-04-06T03:24:06.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:06 smithi153 bash[23852]: cluster 2024-04-06T03:24:05.555871+0000 osd.4 (osd.4) 72 : cluster 0 6.1a scrub ok 2024-04-06T03:24:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.005365+0000 mgr.y (mgr.24368) 120 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:07.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.083205+0000 osd.5 (osd.5) 83 : cluster 0 3.5 scrub starts 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.088275+0000 osd.5 (osd.5) 84 : cluster 0 3.5 scrub ok 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.709919+0000 osd.7 (osd.7) 77 : cluster 0 2.5 deep-scrub starts 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.712067+0000 osd.7 (osd.7) 78 : cluster 0 2.5 deep-scrub ok 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.723302+0000 osd.0 (osd.0) 77 : cluster 0 5.5 scrub starts 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.724636+0000 osd.0 (osd.0) 78 : cluster 0 5.5 scrub ok 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.779018+0000 osd.3 (osd.3) 83 : cluster 0 3.7 scrub starts 2024-04-06T03:24:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:07 smithi153 bash[23852]: cluster 2024-04-06T03:24:06.782111+0000 osd.3 (osd.3) 84 : cluster 0 3.7 scrub ok 2024-04-06T03:24:07.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.005365+0000 mgr.y (mgr.24368) 120 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.083205+0000 osd.5 (osd.5) 83 : cluster 0 3.5 scrub starts 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.088275+0000 osd.5 (osd.5) 84 : cluster 0 3.5 scrub ok 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.709919+0000 osd.7 (osd.7) 77 : cluster 0 2.5 deep-scrub starts 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.712067+0000 osd.7 (osd.7) 78 : cluster 0 2.5 deep-scrub ok 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.723302+0000 osd.0 (osd.0) 77 : cluster 0 5.5 scrub starts 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.724636+0000 osd.0 (osd.0) 78 : cluster 0 5.5 scrub ok 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.779018+0000 osd.3 (osd.3) 83 : cluster 0 3.7 scrub starts 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[21467]: cluster 2024-04-06T03:24:06.782111+0000 osd.3 (osd.3) 84 : cluster 0 3.7 scrub ok 2024-04-06T03:24:07.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.005365+0000 mgr.y (mgr.24368) 120 : cluster 0 pgmap v91: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.083205+0000 osd.5 (osd.5) 83 : cluster 0 3.5 scrub starts 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.088275+0000 osd.5 (osd.5) 84 : cluster 0 3.5 scrub ok 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.709919+0000 osd.7 (osd.7) 77 : cluster 0 2.5 deep-scrub starts 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.712067+0000 osd.7 (osd.7) 78 : cluster 0 2.5 deep-scrub ok 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.723302+0000 osd.0 (osd.0) 77 : cluster 0 5.5 scrub starts 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.724636+0000 osd.0 (osd.0) 78 : cluster 0 5.5 scrub ok 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.779018+0000 osd.3 (osd.3) 83 : cluster 0 3.7 scrub starts 2024-04-06T03:24:07.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:07 smithi050 bash[29221]: cluster 2024-04-06T03:24:06.782111+0000 osd.3 (osd.3) 84 : cluster 0 3.7 scrub ok 2024-04-06T03:24:08.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:08 smithi050 bash[29221]: cluster 2024-04-06T03:24:07.802339+0000 osd.3 (osd.3) 85 : cluster 0 6.6 scrub starts 2024-04-06T03:24:08.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:08 smithi050 bash[29221]: cluster 2024-04-06T03:24:07.804773+0000 osd.3 (osd.3) 86 : cluster 0 6.6 scrub ok 2024-04-06T03:24:08.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:08 smithi050 bash[21467]: cluster 2024-04-06T03:24:07.802339+0000 osd.3 (osd.3) 85 : cluster 0 6.6 scrub starts 2024-04-06T03:24:08.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:08 smithi050 bash[21467]: cluster 2024-04-06T03:24:07.804773+0000 osd.3 (osd.3) 86 : cluster 0 6.6 scrub ok 2024-04-06T03:24:08.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:08 smithi153 bash[23852]: cluster 2024-04-06T03:24:07.802339+0000 osd.3 (osd.3) 85 : cluster 0 6.6 scrub starts 2024-04-06T03:24:08.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:08 smithi153 bash[23852]: cluster 2024-04-06T03:24:07.804773+0000 osd.3 (osd.3) 86 : cluster 0 6.6 scrub ok 2024-04-06T03:24:08.888 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:24:08.888 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-06T03:24:09.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:24:09.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21467]: cluster 2024-04-06T03:24:08.006003+0000 mgr.y (mgr.24368) 121 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:09.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21467]: cluster 2024-04-06T03:24:08.716491+0000 osd.1 (osd.1) 77 : cluster 0 5.c scrub starts 2024-04-06T03:24:09.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21467]: cluster 2024-04-06T03:24:08.718950+0000 osd.1 (osd.1) 78 : cluster 0 5.c scrub ok 2024-04-06T03:24:09.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21467]: cluster 2024-04-06T03:24:08.771455+0000 osd.3 (osd.3) 87 : cluster 0 3.18 scrub starts 2024-04-06T03:24:09.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21467]: cluster 2024-04-06T03:24:08.775521+0000 osd.3 (osd.3) 88 : cluster 0 3.18 scrub ok 2024-04-06T03:24:09.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21467]: audit 2024-04-06T03:24:08.897159+0000 mon.a (mon.0) 964 : audit 0 from='client.? 172.21.15.50:0/935866129' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:24:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[29221]: cluster 2024-04-06T03:24:08.006003+0000 mgr.y (mgr.24368) 121 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[29221]: cluster 2024-04-06T03:24:08.716491+0000 osd.1 (osd.1) 77 : cluster 0 5.c scrub starts 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[29221]: cluster 2024-04-06T03:24:08.718950+0000 osd.1 (osd.1) 78 : cluster 0 5.c scrub ok 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[29221]: cluster 2024-04-06T03:24:08.771455+0000 osd.3 (osd.3) 87 : cluster 0 3.18 scrub starts 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[29221]: cluster 2024-04-06T03:24:08.775521+0000 osd.3 (osd.3) 88 : cluster 0 3.18 scrub ok 2024-04-06T03:24:09.614 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:09 smithi050 bash[29221]: audit 2024-04-06T03:24:08.897159+0000 mon.a (mon.0) 964 : audit 0 from='client.? 172.21.15.50:0/935866129' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T03:24:09.615 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-06T03:24:09.615 INFO:tasks.cephadm:Setup complete, yielding 2024-04-06T03:24:09.616 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-06T03:24:09.634 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-06T03:24:09.634 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9f192350ba5fe6d0a365b251a8c2af61f56a5fa6'}}, 'workunit': {'branch': 'squid', 'sha1': '584a67ccb9eb224c5312dccd96a82e67b0645518'}} 2024-04-06T03:24:09.634 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-06T03:24:09.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-06T03:24:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[23852]: cluster 2024-04-06T03:24:08.006003+0000 mgr.y (mgr.24368) 121 : cluster 0 pgmap v92: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:09.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[23852]: cluster 2024-04-06T03:24:08.716491+0000 osd.1 (osd.1) 77 : cluster 0 5.c scrub starts 2024-04-06T03:24:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[23852]: cluster 2024-04-06T03:24:08.718950+0000 osd.1 (osd.1) 78 : cluster 0 5.c scrub ok 2024-04-06T03:24:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[23852]: cluster 2024-04-06T03:24:08.771455+0000 osd.3 (osd.3) 87 : cluster 0 3.18 scrub starts 2024-04-06T03:24:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[23852]: cluster 2024-04-06T03:24:08.775521+0000 osd.3 (osd.3) 88 : cluster 0 3.18 scrub ok 2024-04-06T03:24:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:09 smithi153 bash[23852]: audit 2024-04-06T03:24:08.897159+0000 mon.a (mon.0) 964 : audit 0 from='client.? 172.21.15.50:0/935866129' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-06T03:24:10.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:10 smithi153 bash[23852]: audit 2024-04-06T03:24:09.087190+0000 mgr.y (mgr.24368) 122 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:10.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:10 smithi153 bash[23852]: cluster 2024-04-06T03:24:09.750642+0000 osd.1 (osd.1) 79 : cluster 0 6.4 scrub starts 2024-04-06T03:24:10.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:10 smithi153 bash[23852]: cluster 2024-04-06T03:24:09.752954+0000 osd.1 (osd.1) 80 : cluster 0 6.4 scrub ok 2024-04-06T03:24:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:10 smithi050 bash[21467]: audit 2024-04-06T03:24:09.087190+0000 mgr.y (mgr.24368) 122 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:10 smithi050 bash[21467]: cluster 2024-04-06T03:24:09.750642+0000 osd.1 (osd.1) 79 : cluster 0 6.4 scrub starts 2024-04-06T03:24:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:10 smithi050 bash[21467]: cluster 2024-04-06T03:24:09.752954+0000 osd.1 (osd.1) 80 : cluster 0 6.4 scrub ok 2024-04-06T03:24:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:10 smithi050 bash[29221]: audit 2024-04-06T03:24:09.087190+0000 mgr.y (mgr.24368) 122 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:10 smithi050 bash[29221]: cluster 2024-04-06T03:24:09.750642+0000 osd.1 (osd.1) 79 : cluster 0 6.4 scrub starts 2024-04-06T03:24:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:10 smithi050 bash[29221]: cluster 2024-04-06T03:24:09.752954+0000 osd.1 (osd.1) 80 : cluster 0 6.4 scrub ok 2024-04-06T03:24:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:11 smithi153 bash[23852]: cluster 2024-04-06T03:24:10.007136+0000 mgr.y (mgr.24368) 123 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:24:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:11 smithi153 bash[23852]: cluster 2024-04-06T03:24:10.105815+0000 osd.5 (osd.5) 85 : cluster 0 6.19 scrub starts 2024-04-06T03:24:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:11 smithi153 bash[23852]: cluster 2024-04-06T03:24:10.108346+0000 osd.5 (osd.5) 86 : cluster 0 6.19 scrub ok 2024-04-06T03:24:11.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:11 smithi153 bash[23852]: cluster 2024-04-06T03:24:10.789564+0000 osd.7 (osd.7) 79 : cluster 0 2.1c scrub starts 2024-04-06T03:24:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:11 smithi153 bash[23852]: cluster 2024-04-06T03:24:10.792031+0000 osd.7 (osd.7) 80 : cluster 0 2.1c scrub ok 2024-04-06T03:24:11.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[29221]: cluster 2024-04-06T03:24:10.007136+0000 mgr.y (mgr.24368) 123 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:24:11.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[29221]: cluster 2024-04-06T03:24:10.105815+0000 osd.5 (osd.5) 85 : cluster 0 6.19 scrub starts 2024-04-06T03:24:11.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[29221]: cluster 2024-04-06T03:24:10.108346+0000 osd.5 (osd.5) 86 : cluster 0 6.19 scrub ok 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[29221]: cluster 2024-04-06T03:24:10.789564+0000 osd.7 (osd.7) 79 : cluster 0 2.1c scrub starts 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[29221]: cluster 2024-04-06T03:24:10.792031+0000 osd.7 (osd.7) 80 : cluster 0 2.1c scrub ok 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[21467]: cluster 2024-04-06T03:24:10.007136+0000 mgr.y (mgr.24368) 123 : cluster 0 pgmap v93: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[21467]: cluster 2024-04-06T03:24:10.105815+0000 osd.5 (osd.5) 85 : cluster 0 6.19 scrub starts 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[21467]: cluster 2024-04-06T03:24:10.108346+0000 osd.5 (osd.5) 86 : cluster 0 6.19 scrub ok 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[21467]: cluster 2024-04-06T03:24:10.789564+0000 osd.7 (osd.7) 79 : cluster 0 2.1c scrub starts 2024-04-06T03:24:11.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:11 smithi050 bash[21467]: cluster 2024-04-06T03:24:10.792031+0000 osd.7 (osd.7) 80 : cluster 0 2.1c scrub ok 2024-04-06T03:24:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:13 smithi153 bash[23852]: cluster 2024-04-06T03:24:12.007804+0000 mgr.y (mgr.24368) 124 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:13 smithi153 bash[23852]: cluster 2024-04-06T03:24:12.538627+0000 osd.2 (osd.2) 59 : cluster 0 6.14 scrub starts 2024-04-06T03:24:13.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:13 smithi153 bash[23852]: cluster 2024-04-06T03:24:12.542062+0000 osd.2 (osd.2) 60 : cluster 0 6.14 scrub ok 2024-04-06T03:24:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:13 smithi050 bash[21467]: cluster 2024-04-06T03:24:12.007804+0000 mgr.y (mgr.24368) 124 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:13 smithi050 bash[21467]: cluster 2024-04-06T03:24:12.538627+0000 osd.2 (osd.2) 59 : cluster 0 6.14 scrub starts 2024-04-06T03:24:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:13 smithi050 bash[21467]: cluster 2024-04-06T03:24:12.542062+0000 osd.2 (osd.2) 60 : cluster 0 6.14 scrub ok 2024-04-06T03:24:13.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:13 smithi050 bash[29221]: cluster 2024-04-06T03:24:12.007804+0000 mgr.y (mgr.24368) 124 : cluster 0 pgmap v94: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:13.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:13 smithi050 bash[29221]: cluster 2024-04-06T03:24:12.538627+0000 osd.2 (osd.2) 59 : cluster 0 6.14 scrub starts 2024-04-06T03:24:13.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:13 smithi050 bash[29221]: cluster 2024-04-06T03:24:12.542062+0000 osd.2 (osd.2) 60 : cluster 0 6.14 scrub ok 2024-04-06T03:24:14.422 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.009232+0000 mgr.y (mgr.24368) 125 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.727429+0000 osd.3 (osd.3) 89 : cluster 0 6.11 scrub starts 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.729857+0000 osd.3 (osd.3) 90 : cluster 0 6.11 scrub ok 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.769119+0000 osd.1 (osd.1) 81 : cluster 0 5.19 scrub starts 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.771829+0000 osd.1 (osd.1) 82 : cluster 0 5.19 scrub ok 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.814889+0000 osd.7 (osd.7) 81 : cluster 0 5.4 deep-scrub starts 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[29221]: cluster 2024-04-06T03:24:14.817078+0000 osd.7 (osd.7) 82 : cluster 0 5.4 deep-scrub ok 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.009232+0000 mgr.y (mgr.24368) 125 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.727429+0000 osd.3 (osd.3) 89 : cluster 0 6.11 scrub starts 2024-04-06T03:24:15.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.729857+0000 osd.3 (osd.3) 90 : cluster 0 6.11 scrub ok 2024-04-06T03:24:15.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.769119+0000 osd.1 (osd.1) 81 : cluster 0 5.19 scrub starts 2024-04-06T03:24:15.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.771829+0000 osd.1 (osd.1) 82 : cluster 0 5.19 scrub ok 2024-04-06T03:24:15.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.814889+0000 osd.7 (osd.7) 81 : cluster 0 5.4 deep-scrub starts 2024-04-06T03:24:15.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:15 smithi050 bash[21467]: cluster 2024-04-06T03:24:14.817078+0000 osd.7 (osd.7) 82 : cluster 0 5.4 deep-scrub ok 2024-04-06T03:24:15.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.009232+0000 mgr.y (mgr.24368) 125 : cluster 0 pgmap v95: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.727429+0000 osd.3 (osd.3) 89 : cluster 0 6.11 scrub starts 2024-04-06T03:24:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.729857+0000 osd.3 (osd.3) 90 : cluster 0 6.11 scrub ok 2024-04-06T03:24:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.769119+0000 osd.1 (osd.1) 81 : cluster 0 5.19 scrub starts 2024-04-06T03:24:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.771829+0000 osd.1 (osd.1) 82 : cluster 0 5.19 scrub ok 2024-04-06T03:24:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.814889+0000 osd.7 (osd.7) 81 : cluster 0 5.4 deep-scrub starts 2024-04-06T03:24:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:15 smithi153 bash[23852]: cluster 2024-04-06T03:24:14.817078+0000 osd.7 (osd.7) 82 : cluster 0 5.4 deep-scrub ok 2024-04-06T03:24:16.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[21467]: cluster 2024-04-06T03:24:15.777684+0000 osd.3 (osd.3) 91 : cluster 0 5.0 scrub starts 2024-04-06T03:24:16.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[21467]: cluster 2024-04-06T03:24:15.781249+0000 osd.3 (osd.3) 92 : cluster 0 5.0 scrub ok 2024-04-06T03:24:16.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[21467]: cluster 2024-04-06T03:24:15.805496+0000 osd.7 (osd.7) 83 : cluster 0 3.15 scrub starts 2024-04-06T03:24:16.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[21467]: cluster 2024-04-06T03:24:15.809764+0000 osd.7 (osd.7) 84 : cluster 0 3.15 scrub ok 2024-04-06T03:24:16.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[29221]: cluster 2024-04-06T03:24:15.777684+0000 osd.3 (osd.3) 91 : cluster 0 5.0 scrub starts 2024-04-06T03:24:16.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[29221]: cluster 2024-04-06T03:24:15.781249+0000 osd.3 (osd.3) 92 : cluster 0 5.0 scrub ok 2024-04-06T03:24:16.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[29221]: cluster 2024-04-06T03:24:15.805496+0000 osd.7 (osd.7) 83 : cluster 0 3.15 scrub starts 2024-04-06T03:24:16.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:16 smithi050 bash[29221]: cluster 2024-04-06T03:24:15.809764+0000 osd.7 (osd.7) 84 : cluster 0 3.15 scrub ok 2024-04-06T03:24:17.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:16 smithi153 bash[23852]: cluster 2024-04-06T03:24:15.777684+0000 osd.3 (osd.3) 91 : cluster 0 5.0 scrub starts 2024-04-06T03:24:17.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:16 smithi153 bash[23852]: cluster 2024-04-06T03:24:15.781249+0000 osd.3 (osd.3) 92 : cluster 0 5.0 scrub ok 2024-04-06T03:24:17.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:16 smithi153 bash[23852]: cluster 2024-04-06T03:24:15.805496+0000 osd.7 (osd.7) 83 : cluster 0 3.15 scrub starts 2024-04-06T03:24:17.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:16 smithi153 bash[23852]: cluster 2024-04-06T03:24:15.809764+0000 osd.7 (osd.7) 84 : cluster 0 3.15 scrub ok 2024-04-06T03:24:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21467]: cluster 2024-04-06T03:24:16.009899+0000 mgr.y (mgr.24368) 126 : cluster 0 pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:17.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21467]: cluster 2024-04-06T03:24:16.787127+0000 osd.3 (osd.3) 93 : cluster 0 6.13 scrub starts 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21467]: cluster 2024-04-06T03:24:16.789532+0000 osd.3 (osd.3) 94 : cluster 0 6.13 scrub ok 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21467]: audit 2024-04-06T03:24:17.479779+0000 mon.a (mon.0) 965 : audit 1 from='client.? 172.21.15.50:0/1011540307' entity='client.admin' 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21467]: audit 2024-04-06T03:24:17.510494+0000 mon.a (mon.0) 966 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[29221]: cluster 2024-04-06T03:24:16.009899+0000 mgr.y (mgr.24368) 126 : cluster 0 pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[29221]: cluster 2024-04-06T03:24:16.787127+0000 osd.3 (osd.3) 93 : cluster 0 6.13 scrub starts 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[29221]: cluster 2024-04-06T03:24:16.789532+0000 osd.3 (osd.3) 94 : cluster 0 6.13 scrub ok 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[29221]: audit 2024-04-06T03:24:17.479779+0000 mon.a (mon.0) 965 : audit 1 from='client.? 172.21.15.50:0/1011540307' entity='client.admin' 2024-04-06T03:24:17.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[29221]: audit 2024-04-06T03:24:17.510494+0000 mon.a (mon.0) 966 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:17.734 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21725]: [06/Apr/2024:03:24:17] ENGINE Bus STOPPING 2024-04-06T03:24:17.734 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21725]: [06/Apr/2024:03:24:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:24:17.734 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21725]: [06/Apr/2024:03:24:17] ENGINE Bus STOPPED 2024-04-06T03:24:17.734 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21725]: [06/Apr/2024:03:24:17] ENGINE Bus STARTING 2024-04-06T03:24:17.734 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21725]: [06/Apr/2024:03:24:17] ENGINE Serving on http://:::9283 2024-04-06T03:24:17.734 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:17 smithi050 bash[21725]: [06/Apr/2024:03:24:17] ENGINE Bus STARTED 2024-04-06T03:24:18.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:17 smithi153 bash[23852]: cluster 2024-04-06T03:24:16.009899+0000 mgr.y (mgr.24368) 126 : cluster 0 pgmap v96: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:18.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:17 smithi153 bash[23852]: cluster 2024-04-06T03:24:16.787127+0000 osd.3 (osd.3) 93 : cluster 0 6.13 scrub starts 2024-04-06T03:24:18.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:17 smithi153 bash[23852]: cluster 2024-04-06T03:24:16.789532+0000 osd.3 (osd.3) 94 : cluster 0 6.13 scrub ok 2024-04-06T03:24:18.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:17 smithi153 bash[23852]: audit 2024-04-06T03:24:17.479779+0000 mon.a (mon.0) 965 : audit 1 from='client.? 172.21.15.50:0/1011540307' entity='client.admin' 2024-04-06T03:24:18.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:17 smithi153 bash[23852]: audit 2024-04-06T03:24:17.510494+0000 mon.a (mon.0) 966 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:24:18.186 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-06T03:24:18.187 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-06T03:24:18.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:24:18.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[21467]: cluster 2024-04-06T03:24:17.846080+0000 osd.7 (osd.7) 85 : cluster 0 5.1a scrub starts 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[21467]: cluster 2024-04-06T03:24:17.848825+0000 osd.7 (osd.7) 86 : cluster 0 5.1a scrub ok 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[21467]: audit 2024-04-06T03:24:18.238452+0000 mon.a (mon.0) 967 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[21467]: audit 2024-04-06T03:24:18.240034+0000 mon.a (mon.0) 968 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[21467]: audit 2024-04-06T03:24:18.247956+0000 mon.a (mon.0) 969 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[21467]: audit 2024-04-06T03:24:18.341590+0000 mon.a (mon.0) 970 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[29221]: cluster 2024-04-06T03:24:17.846080+0000 osd.7 (osd.7) 85 : cluster 0 5.1a scrub starts 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[29221]: cluster 2024-04-06T03:24:17.848825+0000 osd.7 (osd.7) 86 : cluster 0 5.1a scrub ok 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[29221]: audit 2024-04-06T03:24:18.238452+0000 mon.a (mon.0) 967 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[29221]: audit 2024-04-06T03:24:18.240034+0000 mon.a (mon.0) 968 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:24:18.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[29221]: audit 2024-04-06T03:24:18.247956+0000 mon.a (mon.0) 969 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:24:18.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:18 smithi050 bash[29221]: audit 2024-04-06T03:24:18.341590+0000 mon.a (mon.0) 970 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:19.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:18 smithi153 bash[23852]: cluster 2024-04-06T03:24:17.846080+0000 osd.7 (osd.7) 85 : cluster 0 5.1a scrub starts 2024-04-06T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:18 smithi153 bash[23852]: cluster 2024-04-06T03:24:17.848825+0000 osd.7 (osd.7) 86 : cluster 0 5.1a scrub ok 2024-04-06T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:18 smithi153 bash[23852]: audit 2024-04-06T03:24:18.238452+0000 mon.a (mon.0) 967 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:18 smithi153 bash[23852]: audit 2024-04-06T03:24:18.240034+0000 mon.a (mon.0) 968 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:18 smithi153 bash[23852]: audit 2024-04-06T03:24:18.247956+0000 mon.a (mon.0) 969 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:24:19.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:18 smithi153 bash[23852]: audit 2024-04-06T03:24:18.341590+0000 mon.a (mon.0) 970 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:19.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:24:19.557 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:24:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:24:19.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[29221]: cluster 2024-04-06T03:24:18.010531+0000 mgr.y (mgr.24368) 127 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:19.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[29221]: cluster 2024-04-06T03:24:18.858435+0000 osd.7 (osd.7) 87 : cluster 0 1.0 scrub starts 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[29221]: cluster 2024-04-06T03:24:18.862526+0000 osd.7 (osd.7) 88 : cluster 0 1.0 scrub ok 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[29221]: cluster 2024-04-06T03:24:18.883008+0000 osd.6 (osd.6) 51 : cluster 0 2.14 scrub starts 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[29221]: cluster 2024-04-06T03:24:18.885174+0000 osd.6 (osd.6) 52 : cluster 0 2.14 scrub ok 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[29221]: audit 2024-04-06T03:24:19.090143+0000 mgr.y (mgr.24368) 128 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21467]: cluster 2024-04-06T03:24:18.010531+0000 mgr.y (mgr.24368) 127 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21467]: cluster 2024-04-06T03:24:18.858435+0000 osd.7 (osd.7) 87 : cluster 0 1.0 scrub starts 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21467]: cluster 2024-04-06T03:24:18.862526+0000 osd.7 (osd.7) 88 : cluster 0 1.0 scrub ok 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21467]: cluster 2024-04-06T03:24:18.883008+0000 osd.6 (osd.6) 51 : cluster 0 2.14 scrub starts 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21467]: cluster 2024-04-06T03:24:18.885174+0000 osd.6 (osd.6) 52 : cluster 0 2.14 scrub ok 2024-04-06T03:24:19.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:19 smithi050 bash[21467]: audit 2024-04-06T03:24:19.090143+0000 mgr.y (mgr.24368) 128 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:20.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[23852]: cluster 2024-04-06T03:24:18.010531+0000 mgr.y (mgr.24368) 127 : cluster 0 pgmap v97: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:20.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[23852]: cluster 2024-04-06T03:24:18.858435+0000 osd.7 (osd.7) 87 : cluster 0 1.0 scrub starts 2024-04-06T03:24:20.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[23852]: cluster 2024-04-06T03:24:18.862526+0000 osd.7 (osd.7) 88 : cluster 0 1.0 scrub ok 2024-04-06T03:24:20.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[23852]: cluster 2024-04-06T03:24:18.883008+0000 osd.6 (osd.6) 51 : cluster 0 2.14 scrub starts 2024-04-06T03:24:20.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[23852]: cluster 2024-04-06T03:24:18.885174+0000 osd.6 (osd.6) 52 : cluster 0 2.14 scrub ok 2024-04-06T03:24:20.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:19 smithi153 bash[23852]: audit 2024-04-06T03:24:19.090143+0000 mgr.y (mgr.24368) 128 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:20.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[29221]: cluster 2024-04-06T03:24:19.924669+0000 osd.1 (osd.1) 83 : cluster 0 5.12 scrub starts 2024-04-06T03:24:20.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[29221]: cluster 2024-04-06T03:24:19.927092+0000 osd.1 (osd.1) 84 : cluster 0 5.12 scrub ok 2024-04-06T03:24:20.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[29221]: cluster 2024-04-06T03:24:19.937386+0000 osd.0 (osd.0) 79 : cluster 0 6.0 scrub starts 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[29221]: cluster 2024-04-06T03:24:19.939836+0000 osd.0 (osd.0) 80 : cluster 0 6.0 scrub ok 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[29221]: cluster 2024-04-06T03:24:20.011951+0000 mgr.y (mgr.24368) 129 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[21467]: cluster 2024-04-06T03:24:19.924669+0000 osd.1 (osd.1) 83 : cluster 0 5.12 scrub starts 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[21467]: cluster 2024-04-06T03:24:19.927092+0000 osd.1 (osd.1) 84 : cluster 0 5.12 scrub ok 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[21467]: cluster 2024-04-06T03:24:19.937386+0000 osd.0 (osd.0) 79 : cluster 0 6.0 scrub starts 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[21467]: cluster 2024-04-06T03:24:19.939836+0000 osd.0 (osd.0) 80 : cluster 0 6.0 scrub ok 2024-04-06T03:24:20.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:20 smithi050 bash[21467]: cluster 2024-04-06T03:24:20.011951+0000 mgr.y (mgr.24368) 129 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:21.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:20 smithi153 bash[23852]: cluster 2024-04-06T03:24:19.924669+0000 osd.1 (osd.1) 83 : cluster 0 5.12 scrub starts 2024-04-06T03:24:21.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:20 smithi153 bash[23852]: cluster 2024-04-06T03:24:19.927092+0000 osd.1 (osd.1) 84 : cluster 0 5.12 scrub ok 2024-04-06T03:24:21.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:20 smithi153 bash[23852]: cluster 2024-04-06T03:24:19.937386+0000 osd.0 (osd.0) 79 : cluster 0 6.0 scrub starts 2024-04-06T03:24:21.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:20 smithi153 bash[23852]: cluster 2024-04-06T03:24:19.939836+0000 osd.0 (osd.0) 80 : cluster 0 6.0 scrub ok 2024-04-06T03:24:21.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:20 smithi153 bash[23852]: cluster 2024-04-06T03:24:20.011951+0000 mgr.y (mgr.24368) 129 : cluster 0 pgmap v98: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[21467]: cluster 2024-04-06T03:24:20.850479+0000 osd.7 (osd.7) 89 : cluster 0 6.1c scrub starts 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[21467]: cluster 2024-04-06T03:24:20.853451+0000 osd.7 (osd.7) 90 : cluster 0 6.1c scrub ok 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[21467]: cluster 2024-04-06T03:24:20.950730+0000 osd.0 (osd.0) 81 : cluster 0 5.3 scrub starts 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[21467]: cluster 2024-04-06T03:24:20.952987+0000 osd.0 (osd.0) 82 : cluster 0 5.3 scrub ok 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[21467]: cluster 2024-04-06T03:24:21.112993+0000 osd.5 (osd.5) 87 : cluster 0 2.16 scrub starts 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[21467]: cluster 2024-04-06T03:24:21.115179+0000 osd.5 (osd.5) 88 : cluster 0 2.16 scrub ok 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[29221]: cluster 2024-04-06T03:24:20.850479+0000 osd.7 (osd.7) 89 : cluster 0 6.1c scrub starts 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[29221]: cluster 2024-04-06T03:24:20.853451+0000 osd.7 (osd.7) 90 : cluster 0 6.1c scrub ok 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[29221]: cluster 2024-04-06T03:24:20.950730+0000 osd.0 (osd.0) 81 : cluster 0 5.3 scrub starts 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[29221]: cluster 2024-04-06T03:24:20.952987+0000 osd.0 (osd.0) 82 : cluster 0 5.3 scrub ok 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[29221]: cluster 2024-04-06T03:24:21.112993+0000 osd.5 (osd.5) 87 : cluster 0 2.16 scrub starts 2024-04-06T03:24:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:21 smithi050 bash[29221]: cluster 2024-04-06T03:24:21.115179+0000 osd.5 (osd.5) 88 : cluster 0 2.16 scrub ok 2024-04-06T03:24:22.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:21 smithi153 bash[23852]: cluster 2024-04-06T03:24:20.850479+0000 osd.7 (osd.7) 89 : cluster 0 6.1c scrub starts 2024-04-06T03:24:22.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:21 smithi153 bash[23852]: cluster 2024-04-06T03:24:20.853451+0000 osd.7 (osd.7) 90 : cluster 0 6.1c scrub ok 2024-04-06T03:24:22.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:21 smithi153 bash[23852]: cluster 2024-04-06T03:24:20.950730+0000 osd.0 (osd.0) 81 : cluster 0 5.3 scrub starts 2024-04-06T03:24:22.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:21 smithi153 bash[23852]: cluster 2024-04-06T03:24:20.952987+0000 osd.0 (osd.0) 82 : cluster 0 5.3 scrub ok 2024-04-06T03:24:22.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:21 smithi153 bash[23852]: cluster 2024-04-06T03:24:21.112993+0000 osd.5 (osd.5) 87 : cluster 0 2.16 scrub starts 2024-04-06T03:24:22.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:21 smithi153 bash[23852]: cluster 2024-04-06T03:24:21.115179+0000 osd.5 (osd.5) 88 : cluster 0 2.16 scrub ok 2024-04-06T03:24:22.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:22 smithi050 bash[21467]: cluster 2024-04-06T03:24:22.012753+0000 mgr.y (mgr.24368) 130 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:22.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:22 smithi050 bash[29221]: cluster 2024-04-06T03:24:22.012753+0000 mgr.y (mgr.24368) 130 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:22.984 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:23.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:22 smithi153 bash[23852]: cluster 2024-04-06T03:24:22.012753+0000 mgr.y (mgr.24368) 130 : cluster 0 pgmap v99: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:23 smithi050 bash[29221]: cluster 2024-04-06T03:24:22.574977+0000 osd.2 (osd.2) 61 : cluster 0 2.e scrub starts 2024-04-06T03:24:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:23 smithi050 bash[29221]: cluster 2024-04-06T03:24:22.577272+0000 osd.2 (osd.2) 62 : cluster 0 2.e scrub ok 2024-04-06T03:24:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:23 smithi050 bash[21467]: cluster 2024-04-06T03:24:22.574977+0000 osd.2 (osd.2) 61 : cluster 0 2.e scrub starts 2024-04-06T03:24:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:23 smithi050 bash[21467]: cluster 2024-04-06T03:24:22.577272+0000 osd.2 (osd.2) 62 : cluster 0 2.e scrub ok 2024-04-06T03:24:24.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:23 smithi153 bash[23852]: cluster 2024-04-06T03:24:22.574977+0000 osd.2 (osd.2) 61 : cluster 0 2.e scrub starts 2024-04-06T03:24:24.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:23 smithi153 bash[23852]: cluster 2024-04-06T03:24:22.577272+0000 osd.2 (osd.2) 62 : cluster 0 2.e scrub ok 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[21467]: cluster 2024-04-06T03:24:23.877888+0000 osd.7 (osd.7) 91 : cluster 0 6.3 scrub starts 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[21467]: cluster 2024-04-06T03:24:23.880745+0000 osd.7 (osd.7) 92 : cluster 0 6.3 scrub ok 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[21467]: cluster 2024-04-06T03:24:23.885572+0000 osd.3 (osd.3) 95 : cluster 0 5.17 scrub starts 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[21467]: cluster 2024-04-06T03:24:23.887804+0000 osd.3 (osd.3) 96 : cluster 0 5.17 scrub ok 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.013649+0000 mgr.y (mgr.24368) 131 : cluster 0 pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[29221]: cluster 2024-04-06T03:24:23.877888+0000 osd.7 (osd.7) 91 : cluster 0 6.3 scrub starts 2024-04-06T03:24:24.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[29221]: cluster 2024-04-06T03:24:23.880745+0000 osd.7 (osd.7) 92 : cluster 0 6.3 scrub ok 2024-04-06T03:24:24.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[29221]: cluster 2024-04-06T03:24:23.885572+0000 osd.3 (osd.3) 95 : cluster 0 5.17 scrub starts 2024-04-06T03:24:24.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[29221]: cluster 2024-04-06T03:24:23.887804+0000 osd.3 (osd.3) 96 : cluster 0 5.17 scrub ok 2024-04-06T03:24:24.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:24 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.013649+0000 mgr.y (mgr.24368) 131 : cluster 0 pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:25.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:24 smithi153 bash[23852]: cluster 2024-04-06T03:24:23.877888+0000 osd.7 (osd.7) 91 : cluster 0 6.3 scrub starts 2024-04-06T03:24:25.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:24 smithi153 bash[23852]: cluster 2024-04-06T03:24:23.880745+0000 osd.7 (osd.7) 92 : cluster 0 6.3 scrub ok 2024-04-06T03:24:25.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:24 smithi153 bash[23852]: cluster 2024-04-06T03:24:23.885572+0000 osd.3 (osd.3) 95 : cluster 0 5.17 scrub starts 2024-04-06T03:24:25.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:24 smithi153 bash[23852]: cluster 2024-04-06T03:24:23.887804+0000 osd.3 (osd.3) 96 : cluster 0 5.17 scrub ok 2024-04-06T03:24:25.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:24 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.013649+0000 mgr.y (mgr.24368) 131 : cluster 0 pgmap v100: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.877900+0000 osd.1 (osd.1) 85 : cluster 0 2.15 scrub starts 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.879585+0000 osd.1 (osd.1) 86 : cluster 0 2.15 scrub ok 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.884877+0000 osd.7 (osd.7) 93 : cluster 0 2.8 scrub starts 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.886914+0000 osd.7 (osd.7) 94 : cluster 0 2.8 scrub ok 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.914782+0000 osd.3 (osd.3) 97 : cluster 0 5.14 scrub starts 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[21467]: cluster 2024-04-06T03:24:24.916831+0000 osd.3 (osd.3) 98 : cluster 0 5.14 scrub ok 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.877900+0000 osd.1 (osd.1) 85 : cluster 0 2.15 scrub starts 2024-04-06T03:24:25.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.879585+0000 osd.1 (osd.1) 86 : cluster 0 2.15 scrub ok 2024-04-06T03:24:25.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.884877+0000 osd.7 (osd.7) 93 : cluster 0 2.8 scrub starts 2024-04-06T03:24:25.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.886914+0000 osd.7 (osd.7) 94 : cluster 0 2.8 scrub ok 2024-04-06T03:24:25.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.914782+0000 osd.3 (osd.3) 97 : cluster 0 5.14 scrub starts 2024-04-06T03:24:25.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:25 smithi050 bash[29221]: cluster 2024-04-06T03:24:24.916831+0000 osd.3 (osd.3) 98 : cluster 0 5.14 scrub ok 2024-04-06T03:24:26.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:25 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.877900+0000 osd.1 (osd.1) 85 : cluster 0 2.15 scrub starts 2024-04-06T03:24:26.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:25 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.879585+0000 osd.1 (osd.1) 86 : cluster 0 2.15 scrub ok 2024-04-06T03:24:26.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:25 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.884877+0000 osd.7 (osd.7) 93 : cluster 0 2.8 scrub starts 2024-04-06T03:24:26.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:25 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.886914+0000 osd.7 (osd.7) 94 : cluster 0 2.8 scrub ok 2024-04-06T03:24:26.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:25 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.914782+0000 osd.3 (osd.3) 97 : cluster 0 5.14 scrub starts 2024-04-06T03:24:26.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:25 smithi153 bash[23852]: cluster 2024-04-06T03:24:24.916831+0000 osd.3 (osd.3) 98 : cluster 0 5.14 scrub ok 2024-04-06T03:24:26.224 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:24:26.226 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:24:26.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[21467]: cluster 2024-04-06T03:24:25.898053+0000 osd.1 (osd.1) 87 : cluster 0 3.19 scrub starts 2024-04-06T03:24:26.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[21467]: cluster 2024-04-06T03:24:25.901011+0000 osd.1 (osd.1) 88 : cluster 0 3.19 scrub ok 2024-04-06T03:24:26.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[21467]: cluster 2024-04-06T03:24:26.014041+0000 mgr.y (mgr.24368) 132 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[21467]: cluster 2024-04-06T03:24:26.200109+0000 osd.6 (osd.6) 53 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[21467]: cluster 2024-04-06T03:24:26.200203+0000 osd.6 (osd.6) 54 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[21467]: audit 2024-04-06T03:24:26.217938+0000 mgr.y (mgr.24368) 133 : audit 0 from='client.24557 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[29221]: cluster 2024-04-06T03:24:25.898053+0000 osd.1 (osd.1) 87 : cluster 0 3.19 scrub starts 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[29221]: cluster 2024-04-06T03:24:25.901011+0000 osd.1 (osd.1) 88 : cluster 0 3.19 scrub ok 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[29221]: cluster 2024-04-06T03:24:26.014041+0000 mgr.y (mgr.24368) 132 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[29221]: cluster 2024-04-06T03:24:26.200109+0000 osd.6 (osd.6) 53 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[29221]: cluster 2024-04-06T03:24:26.200203+0000 osd.6 (osd.6) 54 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:26.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:26 smithi050 bash[29221]: audit 2024-04-06T03:24:26.217938+0000 mgr.y (mgr.24368) 133 : audit 0 from='client.24557 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:27.033 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-04-06T03:24:26.013830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1429,"num_read_kb":1243,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":881,"ondisk_log_size":881,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248508,"kb_used_data":7420,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712708,"statfs":{"total":767960285184,"available":767705812992,"internally_reserved":0,"allocated":7598080,"data_stored":4804378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005737"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836553+0000","last_change":"2024-04-06T03:23:55.828897+0000","last_active":"2024-04-06T03:23:58.836553+0000","last_peered":"2024-04-06T03:23:58.836553+0000","last_clean":"2024-04-06T03:23:58.836553+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:23:58.836553+0000","last_undegraded":"2024-04-06T03:23:58.836553+0000","last_fullsized":"2024-04-06T03:23:58.836553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.828643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.828643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.076125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:03.509388+0000","last_change":"2024-04-06T03:24:03.509495+0000","last_active":"2024-04-06T03:24:03.509388+0000","last_peered":"2024-04-06T03:24:03.509388+0000","last_clean":"2024-04-06T03:24:03.509388+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:24:03.509388+0000","last_undegraded":"2024-04-06T03:24:03.509388+0000","last_fullsized":"2024-04-06T03:24:03.509388+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:03.509185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:03.509185+0000","last_clean_scrub_stamp":"2024-04-06T03:24:03.509185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:32.625692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.742721+0000","last_change":"2024-04-06T03:23:59.742745+0000","last_active":"2024-04-06T03:23:59.742721+0000","last_peered":"2024-04-06T03:23:59.742721+0000","last_clean":"2024-04-06T03:23:59.742721+0000","last_became_active":"2024-04-06T03:21:35.160225+0000","last_became_peered":"2024-04-06T03:21:35.160225+0000","last_unstale":"2024-04-06T03:23:59.742721+0000","last_undegraded":"2024-04-06T03:23:59.742721+0000","last_fullsized":"2024-04-06T03:23:59.742721+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:59.742638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:59.742638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.865855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"85'16","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837336+0000","last_change":"2024-04-06T03:23:20.747373+0000","last_active":"2024-04-06T03:23:58.837336+0000","last_peered":"2024-04-06T03:23:58.837336+0000","last_clean":"2024-04-06T03:23:58.837336+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:23:58.837336+0000","last_undegraded":"2024-04-06T03:23:58.837336+0000","last_fullsized":"2024-04-06T03:23:58.837336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:23:20.747059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.747059+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.765610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:05.556049+0000","last_change":"2024-04-06T03:24:05.556108+0000","last_active":"2024-04-06T03:24:05.556049+0000","last_peered":"2024-04-06T03:24:05.556049+0000","last_clean":"2024-04-06T03:24:05.556049+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:24:05.556049+0000","last_undegraded":"2024-04-06T03:24:05.556049+0000","last_fullsized":"2024-04-06T03:24:05.556049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:05.555884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:05.555884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.380839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:14.772039+0000","last_change":"2024-04-06T03:24:14.772163+0000","last_active":"2024-04-06T03:24:14.772039+0000","last_peered":"2024-04-06T03:24:14.772039+0000","last_clean":"2024-04-06T03:24:14.772039+0000","last_became_active":"2024-04-06T03:20:55.772175+0000","last_became_peered":"2024-04-06T03:20:55.772175+0000","last_unstale":"2024-04-06T03:24:14.772039+0000","last_undegraded":"2024-04-06T03:24:14.772039+0000","last_fullsized":"2024-04-06T03:24:14.772039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.771844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.771844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.161465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837892+0000","last_change":"2024-04-06T03:23:17.696688+0000","last_active":"2024-04-06T03:23:58.837892+0000","last_peered":"2024-04-06T03:23:58.837892+0000","last_clean":"2024-04-06T03:23:58.837892+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:23:58.837892+0000","last_undegraded":"2024-04-06T03:23:58.837892+0000","last_fullsized":"2024-04-06T03:23:58.837892+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.696473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.696473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.518460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":131,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478758+0000","last_change":"2024-04-06T03:23:52.836932+0000","last_active":"2024-04-06T03:23:58.478758+0000","last_peered":"2024-04-06T03:23:58.478758+0000","last_clean":"2024-04-06T03:23:58.478758+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:23:58.478758+0000","last_undegraded":"2024-04-06T03:23:58.478758+0000","last_fullsized":"2024-04-06T03:23:58.478758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:52.836543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.836543+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:09.578286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:10.108524+0000","last_change":"2024-04-06T03:24:10.108584+0000","last_active":"2024-04-06T03:24:10.108524+0000","last_peered":"2024-04-06T03:24:10.108524+0000","last_clean":"2024-04-06T03:24:10.108524+0000","last_became_active":"2024-04-06T03:20:57.825746+0000","last_became_peered":"2024-04-06T03:20:57.825746+0000","last_unstale":"2024-04-06T03:24:10.108524+0000","last_undegraded":"2024-04-06T03:24:10.108524+0000","last_fullsized":"2024-04-06T03:24:10.108524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:10.108359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:10.108359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.675483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:17.848987+0000","last_change":"2024-04-06T03:24:17.849044+0000","last_active":"2024-04-06T03:24:17.848987+0000","last_peered":"2024-04-06T03:24:17.848987+0000","last_clean":"2024-04-06T03:24:17.848987+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:24:17.848987+0000","last_undegraded":"2024-04-06T03:24:17.848987+0000","last_fullsized":"2024-04-06T03:24:17.848987+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:17.848834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:24:17.848834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.180144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.119727+0000","last_change":"2024-04-06T03:23:21.801571+0000","last_active":"2024-04-06T03:23:59.119727+0000","last_peered":"2024-04-06T03:23:59.119727+0000","last_clean":"2024-04-06T03:23:59.119727+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:23:59.119727+0000","last_undegraded":"2024-04-06T03:23:59.119727+0000","last_fullsized":"2024-04-06T03:23:59.119727+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:21.801341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.801341+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.356673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"112'22","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057866+0000","last_change":"2024-04-06T03:23:36.905749+0000","last_active":"2024-04-06T03:23:59.057866+0000","last_peered":"2024-04-06T03:23:59.057866+0000","last_clean":"2024-04-06T03:23:59.057866+0000","last_became_active":"2024-04-06T03:20:53.352525+0000","last_became_peered":"2024-04-06T03:20:53.352525+0000","last_unstale":"2024-04-06T03:23:59.057866+0000","last_undegraded":"2024-04-06T03:23:59.057866+0000","last_fullsized":"2024-04-06T03:23:59.057866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'22","last_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.905499+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.467751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:04.719237+0000","last_change":"2024-04-06T03:24:04.719301+0000","last_active":"2024-04-06T03:24:04.719237+0000","last_peered":"2024-04-06T03:24:04.719237+0000","last_clean":"2024-04-06T03:24:04.719237+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:24:04.719237+0000","last_undegraded":"2024-04-06T03:24:04.719237+0000","last_fullsized":"2024-04-06T03:24:04.719237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:04.719078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:04.719078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:31.164948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057128+0000","last_change":"2024-04-06T03:23:51.980318+0000","last_active":"2024-04-06T03:23:59.057128+0000","last_peered":"2024-04-06T03:23:59.057128+0000","last_clean":"2024-04-06T03:23:59.057128+0000","last_became_active":"2024-04-06T03:20:55.755058+0000","last_became_peered":"2024-04-06T03:20:55.755058+0000","last_unstale":"2024-04-06T03:23:59.057128+0000","last_undegraded":"2024-04-06T03:23:59.057128+0000","last_fullsized":"2024-04-06T03:23:59.057128+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:51.980107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:51.980107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:15.105749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:10.792224+0000","last_change":"2024-04-06T03:24:10.792283+0000","last_active":"2024-04-06T03:24:10.792224+0000","last_peered":"2024-04-06T03:24:10.792224+0000","last_clean":"2024-04-06T03:24:10.792224+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:24:10.792224+0000","last_undegraded":"2024-04-06T03:24:10.792224+0000","last_fullsized":"2024-04-06T03:24:10.792224+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:10.792042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:17.421734+0000","last_clean_scrub_stamp":"2024-04-06T03:24:10.792042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:29.598568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"105'24","reported_seq":142,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057990+0000","last_change":"2024-04-06T03:23:46.865483+0000","last_active":"2024-04-06T03:23:59.057990+0000","last_peered":"2024-04-06T03:23:59.057990+0000","last_clean":"2024-04-06T03:23:59.057990+0000","last_became_active":"2024-04-06T03:20:53.346381+0000","last_became_peered":"2024-04-06T03:20:53.346381+0000","last_unstale":"2024-04-06T03:23:59.057990+0000","last_undegraded":"2024-04-06T03:23:59.057990+0000","last_fullsized":"2024-04-06T03:23:59.057990+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'24","last_scrub_stamp":"2024-04-06T03:23:46.865084+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.865084+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.280139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.838032+0000","last_change":"2024-04-06T03:23:30.862864+0000","last_active":"2024-04-06T03:23:58.838032+0000","last_peered":"2024-04-06T03:23:58.838032+0000","last_clean":"2024-04-06T03:23:58.838032+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:23:58.838032+0000","last_undegraded":"2024-04-06T03:23:58.838032+0000","last_fullsized":"2024-04-06T03:23:58.838032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:30.862646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:30.862646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.636834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:00.468853+0000","last_change":"2024-04-06T03:24:00.468929+0000","last_active":"2024-04-06T03:24:00.468853+0000","last_peered":"2024-04-06T03:24:00.468853+0000","last_clean":"2024-04-06T03:24:00.468853+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:24:00.468853+0000","last_undegraded":"2024-04-06T03:24:00.468853+0000","last_fullsized":"2024-04-06T03:24:00.468853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:00.468662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.468662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:00.548032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:00.825398+0000","last_change":"2024-04-06T03:24:00.825469+0000","last_active":"2024-04-06T03:24:00.825398+0000","last_peered":"2024-04-06T03:24:00.825398+0000","last_clean":"2024-04-06T03:24:00.825398+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:24:00.825398+0000","last_undegraded":"2024-04-06T03:24:00.825398+0000","last_fullsized":"2024-04-06T03:24:00.825398+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:00.825184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.825184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:18.970342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"111'27","reported_seq":148,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472490+0000","last_change":"2024-04-06T03:23:18.287395+0000","last_active":"2024-04-06T03:23:58.472490+0000","last_peered":"2024-04-06T03:23:58.472490+0000","last_clean":"2024-04-06T03:23:58.472490+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:23:58.472490+0000","last_undegraded":"2024-04-06T03:23:58.472490+0000","last_fullsized":"2024-04-06T03:23:58.472490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'27","last_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.287127+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.693232+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474078+0000","last_change":"2024-04-06T03:23:57.475036+0000","last_active":"2024-04-06T03:23:58.474078+0000","last_peered":"2024-04-06T03:23:58.474078+0000","last_clean":"2024-04-06T03:23:58.474078+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:23:58.474078+0000","last_undegraded":"2024-04-06T03:23:58.474078+0000","last_fullsized":"2024-04-06T03:23:58.474078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:57.474774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:57.474774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.260532+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":89,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.123385+0000","last_change":"2024-04-06T03:23:01.687019+0000","last_active":"2024-04-06T03:23:59.123385+0000","last_peered":"2024-04-06T03:23:59.123385+0000","last_clean":"2024-04-06T03:23:59.123385+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:23:59.123385+0000","last_undegraded":"2024-04-06T03:23:59.123385+0000","last_fullsized":"2024-04-06T03:23:59.123385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_clean_scrub_stamp":"2024-04-06T03:23:01.686796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.412050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769491+0000","last_change":"2024-04-06T03:23:36.848600+0000","last_active":"2024-04-06T03:23:58.769491+0000","last_peered":"2024-04-06T03:23:58.769491+0000","last_clean":"2024-04-06T03:23:58.769491+0000","last_became_active":"2024-04-06T03:20:51.143283+0000","last_became_peered":"2024-04-06T03:20:51.143283+0000","last_unstale":"2024-04-06T03:23:58.769491+0000","last_undegraded":"2024-04-06T03:23:58.769491+0000","last_fullsized":"2024-04-06T03:23:58.769491+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.848364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.848364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.392566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477025+0000","last_change":"2024-04-06T03:23:14.042484+0000","last_active":"2024-04-06T03:23:58.477025+0000","last_peered":"2024-04-06T03:23:58.477025+0000","last_clean":"2024-04-06T03:23:58.477025+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:23:58.477025+0000","last_undegraded":"2024-04-06T03:23:58.477025+0000","last_fullsized":"2024-04-06T03:23:58.477025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:14.042058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:14.042058+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.768307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:03.673346+0000","last_change":"2024-04-06T03:24:03.673472+0000","last_active":"2024-04-06T03:24:03.673346+0000","last_peered":"2024-04-06T03:24:03.673346+0000","last_clean":"2024-04-06T03:24:03.673346+0000","last_became_active":"2024-04-06T03:20:57.845246+0000","last_became_peered":"2024-04-06T03:20:57.845246+0000","last_unstale":"2024-04-06T03:24:03.673346+0000","last_undegraded":"2024-04-06T03:24:03.673346+0000","last_fullsized":"2024-04-06T03:24:03.673346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:03.673150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:03.673150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:33.371338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477684+0000","last_change":"2024-04-06T03:23:34.010043+0000","last_active":"2024-04-06T03:23:58.477684+0000","last_peered":"2024-04-06T03:23:58.477684+0000","last_clean":"2024-04-06T03:23:58.477684+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:23:58.477684+0000","last_undegraded":"2024-04-06T03:23:58.477684+0000","last_fullsized":"2024-04-06T03:23:58.477684+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:23:34.009837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.282584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836179+0000","last_change":"2024-04-06T03:23:43.924012+0000","last_active":"2024-04-06T03:23:58.836179+0000","last_peered":"2024-04-06T03:23:58.836179+0000","last_clean":"2024-04-06T03:23:58.836179+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:23:58.836179+0000","last_undegraded":"2024-04-06T03:23:58.836179+0000","last_fullsized":"2024-04-06T03:23:58.836179+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:43.923791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.923791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:10.650300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"109'20","reported_seq":135,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:08.775685+0000","last_change":"2024-04-06T03:24:08.775767+0000","last_active":"2024-04-06T03:24:08.775685+0000","last_peered":"2024-04-06T03:24:08.775685+0000","last_clean":"2024-04-06T03:24:08.775685+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:24:08.775685+0000","last_undegraded":"2024-04-06T03:24:08.775685+0000","last_fullsized":"2024-04-06T03:24:08.775685+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'20","last_scrub_stamp":"2024-04-06T03:24:08.775535+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:24:08.775535+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:21.755432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:20.853625+0000","last_change":"2024-04-06T03:24:20.853706+0000","last_active":"2024-04-06T03:24:20.853625+0000","last_peered":"2024-04-06T03:24:20.853625+0000","last_clean":"2024-04-06T03:24:20.853625+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:24:20.853625+0000","last_undegraded":"2024-04-06T03:24:20.853625+0000","last_fullsized":"2024-04-06T03:24:20.853625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-06T03:24:20.853465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:20.853465+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:38.127543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"101'12","reported_seq":156,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.370044+0000","last_change":"2024-04-06T03:23:33.869748+0000","last_active":"2024-04-06T03:24:16.370044+0000","last_peered":"2024-04-06T03:24:16.370044+0000","last_clean":"2024-04-06T03:24:16.370044+0000","last_became_active":"2024-04-06T03:20:55.765174+0000","last_became_peered":"2024-04-06T03:20:55.765174+0000","last_unstale":"2024-04-06T03:24:16.370044+0000","last_undegraded":"2024-04-06T03:24:16.370044+0000","last_fullsized":"2024-04-06T03:24:16.370044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-06T03:23:33.869531+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.869531+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.720001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.056374+0000","last_change":"2024-04-06T03:23:19.763448+0000","last_active":"2024-04-06T03:23:59.056374+0000","last_peered":"2024-04-06T03:23:59.056374+0000","last_clean":"2024-04-06T03:23:59.056374+0000","last_became_active":"2024-04-06T03:20:51.144514+0000","last_became_peered":"2024-04-06T03:20:51.144514+0000","last_unstale":"2024-04-06T03:23:59.056374+0000","last_undegraded":"2024-04-06T03:23:59.056374+0000","last_fullsized":"2024-04-06T03:23:59.056374+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.763180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:54.626052+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.763180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:46.950926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"111'50","reported_seq":177,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.122151+0000","last_change":"2024-04-06T03:23:04.830805+0000","last_active":"2024-04-06T03:23:59.122151+0000","last_peered":"2024-04-06T03:23:59.122151+0000","last_clean":"2024-04-06T03:23:59.122151+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:23:59.122151+0000","last_undegraded":"2024-04-06T03:23:59.122151+0000","last_fullsized":"2024-04-06T03:23:59.122151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'50","last_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_clean_scrub_stamp":"2024-04-06T03:23:04.830548+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:25.118064+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.789705+0000","last_change":"2024-04-06T03:24:16.789772+0000","last_active":"2024-04-06T03:24:16.789705+0000","last_peered":"2024-04-06T03:24:16.789705+0000","last_clean":"2024-04-06T03:24:16.789705+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:24:16.789705+0000","last_undegraded":"2024-04-06T03:24:16.789705+0000","last_fullsized":"2024-04-06T03:24:16.789705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:16.789545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:24:16.789545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:19.882917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.119065+0000","last_change":"2024-04-06T03:22:53.518032+0000","last_active":"2024-04-06T03:23:59.119065+0000","last_peered":"2024-04-06T03:23:59.119065+0000","last_clean":"2024-04-06T03:23:59.119065+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:23:59.119065+0000","last_undegraded":"2024-04-06T03:23:59.119065+0000","last_fullsized":"2024-04-06T03:23:59.119065+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_clean_scrub_stamp":"2024-04-06T03:22:53.517730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:13.828631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770383+0000","last_change":"2024-04-06T03:23:26.863931+0000","last_active":"2024-04-06T03:23:58.770383+0000","last_peered":"2024-04-06T03:23:58.770383+0000","last_clean":"2024-04-06T03:23:58.770383+0000","last_became_active":"2024-04-06T03:20:51.142652+0000","last_became_peered":"2024-04-06T03:20:51.142652+0000","last_unstale":"2024-04-06T03:23:58.770383+0000","last_undegraded":"2024-04-06T03:23:58.770383+0000","last_fullsized":"2024-04-06T03:23:58.770383+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:23:26.863691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:41.711589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057802+0000","last_change":"2024-04-06T03:23:47.908697+0000","last_active":"2024-04-06T03:23:59.057802+0000","last_peered":"2024-04-06T03:23:59.057802+0000","last_clean":"2024-04-06T03:23:59.057802+0000","last_became_active":"2024-04-06T03:20:53.350189+0000","last_became_peered":"2024-04-06T03:20:53.350189+0000","last_unstale":"2024-04-06T03:23:59.057802+0000","last_undegraded":"2024-04-06T03:23:59.057802+0000","last_fullsized":"2024-04-06T03:23:59.057802+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:47.908288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.908288+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.038863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:04.681479+0000","last_change":"2024-04-06T03:24:04.681542+0000","last_active":"2024-04-06T03:24:04.681479+0000","last_peered":"2024-04-06T03:24:04.681479+0000","last_clean":"2024-04-06T03:24:04.681479+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:24:04.681479+0000","last_undegraded":"2024-04-06T03:24:04.681479+0000","last_fullsized":"2024-04-06T03:24:04.681479+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:04.681314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_clean_scrub_stamp":"2024-04-06T03:24:04.681314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:26.747454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769129+0000","last_change":"2024-04-06T03:23:25.887364+0000","last_active":"2024-04-06T03:23:58.769129+0000","last_peered":"2024-04-06T03:23:58.769129+0000","last_clean":"2024-04-06T03:23:58.769129+0000","last_became_active":"2024-04-06T03:20:55.769876+0000","last_became_peered":"2024-04-06T03:20:55.769876+0000","last_unstale":"2024-04-06T03:23:58.769129+0000","last_undegraded":"2024-04-06T03:23:58.769129+0000","last_fullsized":"2024-04-06T03:23:58.769129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:25.887152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:25.887152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.529607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:21.115332+0000","last_change":"2024-04-06T03:24:21.115390+0000","last_active":"2024-04-06T03:24:21.115332+0000","last_peered":"2024-04-06T03:24:21.115332+0000","last_clean":"2024-04-06T03:24:21.115332+0000","last_became_active":"2024-04-06T03:20:51.151269+0000","last_became_peered":"2024-04-06T03:20:51.151269+0000","last_unstale":"2024-04-06T03:24:21.115332+0000","last_undegraded":"2024-04-06T03:24:21.115332+0000","last_fullsized":"2024-04-06T03:24:21.115332+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:21.115189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:21.115189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:39.137464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":121,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478669+0000","last_change":"2024-04-06T03:23:18.056122+0000","last_active":"2024-04-06T03:23:58.478669+0000","last_peered":"2024-04-06T03:23:58.478669+0000","last_clean":"2024-04-06T03:23:58.478669+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:23:58.478669+0000","last_undegraded":"2024-04-06T03:23:58.478669+0000","last_fullsized":"2024-04-06T03:23:58.478669+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:18.056024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.056024+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.251393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:14.730027+0000","last_change":"2024-04-06T03:24:14.730089+0000","last_active":"2024-04-06T03:24:14.730027+0000","last_peered":"2024-04-06T03:24:14.730027+0000","last_clean":"2024-04-06T03:24:14.730027+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:24:14.730027+0000","last_undegraded":"2024-04-06T03:24:14.730027+0000","last_fullsized":"2024-04-06T03:24:14.730027+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.729868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.729868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:36.530227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:19.927269+0000","last_change":"2024-04-06T03:24:19.927351+0000","last_active":"2024-04-06T03:24:19.927269+0000","last_peered":"2024-04-06T03:24:19.927269+0000","last_clean":"2024-04-06T03:24:19.927269+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:24:19.927269+0000","last_undegraded":"2024-04-06T03:24:19.927269+0000","last_fullsized":"2024-04-06T03:24:19.927269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:19.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:19.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:45.184511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.123033+0000","last_change":"2024-04-06T03:23:12.824588+0000","last_active":"2024-04-06T03:23:59.123033+0000","last_peered":"2024-04-06T03:23:59.123033+0000","last_clean":"2024-04-06T03:23:59.123033+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:23:59.123033+0000","last_undegraded":"2024-04-06T03:23:59.123033+0000","last_fullsized":"2024-04-06T03:23:59.123033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_clean_scrub_stamp":"2024-04-06T03:23:12.824475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:24.655532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'23","reported_seq":142,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472549+0000","last_change":"2024-04-06T03:23:29.339315+0000","last_active":"2024-04-06T03:23:58.472549+0000","last_peered":"2024-04-06T03:23:58.472549+0000","last_clean":"2024-04-06T03:23:58.472549+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:23:58.472549+0000","last_undegraded":"2024-04-06T03:23:58.472549+0000","last_fullsized":"2024-04-06T03:23:58.472549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-06T03:23:29.339052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.339052+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.010581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474688+0000","last_change":"2024-04-06T03:23:55.780644+0000","last_active":"2024-04-06T03:23:58.474688+0000","last_peered":"2024-04-06T03:23:58.474688+0000","last_clean":"2024-04-06T03:23:58.474688+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:23:58.474688+0000","last_undegraded":"2024-04-06T03:23:58.474688+0000","last_fullsized":"2024-04-06T03:23:58.474688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.780397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.780397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.956035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837343+0000","last_change":"2024-04-06T03:23:52.841531+0000","last_active":"2024-04-06T03:23:58.837343+0000","last_peered":"2024-04-06T03:23:58.837343+0000","last_clean":"2024-04-06T03:23:58.837343+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:23:58.837343+0000","last_undegraded":"2024-04-06T03:23:58.837343+0000","last_fullsized":"2024-04-06T03:23:58.837343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:52.841291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.841291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.389698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:18.885359+0000","last_change":"2024-04-06T03:24:18.885421+0000","last_active":"2024-04-06T03:24:18.885359+0000","last_peered":"2024-04-06T03:24:18.885359+0000","last_clean":"2024-04-06T03:24:18.885359+0000","last_became_active":"2024-04-06T03:20:51.137376+0000","last_became_peered":"2024-04-06T03:20:51.137376+0000","last_unstale":"2024-04-06T03:24:18.885359+0000","last_undegraded":"2024-04-06T03:24:18.885359+0000","last_fullsized":"2024-04-06T03:24:18.885359+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:18.885184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:24:18.885184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:46.543194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:15.809941+0000","last_change":"2024-04-06T03:24:15.810022+0000","last_active":"2024-04-06T03:24:15.809941+0000","last_peered":"2024-04-06T03:24:15.809941+0000","last_clean":"2024-04-06T03:24:15.809941+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:24:15.809941+0000","last_undegraded":"2024-04-06T03:24:15.809941+0000","last_fullsized":"2024-04-06T03:24:15.809941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:24:15.809780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:15.809780+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:37.555195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473782+0000","last_change":"2024-04-06T03:23:19.298817+0000","last_active":"2024-04-06T03:23:58.473782+0000","last_peered":"2024-04-06T03:23:58.473782+0000","last_clean":"2024-04-06T03:23:58.473782+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:23:58.473782+0000","last_undegraded":"2024-04-06T03:23:58.473782+0000","last_fullsized":"2024-04-06T03:23:58.473782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.298498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.298498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.565314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.374213+0000","last_change":"2024-04-06T03:22:59.658875+0000","last_active":"2024-04-06T03:24:16.374213+0000","last_peered":"2024-04-06T03:24:16.374213+0000","last_clean":"2024-04-06T03:24:16.374213+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:24:16.374213+0000","last_undegraded":"2024-04-06T03:24:16.374213+0000","last_fullsized":"2024-04-06T03:24:16.374213+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:22:59.658761+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477187+0000","last_change":"2024-04-06T03:23:40.956392+0000","last_active":"2024-04-06T03:23:58.477187+0000","last_peered":"2024-04-06T03:23:58.477187+0000","last_clean":"2024-04-06T03:23:58.477187+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:23:58.477187+0000","last_undegraded":"2024-04-06T03:23:58.477187+0000","last_fullsized":"2024-04-06T03:23:58.477187+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:40.956167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:40.956167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.850753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:00.754062+0000","last_change":"2024-04-06T03:24:00.754139+0000","last_active":"2024-04-06T03:24:00.754062+0000","last_peered":"2024-04-06T03:24:00.754062+0000","last_clean":"2024-04-06T03:24:00.754062+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:24:00.754062+0000","last_undegraded":"2024-04-06T03:24:00.754062+0000","last_fullsized":"2024-04-06T03:24:00.754062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-06T03:24:00.753905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.753905+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.240141+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.477224+0000","last_change":"2024-04-06T03:23:28.068250+0000","last_active":"2024-04-06T03:23:58.477224+0000","last_peered":"2024-04-06T03:23:58.477224+0000","last_clean":"2024-04-06T03:23:58.477224+0000","last_became_active":"2024-04-06T03:20:57.843310+0000","last_became_peered":"2024-04-06T03:20:57.843310+0000","last_unstale":"2024-04-06T03:23:58.477224+0000","last_undegraded":"2024-04-06T03:23:58.477224+0000","last_fullsized":"2024-04-06T03:23:58.477224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.067970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.747904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"85'11","reported_seq":150,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.370047+0000","last_change":"2024-04-06T03:23:45.891079+0000","last_active":"2024-04-06T03:24:16.370047+0000","last_peered":"2024-04-06T03:24:16.370047+0000","last_clean":"2024-04-06T03:24:16.370047+0000","last_became_active":"2024-04-06T03:20:55.770173+0000","last_became_peered":"2024-04-06T03:20:55.770173+0000","last_unstale":"2024-04-06T03:24:16.370047+0000","last_undegraded":"2024-04-06T03:24:16.370047+0000","last_fullsized":"2024-04-06T03:24:16.370047+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:45.890804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:45.890804+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.597014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.056918+0000","last_change":"2024-04-06T03:23:21.739882+0000","last_active":"2024-04-06T03:23:59.056918+0000","last_peered":"2024-04-06T03:23:59.056918+0000","last_clean":"2024-04-06T03:23:59.056918+0000","last_became_active":"2024-04-06T03:20:51.157201+0000","last_became_peered":"2024-04-06T03:20:51.157201+0000","last_unstale":"2024-04-06T03:23:59.056918+0000","last_undegraded":"2024-04-06T03:23:59.056918+0000","last_fullsized":"2024-04-06T03:23:59.056918+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:21.739671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.739671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.621751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":119,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:00.790379+0000","last_change":"2024-04-06T03:24:00.790460+0000","last_active":"2024-04-06T03:24:00.790379+0000","last_peered":"2024-04-06T03:24:00.790379+0000","last_clean":"2024-04-06T03:24:00.790379+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:24:00.790379+0000","last_undegraded":"2024-04-06T03:24:00.790379+0000","last_fullsized":"2024-04-06T03:24:00.790379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:24:00.790226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.790226+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:16.640308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.119738+0000","last_change":"2024-04-06T03:23:55.806848+0000","last_active":"2024-04-06T03:23:59.119738+0000","last_peered":"2024-04-06T03:23:59.119738+0000","last_clean":"2024-04-06T03:23:59.119738+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:23:59.119738+0000","last_undegraded":"2024-04-06T03:23:59.119738+0000","last_fullsized":"2024-04-06T03:23:59.119738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.806612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.806612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.253217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057951+0000","last_change":"2024-04-06T03:23:31.955588+0000","last_active":"2024-04-06T03:23:59.057951+0000","last_peered":"2024-04-06T03:23:59.057951+0000","last_clean":"2024-04-06T03:23:59.057951+0000","last_became_active":"2024-04-06T03:20:55.756465+0000","last_became_peered":"2024-04-06T03:20:55.756465+0000","last_unstale":"2024-04-06T03:23:59.057951+0000","last_undegraded":"2024-04-06T03:23:59.057951+0000","last_fullsized":"2024-04-06T03:23:59.057951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:31.955371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:31.955371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:56.516085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769295+0000","last_change":"2024-04-06T03:23:35.835087+0000","last_active":"2024-04-06T03:23:58.769295+0000","last_peered":"2024-04-06T03:23:58.769295+0000","last_clean":"2024-04-06T03:23:58.769295+0000","last_became_active":"2024-04-06T03:20:51.145847+0000","last_became_peered":"2024-04-06T03:20:51.145847+0000","last_unstale":"2024-04-06T03:23:58.769295+0000","last_undegraded":"2024-04-06T03:23:58.769295+0000","last_fullsized":"2024-04-06T03:23:58.769295+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.834842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.834842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.169606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"111'6","reported_seq":108,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769153+0000","last_change":"2024-04-06T03:23:21.875463+0000","last_active":"2024-04-06T03:23:58.769153+0000","last_peered":"2024-04-06T03:23:58.769153+0000","last_clean":"2024-04-06T03:23:58.769153+0000","last_became_active":"2024-04-06T03:20:53.348930+0000","last_became_peered":"2024-04-06T03:20:53.348930+0000","last_unstale":"2024-04-06T03:23:58.769153+0000","last_undegraded":"2024-04-06T03:23:58.769153+0000","last_fullsized":"2024-04-06T03:23:58.769153+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6","last_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.875237+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.782953+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:12.542399+0000","last_change":"2024-04-06T03:24:12.542535+0000","last_active":"2024-04-06T03:24:12.542399+0000","last_peered":"2024-04-06T03:24:12.542399+0000","last_clean":"2024-04-06T03:24:12.542399+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:24:12.542399+0000","last_undegraded":"2024-04-06T03:24:12.542399+0000","last_fullsized":"2024-04-06T03:24:12.542399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:24:12.542082+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:24:12.542082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:26.649946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:23.887960+0000","last_change":"2024-04-06T03:24:23.888015+0000","last_active":"2024-04-06T03:24:23.887960+0000","last_peered":"2024-04-06T03:24:23.887960+0000","last_clean":"2024-04-06T03:24:23.887960+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:24:23.887960+0000","last_undegraded":"2024-04-06T03:24:23.887960+0000","last_fullsized":"2024-04-06T03:24:23.887960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:23.887814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:23.887814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:32.529593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473357+0000","last_change":"2024-04-06T03:23:23.589775+0000","last_active":"2024-04-06T03:23:58.473357+0000","last_peered":"2024-04-06T03:23:58.473357+0000","last_clean":"2024-04-06T03:23:58.473357+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:23:58.473357+0000","last_undegraded":"2024-04-06T03:23:58.473357+0000","last_fullsized":"2024-04-06T03:23:58.473357+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:23.589472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.589472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.531624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"111'23","reported_seq":137,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.118434+0000","last_change":"2024-04-06T03:23:20.835769+0000","last_active":"2024-04-06T03:23:59.118434+0000","last_peered":"2024-04-06T03:23:59.118434+0000","last_clean":"2024-04-06T03:23:59.118434+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:23:59.118434+0000","last_undegraded":"2024-04-06T03:23:59.118434+0000","last_fullsized":"2024-04-06T03:23:59.118434+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'23","last_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.835514+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:30.227431+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837493+0000","last_change":"2024-04-06T03:23:44.976222+0000","last_active":"2024-04-06T03:23:58.837493+0000","last_peered":"2024-04-06T03:23:58.837493+0000","last_clean":"2024-04-06T03:23:58.837493+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:23:58.837493+0000","last_undegraded":"2024-04-06T03:23:58.837493+0000","last_fullsized":"2024-04-06T03:23:58.837493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.976015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.976015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.423557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473045+0000","last_change":"2024-04-06T03:23:09.163493+0000","last_active":"2024-04-06T03:23:58.473045+0000","last_peered":"2024-04-06T03:23:58.473045+0000","last_clean":"2024-04-06T03:23:58.473045+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:23:58.473045+0000","last_undegraded":"2024-04-06T03:23:58.473045+0000","last_fullsized":"2024-04-06T03:23:58.473045+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-06T03:23:09.163254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.163254+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.561644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":131,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.119946+0000","last_change":"2024-04-06T03:23:19.822376+0000","last_active":"2024-04-06T03:23:59.119946+0000","last_peered":"2024-04-06T03:23:59.119946+0000","last_clean":"2024-04-06T03:23:59.119946+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:23:59.119946+0000","last_undegraded":"2024-04-06T03:23:59.119946+0000","last_fullsized":"2024-04-06T03:23:59.119946+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:19.822131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.822131+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.155287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473044+0000","last_change":"2024-04-06T03:23:52.629246+0000","last_active":"2024-04-06T03:23:58.473044+0000","last_peered":"2024-04-06T03:23:58.473044+0000","last_clean":"2024-04-06T03:23:58.473044+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:23:58.473044+0000","last_undegraded":"2024-04-06T03:23:58.473044+0000","last_fullsized":"2024-04-06T03:23:58.473044+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:52.629002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.629002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.822619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:19.940038+0000","last_change":"2024-04-06T03:24:19.940098+0000","last_active":"2024-04-06T03:24:19.940038+0000","last_peered":"2024-04-06T03:24:19.940038+0000","last_clean":"2024-04-06T03:24:19.940038+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:24:19.940038+0000","last_undegraded":"2024-04-06T03:24:19.940038+0000","last_fullsized":"2024-04-06T03:24:19.940038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:19.939848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:24:19.939848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.181812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.122682+0000","last_change":"2024-04-06T03:23:50.717593+0000","last_active":"2024-04-06T03:23:59.122682+0000","last_peered":"2024-04-06T03:23:59.122682+0000","last_clean":"2024-04-06T03:23:59.122682+0000","last_became_active":"2024-04-06T03:20:51.148632+0000","last_became_peered":"2024-04-06T03:20:51.148632+0000","last_unstale":"2024-04-06T03:23:59.122682+0000","last_undegraded":"2024-04-06T03:23:59.122682+0000","last_fullsized":"2024-04-06T03:23:59.122682+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:50.717354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:50.717354+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:52.790825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":166,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.370579+0000","last_change":"2024-04-06T03:24:06.088633+0000","last_active":"2024-04-06T03:24:16.370579+0000","last_peered":"2024-04-06T03:24:16.370579+0000","last_clean":"2024-04-06T03:24:16.370579+0000","last_became_active":"2024-04-06T03:20:53.351328+0000","last_became_peered":"2024-04-06T03:20:53.351328+0000","last_unstale":"2024-04-06T03:24:16.370579+0000","last_undegraded":"2024-04-06T03:24:16.370579+0000","last_fullsized":"2024-04-06T03:24:16.370579+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:24:06.088289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.088289+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.093161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":109,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.121362+0000","last_change":"2024-04-06T03:23:56.666850+0000","last_active":"2024-04-06T03:23:59.121362+0000","last_peered":"2024-04-06T03:23:59.121362+0000","last_clean":"2024-04-06T03:23:59.121362+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:23:59.121362+0000","last_undegraded":"2024-04-06T03:23:59.121362+0000","last_fullsized":"2024-04-06T03:23:59.121362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:23:56.666442+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.666442+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:17.097001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":149,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:20.953132+0000","last_change":"2024-04-06T03:24:20.953132+0000","last_active":"2024-04-06T03:24:20.953132+0000","last_peered":"2024-04-06T03:24:20.953132+0000","last_clean":"2024-04-06T03:24:20.953132+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:24:20.953132+0000","last_undegraded":"2024-04-06T03:24:20.953132+0000","last_fullsized":"2024-04-06T03:24:20.953132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:24:20.952992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:20.952992+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:22.514687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:07.804851+0000","last_change":"2024-04-06T03:24:07.804878+0000","last_active":"2024-04-06T03:24:07.804851+0000","last_peered":"2024-04-06T03:24:07.804851+0000","last_clean":"2024-04-06T03:24:07.804851+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:24:07.804851+0000","last_undegraded":"2024-04-06T03:24:07.804851+0000","last_fullsized":"2024-04-06T03:24:07.804851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:07.804777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:07.804777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:14.120558+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.056898+0000","last_change":"2024-04-06T03:23:27.845925+0000","last_active":"2024-04-06T03:23:59.056898+0000","last_peered":"2024-04-06T03:23:59.056898+0000","last_clean":"2024-04-06T03:23:59.056898+0000","last_became_active":"2024-04-06T03:20:51.148366+0000","last_became_peered":"2024-04-06T03:20:51.148366+0000","last_unstale":"2024-04-06T03:23:59.056898+0000","last_undegraded":"2024-04-06T03:23:59.056898+0000","last_fullsized":"2024-04-06T03:23:59.056898+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:27.845710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:27.845710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.306766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'34","reported_seq":159,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.442668+0000","last_change":"2024-04-06T03:23:59.442760+0000","last_active":"2024-04-06T03:23:59.442668+0000","last_peered":"2024-04-06T03:23:59.442668+0000","last_clean":"2024-04-06T03:23:59.442668+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:23:59.442668+0000","last_undegraded":"2024-04-06T03:23:59.442668+0000","last_fullsized":"2024-04-06T03:23:59.442668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'34","last_scrub_stamp":"2024-04-06T03:23:59.442517+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_clean_scrub_stamp":"2024-04-06T03:23:59.442517+0000","objects_scrubbed":9,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:27.492213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:06.724710+0000","last_change":"2024-04-06T03:24:06.724738+0000","last_active":"2024-04-06T03:24:06.724710+0000","last_peered":"2024-04-06T03:24:06.724710+0000","last_clean":"2024-04-06T03:24:06.724710+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:24:06.724710+0000","last_undegraded":"2024-04-06T03:24:06.724710+0000","last_fullsized":"2024-04-06T03:24:06.724710+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:06.724641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.724641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:17.576679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:02.704818+0000","last_change":"2024-04-06T03:24:02.704976+0000","last_active":"2024-04-06T03:24:02.704818+0000","last_peered":"2024-04-06T03:24:02.704818+0000","last_clean":"2024-04-06T03:24:02.704818+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:24:02.704818+0000","last_undegraded":"2024-04-06T03:24:02.704818+0000","last_fullsized":"2024-04-06T03:24:02.704818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:02.704534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:24:02.704534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.684101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":157,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.122152+0000","last_change":"2024-04-06T03:23:35.682185+0000","last_active":"2024-04-06T03:23:59.122152+0000","last_peered":"2024-04-06T03:23:59.122152+0000","last_clean":"2024-04-06T03:23:59.122152+0000","last_became_active":"2024-04-06T03:20:53.358177+0000","last_became_peered":"2024-04-06T03:20:53.358177+0000","last_unstale":"2024-04-06T03:23:59.122152+0000","last_undegraded":"2024-04-06T03:23:59.122152+0000","last_fullsized":"2024-04-06T03:23:59.122152+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-06T03:23:35.681909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.681909+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.508581+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473242+0000","last_change":"2024-04-06T03:23:53.594743+0000","last_active":"2024-04-06T03:23:58.473242+0000","last_peered":"2024-04-06T03:23:58.473242+0000","last_clean":"2024-04-06T03:23:58.473242+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:23:58.473242+0000","last_undegraded":"2024-04-06T03:23:58.473242+0000","last_fullsized":"2024-04-06T03:23:58.473242+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:53.594492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:53.594492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:18.217623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473560+0000","last_change":"2024-04-06T03:23:47.598906+0000","last_active":"2024-04-06T03:23:58.473560+0000","last_peered":"2024-04-06T03:23:58.473560+0000","last_clean":"2024-04-06T03:23:58.473560+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:23:58.473560+0000","last_undegraded":"2024-04-06T03:23:58.473560+0000","last_fullsized":"2024-04-06T03:23:58.473560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:47.598682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.598682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.431027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.058691+0000","last_change":"2024-04-06T03:23:37.854225+0000","last_active":"2024-04-06T03:23:59.058691+0000","last_peered":"2024-04-06T03:23:59.058691+0000","last_clean":"2024-04-06T03:23:59.058691+0000","last_became_active":"2024-04-06T03:20:57.822724+0000","last_became_peered":"2024-04-06T03:20:57.822724+0000","last_unstale":"2024-04-06T03:23:59.058691+0000","last_undegraded":"2024-04-06T03:23:59.058691+0000","last_fullsized":"2024-04-06T03:23:59.058691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:37.854012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.854012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.515577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836547+0000","last_change":"2024-04-06T03:23:51.862357+0000","last_active":"2024-04-06T03:23:58.836547+0000","last_peered":"2024-04-06T03:23:58.836547+0000","last_clean":"2024-04-06T03:23:58.836547+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:23:58.836547+0000","last_undegraded":"2024-04-06T03:23:58.836547+0000","last_fullsized":"2024-04-06T03:23:58.836547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:23:51.861992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:51.861992+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.736112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057025+0000","last_change":"2024-04-06T03:23:20.743012+0000","last_active":"2024-04-06T03:23:59.057025+0000","last_peered":"2024-04-06T03:23:59.057025+0000","last_clean":"2024-04-06T03:23:59.057025+0000","last_became_active":"2024-04-06T03:20:51.151534+0000","last_became_peered":"2024-04-06T03:20:51.151534+0000","last_unstale":"2024-04-06T03:23:59.057025+0000","last_undegraded":"2024-04-06T03:23:59.057025+0000","last_fullsized":"2024-04-06T03:23:59.057025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:20.742768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.742768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:26.458288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":364,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:18.862939+0000","last_change":"2024-04-06T03:24:18.862939+0000","last_active":"2024-04-06T03:24:18.862939+0000","last_peered":"2024-04-06T03:24:18.862939+0000","last_clean":"2024-04-06T03:24:18.862939+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:24:18.862939+0000","last_undegraded":"2024-04-06T03:24:18.862939+0000","last_fullsized":"2024-04-06T03:24:18.862939+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:24:18.862554+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:24:18.862554+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:40.986138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:14.817248+0000","last_change":"2024-04-06T03:24:14.817313+0000","last_active":"2024-04-06T03:24:14.817248+0000","last_peered":"2024-04-06T03:24:14.817248+0000","last_clean":"2024-04-06T03:24:14.817248+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:24:14.817248+0000","last_undegraded":"2024-04-06T03:24:14.817248+0000","last_fullsized":"2024-04-06T03:24:14.817248+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.817089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:43.041041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:09.753140+0000","last_change":"2024-04-06T03:24:09.753198+0000","last_active":"2024-04-06T03:24:09.753140+0000","last_peered":"2024-04-06T03:24:09.753140+0000","last_clean":"2024-04-06T03:24:09.753140+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:24:09.753140+0000","last_undegraded":"2024-04-06T03:24:09.753140+0000","last_fullsized":"2024-04-06T03:24:09.753140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:09.752966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:09.752966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:22.006299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'21","reported_seq":134,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478175+0000","last_change":"2024-04-06T03:23:23.076419+0000","last_active":"2024-04-06T03:23:58.478175+0000","last_peered":"2024-04-06T03:23:58.478175+0000","last_clean":"2024-04-06T03:23:58.478175+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:23:58.478175+0000","last_undegraded":"2024-04-06T03:23:58.478175+0000","last_fullsized":"2024-04-06T03:23:58.478175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:23.076269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.076269+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.958707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.118218+0000","last_change":"2024-04-06T03:22:54.515963+0000","last_active":"2024-04-06T03:23:59.118218+0000","last_peered":"2024-04-06T03:23:59.118218+0000","last_clean":"2024-04-06T03:23:59.118218+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:23:59.118218+0000","last_undegraded":"2024-04-06T03:23:59.118218+0000","last_fullsized":"2024-04-06T03:23:59.118218+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:54.515739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:54.515739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:15.695306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.058545+0000","last_change":"2024-04-06T03:23:28.869298+0000","last_active":"2024-04-06T03:23:59.058545+0000","last_peered":"2024-04-06T03:23:59.058545+0000","last_clean":"2024-04-06T03:23:59.058545+0000","last_became_active":"2024-04-06T03:20:55.754600+0000","last_became_peered":"2024-04-06T03:20:55.754600+0000","last_unstale":"2024-04-06T03:23:59.058545+0000","last_undegraded":"2024-04-06T03:23:59.058545+0000","last_fullsized":"2024-04-06T03:23:59.058545+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.869068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.869068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.304800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.121712+0000","last_change":"2024-04-06T03:23:48.690384+0000","last_active":"2024-04-06T03:23:59.121712+0000","last_peered":"2024-04-06T03:23:59.121712+0000","last_clean":"2024-04-06T03:23:59.121712+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:23:59.121712+0000","last_undegraded":"2024-04-06T03:23:59.121712+0000","last_fullsized":"2024-04-06T03:23:59.121712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:48.690059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.690059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.780733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:06.712257+0000","last_change":"2024-04-06T03:24:06.712317+0000","last_active":"2024-04-06T03:24:06.712257+0000","last_peered":"2024-04-06T03:24:06.712257+0000","last_clean":"2024-04-06T03:24:06.712257+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:24:06.712257+0000","last_undegraded":"2024-04-06T03:24:06.712257+0000","last_fullsized":"2024-04-06T03:24:06.712257+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:06.712077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:06.712077+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.712077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:16.117036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'37","reported_seq":207,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.373423+0000","last_change":"2024-04-06T03:23:09.792573+0000","last_active":"2024-04-06T03:24:16.373423+0000","last_peered":"2024-04-06T03:24:16.373423+0000","last_clean":"2024-04-06T03:24:16.373423+0000","last_became_active":"2024-04-06T03:20:53.352453+0000","last_became_peered":"2024-04-06T03:20:53.352453+0000","last_unstale":"2024-04-06T03:24:16.373423+0000","last_undegraded":"2024-04-06T03:24:16.373423+0000","last_fullsized":"2024-04-06T03:24:16.373423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'37","last_scrub_stamp":"2024-04-06T03:23:09.792373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.792373+0000","objects_scrubbed":11,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.096219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770224+0000","last_change":"2024-04-06T03:23:39.777144+0000","last_active":"2024-04-06T03:23:58.770224+0000","last_peered":"2024-04-06T03:23:58.770224+0000","last_clean":"2024-04-06T03:23:58.770224+0000","last_became_active":"2024-04-06T03:20:55.769795+0000","last_became_peered":"2024-04-06T03:20:55.769795+0000","last_unstale":"2024-04-06T03:23:58.770224+0000","last_undegraded":"2024-04-06T03:23:58.770224+0000","last_fullsized":"2024-04-06T03:23:58.770224+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.776933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.776933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.208938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474197+0000","last_change":"2024-04-06T03:23:35.364623+0000","last_active":"2024-04-06T03:23:58.474197+0000","last_peered":"2024-04-06T03:23:58.474197+0000","last_clean":"2024-04-06T03:23:58.474197+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:23:58.474197+0000","last_undegraded":"2024-04-06T03:23:58.474197+0000","last_fullsized":"2024-04-06T03:23:58.474197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.364381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.332831+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.364381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.739996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.123662+0000","last_change":"2024-04-06T03:23:37.757642+0000","last_active":"2024-04-06T03:23:59.123662+0000","last_peered":"2024-04-06T03:23:59.123662+0000","last_clean":"2024-04-06T03:23:59.123662+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:23:59.123662+0000","last_undegraded":"2024-04-06T03:23:59.123662+0000","last_fullsized":"2024-04-06T03:23:59.123662+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:37.757420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.757420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.290369+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:06.782197+0000","last_change":"2024-04-06T03:24:06.782236+0000","last_active":"2024-04-06T03:24:06.782197+0000","last_peered":"2024-04-06T03:24:06.782197+0000","last_clean":"2024-04-06T03:24:06.782197+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:24:06.782197+0000","last_undegraded":"2024-04-06T03:24:06.782197+0000","last_fullsized":"2024-04-06T03:24:06.782197+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-06T03:24:06.782116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.782116+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:22.152506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":301,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:22.047659+0000","last_change":"2024-04-06T03:23:06.631761+0000","last_active":"2024-04-06T03:24:22.047659+0000","last_peered":"2024-04-06T03:24:22.047659+0000","last_clean":"2024-04-06T03:24:22.047659+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:24:22.047659+0000","last_undegraded":"2024-04-06T03:24:22.047659+0000","last_fullsized":"2024-04-06T03:24:22.047659+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:23:06.631539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:23:06.631539+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:25.562835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":191,"num_read_kb":186,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472783+0000","last_change":"2024-04-06T03:23:53.414430+0000","last_active":"2024-04-06T03:23:58.472783+0000","last_peered":"2024-04-06T03:23:58.472783+0000","last_clean":"2024-04-06T03:23:58.472783+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:23:58.472783+0000","last_undegraded":"2024-04-06T03:23:58.472783+0000","last_fullsized":"2024-04-06T03:23:58.472783+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:53.414220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:53.414220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:07.437199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:23.880898+0000","last_change":"2024-04-06T03:24:23.880954+0000","last_active":"2024-04-06T03:24:23.880898+0000","last_peered":"2024-04-06T03:24:23.880898+0000","last_clean":"2024-04-06T03:24:23.880898+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:24:23.880898+0000","last_undegraded":"2024-04-06T03:24:23.880898+0000","last_fullsized":"2024-04-06T03:24:23.880898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:23.880755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:23.880755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:35.813387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.770152+0000","last_change":"2024-04-06T03:23:17.864400+0000","last_active":"2024-04-06T03:23:58.770152+0000","last_peered":"2024-04-06T03:23:58.770152+0000","last_clean":"2024-04-06T03:23:58.770152+0000","last_became_active":"2024-04-06T03:20:51.134999+0000","last_became_peered":"2024-04-06T03:20:51.134999+0000","last_unstale":"2024-04-06T03:23:58.770152+0000","last_undegraded":"2024-04-06T03:23:58.770152+0000","last_fullsized":"2024-04-06T03:23:58.770152+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.864183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.536200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.478081+0000","last_change":"2024-04-06T03:23:39.032748+0000","last_active":"2024-04-06T03:23:58.478081+0000","last_peered":"2024-04-06T03:23:58.478081+0000","last_clean":"2024-04-06T03:23:58.478081+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:23:58.478081+0000","last_undegraded":"2024-04-06T03:23:58.478081+0000","last_fullsized":"2024-04-06T03:23:58.478081+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:39.032620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.032620+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:39.700020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.472144+0000","last_change":"2024-04-06T03:23:46.374564+0000","last_active":"2024-04-06T03:23:58.472144+0000","last_peered":"2024-04-06T03:23:58.472144+0000","last_clean":"2024-04-06T03:23:58.472144+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:23:58.472144+0000","last_undegraded":"2024-04-06T03:23:58.472144+0000","last_fullsized":"2024-04-06T03:23:58.472144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.374356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.995508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":152,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.373699+0000","last_change":"2024-04-06T03:24:15.781660+0000","last_active":"2024-04-06T03:24:16.373699+0000","last_peered":"2024-04-06T03:24:16.373699+0000","last_clean":"2024-04-06T03:24:16.373699+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:24:16.373699+0000","last_undegraded":"2024-04-06T03:24:16.373699+0000","last_fullsized":"2024-04-06T03:24:16.373699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:24:15.781269+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:24:15.781269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.073494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837001+0000","last_change":"2024-04-06T03:23:38.882403+0000","last_active":"2024-04-06T03:23:58.837001+0000","last_peered":"2024-04-06T03:23:58.837001+0000","last_clean":"2024-04-06T03:23:58.837001+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:23:58.837001+0000","last_undegraded":"2024-04-06T03:23:58.837001+0000","last_fullsized":"2024-04-06T03:23:58.837001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.882269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.592970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.058557+0000","last_change":"2024-04-06T03:23:11.690418+0000","last_active":"2024-04-06T03:23:59.058557+0000","last_peered":"2024-04-06T03:23:59.058557+0000","last_clean":"2024-04-06T03:23:59.058557+0000","last_became_active":"2024-04-06T03:20:55.762055+0000","last_became_peered":"2024-04-06T03:20:55.762055+0000","last_unstale":"2024-04-06T03:23:59.058557+0000","last_undegraded":"2024-04-06T03:23:59.058557+0000","last_fullsized":"2024-04-06T03:23:59.058557+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:11.690170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:11.690170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:35.579295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.118518+0000","last_change":"2024-04-06T03:22:44.643404+0000","last_active":"2024-04-06T03:23:59.118518+0000","last_peered":"2024-04-06T03:23:59.118518+0000","last_clean":"2024-04-06T03:23:59.118518+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:23:59.118518+0000","last_undegraded":"2024-04-06T03:23:59.118518+0000","last_fullsized":"2024-04-06T03:23:59.118518+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:44.643181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:44.643181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:10.998024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":149,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.471804+0000","last_change":"2024-04-06T03:23:56.442918+0000","last_active":"2024-04-06T03:23:58.471804+0000","last_peered":"2024-04-06T03:23:58.471804+0000","last_clean":"2024-04-06T03:23:58.471804+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:23:58.471804+0000","last_undegraded":"2024-04-06T03:23:58.471804+0000","last_fullsized":"2024-04-06T03:23:58.471804+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-06T03:23:56.442661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.442661+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.735553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057597+0000","last_change":"2024-04-06T03:23:38.883987+0000","last_active":"2024-04-06T03:23:59.057597+0000","last_peered":"2024-04-06T03:23:59.057597+0000","last_clean":"2024-04-06T03:23:59.057597+0000","last_became_active":"2024-04-06T03:20:57.845535+0000","last_became_peered":"2024-04-06T03:20:57.845535+0000","last_unstale":"2024-04-06T03:23:59.057597+0000","last_undegraded":"2024-04-06T03:23:59.057597+0000","last_fullsized":"2024-04-06T03:23:59.057597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.883738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.883738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.269641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"85'11","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.370108+0000","last_change":"2024-04-06T03:23:23.295365+0000","last_active":"2024-04-06T03:24:16.370108+0000","last_peered":"2024-04-06T03:24:16.370108+0000","last_clean":"2024-04-06T03:24:16.370108+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:24:16.370108+0000","last_undegraded":"2024-04-06T03:24:16.370108+0000","last_fullsized":"2024-04-06T03:24:16.370108+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:23.295124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.295124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.215136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.665210+0000","last_change":"2024-04-06T03:23:59.665265+0000","last_active":"2024-04-06T03:23:59.665210+0000","last_peered":"2024-04-06T03:23:59.665210+0000","last_clean":"2024-04-06T03:23:59.665210+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:23:59.665210+0000","last_undegraded":"2024-04-06T03:23:59.665210+0000","last_fullsized":"2024-04-06T03:23:59.665210+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:59.665053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:59.665053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:04.127092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"85'27","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.836070+0000","last_change":"2024-04-06T03:23:57.838208+0000","last_active":"2024-04-06T03:23:58.836070+0000","last_peered":"2024-04-06T03:23:58.836070+0000","last_clean":"2024-04-06T03:23:58.836070+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:23:58.836070+0000","last_undegraded":"2024-04-06T03:23:58.836070+0000","last_fullsized":"2024-04-06T03:23:58.836070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'27","last_scrub_stamp":"2024-04-06T03:23:57.837997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:57.837997+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.275078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473984+0000","last_change":"2024-04-06T03:23:36.391113+0000","last_active":"2024-04-06T03:23:58.473984+0000","last_peered":"2024-04-06T03:23:58.473984+0000","last_clean":"2024-04-06T03:23:58.473984+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:23:58.473984+0000","last_undegraded":"2024-04-06T03:23:58.473984+0000","last_fullsized":"2024-04-06T03:23:58.473984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.390806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.390806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:00.809816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":152,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.373548+0000","last_change":"2024-04-06T03:23:29.528196+0000","last_active":"2024-04-06T03:24:16.373548+0000","last_peered":"2024-04-06T03:24:16.373548+0000","last_clean":"2024-04-06T03:24:16.373548+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:24:16.373548+0000","last_undegraded":"2024-04-06T03:24:16.373548+0000","last_fullsized":"2024-04-06T03:24:16.373548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.528005+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:29.753304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.122092+0000","last_change":"2024-04-06T03:23:07.798328+0000","last_active":"2024-04-06T03:23:59.122092+0000","last_peered":"2024-04-06T03:23:59.122092+0000","last_clean":"2024-04-06T03:23:59.122092+0000","last_became_active":"2024-04-06T03:20:51.167121+0000","last_became_peered":"2024-04-06T03:20:51.167121+0000","last_unstale":"2024-04-06T03:23:59.122092+0000","last_undegraded":"2024-04-06T03:23:59.122092+0000","last_fullsized":"2024-04-06T03:23:59.122092+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:07.798091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:07.798091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:28.780888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"85'15","reported_seq":130,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.837140+0000","last_change":"2024-04-06T03:23:24.821416+0000","last_active":"2024-04-06T03:23:58.837140+0000","last_peered":"2024-04-06T03:23:58.837140+0000","last_clean":"2024-04-06T03:23:58.837140+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:23:58.837140+0000","last_undegraded":"2024-04-06T03:23:58.837140+0000","last_fullsized":"2024-04-06T03:23:58.837140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:24.821164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:24.821164+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.815641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.479797+0000","last_change":"2024-04-06T03:23:55.616868+0000","last_active":"2024-04-06T03:23:58.479797+0000","last_peered":"2024-04-06T03:23:58.479797+0000","last_clean":"2024-04-06T03:23:58.479797+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:23:58.479797+0000","last_undegraded":"2024-04-06T03:23:58.479797+0000","last_fullsized":"2024-04-06T03:23:58.479797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.616541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:04.260063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":97,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:08.719053+0000","last_change":"2024-04-06T03:24:08.719094+0000","last_active":"2024-04-06T03:24:08.719053+0000","last_peered":"2024-04-06T03:24:08.719053+0000","last_clean":"2024-04-06T03:24:08.719053+0000","last_became_active":"2024-04-06T03:20:55.758951+0000","last_became_peered":"2024-04-06T03:20:55.758951+0000","last_unstale":"2024-04-06T03:24:08.719053+0000","last_undegraded":"2024-04-06T03:24:08.719053+0000","last_fullsized":"2024-04-06T03:24:08.719053+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:08.718957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:08.718957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:38.145618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.119114+0000","last_change":"2024-04-06T03:23:15.710252+0000","last_active":"2024-04-06T03:23:59.119114+0000","last_peered":"2024-04-06T03:23:59.119114+0000","last_clean":"2024-04-06T03:23:59.119114+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:23:59.119114+0000","last_undegraded":"2024-04-06T03:23:59.119114+0000","last_fullsized":"2024-04-06T03:23:59.119114+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:15.710038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:15.710038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.258607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'25","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.769247+0000","last_change":"2024-04-06T03:23:22.871567+0000","last_active":"2024-04-06T03:23:58.769247+0000","last_peered":"2024-04-06T03:23:58.769247+0000","last_clean":"2024-04-06T03:23:58.769247+0000","last_became_active":"2024-04-06T03:20:53.350784+0000","last_became_peered":"2024-04-06T03:20:53.350784+0000","last_unstale":"2024-04-06T03:23:58.769247+0000","last_undegraded":"2024-04-06T03:23:58.769247+0000","last_fullsized":"2024-04-06T03:23:58.769247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-06T03:23:22.871158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:22.871158+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:30.808433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.118046+0000","last_change":"2024-04-06T03:22:55.509249+0000","last_active":"2024-04-06T03:23:59.118046+0000","last_peered":"2024-04-06T03:23:59.118046+0000","last_clean":"2024-04-06T03:23:59.118046+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:23:59.118046+0000","last_undegraded":"2024-04-06T03:23:59.118046+0000","last_fullsized":"2024-04-06T03:23:59.118046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:55.508950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:55.508950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:15.654754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.473124+0000","last_change":"2024-04-06T03:23:43.566143+0000","last_active":"2024-04-06T03:23:58.473124+0000","last_peered":"2024-04-06T03:23:58.473124+0000","last_clean":"2024-04-06T03:23:58.473124+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:23:58.473124+0000","last_undegraded":"2024-04-06T03:23:58.473124+0000","last_fullsized":"2024-04-06T03:23:58.473124+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.565930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.924455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474970+0000","last_change":"2024-04-06T03:23:56.575312+0000","last_active":"2024-04-06T03:23:58.474970+0000","last_peered":"2024-04-06T03:23:58.474970+0000","last_clean":"2024-04-06T03:23:58.474970+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:23:58.474970+0000","last_undegraded":"2024-04-06T03:23:58.474970+0000","last_fullsized":"2024-04-06T03:23:58.474970+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.575091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.850617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.120026+0000","last_change":"2024-04-06T03:22:57.488426+0000","last_active":"2024-04-06T03:23:59.120026+0000","last_peered":"2024-04-06T03:23:59.120026+0000","last_clean":"2024-04-06T03:23:59.120026+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:23:59.120026+0000","last_undegraded":"2024-04-06T03:23:59.120026+0000","last_fullsized":"2024-04-06T03:23:59.120026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:22:57.488176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:57.488176+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:22.958962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474919+0000","last_change":"2024-04-06T03:23:39.990151+0000","last_active":"2024-04-06T03:23:58.474919+0000","last_peered":"2024-04-06T03:23:58.474919+0000","last_clean":"2024-04-06T03:23:58.474919+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:23:58.474919+0000","last_undegraded":"2024-04-06T03:23:58.474919+0000","last_fullsized":"2024-04-06T03:23:58.474919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.990000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.990000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.631759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:58.474039+0000","last_change":"2024-04-06T03:23:44.525846+0000","last_active":"2024-04-06T03:23:58.474039+0000","last_peered":"2024-04-06T03:23:58.474039+0000","last_clean":"2024-04-06T03:23:58.474039+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:23:58.474039+0000","last_undegraded":"2024-04-06T03:23:58.474039+0000","last_fullsized":"2024-04-06T03:23:58.474039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.525616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.599062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.123318+0000","last_change":"2024-04-06T03:23:16.814953+0000","last_active":"2024-04-06T03:23:59.123318+0000","last_peered":"2024-04-06T03:23:59.123318+0000","last_clean":"2024-04-06T03:23:59.123318+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:23:59.123318+0000","last_undegraded":"2024-04-06T03:23:59.123318+0000","last_fullsized":"2024-04-06T03:23:59.123318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:16.814738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.771476+0000","last_clean_scrub_stamp":"2024-04-06T03:23:16.814738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:27.108754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"85'10","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.057566+0000","last_change":"2024-04-06T03:23:48.924901+0000","last_active":"2024-04-06T03:23:59.057566+0000","last_peered":"2024-04-06T03:23:59.057566+0000","last_clean":"2024-04-06T03:23:59.057566+0000","last_became_active":"2024-04-06T03:20:53.345059+0000","last_became_peered":"2024-04-06T03:20:53.345059+0000","last_unstale":"2024-04-06T03:23:59.057566+0000","last_undegraded":"2024-04-06T03:23:59.057566+0000","last_fullsized":"2024-04-06T03:23:59.057566+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'10","last_scrub_stamp":"2024-04-06T03:23:48.924677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.924677+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.542114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":93,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:00.078350+0000","last_change":"2024-04-06T03:24:00.078410+0000","last_active":"2024-04-06T03:24:00.078350+0000","last_peered":"2024-04-06T03:24:00.078350+0000","last_clean":"2024-04-06T03:24:00.078350+0000","last_became_active":"2024-04-06T03:20:58.128530+0000","last_became_peered":"2024-04-06T03:20:58.128530+0000","last_unstale":"2024-04-06T03:24:00.078350+0000","last_undegraded":"2024-04-06T03:24:00.078350+0000","last_fullsized":"2024-04-06T03:24:00.078350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:00.078196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.078196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.255067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:22.577461+0000","last_change":"2024-04-06T03:24:22.577522+0000","last_active":"2024-04-06T03:24:22.577461+0000","last_peered":"2024-04-06T03:24:22.577461+0000","last_clean":"2024-04-06T03:24:22.577461+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:24:22.577461+0000","last_undegraded":"2024-04-06T03:24:22.577461+0000","last_fullsized":"2024-04-06T03:24:22.577461+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:22.577283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:24:22.577283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:42.933283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":125,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:23:59.120295+0000","last_change":"2024-04-06T03:23:41.707141+0000","last_active":"2024-04-06T03:23:59.120295+0000","last_peered":"2024-04-06T03:23:59.120295+0000","last_clean":"2024-04-06T03:23:59.120295+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:23:59.120295+0000","last_undegraded":"2024-04-06T03:23:59.120295+0000","last_fullsized":"2024-04-06T03:23:59.120295+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:41.706589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:41.706589+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.161763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"85'11","reported_seq":149,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-06T03:24:16.370103+0000","last_change":"2024-04-06T03:23:33.820933+0000","last_active":"2024-04-06T03:24:16.370103+0000","last_peered":"2024-04-06T03:24:16.370103+0000","last_clean":"2024-04-06T03:24:16.370103+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:24:16.370103+0000","last_undegraded":"2024-04-06T03:24:16.370103+0000","last_fullsized":"2024-04-06T03:24:16.370103+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:33.820691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.820691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.287595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":186,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":642,"ondisk_log_size":642,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743477,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32488,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712664,"statfs":{"total":95995035648,"available":95961767936,"internally_reserved":0,"allocated":1343488,"data_stored":971083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135935,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32452,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712700,"statfs":{"total":95995035648,"available":95961804800,"internally_reserved":0,"allocated":1306624,"data_stored":969363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31888,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713264,"statfs":{"total":95995035648,"available":95962382336,"internally_reserved":0,"allocated":729088,"data_stored":378849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36632,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708520,"statfs":{"total":95995035648,"available":95957524480,"internally_reserved":0,"allocated":1327104,"data_stored":970416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378730,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27716,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717436,"statfs":{"total":95995035648,"available":95966654464,"internally_reserved":0,"allocated":716800,"data_stored":378593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084382,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27708,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":708608,"data_stored":378507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31912,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713240,"statfs":{"total":95995035648,"available":95962357760,"internally_reserved":0,"allocated":753664,"data_stored":379078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495691,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":712704,"data_stored":378489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:24:27.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:24:27.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:26 smithi153 bash[23852]: cluster 2024-04-06T03:24:25.898053+0000 osd.1 (osd.1) 87 : cluster 0 3.19 scrub starts 2024-04-06T03:24:27.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:26 smithi153 bash[23852]: cluster 2024-04-06T03:24:25.901011+0000 osd.1 (osd.1) 88 : cluster 0 3.19 scrub ok 2024-04-06T03:24:27.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:26 smithi153 bash[23852]: cluster 2024-04-06T03:24:26.014041+0000 mgr.y (mgr.24368) 132 : cluster 0 pgmap v101: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:27.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:26 smithi153 bash[23852]: cluster 2024-04-06T03:24:26.200109+0000 osd.6 (osd.6) 53 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:27.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:26 smithi153 bash[23852]: cluster 2024-04-06T03:24:26.200203+0000 osd.6 (osd.6) 54 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:27.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:26 smithi153 bash[23852]: audit 2024-04-06T03:24:26.217938+0000 mgr.y (mgr.24368) 133 : audit 0 from='client.24557 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:27.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:27 smithi050 bash[21467]: cluster 2024-04-06T03:24:26.616525+0000 mon.a (mon.0) 971 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-04-06T03:24:27.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:27 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.055907+0000 osd.5 (osd.5) 89 : cluster 0 2.3 scrub starts 2024-04-06T03:24:27.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:27 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.060077+0000 osd.5 (osd.5) 90 : cluster 0 2.3 scrub ok 2024-04-06T03:24:27.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:27 smithi050 bash[29221]: cluster 2024-04-06T03:24:26.616525+0000 mon.a (mon.0) 971 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-04-06T03:24:27.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:27 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.055907+0000 osd.5 (osd.5) 89 : cluster 0 2.3 scrub starts 2024-04-06T03:24:27.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:27 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.060077+0000 osd.5 (osd.5) 90 : cluster 0 2.3 scrub ok 2024-04-06T03:24:28.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:27 smithi153 bash[23852]: cluster 2024-04-06T03:24:26.616525+0000 mon.a (mon.0) 971 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2024-04-06T03:24:28.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:27 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.055907+0000 osd.5 (osd.5) 89 : cluster 0 2.3 scrub starts 2024-04-06T03:24:28.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:27 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.060077+0000 osd.5 (osd.5) 90 : cluster 0 2.3 scrub ok 2024-04-06T03:24:28.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.852071+0000 osd.1 (osd.1) 89 : cluster 0 2.d scrub starts 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.854430+0000 osd.1 (osd.1) 90 : cluster 0 2.d scrub ok 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.907375+0000 osd.7 (osd.7) 95 : cluster 0 5.10 scrub starts 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.909553+0000 osd.7 (osd.7) 96 : cluster 0 5.10 scrub ok 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[21467]: cluster 2024-04-06T03:24:28.014781+0000 mgr.y (mgr.24368) 134 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.852071+0000 osd.1 (osd.1) 89 : cluster 0 2.d scrub starts 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.854430+0000 osd.1 (osd.1) 90 : cluster 0 2.d scrub ok 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.907375+0000 osd.7 (osd.7) 95 : cluster 0 5.10 scrub starts 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.909553+0000 osd.7 (osd.7) 96 : cluster 0 5.10 scrub ok 2024-04-06T03:24:28.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:28 smithi050 bash[29221]: cluster 2024-04-06T03:24:28.014781+0000 mgr.y (mgr.24368) 134 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:29.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:28 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.852071+0000 osd.1 (osd.1) 89 : cluster 0 2.d scrub starts 2024-04-06T03:24:29.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:28 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.854430+0000 osd.1 (osd.1) 90 : cluster 0 2.d scrub ok 2024-04-06T03:24:29.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:28 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.907375+0000 osd.7 (osd.7) 95 : cluster 0 5.10 scrub starts 2024-04-06T03:24:29.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:28 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.909553+0000 osd.7 (osd.7) 96 : cluster 0 5.10 scrub ok 2024-04-06T03:24:29.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:28 smithi153 bash[23852]: cluster 2024-04-06T03:24:28.014781+0000 mgr.y (mgr.24368) 134 : cluster 0 pgmap v103: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:29.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:24:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:24:29.611 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:24:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:24:29.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.770668+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-04-06T03:24:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[21467]: cluster 2024-04-06T03:24:27.773744+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-04-06T03:24:29.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[21467]: audit 2024-04-06T03:24:29.101445+0000 mgr.y (mgr.24368) 135 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:29.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.770668+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-04-06T03:24:29.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[29221]: cluster 2024-04-06T03:24:27.773744+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-04-06T03:24:29.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:29 smithi050 bash[29221]: audit 2024-04-06T03:24:29.101445+0000 mgr.y (mgr.24368) 135 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:30.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:29 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.770668+0000 osd.4 (osd.4) 73 : cluster 0 2.f scrub starts 2024-04-06T03:24:30.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:29 smithi153 bash[23852]: cluster 2024-04-06T03:24:27.773744+0000 osd.4 (osd.4) 74 : cluster 0 2.f scrub ok 2024-04-06T03:24:30.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:29 smithi153 bash[23852]: audit 2024-04-06T03:24:29.101445+0000 mgr.y (mgr.24368) 135 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[21467]: cluster 2024-04-06T03:24:29.963280+0000 osd.3 (osd.3) 99 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:24:30.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[21467]: cluster 2024-04-06T03:24:29.974886+0000 osd.3 (osd.3) 100 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.016279+0000 mgr.y (mgr.24368) 136 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.460581+0000 osd.2 (osd.2) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.460693+0000 osd.2 (osd.2) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[29221]: cluster 2024-04-06T03:24:29.963280+0000 osd.3 (osd.3) 99 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[29221]: cluster 2024-04-06T03:24:29.974886+0000 osd.3 (osd.3) 100 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.016279+0000 mgr.y (mgr.24368) 136 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.460581+0000 osd.2 (osd.2) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:30 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.460693+0000 osd.2 (osd.2) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:31.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:30 smithi153 bash[23852]: cluster 2024-04-06T03:24:29.963280+0000 osd.3 (osd.3) 99 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:24:31.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:30 smithi153 bash[23852]: cluster 2024-04-06T03:24:29.974886+0000 osd.3 (osd.3) 100 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:24:31.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:30 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.016279+0000 mgr.y (mgr.24368) 136 : cluster 0 pgmap v104: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:31.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:30 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.460581+0000 osd.2 (osd.2) 63 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:31.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:30 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.460693+0000 osd.2 (osd.2) 64 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:31.839 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.582376+0000 osd.2 (osd.2) 65 : cluster 0 5.d scrub starts 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.585511+0000 osd.2 (osd.2) 66 : cluster 0 5.d scrub ok 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.637169+0000 mon.a (mon.0) 972 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.963724+0000 osd.6 (osd.6) 55 : cluster 0 3.a scrub starts 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[29221]: cluster 2024-04-06T03:24:30.969528+0000 osd.6 (osd.6) 56 : cluster 0 3.a scrub ok 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.582376+0000 osd.2 (osd.2) 65 : cluster 0 5.d scrub starts 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.585511+0000 osd.2 (osd.2) 66 : cluster 0 5.d scrub ok 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.637169+0000 mon.a (mon.0) 972 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-04-06T03:24:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.963724+0000 osd.6 (osd.6) 55 : cluster 0 3.a scrub starts 2024-04-06T03:24:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:31 smithi050 bash[21467]: cluster 2024-04-06T03:24:30.969528+0000 osd.6 (osd.6) 56 : cluster 0 3.a scrub ok 2024-04-06T03:24:32.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:31 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.582376+0000 osd.2 (osd.2) 65 : cluster 0 5.d scrub starts 2024-04-06T03:24:32.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:31 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.585511+0000 osd.2 (osd.2) 66 : cluster 0 5.d scrub ok 2024-04-06T03:24:32.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:31 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.637169+0000 mon.a (mon.0) 972 : cluster 0 osdmap e119: 8 total, 8 up, 8 in 2024-04-06T03:24:32.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:31 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.963724+0000 osd.6 (osd.6) 55 : cluster 0 3.a scrub starts 2024-04-06T03:24:32.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:31 smithi153 bash[23852]: cluster 2024-04-06T03:24:30.969528+0000 osd.6 (osd.6) 56 : cluster 0 3.a scrub ok 2024-04-06T03:24:32.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:32 smithi050 bash[21467]: cluster 2024-04-06T03:24:32.016965+0000 mgr.y (mgr.24368) 137 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:32.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:32 smithi050 bash[29221]: cluster 2024-04-06T03:24:32.016965+0000 mgr.y (mgr.24368) 137 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:33.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:32 smithi153 bash[23852]: cluster 2024-04-06T03:24:32.016965+0000 mgr.y (mgr.24368) 137 : cluster 0 pgmap v106: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: cluster 2024-04-06T03:24:32.965342+0000 osd.3 (osd.3) 101 : cluster 0 4.0 scrub starts 2024-04-06T03:24:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: cluster 2024-04-06T03:24:32.967381+0000 osd.3 (osd.3) 102 : cluster 0 4.0 scrub ok 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.005206+0000 osd.0 (osd.0) 83 : cluster 0 6.16 scrub starts 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.006640+0000 osd.0 (osd.0) 84 : cluster 0 6.16 scrub ok 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.050629+0000 osd.5 (osd.5) 91 : cluster 0 2.2 scrub starts 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.053040+0000 osd.5 (osd.5) 92 : cluster 0 2.2 scrub ok 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[21467]: audit 2024-04-06T03:24:33.356312+0000 mon.a (mon.0) 973 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: cluster 2024-04-06T03:24:32.965342+0000 osd.3 (osd.3) 101 : cluster 0 4.0 scrub starts 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: cluster 2024-04-06T03:24:32.967381+0000 osd.3 (osd.3) 102 : cluster 0 4.0 scrub ok 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.005206+0000 osd.0 (osd.0) 83 : cluster 0 6.16 scrub starts 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.006640+0000 osd.0 (osd.0) 84 : cluster 0 6.16 scrub ok 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.050629+0000 osd.5 (osd.5) 91 : cluster 0 2.2 scrub starts 2024-04-06T03:24:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.053040+0000 osd.5 (osd.5) 92 : cluster 0 2.2 scrub ok 2024-04-06T03:24:33.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:33 smithi050 bash[29221]: audit 2024-04-06T03:24:33.356312+0000 mon.a (mon.0) 973 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: cluster 2024-04-06T03:24:32.965342+0000 osd.3 (osd.3) 101 : cluster 0 4.0 scrub starts 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: cluster 2024-04-06T03:24:32.967381+0000 osd.3 (osd.3) 102 : cluster 0 4.0 scrub ok 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.005206+0000 osd.0 (osd.0) 83 : cluster 0 6.16 scrub starts 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.006640+0000 osd.0 (osd.0) 84 : cluster 0 6.16 scrub ok 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.050629+0000 osd.5 (osd.5) 91 : cluster 0 2.2 scrub starts 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.053040+0000 osd.5 (osd.5) 92 : cluster 0 2.2 scrub ok 2024-04-06T03:24:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:33 smithi153 bash[23852]: audit 2024-04-06T03:24:33.356312+0000 mon.a (mon.0) 973 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.921575+0000 osd.6 (osd.6) 57 : cluster 0 3.10 scrub starts 2024-04-06T03:24:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.925203+0000 osd.6 (osd.6) 58 : cluster 0 3.10 scrub ok 2024-04-06T03:24:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.953586+0000 osd.3 (osd.3) 103 : cluster 0 6.1f scrub starts 2024-04-06T03:24:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[21467]: cluster 2024-04-06T03:24:33.955920+0000 osd.3 (osd.3) 104 : cluster 0 6.1f scrub ok 2024-04-06T03:24:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[21467]: cluster 2024-04-06T03:24:34.018353+0000 mgr.y (mgr.24368) 138 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:34.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.921575+0000 osd.6 (osd.6) 57 : cluster 0 3.10 scrub starts 2024-04-06T03:24:34.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.925203+0000 osd.6 (osd.6) 58 : cluster 0 3.10 scrub ok 2024-04-06T03:24:34.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.953586+0000 osd.3 (osd.3) 103 : cluster 0 6.1f scrub starts 2024-04-06T03:24:34.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[29221]: cluster 2024-04-06T03:24:33.955920+0000 osd.3 (osd.3) 104 : cluster 0 6.1f scrub ok 2024-04-06T03:24:34.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:34 smithi050 bash[29221]: cluster 2024-04-06T03:24:34.018353+0000 mgr.y (mgr.24368) 138 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:35.031 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:24:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:24:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:34 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.921575+0000 osd.6 (osd.6) 57 : cluster 0 3.10 scrub starts 2024-04-06T03:24:35.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:34 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.925203+0000 osd.6 (osd.6) 58 : cluster 0 3.10 scrub ok 2024-04-06T03:24:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:34 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.953586+0000 osd.3 (osd.3) 103 : cluster 0 6.1f scrub starts 2024-04-06T03:24:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:34 smithi153 bash[23852]: cluster 2024-04-06T03:24:33.955920+0000 osd.3 (osd.3) 104 : cluster 0 6.1f scrub ok 2024-04-06T03:24:35.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:34 smithi153 bash[23852]: cluster 2024-04-06T03:24:34.018353+0000 mgr.y (mgr.24368) 138 : cluster 0 pgmap v107: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:35.822 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-04-06T03:24:34.017259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1439,"num_read_kb":1253,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":887,"ondisk_log_size":887,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":248728,"kb_used_data":7612,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":749712488,"statfs":{"total":767960285184,"available":767705587712,"internally_reserved":0,"allocated":7794688,"data_stored":4927578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003428"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993740+0000","last_change":"2024-04-06T03:23:55.828897+0000","last_active":"2024-04-06T03:24:30.993740+0000","last_peered":"2024-04-06T03:24:30.993740+0000","last_clean":"2024-04-06T03:24:30.993740+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:24:30.993740+0000","last_undegraded":"2024-04-06T03:24:30.993740+0000","last_fullsized":"2024-04-06T03:24:30.993740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.828643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.828643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.076125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632321+0000","last_change":"2024-04-06T03:24:03.509495+0000","last_active":"2024-04-06T03:24:30.632321+0000","last_peered":"2024-04-06T03:24:30.632321+0000","last_clean":"2024-04-06T03:24:30.632321+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:24:30.632321+0000","last_undegraded":"2024-04-06T03:24:30.632321+0000","last_fullsized":"2024-04-06T03:24:30.632321+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:03.509185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:03.509185+0000","last_clean_scrub_stamp":"2024-04-06T03:24:03.509185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:32.625692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":114,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.639665+0000","last_change":"2024-04-06T03:23:59.742745+0000","last_active":"2024-04-06T03:24:30.639665+0000","last_peered":"2024-04-06T03:24:30.639665+0000","last_clean":"2024-04-06T03:24:30.639665+0000","last_became_active":"2024-04-06T03:21:35.160225+0000","last_became_peered":"2024-04-06T03:21:35.160225+0000","last_unstale":"2024-04-06T03:24:30.639665+0000","last_undegraded":"2024-04-06T03:24:30.639665+0000","last_fullsized":"2024-04-06T03:24:30.639665+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:59.742638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:59.742638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.865855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"118'22","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994593+0000","last_change":"2024-04-06T03:24:29.975164+0000","last_active":"2024-04-06T03:24:30.994593+0000","last_peered":"2024-04-06T03:24:30.994593+0000","last_clean":"2024-04-06T03:24:30.994593+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:24:30.994593+0000","last_undegraded":"2024-04-06T03:24:30.994593+0000","last_fullsized":"2024-04-06T03:24:30.994593+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:24:29.974899+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:39.774252+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632665+0000","last_change":"2024-04-06T03:24:05.556108+0000","last_active":"2024-04-06T03:24:30.632665+0000","last_peered":"2024-04-06T03:24:30.632665+0000","last_clean":"2024-04-06T03:24:30.632665+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:24:30.632665+0000","last_undegraded":"2024-04-06T03:24:30.632665+0000","last_fullsized":"2024-04-06T03:24:30.632665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:05.555884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:05.555884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.380839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.858685+0000","last_change":"2024-04-06T03:24:14.772163+0000","last_active":"2024-04-06T03:24:30.858685+0000","last_peered":"2024-04-06T03:24:30.858685+0000","last_clean":"2024-04-06T03:24:30.858685+0000","last_became_active":"2024-04-06T03:20:55.772175+0000","last_became_peered":"2024-04-06T03:20:55.772175+0000","last_unstale":"2024-04-06T03:24:30.858685+0000","last_undegraded":"2024-04-06T03:24:30.858685+0000","last_fullsized":"2024-04-06T03:24:30.858685+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.771844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.771844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.161465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":110,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994913+0000","last_change":"2024-04-06T03:23:17.696688+0000","last_active":"2024-04-06T03:24:30.994913+0000","last_peered":"2024-04-06T03:24:30.994913+0000","last_clean":"2024-04-06T03:24:30.994913+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:24:30.994913+0000","last_undegraded":"2024-04-06T03:24:30.994913+0000","last_fullsized":"2024-04-06T03:24:30.994913+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.696473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.696473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.518460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'17","reported_seq":135,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.639383+0000","last_change":"2024-04-06T03:23:52.836932+0000","last_active":"2024-04-06T03:24:30.639383+0000","last_peered":"2024-04-06T03:24:30.639383+0000","last_clean":"2024-04-06T03:24:30.639383+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:24:30.639383+0000","last_undegraded":"2024-04-06T03:24:30.639383+0000","last_fullsized":"2024-04-06T03:24:30.639383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:52.836543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.836543+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:09.578286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.054019+0000","last_change":"2024-04-06T03:24:10.108584+0000","last_active":"2024-04-06T03:24:31.054019+0000","last_peered":"2024-04-06T03:24:31.054019+0000","last_clean":"2024-04-06T03:24:31.054019+0000","last_became_active":"2024-04-06T03:20:57.825746+0000","last_became_peered":"2024-04-06T03:20:57.825746+0000","last_unstale":"2024-04-06T03:24:31.054019+0000","last_undegraded":"2024-04-06T03:24:31.054019+0000","last_fullsized":"2024-04-06T03:24:31.054019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:10.108359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:10.108359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.675483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073220+0000","last_change":"2024-04-06T03:24:17.849044+0000","last_active":"2024-04-06T03:24:31.073220+0000","last_peered":"2024-04-06T03:24:31.073220+0000","last_clean":"2024-04-06T03:24:31.073220+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:24:31.073220+0000","last_undegraded":"2024-04-06T03:24:31.073220+0000","last_fullsized":"2024-04-06T03:24:31.073220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:17.848834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:24:17.848834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.180144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074434+0000","last_change":"2024-04-06T03:23:21.801571+0000","last_active":"2024-04-06T03:24:31.074434+0000","last_peered":"2024-04-06T03:24:31.074434+0000","last_clean":"2024-04-06T03:24:31.074434+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:24:31.074434+0000","last_undegraded":"2024-04-06T03:24:31.074434+0000","last_fullsized":"2024-04-06T03:24:31.074434+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:21.801341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.801341+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:45.356673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"112'22","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055437+0000","last_change":"2024-04-06T03:23:36.905749+0000","last_active":"2024-04-06T03:24:31.055437+0000","last_peered":"2024-04-06T03:24:31.055437+0000","last_clean":"2024-04-06T03:24:31.055437+0000","last_became_active":"2024-04-06T03:20:53.352525+0000","last_became_peered":"2024-04-06T03:20:53.352525+0000","last_unstale":"2024-04-06T03:24:31.055437+0000","last_undegraded":"2024-04-06T03:24:31.055437+0000","last_fullsized":"2024-04-06T03:24:31.055437+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'22","last_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:23:36.905499+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.905499+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.467751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634391+0000","last_change":"2024-04-06T03:24:04.719301+0000","last_active":"2024-04-06T03:24:30.634391+0000","last_peered":"2024-04-06T03:24:30.634391+0000","last_clean":"2024-04-06T03:24:30.634391+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:24:30.634391+0000","last_undegraded":"2024-04-06T03:24:30.634391+0000","last_fullsized":"2024-04-06T03:24:30.634391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:04.719078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:04.719078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:31.164948+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.054452+0000","last_change":"2024-04-06T03:23:51.980318+0000","last_active":"2024-04-06T03:24:31.054452+0000","last_peered":"2024-04-06T03:24:31.054452+0000","last_clean":"2024-04-06T03:24:31.054452+0000","last_became_active":"2024-04-06T03:20:55.755058+0000","last_became_peered":"2024-04-06T03:20:55.755058+0000","last_unstale":"2024-04-06T03:24:31.054452+0000","last_undegraded":"2024-04-06T03:24:31.054452+0000","last_fullsized":"2024-04-06T03:24:31.054452+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:51.980107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:51.980107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:15.105749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073468+0000","last_change":"2024-04-06T03:24:10.792283+0000","last_active":"2024-04-06T03:24:31.073468+0000","last_peered":"2024-04-06T03:24:31.073468+0000","last_clean":"2024-04-06T03:24:31.073468+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:24:31.073468+0000","last_undegraded":"2024-04-06T03:24:31.073468+0000","last_fullsized":"2024-04-06T03:24:31.073468+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:10.792042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:17.421734+0000","last_clean_scrub_stamp":"2024-04-06T03:24:10.792042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:29.598568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"105'24","reported_seq":146,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055531+0000","last_change":"2024-04-06T03:23:46.865483+0000","last_active":"2024-04-06T03:24:31.055531+0000","last_peered":"2024-04-06T03:24:31.055531+0000","last_clean":"2024-04-06T03:24:31.055531+0000","last_became_active":"2024-04-06T03:20:53.346381+0000","last_became_peered":"2024-04-06T03:20:53.346381+0000","last_unstale":"2024-04-06T03:24:31.055531+0000","last_undegraded":"2024-04-06T03:24:31.055531+0000","last_fullsized":"2024-04-06T03:24:31.055531+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'24","last_scrub_stamp":"2024-04-06T03:23:46.865084+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.865084+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.280139+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":106,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:33.956113+0000","last_change":"2024-04-06T03:24:33.956178+0000","last_active":"2024-04-06T03:24:33.956113+0000","last_peered":"2024-04-06T03:24:33.956113+0000","last_clean":"2024-04-06T03:24:33.956113+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:24:33.956113+0000","last_undegraded":"2024-04-06T03:24:33.956113+0000","last_fullsized":"2024-04-06T03:24:33.956113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:33.955937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:33.955937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:44.590350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633645+0000","last_change":"2024-04-06T03:24:00.468929+0000","last_active":"2024-04-06T03:24:30.633645+0000","last_peered":"2024-04-06T03:24:30.633645+0000","last_clean":"2024-04-06T03:24:30.633645+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:24:30.633645+0000","last_undegraded":"2024-04-06T03:24:30.633645+0000","last_fullsized":"2024-04-06T03:24:30.633645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:00.468662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.468662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:00.548032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":110,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994164+0000","last_change":"2024-04-06T03:24:00.825469+0000","last_active":"2024-04-06T03:24:30.994164+0000","last_peered":"2024-04-06T03:24:30.994164+0000","last_clean":"2024-04-06T03:24:30.994164+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:24:30.994164+0000","last_undegraded":"2024-04-06T03:24:30.994164+0000","last_fullsized":"2024-04-06T03:24:30.994164+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:00.825184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.825184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:18.970342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"111'27","reported_seq":152,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632944+0000","last_change":"2024-04-06T03:23:18.287395+0000","last_active":"2024-04-06T03:24:30.632944+0000","last_peered":"2024-04-06T03:24:30.632944+0000","last_clean":"2024-04-06T03:24:30.632944+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:24:30.632944+0000","last_undegraded":"2024-04-06T03:24:30.632944+0000","last_fullsized":"2024-04-06T03:24:30.632944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'27","last_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.287127+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.693232+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634261+0000","last_change":"2024-04-06T03:23:57.475036+0000","last_active":"2024-04-06T03:24:30.634261+0000","last_peered":"2024-04-06T03:24:30.634261+0000","last_clean":"2024-04-06T03:24:30.634261+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:24:30.634261+0000","last_undegraded":"2024-04-06T03:24:30.634261+0000","last_fullsized":"2024-04-06T03:24:30.634261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:57.474774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:57.474774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.260532+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":93,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.860020+0000","last_change":"2024-04-06T03:23:01.687019+0000","last_active":"2024-04-06T03:24:30.860020+0000","last_peered":"2024-04-06T03:24:30.860020+0000","last_clean":"2024-04-06T03:24:30.860020+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:24:30.860020+0000","last_undegraded":"2024-04-06T03:24:30.860020+0000","last_fullsized":"2024-04-06T03:24:30.860020+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_clean_scrub_stamp":"2024-04-06T03:23:01.686796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:31.412050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.962869+0000","last_change":"2024-04-06T03:23:36.848600+0000","last_active":"2024-04-06T03:24:30.962869+0000","last_peered":"2024-04-06T03:24:30.962869+0000","last_clean":"2024-04-06T03:24:30.962869+0000","last_became_active":"2024-04-06T03:20:51.143283+0000","last_became_peered":"2024-04-06T03:20:51.143283+0000","last_unstale":"2024-04-06T03:24:30.962869+0000","last_undegraded":"2024-04-06T03:24:30.962869+0000","last_fullsized":"2024-04-06T03:24:30.962869+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.848364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.848364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.392566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":126,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.635535+0000","last_change":"2024-04-06T03:23:14.042484+0000","last_active":"2024-04-06T03:24:30.635535+0000","last_peered":"2024-04-06T03:24:30.635535+0000","last_clean":"2024-04-06T03:24:30.635535+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:24:30.635535+0000","last_undegraded":"2024-04-06T03:24:30.635535+0000","last_fullsized":"2024-04-06T03:24:30.635535+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:14.042058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:14.042058+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.768307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.860265+0000","last_change":"2024-04-06T03:24:03.673472+0000","last_active":"2024-04-06T03:24:30.860265+0000","last_peered":"2024-04-06T03:24:30.860265+0000","last_clean":"2024-04-06T03:24:30.860265+0000","last_became_active":"2024-04-06T03:20:57.845246+0000","last_became_peered":"2024-04-06T03:20:57.845246+0000","last_unstale":"2024-04-06T03:24:30.860265+0000","last_undegraded":"2024-04-06T03:24:30.860265+0000","last_fullsized":"2024-04-06T03:24:30.860265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:03.673150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:03.673150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:33.371338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.638150+0000","last_change":"2024-04-06T03:23:34.010043+0000","last_active":"2024-04-06T03:24:30.638150+0000","last_peered":"2024-04-06T03:24:30.638150+0000","last_clean":"2024-04-06T03:24:30.638150+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:24:30.638150+0000","last_undegraded":"2024-04-06T03:24:30.638150+0000","last_fullsized":"2024-04-06T03:24:30.638150+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:23:34.009837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.282584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993502+0000","last_change":"2024-04-06T03:23:43.924012+0000","last_active":"2024-04-06T03:24:30.993502+0000","last_peered":"2024-04-06T03:24:30.993502+0000","last_clean":"2024-04-06T03:24:30.993502+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:24:30.993502+0000","last_undegraded":"2024-04-06T03:24:30.993502+0000","last_fullsized":"2024-04-06T03:24:30.993502+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:43.923791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.923791+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:10.650300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"109'20","reported_seq":139,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993726+0000","last_change":"2024-04-06T03:24:08.775767+0000","last_active":"2024-04-06T03:24:30.993726+0000","last_peered":"2024-04-06T03:24:30.993726+0000","last_clean":"2024-04-06T03:24:30.993726+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:24:30.993726+0000","last_undegraded":"2024-04-06T03:24:30.993726+0000","last_fullsized":"2024-04-06T03:24:30.993726+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'20","last_scrub_stamp":"2024-04-06T03:24:08.775535+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:24:08.775535+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:21.755432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074687+0000","last_change":"2024-04-06T03:24:20.853706+0000","last_active":"2024-04-06T03:24:31.074687+0000","last_peered":"2024-04-06T03:24:31.074687+0000","last_clean":"2024-04-06T03:24:31.074687+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:24:31.074687+0000","last_undegraded":"2024-04-06T03:24:31.074687+0000","last_fullsized":"2024-04-06T03:24:31.074687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-06T03:24:20.853465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:20.853465+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:38.127543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"101'12","reported_seq":162,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.963985+0000","last_change":"2024-04-06T03:23:33.869748+0000","last_active":"2024-04-06T03:24:30.963985+0000","last_peered":"2024-04-06T03:24:30.963985+0000","last_clean":"2024-04-06T03:24:30.963985+0000","last_became_active":"2024-04-06T03:20:55.765174+0000","last_became_peered":"2024-04-06T03:20:55.765174+0000","last_unstale":"2024-04-06T03:24:30.963985+0000","last_undegraded":"2024-04-06T03:24:30.963985+0000","last_fullsized":"2024-04-06T03:24:30.963985+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-06T03:23:33.869531+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.869531+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.720001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":109,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.053890+0000","last_change":"2024-04-06T03:23:19.763448+0000","last_active":"2024-04-06T03:24:31.053890+0000","last_peered":"2024-04-06T03:24:31.053890+0000","last_clean":"2024-04-06T03:24:31.053890+0000","last_became_active":"2024-04-06T03:20:51.144514+0000","last_became_peered":"2024-04-06T03:20:51.144514+0000","last_unstale":"2024-04-06T03:24:31.053890+0000","last_undegraded":"2024-04-06T03:24:31.053890+0000","last_fullsized":"2024-04-06T03:24:31.053890+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.763180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:54.626052+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.763180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:46.950926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"111'50","reported_seq":189,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859006+0000","last_change":"2024-04-06T03:24:25.901138+0000","last_active":"2024-04-06T03:24:30.859006+0000","last_peered":"2024-04-06T03:24:30.859006+0000","last_clean":"2024-04-06T03:24:30.859006+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:24:30.859006+0000","last_undegraded":"2024-04-06T03:24:30.859006+0000","last_fullsized":"2024-04-06T03:24:30.859006+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'50","last_scrub_stamp":"2024-04-06T03:24:25.901018+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_clean_scrub_stamp":"2024-04-06T03:24:25.901018+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:31.937326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994883+0000","last_change":"2024-04-06T03:24:16.789772+0000","last_active":"2024-04-06T03:24:30.994883+0000","last_peered":"2024-04-06T03:24:30.994883+0000","last_clean":"2024-04-06T03:24:30.994883+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:24:30.994883+0000","last_undegraded":"2024-04-06T03:24:30.994883+0000","last_fullsized":"2024-04-06T03:24:30.994883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:16.789545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:24:16.789545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:19.882917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073221+0000","last_change":"2024-04-06T03:24:27.909803+0000","last_active":"2024-04-06T03:24:31.073221+0000","last_peered":"2024-04-06T03:24:31.073221+0000","last_clean":"2024-04-06T03:24:31.073221+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:24:31.073221+0000","last_undegraded":"2024-04-06T03:24:31.073221+0000","last_fullsized":"2024-04-06T03:24:31.073221+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:27.909566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_clean_scrub_stamp":"2024-04-06T03:24:27.909566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:29.227797+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.964110+0000","last_change":"2024-04-06T03:23:26.863931+0000","last_active":"2024-04-06T03:24:30.964110+0000","last_peered":"2024-04-06T03:24:30.964110+0000","last_clean":"2024-04-06T03:24:30.964110+0000","last_became_active":"2024-04-06T03:20:51.142652+0000","last_became_peered":"2024-04-06T03:20:51.142652+0000","last_unstale":"2024-04-06T03:24:30.964110+0000","last_undegraded":"2024-04-06T03:24:30.964110+0000","last_fullsized":"2024-04-06T03:24:30.964110+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:23:26.863691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:41.711589+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":145,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055218+0000","last_change":"2024-04-06T03:23:47.908697+0000","last_active":"2024-04-06T03:24:31.055218+0000","last_peered":"2024-04-06T03:24:31.055218+0000","last_clean":"2024-04-06T03:24:31.055218+0000","last_became_active":"2024-04-06T03:20:53.350189+0000","last_became_peered":"2024-04-06T03:20:53.350189+0000","last_unstale":"2024-04-06T03:24:31.055218+0000","last_undegraded":"2024-04-06T03:24:31.055218+0000","last_fullsized":"2024-04-06T03:24:31.055218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:47.908288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.908288+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.038863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.072888+0000","last_change":"2024-04-06T03:24:04.681542+0000","last_active":"2024-04-06T03:24:31.072888+0000","last_peered":"2024-04-06T03:24:31.072888+0000","last_clean":"2024-04-06T03:24:31.072888+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:24:31.072888+0000","last_undegraded":"2024-04-06T03:24:31.072888+0000","last_fullsized":"2024-04-06T03:24:31.072888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:04.681314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_clean_scrub_stamp":"2024-04-06T03:24:04.681314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:26.747454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.962439+0000","last_change":"2024-04-06T03:23:25.887364+0000","last_active":"2024-04-06T03:24:30.962439+0000","last_peered":"2024-04-06T03:24:30.962439+0000","last_clean":"2024-04-06T03:24:30.962439+0000","last_became_active":"2024-04-06T03:20:55.769876+0000","last_became_peered":"2024-04-06T03:20:55.769876+0000","last_unstale":"2024-04-06T03:24:30.962439+0000","last_undegraded":"2024-04-06T03:24:30.962439+0000","last_fullsized":"2024-04-06T03:24:30.962439+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:25.887152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:25.887152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.529607+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055521+0000","last_change":"2024-04-06T03:24:21.115390+0000","last_active":"2024-04-06T03:24:31.055521+0000","last_peered":"2024-04-06T03:24:31.055521+0000","last_clean":"2024-04-06T03:24:31.055521+0000","last_became_active":"2024-04-06T03:20:51.151269+0000","last_became_peered":"2024-04-06T03:20:51.151269+0000","last_unstale":"2024-04-06T03:24:31.055521+0000","last_undegraded":"2024-04-06T03:24:31.055521+0000","last_fullsized":"2024-04-06T03:24:31.055521+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:21.115189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:21.115189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:39.137464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"85'12","reported_seq":125,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.639216+0000","last_change":"2024-04-06T03:23:18.056122+0000","last_active":"2024-04-06T03:24:30.639216+0000","last_peered":"2024-04-06T03:24:30.639216+0000","last_clean":"2024-04-06T03:24:30.639216+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:24:30.639216+0000","last_undegraded":"2024-04-06T03:24:30.639216+0000","last_fullsized":"2024-04-06T03:24:30.639216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:18.056024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:18.056024+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:40.251393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993246+0000","last_change":"2024-04-06T03:24:14.730089+0000","last_active":"2024-04-06T03:24:30.993246+0000","last_peered":"2024-04-06T03:24:30.993246+0000","last_clean":"2024-04-06T03:24:30.993246+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:24:30.993246+0000","last_undegraded":"2024-04-06T03:24:30.993246+0000","last_fullsized":"2024-04-06T03:24:30.993246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.729868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.729868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:36.530227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859620+0000","last_change":"2024-04-06T03:24:19.927351+0000","last_active":"2024-04-06T03:24:30.859620+0000","last_peered":"2024-04-06T03:24:30.859620+0000","last_clean":"2024-04-06T03:24:30.859620+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:24:30.859620+0000","last_undegraded":"2024-04-06T03:24:30.859620+0000","last_fullsized":"2024-04-06T03:24:30.859620+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:19.927105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:19.927105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:45.184511+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859705+0000","last_change":"2024-04-06T03:24:24.879718+0000","last_active":"2024-04-06T03:24:30.859705+0000","last_peered":"2024-04-06T03:24:30.859705+0000","last_clean":"2024-04-06T03:24:30.859705+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:24:30.859705+0000","last_undegraded":"2024-04-06T03:24:30.859705+0000","last_fullsized":"2024-04-06T03:24:30.859705+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:24.879589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_clean_scrub_stamp":"2024-04-06T03:24:24.879589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:38.589157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'23","reported_seq":146,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633995+0000","last_change":"2024-04-06T03:23:29.339315+0000","last_active":"2024-04-06T03:24:30.633995+0000","last_peered":"2024-04-06T03:24:30.633995+0000","last_clean":"2024-04-06T03:24:30.633995+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:24:30.633995+0000","last_undegraded":"2024-04-06T03:24:30.633995+0000","last_fullsized":"2024-04-06T03:24:30.633995+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-06T03:23:29.339052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:29.339052+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.010581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633283+0000","last_change":"2024-04-06T03:23:55.780644+0000","last_active":"2024-04-06T03:24:30.633283+0000","last_peered":"2024-04-06T03:24:30.633283+0000","last_clean":"2024-04-06T03:24:30.633283+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:24:30.633283+0000","last_undegraded":"2024-04-06T03:24:30.633283+0000","last_fullsized":"2024-04-06T03:24:30.633283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.780397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.780397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.956035+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994551+0000","last_change":"2024-04-06T03:23:52.841531+0000","last_active":"2024-04-06T03:24:30.994551+0000","last_peered":"2024-04-06T03:24:30.994551+0000","last_clean":"2024-04-06T03:24:30.994551+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:24:30.994551+0000","last_undegraded":"2024-04-06T03:24:30.994551+0000","last_fullsized":"2024-04-06T03:24:30.994551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:52.841291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.841291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.389698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.963200+0000","last_change":"2024-04-06T03:24:18.885421+0000","last_active":"2024-04-06T03:24:30.963200+0000","last_peered":"2024-04-06T03:24:30.963200+0000","last_clean":"2024-04-06T03:24:30.963200+0000","last_became_active":"2024-04-06T03:20:51.137376+0000","last_became_peered":"2024-04-06T03:20:51.137376+0000","last_unstale":"2024-04-06T03:24:30.963200+0000","last_undegraded":"2024-04-06T03:24:30.963200+0000","last_fullsized":"2024-04-06T03:24:30.963200+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:18.885184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:24:18.885184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:46.543194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074797+0000","last_change":"2024-04-06T03:24:15.810022+0000","last_active":"2024-04-06T03:24:31.074797+0000","last_peered":"2024-04-06T03:24:31.074797+0000","last_clean":"2024-04-06T03:24:31.074797+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:24:31.074797+0000","last_undegraded":"2024-04-06T03:24:31.074797+0000","last_fullsized":"2024-04-06T03:24:31.074797+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:24:15.809780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:15.809780+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:37.555195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.635000+0000","last_change":"2024-04-06T03:23:19.298817+0000","last_active":"2024-04-06T03:24:30.635000+0000","last_peered":"2024-04-06T03:24:30.635000+0000","last_clean":"2024-04-06T03:24:30.635000+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:24:30.635000+0000","last_undegraded":"2024-04-06T03:24:30.635000+0000","last_fullsized":"2024-04-06T03:24:30.635000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:19.298498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.298498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.565314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":158,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.995272+0000","last_change":"2024-04-06T03:24:24.916947+0000","last_active":"2024-04-06T03:24:30.995272+0000","last_peered":"2024-04-06T03:24:30.995272+0000","last_clean":"2024-04-06T03:24:30.995272+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:24:30.995272+0000","last_undegraded":"2024-04-06T03:24:30.995272+0000","last_fullsized":"2024-04-06T03:24:30.995272+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:24:24.916835+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:24:24.916835+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:44.986604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.635559+0000","last_change":"2024-04-06T03:23:40.956392+0000","last_active":"2024-04-06T03:24:30.635559+0000","last_peered":"2024-04-06T03:24:30.635559+0000","last_clean":"2024-04-06T03:24:30.635559+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:24:30.635559+0000","last_undegraded":"2024-04-06T03:24:30.635559+0000","last_fullsized":"2024-04-06T03:24:30.635559+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:40.956167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:40.956167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.850753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.635039+0000","last_change":"2024-04-06T03:24:00.754139+0000","last_active":"2024-04-06T03:24:30.635039+0000","last_peered":"2024-04-06T03:24:30.635039+0000","last_clean":"2024-04-06T03:24:30.635039+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:24:30.635039+0000","last_undegraded":"2024-04-06T03:24:30.635039+0000","last_fullsized":"2024-04-06T03:24:30.635039+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-06T03:24:00.753905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.753905+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:12.240141+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":104,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:33.006741+0000","last_change":"2024-04-06T03:24:33.006769+0000","last_active":"2024-04-06T03:24:33.006741+0000","last_peered":"2024-04-06T03:24:33.006741+0000","last_clean":"2024-04-06T03:24:33.006741+0000","last_became_active":"2024-04-06T03:20:57.843310+0000","last_became_peered":"2024-04-06T03:20:57.843310+0000","last_unstale":"2024-04-06T03:24:33.006741+0000","last_undegraded":"2024-04-06T03:24:33.006741+0000","last_fullsized":"2024-04-06T03:24:33.006741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:33.006645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:24:33.006645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:42.000072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"85'11","reported_seq":156,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055944+0000","last_change":"2024-04-06T03:23:45.891079+0000","last_active":"2024-04-06T03:24:31.055944+0000","last_peered":"2024-04-06T03:24:31.055944+0000","last_clean":"2024-04-06T03:24:31.055944+0000","last_became_active":"2024-04-06T03:20:55.770173+0000","last_became_peered":"2024-04-06T03:20:55.770173+0000","last_unstale":"2024-04-06T03:24:31.055944+0000","last_undegraded":"2024-04-06T03:24:31.055944+0000","last_fullsized":"2024-04-06T03:24:31.055944+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:45.890804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:45.890804+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:03.597014+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":109,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.054118+0000","last_change":"2024-04-06T03:23:21.739882+0000","last_active":"2024-04-06T03:24:31.054118+0000","last_peered":"2024-04-06T03:24:31.054118+0000","last_clean":"2024-04-06T03:24:31.054118+0000","last_became_active":"2024-04-06T03:20:51.157201+0000","last_became_peered":"2024-04-06T03:20:51.157201+0000","last_unstale":"2024-04-06T03:24:31.054118+0000","last_undegraded":"2024-04-06T03:24:31.054118+0000","last_fullsized":"2024-04-06T03:24:31.054118+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:21.739671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:21.739671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.621751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074133+0000","last_change":"2024-04-06T03:24:00.790460+0000","last_active":"2024-04-06T03:24:31.074133+0000","last_peered":"2024-04-06T03:24:31.074133+0000","last_clean":"2024-04-06T03:24:31.074133+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:24:31.074133+0000","last_undegraded":"2024-04-06T03:24:31.074133+0000","last_fullsized":"2024-04-06T03:24:31.074133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:24:00.790226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.790226+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:16.640308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073675+0000","last_change":"2024-04-06T03:23:55.806848+0000","last_active":"2024-04-06T03:24:31.073675+0000","last_peered":"2024-04-06T03:24:31.073675+0000","last_clean":"2024-04-06T03:24:31.073675+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:24:31.073675+0000","last_undegraded":"2024-04-06T03:24:31.073675+0000","last_fullsized":"2024-04-06T03:24:31.073675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.806612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.806612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.253217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055294+0000","last_change":"2024-04-06T03:23:31.955588+0000","last_active":"2024-04-06T03:24:31.055294+0000","last_peered":"2024-04-06T03:24:31.055294+0000","last_clean":"2024-04-06T03:24:31.055294+0000","last_became_active":"2024-04-06T03:20:55.756465+0000","last_became_peered":"2024-04-06T03:20:55.756465+0000","last_unstale":"2024-04-06T03:24:31.055294+0000","last_undegraded":"2024-04-06T03:24:31.055294+0000","last_fullsized":"2024-04-06T03:24:31.055294+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:31.955371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:31.955371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:56.516085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.962519+0000","last_change":"2024-04-06T03:23:35.835087+0000","last_active":"2024-04-06T03:24:30.962519+0000","last_peered":"2024-04-06T03:24:30.962519+0000","last_clean":"2024-04-06T03:24:30.962519+0000","last_became_active":"2024-04-06T03:20:51.145847+0000","last_became_peered":"2024-04-06T03:20:51.145847+0000","last_unstale":"2024-04-06T03:24:30.962519+0000","last_undegraded":"2024-04-06T03:24:30.962519+0000","last_fullsized":"2024-04-06T03:24:30.962519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.834842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.834842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.169606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"111'6","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:33.925402+0000","last_change":"2024-04-06T03:24:33.925479+0000","last_active":"2024-04-06T03:24:33.925402+0000","last_peered":"2024-04-06T03:24:33.925402+0000","last_clean":"2024-04-06T03:24:33.925402+0000","last_became_active":"2024-04-06T03:20:53.348930+0000","last_became_peered":"2024-04-06T03:20:53.348930+0000","last_unstale":"2024-04-06T03:24:33.925402+0000","last_undegraded":"2024-04-06T03:24:33.925402+0000","last_fullsized":"2024-04-06T03:24:33.925402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6","last_scrub_stamp":"2024-04-06T03:24:33.925216+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:24:33.925216+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:35.832574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.636457+0000","last_change":"2024-04-06T03:24:12.542535+0000","last_active":"2024-04-06T03:24:30.636457+0000","last_peered":"2024-04-06T03:24:30.636457+0000","last_clean":"2024-04-06T03:24:30.636457+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:24:30.636457+0000","last_undegraded":"2024-04-06T03:24:30.636457+0000","last_fullsized":"2024-04-06T03:24:30.636457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:24:12.542082+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:24:12.542082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:26.649946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.995381+0000","last_change":"2024-04-06T03:24:23.888015+0000","last_active":"2024-04-06T03:24:30.995381+0000","last_peered":"2024-04-06T03:24:30.995381+0000","last_clean":"2024-04-06T03:24:30.995381+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:24:30.995381+0000","last_undegraded":"2024-04-06T03:24:30.995381+0000","last_fullsized":"2024-04-06T03:24:30.995381+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:23.887814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:23.887814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:32.529593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":110,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632979+0000","last_change":"2024-04-06T03:23:23.589775+0000","last_active":"2024-04-06T03:24:30.632979+0000","last_peered":"2024-04-06T03:24:30.632979+0000","last_clean":"2024-04-06T03:24:30.632979+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:24:30.632979+0000","last_undegraded":"2024-04-06T03:24:30.632979+0000","last_fullsized":"2024-04-06T03:24:30.632979+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:23.589472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.589472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.531624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"111'23","reported_seq":141,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.072942+0000","last_change":"2024-04-06T03:23:20.835769+0000","last_active":"2024-04-06T03:24:31.072942+0000","last_peered":"2024-04-06T03:24:31.072942+0000","last_clean":"2024-04-06T03:24:31.072942+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:24:31.072942+0000","last_undegraded":"2024-04-06T03:24:31.072942+0000","last_fullsized":"2024-04-06T03:24:31.072942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'23","last_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_clean_scrub_stamp":"2024-04-06T03:23:20.835514+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994614+0000","last_change":"2024-04-06T03:23:44.976222+0000","last_active":"2024-04-06T03:24:30.994614+0000","last_peered":"2024-04-06T03:24:30.994614+0000","last_clean":"2024-04-06T03:24:30.994614+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:24:30.994614+0000","last_undegraded":"2024-04-06T03:24:30.994614+0000","last_fullsized":"2024-04-06T03:24:30.994614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.976015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.976015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.423557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634172+0000","last_change":"2024-04-06T03:24:27.774010+0000","last_active":"2024-04-06T03:24:30.634172+0000","last_peered":"2024-04-06T03:24:30.634172+0000","last_clean":"2024-04-06T03:24:30.634172+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:24:30.634172+0000","last_undegraded":"2024-04-06T03:24:30.634172+0000","last_fullsized":"2024-04-06T03:24:30.634172+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-06T03:24:27.773757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:27.773757+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:47.812084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":135,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073892+0000","last_change":"2024-04-06T03:23:19.822376+0000","last_active":"2024-04-06T03:24:31.073892+0000","last_peered":"2024-04-06T03:24:31.073892+0000","last_clean":"2024-04-06T03:24:31.073892+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:24:31.073892+0000","last_undegraded":"2024-04-06T03:24:31.073892+0000","last_fullsized":"2024-04-06T03:24:31.073892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:23:19.822131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:19.822131+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:36.155287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632891+0000","last_change":"2024-04-06T03:23:52.629246+0000","last_active":"2024-04-06T03:24:30.632891+0000","last_peered":"2024-04-06T03:24:30.632891+0000","last_clean":"2024-04-06T03:24:30.632891+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:24:30.632891+0000","last_undegraded":"2024-04-06T03:24:30.632891+0000","last_fullsized":"2024-04-06T03:24:30.632891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:52.629002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:52.629002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.822619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634097+0000","last_change":"2024-04-06T03:24:19.940098+0000","last_active":"2024-04-06T03:24:30.634097+0000","last_peered":"2024-04-06T03:24:30.634097+0000","last_clean":"2024-04-06T03:24:30.634097+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:24:30.634097+0000","last_undegraded":"2024-04-06T03:24:30.634097+0000","last_fullsized":"2024-04-06T03:24:30.634097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:19.939848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:24:19.939848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.181812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859791+0000","last_change":"2024-04-06T03:23:50.717593+0000","last_active":"2024-04-06T03:24:30.859791+0000","last_peered":"2024-04-06T03:24:30.859791+0000","last_clean":"2024-04-06T03:24:30.859791+0000","last_became_active":"2024-04-06T03:20:51.148632+0000","last_became_peered":"2024-04-06T03:20:51.148632+0000","last_unstale":"2024-04-06T03:24:30.859791+0000","last_undegraded":"2024-04-06T03:24:30.859791+0000","last_fullsized":"2024-04-06T03:24:30.859791+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:50.717354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:50.717354+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:52.790825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"85'16","reported_seq":172,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055240+0000","last_change":"2024-04-06T03:24:06.088633+0000","last_active":"2024-04-06T03:24:31.055240+0000","last_peered":"2024-04-06T03:24:31.055240+0000","last_clean":"2024-04-06T03:24:31.055240+0000","last_became_active":"2024-04-06T03:20:53.351328+0000","last_became_peered":"2024-04-06T03:20:53.351328+0000","last_unstale":"2024-04-06T03:24:31.055240+0000","last_undegraded":"2024-04-06T03:24:31.055240+0000","last_fullsized":"2024-04-06T03:24:31.055240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:24:06.088289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.088289+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.093161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":113,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.858105+0000","last_change":"2024-04-06T03:23:56.666850+0000","last_active":"2024-04-06T03:24:30.858105+0000","last_peered":"2024-04-06T03:24:30.858105+0000","last_clean":"2024-04-06T03:24:30.858105+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:24:30.858105+0000","last_undegraded":"2024-04-06T03:24:30.858105+0000","last_fullsized":"2024-04-06T03:24:30.858105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:23:56.666442+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.666442+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:17.097001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.635471+0000","last_change":"2024-04-06T03:24:20.953132+0000","last_active":"2024-04-06T03:24:30.635471+0000","last_peered":"2024-04-06T03:24:30.635471+0000","last_clean":"2024-04-06T03:24:30.635471+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:24:30.635471+0000","last_undegraded":"2024-04-06T03:24:30.635471+0000","last_fullsized":"2024-04-06T03:24:30.635471+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:24:20.952992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:20.952992+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:22.514687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994802+0000","last_change":"2024-04-06T03:24:07.804878+0000","last_active":"2024-04-06T03:24:30.994802+0000","last_peered":"2024-04-06T03:24:30.994802+0000","last_clean":"2024-04-06T03:24:30.994802+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:24:30.994802+0000","last_undegraded":"2024-04-06T03:24:30.994802+0000","last_fullsized":"2024-04-06T03:24:30.994802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:07.804777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:07.804777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:14.120558+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:33.053208+0000","last_change":"2024-04-06T03:24:33.053264+0000","last_active":"2024-04-06T03:24:33.053208+0000","last_peered":"2024-04-06T03:24:33.053208+0000","last_clean":"2024-04-06T03:24:33.053208+0000","last_became_active":"2024-04-06T03:20:51.148366+0000","last_became_peered":"2024-04-06T03:20:51.148366+0000","last_unstale":"2024-04-06T03:24:33.053208+0000","last_undegraded":"2024-04-06T03:24:33.053208+0000","last_fullsized":"2024-04-06T03:24:33.053208+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:33.053053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:33.053053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:53.675493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"106'34","reported_seq":163,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633099+0000","last_change":"2024-04-06T03:23:59.442760+0000","last_active":"2024-04-06T03:24:30.633099+0000","last_peered":"2024-04-06T03:24:30.633099+0000","last_clean":"2024-04-06T03:24:30.633099+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:24:30.633099+0000","last_undegraded":"2024-04-06T03:24:30.633099+0000","last_fullsized":"2024-04-06T03:24:30.633099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'34","last_scrub_stamp":"2024-04-06T03:23:59.442517+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_clean_scrub_stamp":"2024-04-06T03:23:59.442517+0000","objects_scrubbed":9,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:27.492213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.636256+0000","last_change":"2024-04-06T03:24:06.724738+0000","last_active":"2024-04-06T03:24:30.636256+0000","last_peered":"2024-04-06T03:24:30.636256+0000","last_clean":"2024-04-06T03:24:30.636256+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:24:30.636256+0000","last_undegraded":"2024-04-06T03:24:30.636256+0000","last_fullsized":"2024-04-06T03:24:30.636256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:06.724641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.724641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:17.576679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073858+0000","last_change":"2024-04-06T03:24:02.704976+0000","last_active":"2024-04-06T03:24:31.073858+0000","last_peered":"2024-04-06T03:24:31.073858+0000","last_clean":"2024-04-06T03:24:31.073858+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:24:31.073858+0000","last_undegraded":"2024-04-06T03:24:31.073858+0000","last_fullsized":"2024-04-06T03:24:31.073858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:02.704534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:24:02.704534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:25.684101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":161,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.858847+0000","last_change":"2024-04-06T03:23:35.682185+0000","last_active":"2024-04-06T03:24:30.858847+0000","last_peered":"2024-04-06T03:24:30.858847+0000","last_clean":"2024-04-06T03:24:30.858847+0000","last_became_active":"2024-04-06T03:20:53.358177+0000","last_became_peered":"2024-04-06T03:20:53.358177+0000","last_unstale":"2024-04-06T03:24:30.858847+0000","last_undegraded":"2024-04-06T03:24:30.858847+0000","last_fullsized":"2024-04-06T03:24:30.858847+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-06T03:23:35.681909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.681909+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.508581+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633757+0000","last_change":"2024-04-06T03:23:53.594743+0000","last_active":"2024-04-06T03:24:30.633757+0000","last_peered":"2024-04-06T03:24:30.633757+0000","last_clean":"2024-04-06T03:24:30.633757+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:24:30.633757+0000","last_undegraded":"2024-04-06T03:24:30.633757+0000","last_fullsized":"2024-04-06T03:24:30.633757+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:53.594492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:53.594492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:18.217623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.637204+0000","last_change":"2024-04-06T03:23:47.598906+0000","last_active":"2024-04-06T03:24:30.637204+0000","last_peered":"2024-04-06T03:24:30.637204+0000","last_clean":"2024-04-06T03:24:30.637204+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:24:30.637204+0000","last_undegraded":"2024-04-06T03:24:30.637204+0000","last_fullsized":"2024-04-06T03:24:30.637204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:47.598682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:47.598682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.431027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.056112+0000","last_change":"2024-04-06T03:23:37.854225+0000","last_active":"2024-04-06T03:24:31.056112+0000","last_peered":"2024-04-06T03:24:31.056112+0000","last_clean":"2024-04-06T03:24:31.056112+0000","last_became_active":"2024-04-06T03:20:57.822724+0000","last_became_peered":"2024-04-06T03:20:57.822724+0000","last_unstale":"2024-04-06T03:24:31.056112+0000","last_undegraded":"2024-04-06T03:24:31.056112+0000","last_fullsized":"2024-04-06T03:24:31.056112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:37.854012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.854012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.515577+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":133,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993745+0000","last_change":"2024-04-06T03:23:51.862357+0000","last_active":"2024-04-06T03:24:30.993745+0000","last_peered":"2024-04-06T03:24:30.993745+0000","last_clean":"2024-04-06T03:24:30.993745+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:24:30.993745+0000","last_undegraded":"2024-04-06T03:24:30.993745+0000","last_fullsized":"2024-04-06T03:24:30.993745+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:23:51.861992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:51.861992+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:02.736112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.054368+0000","last_change":"2024-04-06T03:24:27.060413+0000","last_active":"2024-04-06T03:24:31.054368+0000","last_peered":"2024-04-06T03:24:31.054368+0000","last_clean":"2024-04-06T03:24:31.054368+0000","last_became_active":"2024-04-06T03:20:51.151534+0000","last_became_peered":"2024-04-06T03:20:51.151534+0000","last_unstale":"2024-04-06T03:24:31.054368+0000","last_undegraded":"2024-04-06T03:24:31.054368+0000","last_fullsized":"2024-04-06T03:24:31.054368+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:27.060096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:27.060096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:32.788580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":368,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.072648+0000","last_change":"2024-04-06T03:24:18.862939+0000","last_active":"2024-04-06T03:24:31.072648+0000","last_peered":"2024-04-06T03:24:31.072648+0000","last_clean":"2024-04-06T03:24:31.072648+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:24:31.072648+0000","last_undegraded":"2024-04-06T03:24:31.072648+0000","last_fullsized":"2024-04-06T03:24:31.072648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:24:18.862554+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:24:18.862554+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:40.986138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074324+0000","last_change":"2024-04-06T03:24:14.817313+0000","last_active":"2024-04-06T03:24:31.074324+0000","last_peered":"2024-04-06T03:24:31.074324+0000","last_clean":"2024-04-06T03:24:31.074324+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:24:31.074324+0000","last_undegraded":"2024-04-06T03:24:31.074324+0000","last_fullsized":"2024-04-06T03:24:31.074324+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.817089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:43.041041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859150+0000","last_change":"2024-04-06T03:24:09.753198+0000","last_active":"2024-04-06T03:24:30.859150+0000","last_peered":"2024-04-06T03:24:30.859150+0000","last_clean":"2024-04-06T03:24:30.859150+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:24:30.859150+0000","last_undegraded":"2024-04-06T03:24:30.859150+0000","last_fullsized":"2024-04-06T03:24:30.859150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:09.752966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:09.752966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:22.006299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'21","reported_seq":138,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634580+0000","last_change":"2024-04-06T03:23:23.076419+0000","last_active":"2024-04-06T03:24:30.634580+0000","last_peered":"2024-04-06T03:24:30.634580+0000","last_clean":"2024-04-06T03:24:30.634580+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:24:30.634580+0000","last_undegraded":"2024-04-06T03:24:30.634580+0000","last_fullsized":"2024-04-06T03:24:30.634580+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:23:23.076269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.076269+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.958707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.072619+0000","last_change":"2024-04-06T03:22:54.515963+0000","last_active":"2024-04-06T03:24:31.072619+0000","last_peered":"2024-04-06T03:24:31.072619+0000","last_clean":"2024-04-06T03:24:31.072619+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:24:31.072619+0000","last_undegraded":"2024-04-06T03:24:31.072619+0000","last_fullsized":"2024-04-06T03:24:31.072619+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:54.515739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:22:54.515739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055936+0000","last_change":"2024-04-06T03:23:28.869298+0000","last_active":"2024-04-06T03:24:31.055936+0000","last_peered":"2024-04-06T03:24:31.055936+0000","last_clean":"2024-04-06T03:24:31.055936+0000","last_became_active":"2024-04-06T03:20:55.754600+0000","last_became_peered":"2024-04-06T03:20:55.754600+0000","last_unstale":"2024-04-06T03:24:31.055936+0000","last_undegraded":"2024-04-06T03:24:31.055936+0000","last_fullsized":"2024-04-06T03:24:31.055936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:28.869068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:28.869068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.304800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.858609+0000","last_change":"2024-04-06T03:23:48.690384+0000","last_active":"2024-04-06T03:24:30.858609+0000","last_peered":"2024-04-06T03:24:30.858609+0000","last_clean":"2024-04-06T03:24:30.858609+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:24:30.858609+0000","last_undegraded":"2024-04-06T03:24:30.858609+0000","last_fullsized":"2024-04-06T03:24:30.858609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:48.690059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.690059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.780733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074147+0000","last_change":"2024-04-06T03:24:06.712317+0000","last_active":"2024-04-06T03:24:31.074147+0000","last_peered":"2024-04-06T03:24:31.074147+0000","last_clean":"2024-04-06T03:24:31.074147+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:24:31.074147+0000","last_undegraded":"2024-04-06T03:24:31.074147+0000","last_fullsized":"2024-04-06T03:24:31.074147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:06.712077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:06.712077+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.712077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:16.117036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"85'37","reported_seq":213,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859863+0000","last_change":"2024-04-06T03:23:09.792573+0000","last_active":"2024-04-06T03:24:30.859863+0000","last_peered":"2024-04-06T03:24:30.859863+0000","last_clean":"2024-04-06T03:24:30.859863+0000","last_became_active":"2024-04-06T03:20:53.352453+0000","last_became_peered":"2024-04-06T03:20:53.352453+0000","last_unstale":"2024-04-06T03:24:30.859863+0000","last_undegraded":"2024-04-06T03:24:30.859863+0000","last_fullsized":"2024-04-06T03:24:30.859863+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'37","last_scrub_stamp":"2024-04-06T03:23:09.792373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:09.792373+0000","objects_scrubbed":11,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.096219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":100,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.963087+0000","last_change":"2024-04-06T03:23:39.777144+0000","last_active":"2024-04-06T03:24:30.963087+0000","last_peered":"2024-04-06T03:24:30.963087+0000","last_clean":"2024-04-06T03:24:30.963087+0000","last_became_active":"2024-04-06T03:20:55.769795+0000","last_became_peered":"2024-04-06T03:20:55.769795+0000","last_unstale":"2024-04-06T03:24:30.963087+0000","last_undegraded":"2024-04-06T03:24:30.963087+0000","last_fullsized":"2024-04-06T03:24:30.963087+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.776933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.776933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:51.208938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632700+0000","last_change":"2024-04-06T03:23:35.364623+0000","last_active":"2024-04-06T03:24:30.632700+0000","last_peered":"2024-04-06T03:24:30.632700+0000","last_clean":"2024-04-06T03:24:30.632700+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:24:30.632700+0000","last_undegraded":"2024-04-06T03:24:30.632700+0000","last_fullsized":"2024-04-06T03:24:30.632700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:35.364381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.332831+0000","last_clean_scrub_stamp":"2024-04-06T03:23:35.364381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.739996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":110,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.860378+0000","last_change":"2024-04-06T03:23:37.757642+0000","last_active":"2024-04-06T03:24:30.860378+0000","last_peered":"2024-04-06T03:24:30.860378+0000","last_clean":"2024-04-06T03:24:30.860378+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:24:30.860378+0000","last_undegraded":"2024-04-06T03:24:30.860378+0000","last_fullsized":"2024-04-06T03:24:30.860378+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:23:37.757420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:37.757420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:48.290369+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":145,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.995492+0000","last_change":"2024-04-06T03:24:06.782236+0000","last_active":"2024-04-06T03:24:30.995492+0000","last_peered":"2024-04-06T03:24:30.995492+0000","last_clean":"2024-04-06T03:24:30.995492+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:24:30.995492+0000","last_undegraded":"2024-04-06T03:24:30.995492+0000","last_fullsized":"2024-04-06T03:24:30.995492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-06T03:24:06.782116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:06.782116+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:22.152506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":323,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:32.967502+0000","last_change":"2024-04-06T03:24:32.967502+0000","last_active":"2024-04-06T03:24:32.967502+0000","last_peered":"2024-04-06T03:24:32.967502+0000","last_clean":"2024-04-06T03:24:32.967502+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:24:32.967502+0000","last_undegraded":"2024-04-06T03:24:32.967502+0000","last_fullsized":"2024-04-06T03:24:32.967502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:24:32.967386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:24:32.967386+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:44.275071+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":201,"num_read_kb":196,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632498+0000","last_change":"2024-04-06T03:23:53.414430+0000","last_active":"2024-04-06T03:24:30.632498+0000","last_peered":"2024-04-06T03:24:30.632498+0000","last_clean":"2024-04-06T03:24:30.632498+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:24:30.632498+0000","last_undegraded":"2024-04-06T03:24:30.632498+0000","last_fullsized":"2024-04-06T03:24:30.632498+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:53.414220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:53.414220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:07.437199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073316+0000","last_change":"2024-04-06T03:24:23.880954+0000","last_active":"2024-04-06T03:24:31.073316+0000","last_peered":"2024-04-06T03:24:31.073316+0000","last_clean":"2024-04-06T03:24:31.073316+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:24:31.073316+0000","last_undegraded":"2024-04-06T03:24:31.073316+0000","last_fullsized":"2024-04-06T03:24:31.073316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:23.880755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:23.880755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:35.813387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.964056+0000","last_change":"2024-04-06T03:23:17.864400+0000","last_active":"2024-04-06T03:24:30.964056+0000","last_peered":"2024-04-06T03:24:30.964056+0000","last_clean":"2024-04-06T03:24:30.964056+0000","last_became_active":"2024-04-06T03:20:51.134999+0000","last_became_peered":"2024-04-06T03:20:51.134999+0000","last_unstale":"2024-04-06T03:24:30.964056+0000","last_undegraded":"2024-04-06T03:24:30.964056+0000","last_fullsized":"2024-04-06T03:24:30.964056+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:23:17.864183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:32.536200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":122,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.638608+0000","last_change":"2024-04-06T03:23:39.032748+0000","last_active":"2024-04-06T03:24:30.638608+0000","last_peered":"2024-04-06T03:24:30.638608+0000","last_clean":"2024-04-06T03:24:30.638608+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:24:30.638608+0000","last_undegraded":"2024-04-06T03:24:30.638608+0000","last_fullsized":"2024-04-06T03:24:30.638608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:23:39.032620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.032620+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:39.700020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632768+0000","last_change":"2024-04-06T03:23:46.374564+0000","last_active":"2024-04-06T03:24:30.632768+0000","last_peered":"2024-04-06T03:24:30.632768+0000","last_clean":"2024-04-06T03:24:30.632768+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:24:30.632768+0000","last_undegraded":"2024-04-06T03:24:30.632768+0000","last_fullsized":"2024-04-06T03:24:30.632768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:46.374356+0000","last_clean_scrub_stamp":"2024-04-06T03:23:46.374356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:58.995508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":158,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993338+0000","last_change":"2024-04-06T03:24:15.781660+0000","last_active":"2024-04-06T03:24:30.993338+0000","last_peered":"2024-04-06T03:24:30.993338+0000","last_clean":"2024-04-06T03:24:30.993338+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:24:30.993338+0000","last_undegraded":"2024-04-06T03:24:30.993338+0000","last_fullsized":"2024-04-06T03:24:30.993338+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:24:15.781269+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:24:15.781269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:24.073494+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994391+0000","last_change":"2024-04-06T03:23:38.882403+0000","last_active":"2024-04-06T03:24:30.994391+0000","last_peered":"2024-04-06T03:24:30.994391+0000","last_clean":"2024-04-06T03:24:30.994391+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:24:30.994391+0000","last_undegraded":"2024-04-06T03:24:30.994391+0000","last_fullsized":"2024-04-06T03:24:30.994391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.882269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:50.592970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055988+0000","last_change":"2024-04-06T03:23:11.690418+0000","last_active":"2024-04-06T03:24:31.055988+0000","last_peered":"2024-04-06T03:24:31.055988+0000","last_clean":"2024-04-06T03:24:31.055988+0000","last_became_active":"2024-04-06T03:20:55.762055+0000","last_became_peered":"2024-04-06T03:20:55.762055+0000","last_unstale":"2024-04-06T03:24:31.055988+0000","last_undegraded":"2024-04-06T03:24:31.055988+0000","last_fullsized":"2024-04-06T03:24:31.055988+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:11.690170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:11.690170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:35.579295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.072926+0000","last_change":"2024-04-06T03:24:24.887165+0000","last_active":"2024-04-06T03:24:31.072926+0000","last_peered":"2024-04-06T03:24:31.072926+0000","last_clean":"2024-04-06T03:24:31.072926+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:24:31.072926+0000","last_undegraded":"2024-04-06T03:24:31.072926+0000","last_fullsized":"2024-04-06T03:24:31.072926+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:24.886928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:24.886928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:50.371202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"85'24","reported_seq":153,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.632584+0000","last_change":"2024-04-06T03:23:56.442918+0000","last_active":"2024-04-06T03:24:30.632584+0000","last_peered":"2024-04-06T03:24:30.632584+0000","last_clean":"2024-04-06T03:24:30.632584+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:24:30.632584+0000","last_undegraded":"2024-04-06T03:24:30.632584+0000","last_fullsized":"2024-04-06T03:24:30.632584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2024-04-06T03:23:56.442661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.442661+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:57.735553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.054994+0000","last_change":"2024-04-06T03:23:38.883987+0000","last_active":"2024-04-06T03:24:31.054994+0000","last_peered":"2024-04-06T03:24:31.054994+0000","last_clean":"2024-04-06T03:24:31.054994+0000","last_became_active":"2024-04-06T03:20:57.845535+0000","last_became_peered":"2024-04-06T03:20:57.845535+0000","last_unstale":"2024-04-06T03:24:31.054994+0000","last_undegraded":"2024-04-06T03:24:31.054994+0000","last_fullsized":"2024-04-06T03:24:31.054994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:38.883738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:38.883738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:42.269641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"85'11","reported_seq":157,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633741+0000","last_change":"2024-04-06T03:23:23.295365+0000","last_active":"2024-04-06T03:24:30.633741+0000","last_peered":"2024-04-06T03:24:30.633741+0000","last_clean":"2024-04-06T03:24:30.633741+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:24:30.633741+0000","last_undegraded":"2024-04-06T03:24:30.633741+0000","last_fullsized":"2024-04-06T03:24:30.633741+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:23.295124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:23.295124+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:44.215136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":109,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859195+0000","last_change":"2024-04-06T03:23:59.665265+0000","last_active":"2024-04-06T03:24:30.859195+0000","last_peered":"2024-04-06T03:24:30.859195+0000","last_clean":"2024-04-06T03:24:30.859195+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:24:30.859195+0000","last_undegraded":"2024-04-06T03:24:30.859195+0000","last_fullsized":"2024-04-06T03:24:30.859195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:59.665053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:59.665053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:04.127092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"85'27","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.993396+0000","last_change":"2024-04-06T03:23:57.838208+0000","last_active":"2024-04-06T03:24:30.993396+0000","last_peered":"2024-04-06T03:24:30.993396+0000","last_clean":"2024-04-06T03:24:30.993396+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:24:30.993396+0000","last_undegraded":"2024-04-06T03:24:30.993396+0000","last_fullsized":"2024-04-06T03:24:30.993396+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'27","last_scrub_stamp":"2024-04-06T03:23:57.837997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:57.837997+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:59.275078+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634050+0000","last_change":"2024-04-06T03:23:36.391113+0000","last_active":"2024-04-06T03:24:30.634050+0000","last_peered":"2024-04-06T03:24:30.634050+0000","last_clean":"2024-04-06T03:24:30.634050+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:24:30.634050+0000","last_undegraded":"2024-04-06T03:24:30.634050+0000","last_fullsized":"2024-04-06T03:24:30.634050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:36.390806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:36.390806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:00.809816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":166,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.633538+0000","last_change":"2024-04-06T03:24:30.585750+0000","last_active":"2024-04-06T03:24:30.633538+0000","last_peered":"2024-04-06T03:24:30.633538+0000","last_clean":"2024-04-06T03:24:30.633538+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:24:30.633538+0000","last_undegraded":"2024-04-06T03:24:30.633538+0000","last_fullsized":"2024-04-06T03:24:30.633538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:24:30.585522+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:24:30.585522+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:51.300047+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":109,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859088+0000","last_change":"2024-04-06T03:23:07.798328+0000","last_active":"2024-04-06T03:24:30.859088+0000","last_peered":"2024-04-06T03:24:30.859088+0000","last_clean":"2024-04-06T03:24:30.859088+0000","last_became_active":"2024-04-06T03:20:51.167121+0000","last_became_peered":"2024-04-06T03:20:51.167121+0000","last_unstale":"2024-04-06T03:24:30.859088+0000","last_undegraded":"2024-04-06T03:24:30.859088+0000","last_fullsized":"2024-04-06T03:24:30.859088+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:07.798091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:07.798091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"85'15","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.994279+0000","last_change":"2024-04-06T03:23:24.821416+0000","last_active":"2024-04-06T03:24:30.994279+0000","last_peered":"2024-04-06T03:24:30.994279+0000","last_clean":"2024-04-06T03:24:30.994279+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:24:30.994279+0000","last_undegraded":"2024-04-06T03:24:30.994279+0000","last_fullsized":"2024-04-06T03:24:30.994279+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:24.821164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:24.821164+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.815641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":98,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.638319+0000","last_change":"2024-04-06T03:23:55.616868+0000","last_active":"2024-04-06T03:24:30.638319+0000","last_peered":"2024-04-06T03:24:30.638319+0000","last_clean":"2024-04-06T03:24:30.638319+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:24:30.638319+0000","last_undegraded":"2024-04-06T03:24:30.638319+0000","last_fullsized":"2024-04-06T03:24:30.638319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:23:55.616541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:04.260063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859429+0000","last_change":"2024-04-06T03:24:08.719094+0000","last_active":"2024-04-06T03:24:30.859429+0000","last_peered":"2024-04-06T03:24:30.859429+0000","last_clean":"2024-04-06T03:24:30.859429+0000","last_became_active":"2024-04-06T03:20:55.758951+0000","last_became_peered":"2024-04-06T03:20:55.758951+0000","last_unstale":"2024-04-06T03:24:30.859429+0000","last_undegraded":"2024-04-06T03:24:30.859429+0000","last_fullsized":"2024-04-06T03:24:30.859429+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:08.718957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:08.718957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:38.145618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":108,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.073572+0000","last_change":"2024-04-06T03:23:15.710252+0000","last_active":"2024-04-06T03:24:31.073572+0000","last_peered":"2024-04-06T03:24:31.073572+0000","last_clean":"2024-04-06T03:24:31.073572+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:24:31.073572+0000","last_undegraded":"2024-04-06T03:24:31.073572+0000","last_fullsized":"2024-04-06T03:24:31.073572+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:15.710038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:23:15.710038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:34.258607+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'25","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.969697+0000","last_change":"2024-04-06T03:24:30.969790+0000","last_active":"2024-04-06T03:24:30.969697+0000","last_peered":"2024-04-06T03:24:30.969697+0000","last_clean":"2024-04-06T03:24:30.969697+0000","last_became_active":"2024-04-06T03:20:53.350784+0000","last_became_peered":"2024-04-06T03:20:53.350784+0000","last_unstale":"2024-04-06T03:24:30.969697+0000","last_undegraded":"2024-04-06T03:24:30.969697+0000","last_fullsized":"2024-04-06T03:24:30.969697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-06T03:24:30.969540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:30.969540+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:54.231470+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":88,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.072459+0000","last_change":"2024-04-06T03:22:55.509249+0000","last_active":"2024-04-06T03:24:31.072459+0000","last_peered":"2024-04-06T03:24:31.072459+0000","last_clean":"2024-04-06T03:24:31.072459+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:24:31.072459+0000","last_undegraded":"2024-04-06T03:24:31.072459+0000","last_fullsized":"2024-04-06T03:24:31.072459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:22:55.508950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:22:55.508950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.635153+0000","last_change":"2024-04-06T03:23:43.566143+0000","last_active":"2024-04-06T03:24:30.635153+0000","last_peered":"2024-04-06T03:24:30.635153+0000","last_clean":"2024-04-06T03:24:30.635153+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:24:30.635153+0000","last_undegraded":"2024-04-06T03:24:30.635153+0000","last_fullsized":"2024-04-06T03:24:30.635153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:23:43.565930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:43.924455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":110,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.636036+0000","last_change":"2024-04-06T03:23:56.575312+0000","last_active":"2024-04-06T03:24:30.636036+0000","last_peered":"2024-04-06T03:24:30.636036+0000","last_clean":"2024-04-06T03:24:30.636036+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:24:30.636036+0000","last_undegraded":"2024-04-06T03:24:30.636036+0000","last_fullsized":"2024-04-06T03:24:30.636036+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:23:56.575091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.850617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074619+0000","last_change":"2024-04-06T03:22:57.488426+0000","last_active":"2024-04-06T03:24:31.074619+0000","last_peered":"2024-04-06T03:24:31.074619+0000","last_clean":"2024-04-06T03:24:31.074619+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:24:31.074619+0000","last_undegraded":"2024-04-06T03:24:31.074619+0000","last_fullsized":"2024-04-06T03:24:31.074619+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:22:57.488176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:22:57.488176+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":96,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634291+0000","last_change":"2024-04-06T03:23:39.990151+0000","last_active":"2024-04-06T03:24:30.634291+0000","last_peered":"2024-04-06T03:24:30.634291+0000","last_clean":"2024-04-06T03:24:30.634291+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:24:30.634291+0000","last_undegraded":"2024-04-06T03:24:30.634291+0000","last_fullsized":"2024-04-06T03:24:30.634291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:39.990000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:23:39.990000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:55.631759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":102,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.634730+0000","last_change":"2024-04-06T03:23:44.525846+0000","last_active":"2024-04-06T03:24:30.634730+0000","last_peered":"2024-04-06T03:24:30.634730+0000","last_clean":"2024-04-06T03:24:30.634730+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:24:30.634730+0000","last_undegraded":"2024-04-06T03:24:30.634730+0000","last_fullsized":"2024-04-06T03:24:30.634730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_clean_scrub_stamp":"2024-04-06T03:23:44.525616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:05.599062+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.859876+0000","last_change":"2024-04-06T03:24:27.854683+0000","last_active":"2024-04-06T03:24:30.859876+0000","last_peered":"2024-04-06T03:24:30.859876+0000","last_clean":"2024-04-06T03:24:30.859876+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:24:30.859876+0000","last_undegraded":"2024-04-06T03:24:30.859876+0000","last_fullsized":"2024-04-06T03:24:30.859876+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:27.854439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.771476+0000","last_clean_scrub_stamp":"2024-04-06T03:24:27.854439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:30.174933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"85'10","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.055080+0000","last_change":"2024-04-06T03:23:48.924901+0000","last_active":"2024-04-06T03:24:31.055080+0000","last_peered":"2024-04-06T03:24:31.055080+0000","last_clean":"2024-04-06T03:24:31.055080+0000","last_became_active":"2024-04-06T03:20:53.345059+0000","last_became_peered":"2024-04-06T03:20:53.345059+0000","last_unstale":"2024-04-06T03:24:31.055080+0000","last_undegraded":"2024-04-06T03:24:31.055080+0000","last_fullsized":"2024-04-06T03:24:31.055080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'10","last_scrub_stamp":"2024-04-06T03:23:48.924677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:48.924677+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:13.542114+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":97,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.053963+0000","last_change":"2024-04-06T03:24:00.078410+0000","last_active":"2024-04-06T03:24:31.053963+0000","last_peered":"2024-04-06T03:24:31.053963+0000","last_clean":"2024-04-06T03:24:31.053963+0000","last_became_active":"2024-04-06T03:20:58.128530+0000","last_became_peered":"2024-04-06T03:20:58.128530+0000","last_unstale":"2024-04-06T03:24:31.053963+0000","last_undegraded":"2024-04-06T03:24:31.053963+0000","last_fullsized":"2024-04-06T03:24:31.053963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:00.078196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:24:00.078196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:23.255067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":118,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:30.638758+0000","last_change":"2024-04-06T03:24:22.577522+0000","last_active":"2024-04-06T03:24:30.638758+0000","last_peered":"2024-04-06T03:24:30.638758+0000","last_clean":"2024-04-06T03:24:30.638758+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:24:30.638758+0000","last_undegraded":"2024-04-06T03:24:30.638758+0000","last_fullsized":"2024-04-06T03:24:30.638758+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:22.577283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:24:22.577283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:42.933283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":129,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074183+0000","last_change":"2024-04-06T03:23:41.707141+0000","last_active":"2024-04-06T03:24:31.074183+0000","last_peered":"2024-04-06T03:24:31.074183+0000","last_clean":"2024-04-06T03:24:31.074183+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:24:31.074183+0000","last_undegraded":"2024-04-06T03:24:31.074183+0000","last_fullsized":"2024-04-06T03:24:31.074183+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:23:41.706589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:23:41.706589+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:25:08.161763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"85'11","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-04-06T03:24:31.074182+0000","last_change":"2024-04-06T03:23:33.820933+0000","last_active":"2024-04-06T03:24:31.074182+0000","last_peered":"2024-04-06T03:24:31.074182+0000","last_clean":"2024-04-06T03:24:31.074182+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:24:31.074182+0000","last_undegraded":"2024-04-06T03:24:31.074182+0000","last_fullsized":"2024-04-06T03:24:31.074182+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:23:33.820691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:23:33.820691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:24:37.287595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":201,"num_read_kb":196,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":648,"ondisk_log_size":648,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743479,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712636,"statfs":{"total":95995035648,"available":95961739264,"internally_reserved":0,"allocated":1368064,"data_stored":986483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135937,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32480,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712672,"statfs":{"total":95995035648,"available":95961776128,"internally_reserved":0,"allocated":1331200,"data_stored":984763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":753664,"data_stored":394249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36656,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708496,"statfs":{"total":95995035648,"available":95957499904,"internally_reserved":0,"allocated":1351680,"data_stored":985816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378732,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27744,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717408,"statfs":{"total":95995035648,"available":95966625792,"internally_reserved":0,"allocated":741376,"data_stored":393993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084384,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":393907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855447,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":778240,"data_stored":394478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495693,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":737280,"data_stored":393889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:24:35.830 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-06T03:24:35.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump 2024-04-06T03:24:35.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: cluster 2024-04-06T03:24:34.891411+0000 osd.6 (osd.6) 59 : cluster 0 2.7 scrub starts 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: cluster 2024-04-06T03:24:34.893723+0000 osd.6 (osd.6) 60 : cluster 0 2.7 scrub ok 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: cluster 2024-04-06T03:24:34.988338+0000 osd.5 (osd.5) 93 : cluster 0 5.7 scrub starts 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: cluster 2024-04-06T03:24:34.990666+0000 osd.5 (osd.5) 94 : cluster 0 5.7 scrub ok 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: audit 2024-04-06T03:24:35.026109+0000 mgr.y (mgr.24368) 139 : audit 0 from='client.14688 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.151076+0000 osd.5 (osd.5) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.151209+0000 osd.5 (osd.5) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: cluster 2024-04-06T03:24:34.891411+0000 osd.6 (osd.6) 59 : cluster 0 2.7 scrub starts 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: cluster 2024-04-06T03:24:34.893723+0000 osd.6 (osd.6) 60 : cluster 0 2.7 scrub ok 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: cluster 2024-04-06T03:24:34.988338+0000 osd.5 (osd.5) 93 : cluster 0 5.7 scrub starts 2024-04-06T03:24:35.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: cluster 2024-04-06T03:24:34.990666+0000 osd.5 (osd.5) 94 : cluster 0 5.7 scrub ok 2024-04-06T03:24:35.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: audit 2024-04-06T03:24:35.026109+0000 mgr.y (mgr.24368) 139 : audit 0 from='client.14688 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:35.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.151076+0000 osd.5 (osd.5) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:35.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:35 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.151209+0000 osd.5 (osd.5) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: cluster 2024-04-06T03:24:34.891411+0000 osd.6 (osd.6) 59 : cluster 0 2.7 scrub starts 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: cluster 2024-04-06T03:24:34.893723+0000 osd.6 (osd.6) 60 : cluster 0 2.7 scrub ok 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: cluster 2024-04-06T03:24:34.988338+0000 osd.5 (osd.5) 93 : cluster 0 5.7 scrub starts 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: cluster 2024-04-06T03:24:34.990666+0000 osd.5 (osd.5) 94 : cluster 0 5.7 scrub ok 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: audit 2024-04-06T03:24:35.026109+0000 mgr.y (mgr.24368) 139 : audit 0 from='client.14688 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.151076+0000 osd.5 (osd.5) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:36.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:35 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.151209+0000 osd.5 (osd.5) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.658703+0000 mon.a (mon.0) 974 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.826382+0000 osd.7 (osd.7) 97 : cluster 0 6.8 scrub starts 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.830380+0000 osd.7 (osd.7) 98 : cluster 0 6.8 scrub ok 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.880253+0000 osd.1 (osd.1) 91 : cluster 0 2.a scrub starts 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[21467]: cluster 2024-04-06T03:24:35.881855+0000 osd.1 (osd.1) 92 : cluster 0 2.a scrub ok 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.018935+0000 mgr.y (mgr.24368) 140 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.658703+0000 mon.a (mon.0) 974 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.826382+0000 osd.7 (osd.7) 97 : cluster 0 6.8 scrub starts 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.830380+0000 osd.7 (osd.7) 98 : cluster 0 6.8 scrub ok 2024-04-06T03:24:36.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.880253+0000 osd.1 (osd.1) 91 : cluster 0 2.a scrub starts 2024-04-06T03:24:36.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[29221]: cluster 2024-04-06T03:24:35.881855+0000 osd.1 (osd.1) 92 : cluster 0 2.a scrub ok 2024-04-06T03:24:36.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:36 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.018935+0000 mgr.y (mgr.24368) 140 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:36 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.658703+0000 mon.a (mon.0) 974 : cluster 0 osdmap e120: 8 total, 8 up, 8 in 2024-04-06T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:36 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.826382+0000 osd.7 (osd.7) 97 : cluster 0 6.8 scrub starts 2024-04-06T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:36 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.830380+0000 osd.7 (osd.7) 98 : cluster 0 6.8 scrub ok 2024-04-06T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:36 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.880253+0000 osd.1 (osd.1) 91 : cluster 0 2.a scrub starts 2024-04-06T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:36 smithi153 bash[23852]: cluster 2024-04-06T03:24:35.881855+0000 osd.1 (osd.1) 92 : cluster 0 2.a scrub ok 2024-04-06T03:24:37.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:36 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.018935+0000 mgr.y (mgr.24368) 140 : cluster 0 pgmap v109: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:37.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.583387+0000 osd.2 (osd.2) 67 : cluster 0 2.10 scrub starts 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.585798+0000 osd.2 (osd.2) 68 : cluster 0 2.10 scrub ok 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.792480+0000 osd.7 (osd.7) 99 : cluster 0 2.0 scrub starts 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.794950+0000 osd.7 (osd.7) 100 : cluster 0 2.0 scrub ok 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.843084+0000 osd.1 (osd.1) 93 : cluster 0 5.1d scrub starts 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[21467]: cluster 2024-04-06T03:24:36.845748+0000 osd.1 (osd.1) 94 : cluster 0 5.1d scrub ok 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.583387+0000 osd.2 (osd.2) 67 : cluster 0 2.10 scrub starts 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.585798+0000 osd.2 (osd.2) 68 : cluster 0 2.10 scrub ok 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.792480+0000 osd.7 (osd.7) 99 : cluster 0 2.0 scrub starts 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.794950+0000 osd.7 (osd.7) 100 : cluster 0 2.0 scrub ok 2024-04-06T03:24:37.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.843084+0000 osd.1 (osd.1) 93 : cluster 0 5.1d scrub starts 2024-04-06T03:24:37.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:37 smithi050 bash[29221]: cluster 2024-04-06T03:24:36.845748+0000 osd.1 (osd.1) 94 : cluster 0 5.1d scrub ok 2024-04-06T03:24:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:37 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.583387+0000 osd.2 (osd.2) 67 : cluster 0 2.10 scrub starts 2024-04-06T03:24:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:37 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.585798+0000 osd.2 (osd.2) 68 : cluster 0 2.10 scrub ok 2024-04-06T03:24:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:37 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.792480+0000 osd.7 (osd.7) 99 : cluster 0 2.0 scrub starts 2024-04-06T03:24:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:37 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.794950+0000 osd.7 (osd.7) 100 : cluster 0 2.0 scrub ok 2024-04-06T03:24:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:37 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.843084+0000 osd.1 (osd.1) 93 : cluster 0 5.1d scrub starts 2024-04-06T03:24:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:37 smithi153 bash[23852]: cluster 2024-04-06T03:24:36.845748+0000 osd.1 (osd.1) 94 : cluster 0 5.1d scrub ok 2024-04-06T03:24:38.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:37.747943+0000 osd.7 (osd.7) 101 : cluster 0 3.d scrub starts 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:37.752701+0000 osd.7 (osd.7) 102 : cluster 0 3.d scrub ok 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:38.019613+0000 mgr.y (mgr.24368) 141 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:38.039396+0000 osd.5 (osd.5) 97 : cluster 0 5.f scrub starts 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:38.042023+0000 osd.5 (osd.5) 98 : cluster 0 5.f scrub ok 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:38.374093+0000 osd.1 (osd.1) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[21467]: cluster 2024-04-06T03:24:38.374203+0000 osd.1 (osd.1) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:37.747943+0000 osd.7 (osd.7) 101 : cluster 0 3.d scrub starts 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:37.752701+0000 osd.7 (osd.7) 102 : cluster 0 3.d scrub ok 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:38.019613+0000 mgr.y (mgr.24368) 141 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:38.039396+0000 osd.5 (osd.5) 97 : cluster 0 5.f scrub starts 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:38.042023+0000 osd.5 (osd.5) 98 : cluster 0 5.f scrub ok 2024-04-06T03:24:38.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:38.374093+0000 osd.1 (osd.1) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:38.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:38 smithi050 bash[29221]: cluster 2024-04-06T03:24:38.374203+0000 osd.1 (osd.1) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:37.747943+0000 osd.7 (osd.7) 101 : cluster 0 3.d scrub starts 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:37.752701+0000 osd.7 (osd.7) 102 : cluster 0 3.d scrub ok 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:38.019613+0000 mgr.y (mgr.24368) 141 : cluster 0 pgmap v110: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:38.039396+0000 osd.5 (osd.5) 97 : cluster 0 5.f scrub starts 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:38.042023+0000 osd.5 (osd.5) 98 : cluster 0 5.f scrub ok 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:38.374093+0000 osd.1 (osd.1) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:24:39.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:38 smithi153 bash[23852]: cluster 2024-04-06T03:24:38.374203+0000 osd.1 (osd.1) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:24:39.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:24:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:24:39.675 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:24:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:24:39.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:39 smithi050 bash[21467]: cluster 2024-04-06T03:24:38.690599+0000 mon.a (mon.0) 975 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-04-06T03:24:39.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:39 smithi050 bash[21467]: audit 2024-04-06T03:24:39.111375+0000 mgr.y (mgr.24368) 142 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:39.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:39 smithi050 bash[29221]: cluster 2024-04-06T03:24:38.690599+0000 mon.a (mon.0) 975 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-04-06T03:24:39.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:39 smithi050 bash[29221]: audit 2024-04-06T03:24:39.111375+0000 mgr.y (mgr.24368) 142 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:40.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:39 smithi153 bash[23852]: cluster 2024-04-06T03:24:38.690599+0000 mon.a (mon.0) 975 : cluster 0 osdmap e121: 8 total, 8 up, 8 in 2024-04-06T03:24:40.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:39 smithi153 bash[23852]: audit 2024-04-06T03:24:39.111375+0000 mgr.y (mgr.24368) 142 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:40.626 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:40 smithi050 bash[29221]: cluster 2024-04-06T03:24:39.922318+0000 osd.1 (osd.1) 97 : cluster 0 3.4 scrub starts 2024-04-06T03:24:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:40 smithi050 bash[29221]: cluster 2024-04-06T03:24:39.927664+0000 osd.1 (osd.1) 98 : cluster 0 3.4 scrub ok 2024-04-06T03:24:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:40 smithi050 bash[29221]: cluster 2024-04-06T03:24:40.021133+0000 mgr.y (mgr.24368) 143 : cluster 0 pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:40 smithi050 bash[21467]: cluster 2024-04-06T03:24:39.922318+0000 osd.1 (osd.1) 97 : cluster 0 3.4 scrub starts 2024-04-06T03:24:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:40 smithi050 bash[21467]: cluster 2024-04-06T03:24:39.927664+0000 osd.1 (osd.1) 98 : cluster 0 3.4 scrub ok 2024-04-06T03:24:40.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:40 smithi050 bash[21467]: cluster 2024-04-06T03:24:40.021133+0000 mgr.y (mgr.24368) 143 : cluster 0 pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:41.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:40 smithi153 bash[23852]: cluster 2024-04-06T03:24:39.922318+0000 osd.1 (osd.1) 97 : cluster 0 3.4 scrub starts 2024-04-06T03:24:41.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:40 smithi153 bash[23852]: cluster 2024-04-06T03:24:39.927664+0000 osd.1 (osd.1) 98 : cluster 0 3.4 scrub ok 2024-04-06T03:24:41.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:40 smithi153 bash[23852]: cluster 2024-04-06T03:24:40.021133+0000 mgr.y (mgr.24368) 143 : cluster 0 pgmap v112: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:41.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:41 smithi050 bash[21467]: cluster 2024-04-06T03:24:40.804122+0000 osd.7 (osd.7) 103 : cluster 0 3.11 scrub starts 2024-04-06T03:24:41.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:41 smithi050 bash[21467]: cluster 2024-04-06T03:24:40.808615+0000 osd.7 (osd.7) 104 : cluster 0 3.11 scrub ok 2024-04-06T03:24:41.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:41 smithi050 bash[29221]: cluster 2024-04-06T03:24:40.804122+0000 osd.7 (osd.7) 103 : cluster 0 3.11 scrub starts 2024-04-06T03:24:41.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:41 smithi050 bash[29221]: cluster 2024-04-06T03:24:40.808615+0000 osd.7 (osd.7) 104 : cluster 0 3.11 scrub ok 2024-04-06T03:24:42.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:41 smithi153 bash[23852]: cluster 2024-04-06T03:24:40.804122+0000 osd.7 (osd.7) 103 : cluster 0 3.11 scrub starts 2024-04-06T03:24:42.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:41 smithi153 bash[23852]: cluster 2024-04-06T03:24:40.808615+0000 osd.7 (osd.7) 104 : cluster 0 3.11 scrub ok 2024-04-06T03:24:42.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:42 smithi050 bash[21467]: cluster 2024-04-06T03:24:42.021804+0000 mgr.y (mgr.24368) 144 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:42.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:42 smithi050 bash[29221]: cluster 2024-04-06T03:24:42.021804+0000 mgr.y (mgr.24368) 144 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:43.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:42 smithi153 bash[23852]: cluster 2024-04-06T03:24:42.021804+0000 mgr.y (mgr.24368) 144 : cluster 0 pgmap v113: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:24:43.920 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 121 2024-04-06T03:24:43.920 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:24:43.920 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-04-06T03:10:26.969128+0000 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:modified 2024-04-06T03:24:38.673418+0000 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:crush_version 18 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:full_ratio 0.95 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:backfillfull_ratio 0.9 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:nearfull_ratio 0.85 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:require_min_compat_client luminous 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:min_compat_client luminous 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:require_osd_release squid 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:stretch_mode_enabled false 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-06T03:24:43.921 INFO:teuthology.orchestra.run.smithi050.stdout:pool 4 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:max_osd 8 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0 up in weight 1 up_from 8 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646] [v2:172.21.15.50:6804/3531754646,v1:172.21.15.50:6805/3531754646] exists,up e607f527-4592-4acd-a47f-76575615ae1b 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069] [v2:172.21.15.50:6812/4128410069,v1:172.21.15.50:6813/4128410069] exists,up 2ae8f784-d36a-4633-8630-2aed04e4ff10 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2 up in weight 1 up_from 19 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904] [v2:172.21.15.50:6820/1307247904,v1:172.21.15.50:6821/1307247904] exists,up 43f31d09-5c16-4aea-8822-e0d877cc931b 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3 up in weight 1 up_from 28 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508] [v2:172.21.15.50:6828/1397888508,v1:172.21.15.50:6829/1397888508] exists,up 4c913325-442e-4ee5-94f1-8b5bca0c3240 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4 up in weight 1 up_from 35 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473] [v2:172.21.15.153:6802/3377683473,v1:172.21.15.153:6803/3377683473] exists,up 50c02bfc-c2f2-4aae-b1d5-c38a86726b48 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117] [v2:172.21.15.153:6810/357179117,v1:172.21.15.153:6811/357179117] exists,up c3e8194b-2c20-4ffb-859d-00c33902b241 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6 up in weight 1 up_from 57 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208] [v2:172.21.15.153:6818/1552538208,v1:172.21.15.153:6819/1552538208] exists,up 98027d53-4324-44d1-89ce-8da85be58efb 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7 up in weight 1 up_from 69 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924] [v2:172.21.15.153:6826/2479193924,v1:172.21.15.153:6827/2479193924] exists,up 6b47d95a-1228-4fb1-a315-5371d9b42bbd 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 2.1f [7,2] 2024-04-06T03:24:43.922 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/283275195 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2738019433 expires 2024-04-07T03:11:33.442338+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/889382629 expires 2024-04-07T03:10:58.614019+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/1268273764 expires 2024-04-07T03:10:58.614019+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/1263219005 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/567103210 expires 2024-04-07T03:10:58.614019+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2891059051 expires 2024-04-07T03:11:33.442338+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2170219983 expires 2024-04-07T03:11:33.442338+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/132418263 expires 2024-04-07T03:11:33.442338+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/4082841962 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/1263219005 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/570623774 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/889382629 expires 2024-04-07T03:10:58.614019+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/1379780741 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2880368026 expires 2024-04-07T03:10:58.614019+0000 2024-04-06T03:24:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/132418263 expires 2024-04-07T03:11:33.442338+0000 2024-04-06T03:24:43.924 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2044358308 expires 2024-04-07T03:21:47.915758+0000 2024-04-06T03:24:43.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[29221]: cluster 2024-04-06T03:24:42.766515+0000 osd.4 (osd.4) 75 : cluster 0 3.1a scrub starts 2024-04-06T03:24:43.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[29221]: cluster 2024-04-06T03:24:42.772805+0000 osd.4 (osd.4) 76 : cluster 0 3.1a scrub ok 2024-04-06T03:24:43.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[29221]: cluster 2024-04-06T03:24:43.020033+0000 osd.5 (osd.5) 99 : cluster 0 6.d scrub starts 2024-04-06T03:24:43.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[29221]: cluster 2024-04-06T03:24:43.022237+0000 osd.5 (osd.5) 100 : cluster 0 6.d scrub ok 2024-04-06T03:24:43.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[21467]: cluster 2024-04-06T03:24:42.766515+0000 osd.4 (osd.4) 75 : cluster 0 3.1a scrub starts 2024-04-06T03:24:43.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[21467]: cluster 2024-04-06T03:24:42.772805+0000 osd.4 (osd.4) 76 : cluster 0 3.1a scrub ok 2024-04-06T03:24:43.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[21467]: cluster 2024-04-06T03:24:43.020033+0000 osd.5 (osd.5) 99 : cluster 0 6.d scrub starts 2024-04-06T03:24:43.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:43 smithi050 bash[21467]: cluster 2024-04-06T03:24:43.022237+0000 osd.5 (osd.5) 100 : cluster 0 6.d scrub ok 2024-04-06T03:24:44.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:43 smithi153 bash[23852]: cluster 2024-04-06T03:24:42.766515+0000 osd.4 (osd.4) 75 : cluster 0 3.1a scrub starts 2024-04-06T03:24:44.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:43 smithi153 bash[23852]: cluster 2024-04-06T03:24:42.772805+0000 osd.4 (osd.4) 76 : cluster 0 3.1a scrub ok 2024-04-06T03:24:44.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:43 smithi153 bash[23852]: cluster 2024-04-06T03:24:43.020033+0000 osd.5 (osd.5) 99 : cluster 0 6.d scrub starts 2024-04-06T03:24:44.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:43 smithi153 bash[23852]: cluster 2024-04-06T03:24:43.022237+0000 osd.5 (osd.5) 100 : cluster 0 6.d scrub ok 2024-04-06T03:24:44.745 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 96 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6802/3531754646,v1:172.21.15.50:6803/3531754646] [v2:172.21.15.50:6804/3531754646,v1:172.21.15.50:6805/3531754646] exists,up e607f527-4592-4acd-a47f-76575615ae1b', 'osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6810/4128410069,v1:172.21.15.50:6811/4128410069] [v2:172.21.15.50:6812/4128410069,v1:172.21.15.50:6813/4128410069] exists,up 2ae8f784-d36a-4633-8630-2aed04e4ff10', 'osd.2 up in weight 1 up_from 19 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6818/1307247904,v1:172.21.15.50:6819/1307247904] [v2:172.21.15.50:6820/1307247904,v1:172.21.15.50:6821/1307247904] exists,up 43f31d09-5c16-4aea-8822-e0d877cc931b', 'osd.3 up in weight 1 up_from 28 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6826/1397888508,v1:172.21.15.50:6827/1397888508] [v2:172.21.15.50:6828/1397888508,v1:172.21.15.50:6829/1397888508] exists,up 4c913325-442e-4ee5-94f1-8b5bca0c3240', 'osd.4 up in weight 1 up_from 35 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/3377683473,v1:172.21.15.153:6801/3377683473] [v2:172.21.15.153:6802/3377683473,v1:172.21.15.153:6803/3377683473] exists,up 50c02bfc-c2f2-4aae-b1d5-c38a86726b48', 'osd.5 up in weight 1 up_from 46 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/357179117,v1:172.21.15.153:6809/357179117] [v2:172.21.15.153:6810/357179117,v1:172.21.15.153:6811/357179117] exists,up c3e8194b-2c20-4ffb-859d-00c33902b241', 'osd.6 up in weight 1 up_from 57 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/1552538208,v1:172.21.15.153:6817/1552538208] [v2:172.21.15.153:6818/1552538208,v1:172.21.15.153:6819/1552538208] exists,up 98027d53-4324-44d1-89ce-8da85be58efb', 'osd.7 up in weight 1 up_from 69 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/2479193924,v1:172.21.15.153:6825/2479193924] [v2:172.21.15.153:6826/2479193924,v1:172.21.15.153:6827/2479193924] exists,up 6b47d95a-1228-4fb1-a315-5371d9b42bbd'] 2024-04-06T03:24:44.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon mon.a config show 2024-04-06T03:24:44.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[21467]: audit 2024-04-06T03:24:43.928446+0000 mon.c (mon.2) 30 : audit 0 from='client.? 172.21.15.50:0/1759925086' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-06T03:24:44.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[21467]: cluster 2024-04-06T03:24:44.022546+0000 mgr.y (mgr.24368) 145 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:44.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[21467]: cluster 2024-04-06T03:24:44.081451+0000 osd.0 (osd.0) 85 : cluster 0 3.6 scrub starts 2024-04-06T03:24:44.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[21467]: cluster 2024-04-06T03:24:44.084524+0000 osd.0 (osd.0) 86 : cluster 0 3.6 scrub ok 2024-04-06T03:24:44.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[29221]: audit 2024-04-06T03:24:43.928446+0000 mon.c (mon.2) 30 : audit 0 from='client.? 172.21.15.50:0/1759925086' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-06T03:24:44.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[29221]: cluster 2024-04-06T03:24:44.022546+0000 mgr.y (mgr.24368) 145 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:44.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[29221]: cluster 2024-04-06T03:24:44.081451+0000 osd.0 (osd.0) 85 : cluster 0 3.6 scrub starts 2024-04-06T03:24:44.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:44 smithi050 bash[29221]: cluster 2024-04-06T03:24:44.084524+0000 osd.0 (osd.0) 86 : cluster 0 3.6 scrub ok 2024-04-06T03:24:45.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:44 smithi153 bash[23852]: audit 2024-04-06T03:24:43.928446+0000 mon.c (mon.2) 30 : audit 0 from='client.? 172.21.15.50:0/1759925086' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-06T03:24:45.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:44 smithi153 bash[23852]: cluster 2024-04-06T03:24:44.022546+0000 mgr.y (mgr.24368) 145 : cluster 0 pgmap v114: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:45.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:44 smithi153 bash[23852]: cluster 2024-04-06T03:24:44.081451+0000 osd.0 (osd.0) 85 : cluster 0 3.6 scrub starts 2024-04-06T03:24:45.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:44 smithi153 bash[23852]: cluster 2024-04-06T03:24:44.084524+0000 osd.0 (osd.0) 86 : cluster 0 3.6 scrub ok 2024-04-06T03:24:47.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[21467]: cluster 2024-04-06T03:24:45.752092+0000 osd.4 (osd.4) 77 : cluster 0 6.17 scrub starts 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[21467]: cluster 2024-04-06T03:24:45.754455+0000 osd.4 (osd.4) 78 : cluster 0 6.17 scrub ok 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[21467]: cluster 2024-04-06T03:24:45.909470+0000 osd.7 (osd.7) 105 : cluster 0 2.b scrub starts 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[21467]: cluster 2024-04-06T03:24:45.911972+0000 osd.7 (osd.7) 106 : cluster 0 2.b scrub ok 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.023101+0000 mgr.y (mgr.24368) 146 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[29221]: cluster 2024-04-06T03:24:45.752092+0000 osd.4 (osd.4) 77 : cluster 0 6.17 scrub starts 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[29221]: cluster 2024-04-06T03:24:45.754455+0000 osd.4 (osd.4) 78 : cluster 0 6.17 scrub ok 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[29221]: cluster 2024-04-06T03:24:45.909470+0000 osd.7 (osd.7) 105 : cluster 0 2.b scrub starts 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[29221]: cluster 2024-04-06T03:24:45.911972+0000 osd.7 (osd.7) 106 : cluster 0 2.b scrub ok 2024-04-06T03:24:47.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:47 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.023101+0000 mgr.y (mgr.24368) 146 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:47.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:47 smithi153 bash[23852]: cluster 2024-04-06T03:24:45.752092+0000 osd.4 (osd.4) 77 : cluster 0 6.17 scrub starts 2024-04-06T03:24:47.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:47 smithi153 bash[23852]: cluster 2024-04-06T03:24:45.754455+0000 osd.4 (osd.4) 78 : cluster 0 6.17 scrub ok 2024-04-06T03:24:47.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:47 smithi153 bash[23852]: cluster 2024-04-06T03:24:45.909470+0000 osd.7 (osd.7) 105 : cluster 0 2.b scrub starts 2024-04-06T03:24:47.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:47 smithi153 bash[23852]: cluster 2024-04-06T03:24:45.911972+0000 osd.7 (osd.7) 106 : cluster 0 2.b scrub ok 2024-04-06T03:24:47.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:47 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.023101+0000 mgr.y (mgr.24368) 146 : cluster 0 pgmap v115: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.150745+0000 osd.0 (osd.0) 87 : cluster 0 3.17 deep-scrub starts 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.160695+0000 osd.0 (osd.0) 88 : cluster 0 3.17 deep-scrub ok 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.959368+0000 osd.5 (osd.5) 101 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.961624+0000 osd.5 (osd.5) 102 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.995879+0000 osd.6 (osd.6) 61 : cluster 0 5.11 scrub starts 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:46.998335+0000 osd.6 (osd.6) 62 : cluster 0 5.11 scrub ok 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:47.064020+0000 osd.3 (osd.3) 105 : cluster 0 3.b scrub starts 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:47.067930+0000 osd.3 (osd.3) 106 : cluster 0 3.b scrub ok 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:47.113993+0000 osd.0 (osd.0) 89 : cluster 0 3.1b scrub starts 2024-04-06T03:24:48.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[29221]: cluster 2024-04-06T03:24:47.117817+0000 osd.0 (osd.0) 90 : cluster 0 3.1b scrub ok 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.150745+0000 osd.0 (osd.0) 87 : cluster 0 3.17 deep-scrub starts 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.160695+0000 osd.0 (osd.0) 88 : cluster 0 3.17 deep-scrub ok 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.959368+0000 osd.5 (osd.5) 101 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.961624+0000 osd.5 (osd.5) 102 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.995879+0000 osd.6 (osd.6) 61 : cluster 0 5.11 scrub starts 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:46.998335+0000 osd.6 (osd.6) 62 : cluster 0 5.11 scrub ok 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:47.064020+0000 osd.3 (osd.3) 105 : cluster 0 3.b scrub starts 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:47.067930+0000 osd.3 (osd.3) 106 : cluster 0 3.b scrub ok 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:47.113993+0000 osd.0 (osd.0) 89 : cluster 0 3.1b scrub starts 2024-04-06T03:24:48.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:48 smithi050 bash[21467]: cluster 2024-04-06T03:24:47.117817+0000 osd.0 (osd.0) 90 : cluster 0 3.1b scrub ok 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.150745+0000 osd.0 (osd.0) 87 : cluster 0 3.17 deep-scrub starts 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.160695+0000 osd.0 (osd.0) 88 : cluster 0 3.17 deep-scrub ok 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.959368+0000 osd.5 (osd.5) 101 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.961624+0000 osd.5 (osd.5) 102 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.995879+0000 osd.6 (osd.6) 61 : cluster 0 5.11 scrub starts 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:46.998335+0000 osd.6 (osd.6) 62 : cluster 0 5.11 scrub ok 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:47.064020+0000 osd.3 (osd.3) 105 : cluster 0 3.b scrub starts 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:47.067930+0000 osd.3 (osd.3) 106 : cluster 0 3.b scrub ok 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:47.113993+0000 osd.0 (osd.0) 89 : cluster 0 3.1b scrub starts 2024-04-06T03:24:48.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:48 smithi153 bash[23852]: cluster 2024-04-06T03:24:47.117817+0000 osd.0 (osd.0) 90 : cluster 0 3.1b scrub ok 2024-04-06T03:24:49.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:49 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.023796+0000 mgr.y (mgr.24368) 147 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:49.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:49 smithi050 bash[21467]: audit 2024-04-06T03:24:48.389762+0000 mon.a (mon.0) 976 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:49 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.023796+0000 mgr.y (mgr.24368) 147 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:49.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:49 smithi050 bash[29221]: audit 2024-04-06T03:24:48.389762+0000 mon.a (mon.0) 976 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:49.483 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:24:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:24:49.540 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:49.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:24:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:24:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:49 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.023796+0000 mgr.y (mgr.24368) 147 : cluster 0 pgmap v116: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:24:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:49 smithi153 bash[23852]: audit 2024-04-06T03:24:48.389762+0000 mon.a (mon.0) 976 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.735458+0000 osd.4 (osd.4) 79 : cluster 0 3.14 scrub starts 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.740361+0000 osd.4 (osd.4) 80 : cluster 0 3.14 scrub ok 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.864143+0000 osd.7 (osd.7) 107 : cluster 0 3.e scrub starts 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.868532+0000 osd.7 (osd.7) 108 : cluster 0 3.e scrub ok 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.946708+0000 osd.6 (osd.6) 63 : cluster 0 2.17 scrub starts 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: cluster 2024-04-06T03:24:48.948985+0000 osd.6 (osd.6) 64 : cluster 0 2.17 scrub ok 2024-04-06T03:24:50.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[29221]: audit 2024-04-06T03:24:49.122938+0000 mgr.y (mgr.24368) 148 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.735458+0000 osd.4 (osd.4) 79 : cluster 0 3.14 scrub starts 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.740361+0000 osd.4 (osd.4) 80 : cluster 0 3.14 scrub ok 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.864143+0000 osd.7 (osd.7) 107 : cluster 0 3.e scrub starts 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.868532+0000 osd.7 (osd.7) 108 : cluster 0 3.e scrub ok 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.946708+0000 osd.6 (osd.6) 63 : cluster 0 2.17 scrub starts 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: cluster 2024-04-06T03:24:48.948985+0000 osd.6 (osd.6) 64 : cluster 0 2.17 scrub ok 2024-04-06T03:24:50.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:50 smithi050 bash[21467]: audit 2024-04-06T03:24:49.122938+0000 mgr.y (mgr.24368) 148 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.735458+0000 osd.4 (osd.4) 79 : cluster 0 3.14 scrub starts 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.740361+0000 osd.4 (osd.4) 80 : cluster 0 3.14 scrub ok 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.864143+0000 osd.7 (osd.7) 107 : cluster 0 3.e scrub starts 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.868532+0000 osd.7 (osd.7) 108 : cluster 0 3.e scrub ok 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.946708+0000 osd.6 (osd.6) 63 : cluster 0 2.17 scrub starts 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: cluster 2024-04-06T03:24:48.948985+0000 osd.6 (osd.6) 64 : cluster 0 2.17 scrub ok 2024-04-06T03:24:50.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:50 smithi153 bash[23852]: audit 2024-04-06T03:24:49.122938+0000 mgr.y (mgr.24368) 148 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:24:51.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:51 smithi050 bash[21467]: cluster 2024-04-06T03:24:50.025283+0000 mgr.y (mgr.24368) 149 : cluster 0 pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:24:51.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:51 smithi050 bash[29221]: cluster 2024-04-06T03:24:50.025283+0000 mgr.y (mgr.24368) 149 : cluster 0 pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:24:51.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:51 smithi153 bash[23852]: cluster 2024-04-06T03:24:50.025283+0000 mgr.y (mgr.24368) 149 : cluster 0 pgmap v117: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[21467]: cluster 2024-04-06T03:24:50.542528+0000 osd.2 (osd.2) 69 : cluster 0 5.b scrub starts 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[21467]: cluster 2024-04-06T03:24:50.545063+0000 osd.2 (osd.2) 70 : cluster 0 5.b scrub ok 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.041141+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.044290+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.107850+0000 osd.3 (osd.3) 107 : cluster 0 2.1e scrub starts 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.110541+0000 osd.3 (osd.3) 108 : cluster 0 2.1e scrub ok 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[29221]: cluster 2024-04-06T03:24:50.542528+0000 osd.2 (osd.2) 69 : cluster 0 5.b scrub starts 2024-04-06T03:24:52.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[29221]: cluster 2024-04-06T03:24:50.545063+0000 osd.2 (osd.2) 70 : cluster 0 5.b scrub ok 2024-04-06T03:24:52.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.041141+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-04-06T03:24:52.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.044290+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-04-06T03:24:52.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.107850+0000 osd.3 (osd.3) 107 : cluster 0 2.1e scrub starts 2024-04-06T03:24:52.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:52 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.110541+0000 osd.3 (osd.3) 108 : cluster 0 2.1e scrub ok 2024-04-06T03:24:52.542 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:52 smithi153 bash[23852]: cluster 2024-04-06T03:24:50.542528+0000 osd.2 (osd.2) 69 : cluster 0 5.b scrub starts 2024-04-06T03:24:52.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:52 smithi153 bash[23852]: cluster 2024-04-06T03:24:50.545063+0000 osd.2 (osd.2) 70 : cluster 0 5.b scrub ok 2024-04-06T03:24:52.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:52 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.041141+0000 osd.1 (osd.1) 99 : cluster 0 2.6 scrub starts 2024-04-06T03:24:52.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:52 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.044290+0000 osd.1 (osd.1) 100 : cluster 0 2.6 scrub ok 2024-04-06T03:24:52.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:52 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.107850+0000 osd.3 (osd.3) 107 : cluster 0 2.1e scrub starts 2024-04-06T03:24:52.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:52 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.110541+0000 osd.3 (osd.3) 108 : cluster 0 2.1e scrub ok 2024-04-06T03:24:52.609 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mon.a", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster": "ceph", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "admin_socket_mode": "", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_client_required": "cephx, none", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_cluster_required": "cephx", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_debug": "true", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_service_required": "cephx", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-06T03:24:52.610 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_supported": "", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio": "true", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_poll_ms": "250", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_reap_max": "16", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_async_discard": "false", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_block_size": "4096", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio": "false", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_inflight_ios": "false", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_enable_discard": "false", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_flock_retry": "3", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-06T03:24:52.611 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_inject_crash": "0", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring_hipri": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_type": "", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_alloc_size": "1048576", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_allocator": "hybrid", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_buffered_io": "true", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_for_zeros": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-06T03:24:52.612 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_compact_log_sync": "false", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_max_log_runway": "4194304", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_max_prefetch": "1048576", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_min_flush_size": "524288", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_min_log_runway": "1048576", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_replay_recovery": "false", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-06T03:24:52.613 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_sync_write": "false", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_allocation_from_file": "true", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_allocator": "hybrid", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-06T03:24:52.614 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_create": "true", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_create": "false", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_path": "", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_size": "0", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_path": "", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_preallocate_file": "false", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_size": "107374182400", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_create": "false", 2024-04-06T03:24:52.615 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_path": "", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_size": "100663296", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs": "true", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-06T03:24:52.616 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_autotune": "true", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size": "0", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_type": "2q", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-06T03:24:52.617 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_clone_cow": "true", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_mode": "none", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_csum_type": "crc32c", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_freelist": "false", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-06T03:24:52.618 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_small_allocations": "0", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_default_buffered_read": "true", 2024-04-06T03:24:52.619 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_default_buffered_write": "false", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fail_eio": "false", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-06T03:24:52.620 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mount": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_umount": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_ignore_data_csum": "false", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-06T03:24:52.621 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_op_age": "5.000000", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_alloc_size": "0", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size": "0", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_deferred_txc": "32", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size": "0", 2024-04-06T03:24:52.622 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_nid_prealloc": "1024", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_retry_disk_reads": "3", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_cf": "true", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-06T03:24:52.623 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_mem": "512", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_tracing": "false", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-06T03:24:52.624 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_zero_block_detection": "false", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-06T03:24:52.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_cluster_require_signatures": "false", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_cluster_require_version": "2", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_require_signatures": "false", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_require_version": "2", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_service_require_signatures": "false", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_service_require_version": "2", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_sign_messages": "true", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "chdir": "", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_acl_type": "", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_asio_thread_count": "2", 2024-04-06T03:24:52.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_cache_mid": "0.750000", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_cache_size": "16384", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_caps_release_delay": "5", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_check_pool_perm": "true", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_force_sync_read": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_getattr_caps": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_inject_tick_delay": "0", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_die_on_failed_remount": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_dirsize_rbytes": "true", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_force_lazyio": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_fs": "", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_inject_release_failure": "false", 2024-04-06T03:24:52.627 INFO:teuthology.orchestra.run.smithi050.stdout: "client_max_inline_size": "4096", 2024-04-06T03:24:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-06T03:24:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mds_namespace": "", 2024-04-06T03:24:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_metadata": "", 2024-04-06T03:24:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_gid": "-1", 2024-04-06T03:24:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_timeout": "300", 2024-04-06T03:24:52.628 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_uid": "-1", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mountpoint": "/", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_notify_timeout": "10", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc": "true", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_dirty": "104857600", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_objects": "1000", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_size": "209715200", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_target_dirty": "8388608", 2024-04-06T03:24:52.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_permissions": "true", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_quota": "true", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_quota_df": "true", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_max_bytes": "0", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_max_periods": "4", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_min": "131072", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_reconnect_stale": "false", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_shutdown_timeout": "30", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_snapdir": ".snap", 2024-04-06T03:24:52.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_tick_interval": "1", 2024-04-06T03:24:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_trace": "", 2024-04-06T03:24:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_try_dentry_invalidate": "false", 2024-04-06T03:24:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_use_faked_inos": "false", 2024-04-06T03:24:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_use_random_mds": "false", 2024-04-06T03:24:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog": "false", 2024-04-06T03:24:52.631 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog_port": "12201", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_monitors": "default=true", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog": "false", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog_level": "info", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "-", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_network": "", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_network_interface": "", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_isal": "false", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_level": "5", 2024-04-06T03:24:52.632 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_winsize": "-15", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zstd_level": "1", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a592ef0f2cc60f3f7bee762d48a80bb237e35213427028fdb29a2b5d27584e3c", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_alien_op_num_threads": "6", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_seastar_cpu_cores": "", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location_hook": "", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location_hook_timeout": "10", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "daemonize": "false", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "daos_pool": "tank", 2024-04-06T03:24:52.633 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_alienstore": "0/5", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_allow_any_pool_priority": "false", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asok": "1/5", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asok_assert_abort": "false", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asserts_on_shutdown": "true", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_auth": "1/5", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bdev": "1/3", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bluefs": "1/5", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bluestore": "1/5", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_buffer": "0/1", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ceph_exporter": "1/5", 2024-04-06T03:24:52.634 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cephfs_mirror": "0/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cephsqlite": "0/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_client": "0/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_compressor": "1/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_context": "0/1", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crimson_interrupt": "0/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crush": "1/1", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crypto": "1/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cyanstore": "0/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_deliberately_leak_memory": "false", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_disable_randomized_ping": "false", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_dpdk": "1/5", 2024-04-06T03:24:52.635 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_eventtrace": "1/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_filer": "0/1", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_filestore": "1/3", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_finisher": "1/1", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_fuse": "1/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_heartbeat_testing_span": "0", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_heartbeatmap": "1/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_javaclient": "1/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_journal": "1/3", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_journaler": "0/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_kstore": "1/5", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_lockdep": "0/1", 2024-04-06T03:24:52.636 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mclock": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_balancer": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_locker": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_log": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_log_expire": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_migrator": "1/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_quiesce": "3/5", 2024-04-06T03:24:52.637 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_memstore": "1/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mgr": "2/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mgrc": "1/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mon": "20/20", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_monc": "0/10", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ms": "1/1", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_none": "0/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objclass": "0/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objectcacher": "0/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objecter": "0/1", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_optracker": "0/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_osd": "1/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_paxos": "20/20", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_perfcounter": "1/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_prioritycache": "1/5", 2024-04-06T03:24:52.638 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rados": "0/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd": "0/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_mirror": "0/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_pwl": "0/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_replay": "0/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_refs": "0/0", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_reserver": "1/1", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw": "1/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_access": "1/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_datacache": "1/5", 2024-04-06T03:24:52.639 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_dbstore": "1/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_flight": "1/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_sync": "1/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rocksdb": "4/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_backref": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_cache": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_cleaner": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_device": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_epm": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_journal": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_lba": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_odata": "0/5", 2024-04-06T03:24:52.640 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_omap": "0/5", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_onode": "0/5", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_t": "0/5", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_tm": "0/5", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_striper": "0/1", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_test": "0/5", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_throttle": "1/1", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_timer": "0/1", 2024-04-06T03:24:52.641 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_tp": "0/5", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_trace": "1/5", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "device_failure_prediction_mode": "none", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_graylog": "false", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_journald": "false", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_stderr": "true", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_syslog": "false", 2024-04-06T03:24:52.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event_tracing": "false", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_addr": "0.0.0.0", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_http_port": "9926", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_prio_limit": "5", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_sort_metrics": "true", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_stats_period": "5", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "fake_statfs_for_testing": "0", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "fatal_signal_handlers": "true", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filer_max_purge_ops": "10", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filer_max_truncate_ops": "128", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_apply_finisher_threads": "1", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_blackhole": "false", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_btrfs_clone_range": "true", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_btrfs_snap": "true", 2024-04-06T03:24:52.643 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_caller_concurrency": "10", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_collect_device_partition_information": "true", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_commit_timeout": "600.000000", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_inject_read_err": "false", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_omap_check": "false", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_verify_split": "false", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_dump_file": "", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fadvise": "true", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fail_eio": "true", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fd_cache_shards": "16", 2024-04-06T03:24:52.644 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fd_cache_size": "128", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fiemap": "false", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fiemap_threshold": "4096", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_inject_stall": "0", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_parallel": "false", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_trailing": "false", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_writeahead": "false", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_kill_at": "0", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-06T03:24:52.645 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs": "0", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size": "0", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_merge_threshold": "-10", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_odsync_write": "false", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_backend": "rocksdb", 2024-04-06T03:24:52.646 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_backend_path": "", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_thread_timeout": "60", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_threads": "2", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_punch_hole": "false", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-06T03:24:52.647 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_ops": "50", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_seek_data_hole": "false", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_sloppy_crc": "false", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_splice": "false", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_split_multiple": "2", 2024-04-06T03:24:52.648 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_split_rand_factor": "20", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_update_to": "1000", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_enable": "true", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-06T03:24:52.649 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_xfs_extsize": "false", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_zfs_snap": "false", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fio_dir": "/tmp/fio", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_allow_other": "true", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_atomic_o_trunc": "true", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_big_writes": "true", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_debug": "false", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_default_permissions": "false", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_disable_pagecache": "false", 2024-04-06T03:24:52.650 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_max_write": "0", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_multithreaded": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_require_active_mds": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_set_user_groups": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_move": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_read": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_write": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_use_invalidate_cb": "true", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "gss_target_name": "ceph", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_file": "", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_inject_failure": "0", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_interval": "5", 2024-04-06T03:24:52.651 INFO:teuthology.orchestra.run.smithi050.stdout: "host": "smithi050", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-06T03:24:52.652 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "inject_early_sigterm": "false", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "jaeger_agent_port": "6799", 2024-04-06T03:24:52.653 INFO:teuthology.orchestra.run.smithi050.stdout: "jaeger_tracing_enable": "false", 2024-04-06T03:24:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_aio": "true", 2024-04-06T03:24:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_align_min_size": "65536", 2024-04-06T03:24:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_block_align": "true", 2024-04-06T03:24:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_block_size": "4096", 2024-04-06T03:24:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_dio": "true", 2024-04-06T03:24:52.654 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_discard": "false", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_force_aio": "false", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_ignore_corruption": "false", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_max_write_bytes": "10485760", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_max_write_entries": "100", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_replay_from": "0", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_write_header_frequency": "0", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_zero_on_create": "false", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_prefetch_periods": "10", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_prezero_periods": "5", 2024-04-06T03:24:52.655 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_write_head_interval": "15", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "keyfile": "", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_backend": "rocksdb", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_default_stripe_size": "65536", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_fsck_on_mount": "false", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_max_bytes": "67108864", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_max_ops": "512", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_nid_prealloc": "1024", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_onode_map_size": "1024", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_sync_submit_transaction": "false", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_sync_transaction": "false", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "librados_thread_count": "2", 2024-04-06T03:24:52.656 INFO:teuthology.orchestra.run.smithi050.stdout: "lockdep": "false", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "lockdep_force_backtrace": "false", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_coarse_timestamps": "true", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_flush_on_exit": "false", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_graylog_host": "127.0.0.1", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_graylog_port": "12201", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_max_new": "1000", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_max_recent": "10000", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_stderr_prefix": "debug ", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_stop_at_utilization": "0.970000", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": "true", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_graylog": "false", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_journald": "false", 2024-04-06T03:24:52.657 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_stderr": "false", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_syslog": "false", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "max_rotating_auth_attempts": "10", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_action_on_write_error": "1", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_alternate_name_max": "8192", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_asio_thread_count": "2", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_export_pin": "true", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_dirs": "true", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_interval": "5", 2024-04-06T03:24:52.658 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_interval": "10", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_max": "-1", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_max_until": "-1", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_merge_size": "50", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_midchunk": "0.300000", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_min_start": "0.200000", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_minchunk": "0.001000", 2024-04-06T03:24:52.659 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_mode": "0", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_need_max": "1.200000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_need_min": "0.800000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_overload_epochs": "2", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_bits": "3", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_size": "10000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_target_decay": "10.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_grace": "15.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_interval": "4.000000", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-06T03:24:52.660 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_mid": "0.700000", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_delay": "0", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_release_free_interval": "10", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_reservation": "0.050000", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_interval": "1", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_threshold": "262144", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-06T03:24:52.661 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_prealloc_inos": "1000", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_connect_bootstrapping": "false", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_damage_table_max_entries": "10000", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_auth_pins": "false", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_frag": "false", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_scatterstat": "false", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_subtrees": "false", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_decay_halflife": "5.000000", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_default_dir_hash": "2", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_defer_session_stale": "true", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_deny_all_reconnect": "false", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_keys_per_op": "16384", 2024-04-06T03:24:52.662 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_max_commit_size": "10", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_max_entries": "0", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_prefetch": "true", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_on_map": "false", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_early_reply": "true", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_enable_op_tracker": "true", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_enforce_unique_name": "true", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_random": "true", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-06T03:24:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_health_summarize_threshold": "10", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_health_dummy": "false", 2024-04-06T03:24:52.664 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_migrator_session_race": "false", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_join_fs": "", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_journal_format": "1", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_create_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_export_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_import_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_expire_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_replay_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_link_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_mdstable_at": "0", 2024-04-06T03:24:52.665 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_openc_at": "0", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_rename_at": "0", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_shutdown_at": "0", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_event_large_threshold": "524288", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_events_per_segment": "1024", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_max_events": "-1", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_max_segments": "128", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_pause": "false", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_segment_size": "0", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-06T03:24:52.666 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_trim_threshold": "128", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_warn_factor": "2.000000", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_caps_per_client": "1048576", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_completed_flushes": "100000", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_completed_requests": "100000", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_export_size": "20971520", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_file_recover": "32", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_files": "64", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_ops": "8192", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-06T03:24:52.667 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_snaps_per_dir": "100", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_metrics_update_interval": "2", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_min_caps_per_client": "100", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_min_caps_working_set": "10000", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_numa_node": "-1", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_complaint_time": "30.000000", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_duration": "600", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_size": "20", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_slow_op_size": "20", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_log_threshold": "5", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_ping_grace": "15", 2024-04-06T03:24:52.668 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_ping_interval": "5", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_caps": "30000", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_warning_threshold": "262144", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_replay_interval": "1.000000", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-06T03:24:52.669 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_root_ino_gid": "0", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_root_ino_uid": "0", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_shutdown_check": "0", 2024-04-06T03:24:52.670 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_skip_ino": "0", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_max_uid": "4294967294", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_min_uid": "0", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_rstat": "false", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_standby_replay_damaged": "false", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_symlink_recovery": "true", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_thrash_exports": "0", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_thrash_fragments": "0", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_tick_interval": "5.000000", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_valgrind_exit": "false", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_verify_backtrace": "1", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_verify_scatter": "false", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_wipe_sessions": "false", 2024-04-06T03:24:52.671 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool_debug": "false", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_device_bytes": "1073741824", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_page_set": "false", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_page_size": "65536", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_bytes": "134217728", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_messages": "512", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_disabled_modules": "", 2024-04-06T03:24:52.672 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_max_pg_num_change": "128", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mds_bytes": "134217728", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mds_messages": "128", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mon_bytes": "134217728", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mon_messages": "128", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_osd_bytes": "536870912", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_osd_messages": "8192", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_pool": "true", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_standby_modules": "true", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_stats_period": "5", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_stats_threshold": "5", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_tick_period": "2", 2024-04-06T03:24:52.673 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_allow_pool_delete": "true", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_allow_pool_size_one": "false", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_auth_validate_all_caps": "true", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_bytes": "104857600", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_directed_command_retry": "2", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_parallel": "3", 2024-04-06T03:24:52.674 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_log_interval": "1.000000", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_ping_interval": "10.000000", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_target_rank": "-1", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_file_level": "debug", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_file": "true", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_journald": "false", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_stderr": "true", 2024-04-06T03:24:52.675 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_bootstrap": "false", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_start": "false", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_trim": "true", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cpu_threads": "4", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_crush_min_required_version": "hammer", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_daemon_bytes": "419430400", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_avail_crit": "5", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_avail_warn": "5", 2024-04-06T03:24:52.676 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_size_warn": "16106127360", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_json": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_transactions": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_extra_checks": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_reef": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_squid": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_down_mkfs_grace": "60", 2024-04-06T03:24:52.677 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_down_uptime_grace": "60", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_election_default_strategy": "1", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_election_timeout": "5.000000", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ping_divisor": "2", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_enable_op_tracker": "true", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_fake_pool_delete": "false", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_force_quorum_join": "false", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_globalid_prealloc": "10000", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_detail_to_clog": "true", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_log_update_period": "5", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_max_detail": "50", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog": "true", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog_interval": "600", 2024-04-06T03:24:52.678 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_host": "[v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0]", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_host_override": "", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_initial_members": "", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease": "5.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_full_interval": "50", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_max": "10000", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_max_summary": "50", 2024-04-06T03:24:52.679 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_log_epochs": "500", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_osd": "10000", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_pg_per_osd": "10000", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_pool_pg_num": "65536", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_force_trim_to": "0", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_skip_sanity": "false", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_memory_autotune": "true", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_memory_target": "2147483648", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_beacon_grace": "30", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_digest_period": "5", 2024-04-06T03:24:52.680 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_inactive_grace": "60", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_min_osdmap_epochs": "50", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_complaint_time": "30", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_duration": "600", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_size": "20", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_slow_op_size": "20", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_log_threshold": "5", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_in": "false", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-06T03:24:52.681 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_cache_size": "500", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_down_out_interval": "600", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_force_trim_to": "0", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-06T03:24:52.682 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_down_reporters": "2", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_report_timeout": "900", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-06T03:24:52.683 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_stuck_threshold": "60", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_probe_timeout": "2.000000", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_max_change": "0.050000", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_max_osds": "4", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-06T03:24:52.684 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_interval": "86400", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_max_keys": "100", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_timeout": "300", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_session_timeout": "300", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_smart_report_timeout": "5", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stat_smooth_intervals": "6", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_pool_min_size": "2", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_pool_size": "4", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-06T03:24:52.685 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_debug": "false", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_provider_kill_at": "0", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_requester_kill_at": "0", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_timeout": "60.000000", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_target_pg_per_osd": "100", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_tick_interval": "5", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_timecheck_interval": "300.000000", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_use_min_delay_socket": "false", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_older_version_delay": "604800", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-06T03:24:52.686 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_misplaced": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_older_version": "true", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-06T03:24:52.687 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": "", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_tracing_enabled": "false", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_op_threads": "3", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_cm": "false", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_device_name": "", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_dscp": "96", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-06T03:24:52.688 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_local_gid": "", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_port_num": "1", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_sl": "3", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_support_srq": "true", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_type": "ib", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_reap_threshold": "5", 2024-04-06T03:24:52.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_before_connect": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_ipv4": "true", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_ipv6": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_msgr1": "true", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_msgr2": "true", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_port_max": "7568", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_port_min": "6800", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_retry_count": "3", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_retry_delay": "5", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_client": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mds": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mgr": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mon": "false", 2024-04-06T03:24:52.690 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_osd": "false", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_client_mode": "crc secure", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_cluster_mode": "crc secure", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_cluster_type": "", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_compress_secure": "false", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_connection_idle_timeout": "900", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_connection_ready_timeout": "10", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_crc_data": "true", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_crc_header": "true", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_bad_msg": "false", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_bug": "true", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_old_message": "true", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_skipped_message": "false", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-06T03:24:52.691 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_coremask": "0xF", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_enable_tso": "true", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hugepages": "", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_lro": "true", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_memory_channel": "4", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_pmd": "", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_port_id": "0", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-06T03:24:52.692 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dump_on_send": "false", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_initial_backoff": "0.200000", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_max": "1.000000", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_type": "", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_network_congestion": "0", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_socket_failures": "0", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_learn_addr_from_peer": "true", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_max_accept_failures": "4", 2024-04-06T03:24:52.693 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_max_backoff": "15.000000", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_client_mode": "secure crc", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_service_mode": "secure crc", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compress_min_size": "1024", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compress_mode": "none", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_pq_min_cost": "65536", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_public_type": "", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_service_mode": "crc secure", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_listen_backlog": "512", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_nodelay": "true", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_rcvbuf": "0", 2024-04-06T03:24:52.694 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_type": "async+posix", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "no_config_file": "false", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_completion_locks_per_session": "32", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inflight_ops": "1024", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_tick_interval": "5.000000", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_timeout": "10.000000", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objectstore_blackhole": "false", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "openssl_engine_opts": "", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_delay_time": "5.000000", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_hist_halflife": "1000", 2024-04-06T03:24:52.695 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_max_low_ops": "2", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_max_ops": "4", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_slop": "0.020000", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_asio_thread_count": "2", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_async_recovery_min_cost": "100", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_scan_max": "512", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_scan_min": "64", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_degraded": "false", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_peering": "false", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_unfound": "true", 2024-04-06T03:24:52.696 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_beacon_report_interval": "300", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_duration": "30", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_max_block_size": "67108864", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_blkin_trace_all": "false", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_check_for_log_corruption": "false", 2024-04-06T03:24:52.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_update_on_start": "true", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_message_cap": "256", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_message_size_cap": "524288000", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_op_priority": "63", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_watch_timeout": "30", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_command_max_records": "256", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_compact_on_start": "false", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-06T03:24:52.698 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_update_on_start": "true", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_update_weight_set": "true", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_feed_pullee": "-1", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_misdirected_ops": "false", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_no_acting_change": "false", 2024-04-06T03:24:52.699 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_no_purge_strays": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_op_order": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_shutdown": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_trim_objects": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_snaps": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-06T03:24:52.700 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_keys": "1024", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_stride": "524288", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_default_notify_timeout": "30", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep": "0.000000", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_discard_disconnected_ops": "true", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_enable_op_tracker": "true", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-06T03:24:52.701 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_info": "true", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown": "true", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_function_tracing": "false", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_grace": "20", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_interval": "6", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_peers": "10", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_size": "2000", 2024-04-06T03:24:52.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_stale": "600", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_max_size": "100000", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_min_size": "1000", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal_size": "5120", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_kill_backfill_at": "0", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_cache_size": "50", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_dedup": "true", 2024-04-06T03:24:52.703 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_message_max": "40", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_share_max_epochs": "40", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_attr_name_len": "100", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_attr_size": "0", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_backfills": "1", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_markdown_count": "5", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_markdown_period": "600", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_name_len": "2048", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_namespace_len": "256", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_size": "134217728", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_blocked_by": "16", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_log_entries": "10000", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-06T03:24:52.704 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pgls": "1024", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_push_cost": "8388608", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_push_objects": "10", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_scrubs": "3", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_trimming_pgs": "2", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_write_op_reply_len": "64", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_write_size": "90", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-06T03:24:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_profile": "balanced", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_base": "805306368", 2024-04-06T03:24:52.706 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_cache_min": "134217728", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target": "4294967296", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target_autotune": "false", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_min_pg_log_entries": "250", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_min_recovery_priority": "0", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_report_interval": "5", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_num_cache_shards": "32", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_num_op_tracker_shard": "32", 2024-04-06T03:24:52.707 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_auto_affinity": "true", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_node": "-1", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_prefer_iface": "true", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objecter_finishers": "1", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore": "bluestore", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore_fuse": "false", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore_tracing": "false", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_complaint_time": "30.000000", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_duration": "600", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_size": "20", 2024-04-06T03:24:52.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_slow_op_size": "20", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_log_threshold": "5", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards": "0", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards_hdd": "5", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards_ssd": "8", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_pq_min_cost": "65536", 2024-04-06T03:24:52.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-06T03:24:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_queue_cut_off": "high", 2024-04-06T03:24:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-06T03:24:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_thread_timeout": "15", 2024-04-06T03:24:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_open_classes_on_start": "true", 2024-04-06T03:24:52.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_os_flags": "0", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_peering_op_priority": "255", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_delete_cost": "1048576", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_delete_priority": "5", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_trim_max": "10000", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_trim_min": "100", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-06T03:24:52.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_crimson": "false", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flags": "0", 2024-04-06T03:24:52.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_min_size": "0", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pg_num": "32", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pgp_num": "0", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_read_ratio": "70", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_size": "3", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_type": "replicated", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-06T03:24:52.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_push_per_object_cost": "1000", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recover_clone_overlap": "true", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_cost": "20971520", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active": "0", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_single_start": "1", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_op_priority": "3", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-06T03:24:52.714 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_priority": "5", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep": "0.000000", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_repair_during_recovery": "false", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_requested_scrub_priority": "120", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_auto_repair": "false", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_begin_hour": "0", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_begin_week_day": "0", 2024-04-06T03:24:52.715 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_chunk_max": "25", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_chunk_min": "5", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_cost": "52428800", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_during_recovery": "false", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_end_hour": "0", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_end_week_day": "0", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_event_cost": "4096", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_invalid_stats": "true", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_max_preemptions": "5", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-06T03:24:52.716 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_priority": "5", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_sleep": "0.000000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_skip_data_digest": "false", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_smart_report_timeout": "5", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_cost": "1048576", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_priority": "5", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-06T03:24:52.717 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_target_transaction_size": "30", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-06T03:24:52.718 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tracing": "false", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_use_stale_snap": "false", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "osdc_blkin_trace_all": "false", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_kill_at": "0", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_max_join_drift": "10", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_min": "500", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_min_wait": "0.050000", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_propose_interval": "1.000000", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_max": "500", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_min": "10", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_stash_full_interval": "25", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_trim_max": "500", 2024-04-06T03:24:52.719 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_trim_min": "250", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "perf": "true", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "pid_file": "", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "-", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrv": "", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_bind_addr": "-", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_network": "", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "public_network_interface": "", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "qat_compressor_enabled": "false", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "qat_compressor_session_max_number": "256", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_mon_op_timeout": "0", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_osd_op_timeout": "0", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_tracing": "false", 2024-04-06T03:24:52.720 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_atime_update_interval": "60", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_balance_parent_reads": "false", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_balance_snap_reads": "false", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blkin_trace_all": "false", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache": "true", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_policy": "writearound", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_size": "33554432", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-06T03:24:52.721 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_clone_copy_on_read": "false", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_compression_hint": "none", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_concurrent_management_ops": "10", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_clone_format": "auto", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_data_pool": "", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_features": "61", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_format": "2", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_map_options": "", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_order": "22", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_pool": "rbd", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_stripe_count": "0", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_stripe_unit": "0", 2024-04-06T03:24:52.722 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_enable_alloc_hint": "true", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_io_scheduler": "simple", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-06T03:24:52.723 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_order": "24", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_pool": "", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_splay_width": "4", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_localize_parent_reads": "false", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_localize_snap_reads": "false", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-06T03:24:52.724 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-06T03:24:52.725 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mtime_update_interval": "60", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_non_blocking_aio": "true", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_op_thread_timeout": "60", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_op_threads": "1", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_parent_cache_enabled": "false", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_plugins": "", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_burst": "0", 2024-04-06T03:24:52.726 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_limit": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_exclude_ops": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_burst": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_limit": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-06T03:24:52.727 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_read_from_replica_policy": "default", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_skip_partial_discard": "true", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_tracing": "false", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_validate_names": "true", 2024-04-06T03:24:52.728 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_validate_pool": "true", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "restapi_base_url": "", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "restapi_log_level": "", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_acl_grants_max_num": "100", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_admin_entry": "admin", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_backend_store": "rados", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_barbican_url": "", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_beast_enable_async": "true", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_counters_cache": "false", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-06T03:24:52.729 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_enabled": "true", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_expiry_interval": "900", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_lru_size": "10000", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_config_store": "rados", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_content_length_compat": "false", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_copy_obj_progress": "true", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cors_rules_max_num": "100", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cross_domain_policy": "", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_addr": "", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-06T03:24:52.730 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_password": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_username": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_require_ssl": "true", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-06T03:24:52.731 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_addr": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_auth": "token", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_namespace": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_prefix": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-06T03:24:52.732 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_token_file": "", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_buffersize": "524288", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_low_speed_time": "300", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-06T03:24:52.733 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_port": "6379", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_changes_size": "1000", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_num_shards": "128", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_window": "30", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-06T03:24:52.734 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-06T03:24:52.735 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dns_name": "", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dns_s3website_name": "", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dynamic_resharding": "true", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_gc_threads": "true", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_lc_threads": "true", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_ops_log": "false", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_quota_threads": "true", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_static_website": "false", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_usage_log": "false", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enforce_swift_acls": "true", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_exit_timeout_secs": "120", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_expose_bucket": "false", 2024-04-06T03:24:52.736 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_extended_http_attrs": "", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_filter": "none", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_frontends": "beast port=7480", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_deferred": "50", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_objs": "32", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_processor_period": "3600", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-06T03:24:52.737 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_init_timeout": "300", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_kafka_connection_idle": "300", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_domain": "", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_password": "", 2024-04-06T03:24:52.738 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_password_path": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_project": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_tenant": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_token": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_token_path": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_user": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_api_version": "2", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_domain": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_password": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_project": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_user": "", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-06T03:24:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_url": "", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_debug_interval": "-1", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_lock_max_time": "90", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_objs": "32", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_rules": "1000", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_worker": "3", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_thread_delay": "0", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_dnattr": "uid", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_searchfilter": "", 2024-04-06T03:24:52.740 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_uri": "ldaps://", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_http_headers": "", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_object_name_utc": "false", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attr_name_len": "0", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attr_size": "0", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-06T03:24:52.741 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_chunk_size": "4194304", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_listing_results": "1000", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_notify_retries": "10", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_put_param_size": "1048576", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_put_size": "5368709120", 2024-04-06T03:24:52.742 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_slo_entries": "1000", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_md_log_max_shards": "64", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_mp_lock_max_time": "600", 2024-04-06T03:24:52.745 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_max_gc": "300", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-06T03:24:52.746 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_num_async_rados_threads": "32", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_num_control_oids": "8", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_numa_node": "-1", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_chunk_size": "100", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_gc_interval": "600", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_thread_timeout": "600", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_tracing": "false", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_token": "", 2024-04-06T03:24:52.747 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_url": "", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_verify_ssl": "true", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_rados": "false", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_socket_path": "", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_push_interval": "2.000000", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-06T03:24:52.748 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_lanes": "3", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_partitions": "3", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_print_continue": "true", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_tracing": "false", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm": "", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm_id": "", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_region": "", 2024-04-06T03:24:52.749 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_request_uri": "", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_batch_size": "64", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_max_aio": "128", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_num_logs": "16", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_thread_interval": "600", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_resolve_cname": "false", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_run_sync_thread": "true", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-06T03:24:52.750 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_scheduler_type": "throttler", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_script_uri": "", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_service_provider_name": "", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_client_id": "", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_client_secret": "", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_entry": "sts", 2024-04-06T03:24:52.751 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_key": "sts", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_min_session_duration": "900", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_token_introspection_url": "", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_account_in_url": "false", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_auth_entry": "auth", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_auth_url": "", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_custom_header": "", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_need_stats": "true", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_tenant_name": "", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_token_expiration": "86400", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_url": "", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_url_prefix": "swift", 2024-04-06T03:24:52.752 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_lease_period": "120", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_thread_pool_size": "512", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-06T03:24:52.753 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_comment": "", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_createby": "", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_encoding": "", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_flag": "false", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_origin": "", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_tracker": "", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_trust_forwarded_https": "false", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-06T03:24:52.754 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_max_shards": "32", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_max_user_shards": "1", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_use_opa_authz": "false", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_counters_cache": "false", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_max_buckets": "1000", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_policies_max_num": "100", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_unique_email": "true", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_verify_ssl": "true", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-06T03:24:52.755 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone": "", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone_id": "", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup": "", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup_id": "", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgwlc_auto_session_clear": "true", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_block_size": "4096", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_size": "536870912", 2024-04-06T03:24:52.756 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_index_type": "binary_search", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_partition_filters": "false", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_perf": "false", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-06T03:24:52.757 INFO:teuthology.orchestra.run.smithi050.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "run_dir": "/var/run/ceph", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_block_create": "true", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_cache_lru_size": "67108864", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_default_max_object_size": "16777216", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_device_size": "53687091200", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_capacity": "16", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-06T03:24:52.758 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_main_device_type": "SSD", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_segment_size": "67108864", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "set_keepcaps": "false", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "setgroup": "ceph", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "setuser": "ceph", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "setuser_match_path": "", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "thp": "false", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "threadpool_default_timeout": "60", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout: "throttler_perf_counter": "true" 2024-04-06T03:24:52.759 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:24:53.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.193961+0000 osd.0 (osd.0) 91 : cluster 0 3.1 scrub starts 2024-04-06T03:24:53.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.199199+0000 osd.0 (osd.0) 92 : cluster 0 3.1 scrub ok 2024-04-06T03:24:53.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.910068+0000 osd.6 (osd.6) 65 : cluster 0 5.1f scrub starts 2024-04-06T03:24:53.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[29221]: cluster 2024-04-06T03:24:51.913167+0000 osd.6 (osd.6) 66 : cluster 0 5.1f scrub ok 2024-04-06T03:24:53.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[29221]: cluster 2024-04-06T03:24:52.026018+0000 mgr.y (mgr.24368) 150 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:53.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.193961+0000 osd.0 (osd.0) 91 : cluster 0 3.1 scrub starts 2024-04-06T03:24:53.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.199199+0000 osd.0 (osd.0) 92 : cluster 0 3.1 scrub ok 2024-04-06T03:24:53.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.910068+0000 osd.6 (osd.6) 65 : cluster 0 5.1f scrub starts 2024-04-06T03:24:53.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[21467]: cluster 2024-04-06T03:24:51.913167+0000 osd.6 (osd.6) 66 : cluster 0 5.1f scrub ok 2024-04-06T03:24:53.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:53 smithi050 bash[21467]: cluster 2024-04-06T03:24:52.026018+0000 mgr.y (mgr.24368) 150 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:53.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-06T03:24:53.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:53 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.193961+0000 osd.0 (osd.0) 91 : cluster 0 3.1 scrub starts 2024-04-06T03:24:53.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:53 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.199199+0000 osd.0 (osd.0) 92 : cluster 0 3.1 scrub ok 2024-04-06T03:24:53.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:53 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.910068+0000 osd.6 (osd.6) 65 : cluster 0 5.1f scrub starts 2024-04-06T03:24:53.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:53 smithi153 bash[23852]: cluster 2024-04-06T03:24:51.913167+0000 osd.6 (osd.6) 66 : cluster 0 5.1f scrub ok 2024-04-06T03:24:53.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:53 smithi153 bash[23852]: cluster 2024-04-06T03:24:52.026018+0000 mgr.y (mgr.24368) 150 : cluster 0 pgmap v118: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:54.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[29221]: cluster 2024-04-06T03:24:52.552149+0000 osd.2 (osd.2) 71 : cluster 0 5.6 scrub starts 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[29221]: cluster 2024-04-06T03:24:52.553895+0000 osd.2 (osd.2) 72 : cluster 0 5.6 scrub ok 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[29221]: cluster 2024-04-06T03:24:53.857731+0000 osd.7 (osd.7) 109 : cluster 0 5.9 scrub starts 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[29221]: cluster 2024-04-06T03:24:53.860945+0000 osd.7 (osd.7) 110 : cluster 0 5.9 scrub ok 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.155594+0000 osd.3 (osd.3) 109 : cluster 0 6.c scrub starts 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.157965+0000 osd.3 (osd.3) 110 : cluster 0 6.c scrub ok 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[21467]: cluster 2024-04-06T03:24:52.552149+0000 osd.2 (osd.2) 71 : cluster 0 5.6 scrub starts 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[21467]: cluster 2024-04-06T03:24:52.553895+0000 osd.2 (osd.2) 72 : cluster 0 5.6 scrub ok 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[21467]: cluster 2024-04-06T03:24:53.857731+0000 osd.7 (osd.7) 109 : cluster 0 5.9 scrub starts 2024-04-06T03:24:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[21467]: cluster 2024-04-06T03:24:53.860945+0000 osd.7 (osd.7) 110 : cluster 0 5.9 scrub ok 2024-04-06T03:24:54.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.155594+0000 osd.3 (osd.3) 109 : cluster 0 6.c scrub starts 2024-04-06T03:24:54.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:54 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.157965+0000 osd.3 (osd.3) 110 : cluster 0 6.c scrub ok 2024-04-06T03:24:54.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:54 smithi153 bash[23852]: cluster 2024-04-06T03:24:52.552149+0000 osd.2 (osd.2) 71 : cluster 0 5.6 scrub starts 2024-04-06T03:24:54.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:54 smithi153 bash[23852]: cluster 2024-04-06T03:24:52.553895+0000 osd.2 (osd.2) 72 : cluster 0 5.6 scrub ok 2024-04-06T03:24:54.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:54 smithi153 bash[23852]: cluster 2024-04-06T03:24:53.857731+0000 osd.7 (osd.7) 109 : cluster 0 5.9 scrub starts 2024-04-06T03:24:54.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:54 smithi153 bash[23852]: cluster 2024-04-06T03:24:53.860945+0000 osd.7 (osd.7) 110 : cluster 0 5.9 scrub ok 2024-04-06T03:24:54.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:54 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.155594+0000 osd.3 (osd.3) 109 : cluster 0 6.c scrub starts 2024-04-06T03:24:54.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:54 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.157965+0000 osd.3 (osd.3) 110 : cluster 0 6.c scrub ok 2024-04-06T03:24:55.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:53.792745+0000 osd.4 (osd.4) 81 : cluster 0 5.e scrub starts 2024-04-06T03:24:55.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:53.796035+0000 osd.4 (osd.4) 82 : cluster 0 5.e scrub ok 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:53.888385+0000 osd.6 (osd.6) 67 : cluster 0 2.1a scrub starts 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:53.890850+0000 osd.6 (osd.6) 68 : cluster 0 2.1a scrub ok 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.027414+0000 mgr.y (mgr.24368) 151 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.844684+0000 osd.7 (osd.7) 111 : cluster 0 2.1d scrub starts 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.847884+0000 osd.7 (osd.7) 112 : cluster 0 2.1d scrub ok 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:53.792745+0000 osd.4 (osd.4) 81 : cluster 0 5.e scrub starts 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:53.796035+0000 osd.4 (osd.4) 82 : cluster 0 5.e scrub ok 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:53.888385+0000 osd.6 (osd.6) 67 : cluster 0 2.1a scrub starts 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:53.890850+0000 osd.6 (osd.6) 68 : cluster 0 2.1a scrub ok 2024-04-06T03:24:55.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.027414+0000 mgr.y (mgr.24368) 151 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:55.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.844684+0000 osd.7 (osd.7) 111 : cluster 0 2.1d scrub starts 2024-04-06T03:24:55.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:55 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.847884+0000 osd.7 (osd.7) 112 : cluster 0 2.1d scrub ok 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:53.792745+0000 osd.4 (osd.4) 81 : cluster 0 5.e scrub starts 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:53.796035+0000 osd.4 (osd.4) 82 : cluster 0 5.e scrub ok 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:53.888385+0000 osd.6 (osd.6) 67 : cluster 0 2.1a scrub starts 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:53.890850+0000 osd.6 (osd.6) 68 : cluster 0 2.1a scrub ok 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.027414+0000 mgr.y (mgr.24368) 151 : cluster 0 pgmap v119: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.844684+0000 osd.7 (osd.7) 111 : cluster 0 2.1d scrub starts 2024-04-06T03:24:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:55 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.847884+0000 osd.7 (osd.7) 112 : cluster 0 2.1d scrub ok 2024-04-06T03:24:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:56 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.823493+0000 osd.4 (osd.4) 83 : cluster 0 6.2 deep-scrub starts 2024-04-06T03:24:56.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:56 smithi050 bash[29221]: cluster 2024-04-06T03:24:54.826468+0000 osd.4 (osd.4) 84 : cluster 0 6.2 deep-scrub ok 2024-04-06T03:24:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:56 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.823493+0000 osd.4 (osd.4) 83 : cluster 0 6.2 deep-scrub starts 2024-04-06T03:24:56.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:56 smithi050 bash[21467]: cluster 2024-04-06T03:24:54.826468+0000 osd.4 (osd.4) 84 : cluster 0 6.2 deep-scrub ok 2024-04-06T03:24:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:56 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.823493+0000 osd.4 (osd.4) 83 : cluster 0 6.2 deep-scrub starts 2024-04-06T03:24:56.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:56 smithi153 bash[23852]: cluster 2024-04-06T03:24:54.826468+0000 osd.4 (osd.4) 84 : cluster 0 6.2 deep-scrub ok 2024-04-06T03:24:57.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[21467]: cluster 2024-04-06T03:24:55.996890+0000 osd.5 (osd.5) 103 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:24:57.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[21467]: cluster 2024-04-06T03:24:55.999173+0000 osd.5 (osd.5) 104 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[21467]: cluster 2024-04-06T03:24:56.028125+0000 mgr.y (mgr.24368) 152 : cluster 0 pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[21467]: cluster 2024-04-06T03:24:57.136975+0000 osd.3 (osd.3) 111 : cluster 0 6.1b scrub starts 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[21467]: cluster 2024-04-06T03:24:57.139417+0000 osd.3 (osd.3) 112 : cluster 0 6.1b scrub ok 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[29221]: cluster 2024-04-06T03:24:55.996890+0000 osd.5 (osd.5) 103 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[29221]: cluster 2024-04-06T03:24:55.999173+0000 osd.5 (osd.5) 104 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[29221]: cluster 2024-04-06T03:24:56.028125+0000 mgr.y (mgr.24368) 152 : cluster 0 pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[29221]: cluster 2024-04-06T03:24:57.136975+0000 osd.3 (osd.3) 111 : cluster 0 6.1b scrub starts 2024-04-06T03:24:57.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:57 smithi050 bash[29221]: cluster 2024-04-06T03:24:57.139417+0000 osd.3 (osd.3) 112 : cluster 0 6.1b scrub ok 2024-04-06T03:24:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:57 smithi153 bash[23852]: cluster 2024-04-06T03:24:55.996890+0000 osd.5 (osd.5) 103 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:24:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:57 smithi153 bash[23852]: cluster 2024-04-06T03:24:55.999173+0000 osd.5 (osd.5) 104 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:24:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:57 smithi153 bash[23852]: cluster 2024-04-06T03:24:56.028125+0000 mgr.y (mgr.24368) 152 : cluster 0 pgmap v120: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:57 smithi153 bash[23852]: cluster 2024-04-06T03:24:57.136975+0000 osd.3 (osd.3) 111 : cluster 0 6.1b scrub starts 2024-04-06T03:24:57.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:57 smithi153 bash[23852]: cluster 2024-04-06T03:24:57.139417+0000 osd.3 (osd.3) 112 : cluster 0 6.1b scrub ok 2024-04-06T03:24:58.203 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:24:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[29221]: cluster 2024-04-06T03:24:56.932997+0000 osd.6 (osd.6) 69 : cluster 0 5.2 scrub starts 2024-04-06T03:24:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[29221]: cluster 2024-04-06T03:24:56.935315+0000 osd.6 (osd.6) 70 : cluster 0 5.2 scrub ok 2024-04-06T03:24:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[29221]: cluster 2024-04-06T03:24:58.073505+0000 osd.0 (osd.0) 93 : cluster 0 6.9 scrub starts 2024-04-06T03:24:58.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[29221]: cluster 2024-04-06T03:24:58.075931+0000 osd.0 (osd.0) 94 : cluster 0 6.9 scrub ok 2024-04-06T03:24:58.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[21467]: cluster 2024-04-06T03:24:56.932997+0000 osd.6 (osd.6) 69 : cluster 0 5.2 scrub starts 2024-04-06T03:24:58.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[21467]: cluster 2024-04-06T03:24:56.935315+0000 osd.6 (osd.6) 70 : cluster 0 5.2 scrub ok 2024-04-06T03:24:58.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[21467]: cluster 2024-04-06T03:24:58.073505+0000 osd.0 (osd.0) 93 : cluster 0 6.9 scrub starts 2024-04-06T03:24:58.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:58 smithi050 bash[21467]: cluster 2024-04-06T03:24:58.075931+0000 osd.0 (osd.0) 94 : cluster 0 6.9 scrub ok 2024-04-06T03:24:58.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:58 smithi153 bash[23852]: cluster 2024-04-06T03:24:56.932997+0000 osd.6 (osd.6) 69 : cluster 0 5.2 scrub starts 2024-04-06T03:24:58.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:58 smithi153 bash[23852]: cluster 2024-04-06T03:24:56.935315+0000 osd.6 (osd.6) 70 : cluster 0 5.2 scrub ok 2024-04-06T03:24:58.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:58 smithi153 bash[23852]: cluster 2024-04-06T03:24:58.073505+0000 osd.0 (osd.0) 93 : cluster 0 6.9 scrub starts 2024-04-06T03:24:58.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:58 smithi153 bash[23852]: cluster 2024-04-06T03:24:58.075931+0000 osd.0 (osd.0) 94 : cluster 0 6.9 scrub ok 2024-04-06T03:24:59.420 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:24:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:24:59.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[21467]: cluster 2024-04-06T03:24:58.028844+0000 mgr.y (mgr.24368) 153 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:59.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[21467]: cluster 2024-04-06T03:24:59.152866+0000 osd.3 (osd.3) 113 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:24:59.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[21467]: cluster 2024-04-06T03:24:59.155792+0000 osd.3 (osd.3) 114 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:24:59.570 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:24:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:24:59.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[29221]: cluster 2024-04-06T03:24:58.028844+0000 mgr.y (mgr.24368) 153 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:59.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[29221]: cluster 2024-04-06T03:24:59.152866+0000 osd.3 (osd.3) 113 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:24:59.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:24:59 smithi050 bash[29221]: cluster 2024-04-06T03:24:59.155792+0000 osd.3 (osd.3) 114 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:24:59.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:59 smithi153 bash[23852]: cluster 2024-04-06T03:24:58.028844+0000 mgr.y (mgr.24368) 153 : cluster 0 pgmap v121: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:24:59.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:59 smithi153 bash[23852]: cluster 2024-04-06T03:24:59.152866+0000 osd.3 (osd.3) 113 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:24:59.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:24:59 smithi153 bash[23852]: cluster 2024-04-06T03:24:59.155792+0000 osd.3 (osd.3) 114 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:25:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: cluster 2024-04-06T03:24:58.564237+0000 osd.2 (osd.2) 73 : cluster 0 5.8 scrub starts 2024-04-06T03:25:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: cluster 2024-04-06T03:24:58.566513+0000 osd.2 (osd.2) 74 : cluster 0 5.8 scrub ok 2024-04-06T03:25:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: audit 2024-04-06T03:24:59.134435+0000 mgr.y (mgr.24368) 154 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.025839+0000 osd.1 (osd.1) 101 : cluster 0 3.0 scrub starts 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.029137+0000 osd.1 (osd.1) 102 : cluster 0 3.0 scrub ok 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.148812+0000 osd.3 (osd.3) 115 : cluster 0 3.8 scrub starts 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.152141+0000 osd.3 (osd.3) 116 : cluster 0 3.8 scrub ok 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: cluster 2024-04-06T03:24:58.564237+0000 osd.2 (osd.2) 73 : cluster 0 5.8 scrub starts 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: cluster 2024-04-06T03:24:58.566513+0000 osd.2 (osd.2) 74 : cluster 0 5.8 scrub ok 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: audit 2024-04-06T03:24:59.134435+0000 mgr.y (mgr.24368) 154 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.025839+0000 osd.1 (osd.1) 101 : cluster 0 3.0 scrub starts 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.029137+0000 osd.1 (osd.1) 102 : cluster 0 3.0 scrub ok 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.148812+0000 osd.3 (osd.3) 115 : cluster 0 3.8 scrub starts 2024-04-06T03:25:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:00 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.152141+0000 osd.3 (osd.3) 116 : cluster 0 3.8 scrub ok 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: cluster 2024-04-06T03:24:58.564237+0000 osd.2 (osd.2) 73 : cluster 0 5.8 scrub starts 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: cluster 2024-04-06T03:24:58.566513+0000 osd.2 (osd.2) 74 : cluster 0 5.8 scrub ok 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: audit 2024-04-06T03:24:59.134435+0000 mgr.y (mgr.24368) 154 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.025839+0000 osd.1 (osd.1) 101 : cluster 0 3.0 scrub starts 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.029137+0000 osd.1 (osd.1) 102 : cluster 0 3.0 scrub ok 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.148812+0000 osd.3 (osd.3) 115 : cluster 0 3.8 scrub starts 2024-04-06T03:25:00.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:00 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.152141+0000 osd.3 (osd.3) 116 : cluster 0 3.8 scrub ok 2024-04-06T03:25:01.225 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a: {} 2024-04-06T03:25:01.240 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b: {} 2024-04-06T03:25:01.251 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c: {} 2024-04-06T03:25:01.252 INFO:teuthology.orchestra.run.smithi050.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-06T03:25:01.252 INFO:teuthology.orchestra.run.smithi050.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-06T03:25:01.252 INFO:teuthology.orchestra.run.smithi050.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-06T03:25:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.020480+0000 osd.5 (osd.5) 105 : cluster 0 5.16 scrub starts 2024-04-06T03:25:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.022308+0000 osd.5 (osd.5) 106 : cluster 0 5.16 scrub ok 2024-04-06T03:25:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[29221]: cluster 2024-04-06T03:25:00.030437+0000 mgr.y (mgr.24368) 155 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[29221]: audit 2024-04-06T03:25:01.222025+0000 mon.a (mon.0) 977 : audit 0 from='client.? 172.21.15.50:0/217877917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:25:01.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.020480+0000 osd.5 (osd.5) 105 : cluster 0 5.16 scrub starts 2024-04-06T03:25:01.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.022308+0000 osd.5 (osd.5) 106 : cluster 0 5.16 scrub ok 2024-04-06T03:25:01.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[21467]: cluster 2024-04-06T03:25:00.030437+0000 mgr.y (mgr.24368) 155 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:01.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:01 smithi050 bash[21467]: audit 2024-04-06T03:25:01.222025+0000 mon.a (mon.0) 977 : audit 0 from='client.? 172.21.15.50:0/217877917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:25:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:01 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.020480+0000 osd.5 (osd.5) 105 : cluster 0 5.16 scrub starts 2024-04-06T03:25:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:01 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.022308+0000 osd.5 (osd.5) 106 : cluster 0 5.16 scrub ok 2024-04-06T03:25:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:01 smithi153 bash[23852]: cluster 2024-04-06T03:25:00.030437+0000 mgr.y (mgr.24368) 155 : cluster 0 pgmap v122: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:01.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:01 smithi153 bash[23852]: audit 2024-04-06T03:25:01.222025+0000 mon.a (mon.0) 977 : audit 0 from='client.? 172.21.15.50:0/217877917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:25:01.989 INFO:teuthology.run_tasks:Running task full_sequential... 2024-04-06T03:25:02.025 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-06T03:25:02.033 INFO:tasks.radosbench:Beginning radosbench... 2024-04-06T03:25:02.034 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-06T03:25:02.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool create unique_pool_0 16 2024-04-06T03:25:02.035 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-06T03:25:02.035 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:25:02.035 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:25:02.035 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-06T03:25:02.035 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.740165 2024-04-06T03:25:02.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 1 0.7401647762640394 2024-04-06T03:25:02.036 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-06T03:25:02.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:02.036 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-06T03:25:02.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:25:02.037 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-06T03:25:02.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-06T03:25:02.037 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-06T03:25:02.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:25:02.066 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:02.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:02.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:02.193 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:02.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:02.320 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:02.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:02.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:02.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:02.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:02.713 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:02.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:02.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:02.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:02.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:03.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:03.106 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:03.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:03.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:03.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:03.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:03.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:03 smithi050 bash[21467]: cluster 2024-04-06T03:25:02.031115+0000 mgr.y (mgr.24368) 156 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:03.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:03 smithi050 bash[21467]: audit 2024-04-06T03:25:03.420673+0000 mon.a (mon.0) 978 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:03.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:03 smithi050 bash[29221]: cluster 2024-04-06T03:25:02.031115+0000 mgr.y (mgr.24368) 156 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:03.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:03 smithi050 bash[29221]: audit 2024-04-06T03:25:03.420673+0000 mon.a (mon.0) 978 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:03.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:03.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:03 smithi153 bash[23852]: cluster 2024-04-06T03:25:02.031115+0000 mgr.y (mgr.24368) 156 : cluster 0 pgmap v123: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:03.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:03 smithi153 bash[23852]: audit 2024-04-06T03:25:03.420673+0000 mon.a (mon.0) 978 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:03.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:03.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:03.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:03.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:03.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:03.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:03.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:04.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:04.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:04.154 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:04.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:04.286 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:04.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:04.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:04.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:04.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:04 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.051327+0000 osd.1 (osd.1) 103 : cluster 0 2.4 deep-scrub starts 2024-04-06T03:25:04.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:04 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.056562+0000 osd.1 (osd.1) 104 : cluster 0 2.4 deep-scrub ok 2024-04-06T03:25:04.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:04.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:04.677 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:04 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.051327+0000 osd.1 (osd.1) 103 : cluster 0 2.4 deep-scrub starts 2024-04-06T03:25:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:04 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.056562+0000 osd.1 (osd.1) 104 : cluster 0 2.4 deep-scrub ok 2024-04-06T03:25:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:04 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.051327+0000 osd.1 (osd.1) 103 : cluster 0 2.4 deep-scrub starts 2024-04-06T03:25:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:04 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.056562+0000 osd.1 (osd.1) 104 : cluster 0 2.4 deep-scrub ok 2024-04-06T03:25:04.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:04.805 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:04.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:04.937 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:04.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:05.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:05.065 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:05.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:05.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:05.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:05.322 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:05.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:05.454 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:05.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:05.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:05.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:03.906305+0000 osd.6 (osd.6) 71 : cluster 0 2.11 scrub starts 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:03.908645+0000 osd.6 (osd.6) 72 : cluster 0 2.11 scrub ok 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.032553+0000 mgr.y (mgr.24368) 157 : cluster 0 pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.471135+0000 osd.2 (osd.2) 75 : cluster 0 6.f scrub starts 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.473357+0000 osd.2 (osd.2) 76 : cluster 0 6.f scrub ok 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.897441+0000 osd.0 (osd.0) 95 : cluster 0 5.1e scrub starts 2024-04-06T03:25:05.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:04.898914+0000 osd.0 (osd.0) 96 : cluster 0 5.1e scrub ok 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.069385+0000 osd.1 (osd.1) 105 : cluster 0 6.1 scrub starts 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.071492+0000 osd.1 (osd.1) 106 : cluster 0 6.1 scrub ok 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:03.906305+0000 osd.6 (osd.6) 71 : cluster 0 2.11 scrub starts 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:03.908645+0000 osd.6 (osd.6) 72 : cluster 0 2.11 scrub ok 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.032553+0000 mgr.y (mgr.24368) 157 : cluster 0 pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.471135+0000 osd.2 (osd.2) 75 : cluster 0 6.f scrub starts 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.473357+0000 osd.2 (osd.2) 76 : cluster 0 6.f scrub ok 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.897441+0000 osd.0 (osd.0) 95 : cluster 0 5.1e scrub starts 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:04.898914+0000 osd.0 (osd.0) 96 : cluster 0 5.1e scrub ok 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.069385+0000 osd.1 (osd.1) 105 : cluster 0 6.1 scrub starts 2024-04-06T03:25:05.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:05 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.071492+0000 osd.1 (osd.1) 106 : cluster 0 6.1 scrub ok 2024-04-06T03:25:05.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:03.906305+0000 osd.6 (osd.6) 71 : cluster 0 2.11 scrub starts 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:03.908645+0000 osd.6 (osd.6) 72 : cluster 0 2.11 scrub ok 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.032553+0000 mgr.y (mgr.24368) 157 : cluster 0 pgmap v124: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.471135+0000 osd.2 (osd.2) 75 : cluster 0 6.f scrub starts 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.473357+0000 osd.2 (osd.2) 76 : cluster 0 6.f scrub ok 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.897441+0000 osd.0 (osd.0) 95 : cluster 0 5.1e scrub starts 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:04.898914+0000 osd.0 (osd.0) 96 : cluster 0 5.1e scrub ok 2024-04-06T03:25:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.069385+0000 osd.1 (osd.1) 105 : cluster 0 6.1 scrub starts 2024-04-06T03:25:05.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:05 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.071492+0000 osd.1 (osd.1) 106 : cluster 0 6.1 scrub ok 2024-04-06T03:25:05.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:05.846 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:05.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:05.978 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:05.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:06.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:06.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:06.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:06.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:06.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:06.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:06.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:06.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:06.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:06 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.059033+0000 osd.5 (osd.5) 107 : cluster 0 6.7 scrub starts 2024-04-06T03:25:06.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:06 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.061447+0000 osd.5 (osd.5) 108 : cluster 0 6.7 scrub ok 2024-04-06T03:25:06.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:06 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.216023+0000 osd.3 (osd.3) 117 : cluster 0 6.b scrub starts 2024-04-06T03:25:06.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:06 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.218270+0000 osd.3 (osd.3) 118 : cluster 0 6.b scrub ok 2024-04-06T03:25:06.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:06.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.059033+0000 osd.5 (osd.5) 107 : cluster 0 6.7 scrub starts 2024-04-06T03:25:06.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.061447+0000 osd.5 (osd.5) 108 : cluster 0 6.7 scrub ok 2024-04-06T03:25:06.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.216023+0000 osd.3 (osd.3) 117 : cluster 0 6.b scrub starts 2024-04-06T03:25:06.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.218270+0000 osd.3 (osd.3) 118 : cluster 0 6.b scrub ok 2024-04-06T03:25:06.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.059033+0000 osd.5 (osd.5) 107 : cluster 0 6.7 scrub starts 2024-04-06T03:25:06.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.061447+0000 osd.5 (osd.5) 108 : cluster 0 6.7 scrub ok 2024-04-06T03:25:06.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.216023+0000 osd.3 (osd.3) 117 : cluster 0 6.b scrub starts 2024-04-06T03:25:06.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:06 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.218270+0000 osd.3 (osd.3) 118 : cluster 0 6.b scrub ok 2024-04-06T03:25:06.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:06.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:06.829 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:06.830 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:06.830 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:06.830 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:06.831 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:06.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:06.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:06.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:06.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:07.015 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:07.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:07.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:07.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:07.294 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:07.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:07.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:07.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:07.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:07 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.744003+0000 osd.4 (osd.4) 85 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:07.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:07 smithi153 bash[23852]: cluster 2024-04-06T03:25:05.744127+0000 osd.4 (osd.4) 86 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:07.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:07 smithi153 bash[23852]: cluster 2024-04-06T03:25:06.033234+0000 mgr.y (mgr.24368) 158 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:07.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:07 smithi153 bash[23852]: cluster 2024-04-06T03:25:06.089679+0000 osd.5 (osd.5) 109 : cluster 0 3.1c scrub starts 2024-04-06T03:25:07.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:07 smithi153 bash[23852]: cluster 2024-04-06T03:25:06.094332+0000 osd.5 (osd.5) 110 : cluster 0 3.1c scrub ok 2024-04-06T03:25:07.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:07 smithi153 bash[23852]: cluster 2024-04-06T03:25:06.473364+0000 mon.a (mon.0) 979 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-04-06T03:25:07.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:07.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:07.687 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:07.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.744003+0000 osd.4 (osd.4) 85 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:07.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[29221]: cluster 2024-04-06T03:25:05.744127+0000 osd.4 (osd.4) 86 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[29221]: cluster 2024-04-06T03:25:06.033234+0000 mgr.y (mgr.24368) 158 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[29221]: cluster 2024-04-06T03:25:06.089679+0000 osd.5 (osd.5) 109 : cluster 0 3.1c scrub starts 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[29221]: cluster 2024-04-06T03:25:06.094332+0000 osd.5 (osd.5) 110 : cluster 0 3.1c scrub ok 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[29221]: cluster 2024-04-06T03:25:06.473364+0000 mon.a (mon.0) 979 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.744003+0000 osd.4 (osd.4) 85 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[21467]: cluster 2024-04-06T03:25:05.744127+0000 osd.4 (osd.4) 86 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[21467]: cluster 2024-04-06T03:25:06.033234+0000 mgr.y (mgr.24368) 158 : cluster 0 pgmap v125: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[21467]: cluster 2024-04-06T03:25:06.089679+0000 osd.5 (osd.5) 109 : cluster 0 3.1c scrub starts 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[21467]: cluster 2024-04-06T03:25:06.094332+0000 osd.5 (osd.5) 110 : cluster 0 3.1c scrub ok 2024-04-06T03:25:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:07 smithi050 bash[21467]: cluster 2024-04-06T03:25:06.473364+0000 mon.a (mon.0) 979 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2024-04-06T03:25:07.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:07.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:07.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:07.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:07.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:08.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:08.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:08.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:08.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:08.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:08.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:08.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.582 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:08 smithi153 bash[23852]: cluster 2024-04-06T03:25:06.854039+0000 osd.4 (osd.4) 87 : cluster 0 3.9 deep-scrub starts 2024-04-06T03:25:08.582 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:08 smithi153 bash[23852]: cluster 2024-04-06T03:25:06.862407+0000 osd.4 (osd.4) 88 : cluster 0 3.9 deep-scrub ok 2024-04-06T03:25:08.582 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:08 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.173730+0000 osd.1 (osd.1) 107 : cluster 0 2.9 scrub starts 2024-04-06T03:25:08.583 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:08 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.175985+0000 osd.1 (osd.1) 108 : cluster 0 2.9 scrub ok 2024-04-06T03:25:08.599 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:08.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:08.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[29221]: cluster 2024-04-06T03:25:06.854039+0000 osd.4 (osd.4) 87 : cluster 0 3.9 deep-scrub starts 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[29221]: cluster 2024-04-06T03:25:06.862407+0000 osd.4 (osd.4) 88 : cluster 0 3.9 deep-scrub ok 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.173730+0000 osd.1 (osd.1) 107 : cluster 0 2.9 scrub starts 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.175985+0000 osd.1 (osd.1) 108 : cluster 0 2.9 scrub ok 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[21467]: cluster 2024-04-06T03:25:06.854039+0000 osd.4 (osd.4) 87 : cluster 0 3.9 deep-scrub starts 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[21467]: cluster 2024-04-06T03:25:06.862407+0000 osd.4 (osd.4) 88 : cluster 0 3.9 deep-scrub ok 2024-04-06T03:25:08.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.173730+0000 osd.1 (osd.1) 107 : cluster 0 2.9 scrub starts 2024-04-06T03:25:08.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:08 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.175985+0000 osd.1 (osd.1) 108 : cluster 0 2.9 scrub ok 2024-04-06T03:25:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:08.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:08.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:08.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:08.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:09.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:09.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:09.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:09.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:09.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:09.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:09.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:25:09.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:09.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:09.467 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:25:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:25:09.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:07.822218+0000 osd.4 (osd.4) 89 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:25:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:07.824545+0000 osd.4 (osd.4) 90 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:25:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.033667+0000 mgr.y (mgr.24368) 159 : cluster 0 pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.224443+0000 osd.3 (osd.3) 119 : cluster 0 3.2 scrub starts 2024-04-06T03:25:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.227336+0000 osd.3 (osd.3) 120 : cluster 0 3.2 scrub ok 2024-04-06T03:25:09.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.945939+0000 osd.7 (osd.7) 113 : cluster 0 3.f scrub starts 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[21467]: cluster 2024-04-06T03:25:08.950479+0000 osd.7 (osd.7) 114 : cluster 0 3.f scrub ok 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:07.822218+0000 osd.4 (osd.4) 89 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:07.824545+0000 osd.4 (osd.4) 90 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.033667+0000 mgr.y (mgr.24368) 159 : cluster 0 pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.224443+0000 osd.3 (osd.3) 119 : cluster 0 3.2 scrub starts 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.227336+0000 osd.3 (osd.3) 120 : cluster 0 3.2 scrub ok 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.945939+0000 osd.7 (osd.7) 113 : cluster 0 3.f scrub starts 2024-04-06T03:25:09.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:09 smithi050 bash[29221]: cluster 2024-04-06T03:25:08.950479+0000 osd.7 (osd.7) 114 : cluster 0 3.f scrub ok 2024-04-06T03:25:09.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:09.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:09.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:09.630 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:09.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:09.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:07.822218+0000 osd.4 (osd.4) 89 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:07.824545+0000 osd.4 (osd.4) 90 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.033667+0000 mgr.y (mgr.24368) 159 : cluster 0 pgmap v127: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.224443+0000 osd.3 (osd.3) 119 : cluster 0 3.2 scrub starts 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.227336+0000 osd.3 (osd.3) 120 : cluster 0 3.2 scrub ok 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.945939+0000 osd.7 (osd.7) 113 : cluster 0 3.f scrub starts 2024-04-06T03:25:09.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:09 smithi153 bash[23852]: cluster 2024-04-06T03:25:08.950479+0000 osd.7 (osd.7) 114 : cluster 0 3.f scrub ok 2024-04-06T03:25:09.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:09.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:09.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:09.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:10.022 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:10.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:10.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:10.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:10.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:10.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:10.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:10.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:10.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:10.543 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:10.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.061192+0000 osd.5 (osd.5) 111 : cluster 0 5.15 scrub starts 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.064588+0000 osd.5 (osd.5) 112 : cluster 0 5.15 scrub ok 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: audit 2024-04-06T03:25:09.143533+0000 mgr.y (mgr.24368) 160 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.683848+0000 osd.0 (osd.0) 97 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.683962+0000 osd.0 (osd.0) 98 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.934874+0000 osd.0 (osd.0) 99 : cluster 0 2.13 scrub starts 2024-04-06T03:25:10.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:10 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.937185+0000 osd.0 (osd.0) 100 : cluster 0 2.13 scrub ok 2024-04-06T03:25:10.674 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:10.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.061192+0000 osd.5 (osd.5) 111 : cluster 0 5.15 scrub starts 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.064588+0000 osd.5 (osd.5) 112 : cluster 0 5.15 scrub ok 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: audit 2024-04-06T03:25:09.143533+0000 mgr.y (mgr.24368) 160 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.683848+0000 osd.0 (osd.0) 97 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.683962+0000 osd.0 (osd.0) 98 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.934874+0000 osd.0 (osd.0) 99 : cluster 0 2.13 scrub starts 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.937185+0000 osd.0 (osd.0) 100 : cluster 0 2.13 scrub ok 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.061192+0000 osd.5 (osd.5) 111 : cluster 0 5.15 scrub starts 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.064588+0000 osd.5 (osd.5) 112 : cluster 0 5.15 scrub ok 2024-04-06T03:25:10.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: audit 2024-04-06T03:25:09.143533+0000 mgr.y (mgr.24368) 160 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:10.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.683848+0000 osd.0 (osd.0) 97 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:10.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.683962+0000 osd.0 (osd.0) 98 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:10.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.934874+0000 osd.0 (osd.0) 99 : cluster 0 2.13 scrub starts 2024-04-06T03:25:10.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:10 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.937185+0000 osd.0 (osd.0) 100 : cluster 0 2.13 scrub ok 2024-04-06T03:25:10.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:10.803 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:10.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:10.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:10.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:11.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:11.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:11.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:11.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:11.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:11.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:11.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:11.436 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:11.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:11.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:11.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:11.703 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:11.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.805073+0000 osd.4 (osd.4) 91 : cluster 0 5.1c scrub starts 2024-04-06T03:25:11.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[21467]: cluster 2024-04-06T03:25:09.806919+0000 osd.4 (osd.4) 92 : cluster 0 5.1c scrub ok 2024-04-06T03:25:11.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[21467]: cluster 2024-04-06T03:25:10.012238+0000 osd.5 (osd.5) 113 : cluster 0 3.1d scrub starts 2024-04-06T03:25:11.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[21467]: cluster 2024-04-06T03:25:10.015557+0000 osd.5 (osd.5) 114 : cluster 0 3.1d scrub ok 2024-04-06T03:25:11.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[21467]: cluster 2024-04-06T03:25:10.035073+0000 mgr.y (mgr.24368) 161 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[21467]: cluster 2024-04-06T03:25:10.489519+0000 mon.a (mon.0) 980 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.805073+0000 osd.4 (osd.4) 91 : cluster 0 5.1c scrub starts 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[29221]: cluster 2024-04-06T03:25:09.806919+0000 osd.4 (osd.4) 92 : cluster 0 5.1c scrub ok 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[29221]: cluster 2024-04-06T03:25:10.012238+0000 osd.5 (osd.5) 113 : cluster 0 3.1d scrub starts 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[29221]: cluster 2024-04-06T03:25:10.015557+0000 osd.5 (osd.5) 114 : cluster 0 3.1d scrub ok 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[29221]: cluster 2024-04-06T03:25:10.035073+0000 mgr.y (mgr.24368) 161 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:11.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:11 smithi050 bash[29221]: cluster 2024-04-06T03:25:10.489519+0000 mon.a (mon.0) 980 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-04-06T03:25:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:11 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.805073+0000 osd.4 (osd.4) 91 : cluster 0 5.1c scrub starts 2024-04-06T03:25:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:11 smithi153 bash[23852]: cluster 2024-04-06T03:25:09.806919+0000 osd.4 (osd.4) 92 : cluster 0 5.1c scrub ok 2024-04-06T03:25:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:11 smithi153 bash[23852]: cluster 2024-04-06T03:25:10.012238+0000 osd.5 (osd.5) 113 : cluster 0 3.1d scrub starts 2024-04-06T03:25:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:11 smithi153 bash[23852]: cluster 2024-04-06T03:25:10.015557+0000 osd.5 (osd.5) 114 : cluster 0 3.1d scrub ok 2024-04-06T03:25:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:11 smithi153 bash[23852]: cluster 2024-04-06T03:25:10.035073+0000 mgr.y (mgr.24368) 161 : cluster 0 pgmap v128: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:11.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:11 smithi153 bash[23852]: cluster 2024-04-06T03:25:10.489519+0000 mon.a (mon.0) 980 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2024-04-06T03:25:11.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:11.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:11.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:11.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:11.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:12.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:12.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:12.200 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:12.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:12.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:12.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:12.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:12.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:12.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:12.623 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:12.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[21467]: cluster 2024-04-06T03:25:11.206771+0000 osd.3 (osd.3) 121 : cluster 0 2.19 scrub starts 2024-04-06T03:25:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[21467]: cluster 2024-04-06T03:25:11.208309+0000 osd.3 (osd.3) 122 : cluster 0 2.19 scrub ok 2024-04-06T03:25:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[21467]: cluster 2024-04-06T03:25:11.957282+0000 osd.3 (osd.3) 123 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:12.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[21467]: cluster 2024-04-06T03:25:11.957434+0000 osd.3 (osd.3) 124 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:12.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[29221]: cluster 2024-04-06T03:25:11.206771+0000 osd.3 (osd.3) 121 : cluster 0 2.19 scrub starts 2024-04-06T03:25:12.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[29221]: cluster 2024-04-06T03:25:11.208309+0000 osd.3 (osd.3) 122 : cluster 0 2.19 scrub ok 2024-04-06T03:25:12.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[29221]: cluster 2024-04-06T03:25:11.957282+0000 osd.3 (osd.3) 123 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:12.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:12 smithi050 bash[29221]: cluster 2024-04-06T03:25:11.957434+0000 osd.3 (osd.3) 124 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:12.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:12 smithi153 bash[23852]: cluster 2024-04-06T03:25:11.206771+0000 osd.3 (osd.3) 121 : cluster 0 2.19 scrub starts 2024-04-06T03:25:12.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:12 smithi153 bash[23852]: cluster 2024-04-06T03:25:11.208309+0000 osd.3 (osd.3) 122 : cluster 0 2.19 scrub ok 2024-04-06T03:25:12.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:12 smithi153 bash[23852]: cluster 2024-04-06T03:25:11.957282+0000 osd.3 (osd.3) 123 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:12.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:12 smithi153 bash[23852]: cluster 2024-04-06T03:25:11.957434+0000 osd.3 (osd.3) 124 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:12.754 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:12.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:12.883 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:12.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:12.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:13.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:13.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:13.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:13.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:13.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:13.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:13.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:13.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:13 smithi050 bash[29221]: cluster 2024-04-06T03:25:12.035761+0000 mgr.y (mgr.24368) 162 : cluster 0 pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:25:13.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:13 smithi050 bash[29221]: cluster 2024-04-06T03:25:12.498820+0000 mon.a (mon.0) 981 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-04-06T03:25:13.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:13 smithi050 bash[21467]: cluster 2024-04-06T03:25:12.035761+0000 mgr.y (mgr.24368) 162 : cluster 0 pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:25:13.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:13 smithi050 bash[21467]: cluster 2024-04-06T03:25:12.498820+0000 mon.a (mon.0) 981 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-04-06T03:25:13.542 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:13.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:13.671 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:13.774 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:13.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:13 smithi153 bash[23852]: cluster 2024-04-06T03:25:12.035761+0000 mgr.y (mgr.24368) 162 : cluster 0 pgmap v130: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:25:13.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:13 smithi153 bash[23852]: cluster 2024-04-06T03:25:12.498820+0000 mon.a (mon.0) 981 : cluster 0 osdmap e124: 8 total, 8 up, 8 in 2024-04-06T03:25:13.801 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:13.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:13.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:13.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:14.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:14.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:14.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:14.189 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:14.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:14.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:14.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:14.451 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:14.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:14.582 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:14.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:14.714 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[29221]: cluster 2024-04-06T03:25:13.820694+0000 osd.7 (osd.7) 115 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[29221]: cluster 2024-04-06T03:25:13.820783+0000 osd.7 (osd.7) 116 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:14.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.196385+0000 osd.3 (osd.3) 125 : cluster 0 6.6 scrub starts 2024-04-06T03:25:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.197738+0000 osd.3 (osd.3) 126 : cluster 0 6.6 scrub ok 2024-04-06T03:25:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[21467]: cluster 2024-04-06T03:25:13.820694+0000 osd.7 (osd.7) 115 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[21467]: cluster 2024-04-06T03:25:13.820783+0000 osd.7 (osd.7) 116 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.196385+0000 osd.3 (osd.3) 125 : cluster 0 6.6 scrub starts 2024-04-06T03:25:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:14 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.197738+0000 osd.3 (osd.3) 126 : cluster 0 6.6 scrub ok 2024-04-06T03:25:14.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:14 smithi153 bash[23852]: cluster 2024-04-06T03:25:13.820694+0000 osd.7 (osd.7) 115 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:14.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:14 smithi153 bash[23852]: cluster 2024-04-06T03:25:13.820783+0000 osd.7 (osd.7) 116 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:14.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:14 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.196385+0000 osd.3 (osd.3) 125 : cluster 0 6.6 scrub starts 2024-04-06T03:25:14.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:14 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.197738+0000 osd.3 (osd.3) 126 : cluster 0 6.6 scrub ok 2024-04-06T03:25:14.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:14.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:14.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:14.972 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:14.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:15.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:15.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:15.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:15.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:15.325 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:25:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:25:15.325 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-06T03:25:15.325 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:25:15.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:15.355 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:15.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:15.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:15.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:15.602 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:15 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.037312+0000 mgr.y (mgr.24368) 163 : cluster 0 pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:25:15.602 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:15 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.510232+0000 mon.a (mon.0) 982 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-04-06T03:25:15.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:15 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.037312+0000 mgr.y (mgr.24368) 163 : cluster 0 pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:25:15.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:15 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.510232+0000 mon.a (mon.0) 982 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-04-06T03:25:15.619 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:15.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:15.750 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:15 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.037312+0000 mgr.y (mgr.24368) 163 : cluster 0 pgmap v132: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:25:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:15 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.510232+0000 mon.a (mon.0) 982 : cluster 0 osdmap e125: 8 total, 8 up, 8 in 2024-04-06T03:25:15.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:15.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:15.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:15.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:16.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:16.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:16.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:16.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:16.268 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:16.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:16.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:16.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:16.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:16.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:16.659 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.524481+0000 osd.2 (osd.2) 77 : cluster 0 5.a scrub starts 2024-04-06T03:25:16.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.526295+0000 osd.2 (osd.2) 78 : cluster 0 5.a scrub ok 2024-04-06T03:25:16.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.973264+0000 osd.5 (osd.5) 115 : cluster 0 3.16 scrub starts 2024-04-06T03:25:16.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[21467]: cluster 2024-04-06T03:25:14.975597+0000 osd.5 (osd.5) 116 : cluster 0 3.16 scrub ok 2024-04-06T03:25:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.524481+0000 osd.2 (osd.2) 77 : cluster 0 5.a scrub starts 2024-04-06T03:25:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.526295+0000 osd.2 (osd.2) 78 : cluster 0 5.a scrub ok 2024-04-06T03:25:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.973264+0000 osd.5 (osd.5) 115 : cluster 0 3.16 scrub starts 2024-04-06T03:25:16.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:16 smithi050 bash[29221]: cluster 2024-04-06T03:25:14.975597+0000 osd.5 (osd.5) 116 : cluster 0 3.16 scrub ok 2024-04-06T03:25:16.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:16.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:16 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.524481+0000 osd.2 (osd.2) 77 : cluster 0 5.a scrub starts 2024-04-06T03:25:16.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:16 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.526295+0000 osd.2 (osd.2) 78 : cluster 0 5.a scrub ok 2024-04-06T03:25:16.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:16 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.973264+0000 osd.5 (osd.5) 115 : cluster 0 3.16 scrub starts 2024-04-06T03:25:16.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:16 smithi153 bash[23852]: cluster 2024-04-06T03:25:14.975597+0000 osd.5 (osd.5) 116 : cluster 0 3.16 scrub ok 2024-04-06T03:25:16.788 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:16.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:16.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:16.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:17.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:17.048 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:17.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:17.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:17.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:17.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:17.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:17.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:17.434 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:17.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:17.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:17.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:17.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:17.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:17.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:15.708954+0000 osd.4 (osd.4) 93 : cluster 0 6.e scrub starts 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:15.710848+0000 osd.4 (osd.4) 94 : cluster 0 6.e scrub ok 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:15.963750+0000 osd.5 (osd.5) 117 : cluster 0 3.c deep-scrub starts 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:15.970073+0000 osd.5 (osd.5) 118 : cluster 0 3.c deep-scrub ok 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:16.038047+0000 mgr.y (mgr.24368) 164 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:17.006321+0000 osd.0 (osd.0) 101 : cluster 0 3.1f deep-scrub starts 2024-04-06T03:25:17.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:17 smithi153 bash[23852]: cluster 2024-04-06T03:25:17.013818+0000 osd.0 (osd.0) 102 : cluster 0 3.1f deep-scrub ok 2024-04-06T03:25:17.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:15.708954+0000 osd.4 (osd.4) 93 : cluster 0 6.e scrub starts 2024-04-06T03:25:17.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:15.710848+0000 osd.4 (osd.4) 94 : cluster 0 6.e scrub ok 2024-04-06T03:25:17.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:15.963750+0000 osd.5 (osd.5) 117 : cluster 0 3.c deep-scrub starts 2024-04-06T03:25:17.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:15.970073+0000 osd.5 (osd.5) 118 : cluster 0 3.c deep-scrub ok 2024-04-06T03:25:17.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:16.038047+0000 mgr.y (mgr.24368) 164 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:17.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:17.006321+0000 osd.0 (osd.0) 101 : cluster 0 3.1f deep-scrub starts 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[29221]: cluster 2024-04-06T03:25:17.013818+0000 osd.0 (osd.0) 102 : cluster 0 3.1f deep-scrub ok 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:15.708954+0000 osd.4 (osd.4) 93 : cluster 0 6.e scrub starts 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:15.710848+0000 osd.4 (osd.4) 94 : cluster 0 6.e scrub ok 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:15.963750+0000 osd.5 (osd.5) 117 : cluster 0 3.c deep-scrub starts 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:15.970073+0000 osd.5 (osd.5) 118 : cluster 0 3.c deep-scrub ok 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:16.038047+0000 mgr.y (mgr.24368) 164 : cluster 0 pgmap v134: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:17.006321+0000 osd.0 (osd.0) 101 : cluster 0 3.1f deep-scrub starts 2024-04-06T03:25:17.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:17 smithi050 bash[21467]: cluster 2024-04-06T03:25:17.013818+0000 osd.0 (osd.0) 102 : cluster 0 3.1f deep-scrub ok 2024-04-06T03:25:17.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:17.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:17.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:18.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:18.078 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:18.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:18.208 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:18.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:18.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:18.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-06T03:25:18.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:18.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:18.531 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.1 primary-affinity to 0.740165 (8485072) 2024-04-06T03:25:18.538 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:25:18.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:18.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: audit 2024-04-06T03:25:17.599127+0000 mon.c (mon.2) 31 : audit 1 from='client.? 172.21.15.50:0/3712504015' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: audit 2024-04-06T03:25:17.599406+0000 mon.a (mon.0) 983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: audit 2024-04-06T03:25:17.715285+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.50:0/3745710848' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]: dispatch 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: audit 2024-04-06T03:25:17.715633+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]: dispatch 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: cluster 2024-04-06T03:25:17.977200+0000 osd.0 (osd.0) 103 : cluster 0 3.12 scrub starts 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: cluster 2024-04-06T03:25:17.979573+0000 osd.0 (osd.0) 104 : cluster 0 3.12 scrub ok 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.056219+0000 osd.1 (osd.1) 109 : cluster 0 4.2 scrub starts 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.059051+0000 osd.1 (osd.1) 110 : cluster 0 4.2 scrub ok 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: audit 2024-04-06T03:25:18.341233+0000 mon.a (mon.0) 985 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:18.581 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:18 smithi153 bash[23852]: audit 2024-04-06T03:25:18.453086+0000 mon.a (mon.0) 986 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:18.592 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:18.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: audit 2024-04-06T03:25:17.599127+0000 mon.c (mon.2) 31 : audit 1 from='client.? 172.21.15.50:0/3712504015' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: audit 2024-04-06T03:25:17.599406+0000 mon.a (mon.0) 983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: audit 2024-04-06T03:25:17.715285+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.50:0/3745710848' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: audit 2024-04-06T03:25:17.715633+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: cluster 2024-04-06T03:25:17.977200+0000 osd.0 (osd.0) 103 : cluster 0 3.12 scrub starts 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: cluster 2024-04-06T03:25:17.979573+0000 osd.0 (osd.0) 104 : cluster 0 3.12 scrub ok 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.056219+0000 osd.1 (osd.1) 109 : cluster 0 4.2 scrub starts 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.059051+0000 osd.1 (osd.1) 110 : cluster 0 4.2 scrub ok 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: audit 2024-04-06T03:25:18.341233+0000 mon.a (mon.0) 985 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[21467]: audit 2024-04-06T03:25:18.453086+0000 mon.a (mon.0) 986 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: audit 2024-04-06T03:25:17.599127+0000 mon.c (mon.2) 31 : audit 1 from='client.? 172.21.15.50:0/3712504015' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: audit 2024-04-06T03:25:17.599406+0000 mon.a (mon.0) 983 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:25:18.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: audit 2024-04-06T03:25:17.715285+0000 mon.c (mon.2) 32 : audit 1 from='client.? 172.21.15.50:0/3745710848' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]: dispatch 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: audit 2024-04-06T03:25:17.715633+0000 mon.a (mon.0) 984 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]: dispatch 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: cluster 2024-04-06T03:25:17.977200+0000 osd.0 (osd.0) 103 : cluster 0 3.12 scrub starts 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: cluster 2024-04-06T03:25:17.979573+0000 osd.0 (osd.0) 104 : cluster 0 3.12 scrub ok 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.056219+0000 osd.1 (osd.1) 109 : cluster 0 4.2 scrub starts 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.059051+0000 osd.1 (osd.1) 110 : cluster 0 4.2 scrub ok 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: audit 2024-04-06T03:25:18.341233+0000 mon.a (mon.0) 985 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:25:18.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:18 smithi050 bash[29221]: audit 2024-04-06T03:25:18.453086+0000 mon.a (mon.0) 986 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:18.640 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:25:18.681 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:25:18.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:18.713 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:18.722 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:25:18.767 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:25:18.802 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:25:18.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:18.839 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:18.844 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:25:18.866 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:25:18.887 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:25:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:25:18.889 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:25:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:25:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:25:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:25:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:25:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:25:18.890 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:25:18.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:18.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:19.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:19.200 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:19.225 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:19.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:19.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:19.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:19.463 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:25:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:25:19.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:19.516 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:25:19.537 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_0' created 2024-04-06T03:25:19.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:17.734768+0000 osd.4 (osd.4) 95 : cluster 0 5.1 scrub starts 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:17.736838+0000 osd.4 (osd.4) 96 : cluster 0 5.1 scrub ok 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.038843+0000 mgr.y (mgr.24368) 165 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 0 op/s 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.461283+0000 osd.2 (osd.2) 79 : cluster 0 2.1 scrub starts 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.463302+0000 osd.2 (osd.2) 80 : cluster 0 2.1 scrub ok 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.521866+0000 mon.a (mon.0) 987 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:18.530317+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:18.530422+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]': finished 2024-04-06T03:25:19.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: cluster 2024-04-06T03:25:18.534156+0000 mon.a (mon.0) 990 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:18.627699+0000 mon.c (mon.2) 33 : audit 0 from='client.? 172.21.15.50:0/1973625701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:18.982370+0000 mon.c (mon.2) 34 : audit 1 from='client.? 172.21.15.50:0/2392921383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:18.982798+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:19.408048+0000 mon.a (mon.0) 992 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:19.409030+0000 mon.a (mon.0) 993 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[21467]: audit 2024-04-06T03:25:19.415799+0000 mon.a (mon.0) 994 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:17.734768+0000 osd.4 (osd.4) 95 : cluster 0 5.1 scrub starts 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:17.736838+0000 osd.4 (osd.4) 96 : cluster 0 5.1 scrub ok 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.038843+0000 mgr.y (mgr.24368) 165 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 0 op/s 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.461283+0000 osd.2 (osd.2) 79 : cluster 0 2.1 scrub starts 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.463302+0000 osd.2 (osd.2) 80 : cluster 0 2.1 scrub ok 2024-04-06T03:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.521866+0000 mon.a (mon.0) 987 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:18.530317+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:18.530422+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]': finished 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: cluster 2024-04-06T03:25:18.534156+0000 mon.a (mon.0) 990 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:18.627699+0000 mon.c (mon.2) 33 : audit 0 from='client.? 172.21.15.50:0/1973625701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:18.982370+0000 mon.c (mon.2) 34 : audit 1 from='client.? 172.21.15.50:0/2392921383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:18.982798+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:19.408048+0000 mon.a (mon.0) 992 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:19.409030+0000 mon.a (mon.0) 993 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:25:19.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:19 smithi050 bash[29221]: audit 2024-04-06T03:25:19.415799+0000 mon.a (mon.0) 994 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:19.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:19.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:19.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:17.734768+0000 osd.4 (osd.4) 95 : cluster 0 5.1 scrub starts 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:17.736838+0000 osd.4 (osd.4) 96 : cluster 0 5.1 scrub ok 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.038843+0000 mgr.y (mgr.24368) 165 : cluster 0 pgmap v135: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 677 B/s rd, 0 op/s 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.461283+0000 osd.2 (osd.2) 79 : cluster 0 2.1 scrub starts 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.463302+0000 osd.2 (osd.2) 80 : cluster 0 2.1 scrub ok 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.521866+0000 mon.a (mon.0) 987 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:25:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:18.530317+0000 mon.a (mon.0) 988 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:18.530422+0000 mon.a (mon.0) 989 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.7401647762640394}]': finished 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: cluster 2024-04-06T03:25:18.534156+0000 mon.a (mon.0) 990 : cluster 0 osdmap e126: 8 total, 8 up, 8 in 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:18.627699+0000 mon.c (mon.2) 33 : audit 0 from='client.? 172.21.15.50:0/1973625701' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:18.982370+0000 mon.c (mon.2) 34 : audit 1 from='client.? 172.21.15.50:0/2392921383' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:18.982798+0000 mon.a (mon.0) 991 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:19.408048+0000 mon.a (mon.0) 992 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:19.409030+0000 mon.a (mon.0) 993 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:25:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:19 smithi153 bash[23852]: audit 2024-04-06T03:25:19.415799+0000 mon.a (mon.0) 994 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:19.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:19.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:19.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:19.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:19.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:20.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:20.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:20.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:20.258 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:20.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:20.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:20.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:20.518 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:20.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:20.649 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:20 smithi050 bash[21467]: audit 2024-04-06T03:25:19.151571+0000 mgr.y (mgr.24368) 166 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:20.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:20 smithi050 bash[21467]: audit 2024-04-06T03:25:19.534139+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-06T03:25:20.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:20 smithi050 bash[21467]: cluster 2024-04-06T03:25:19.537143+0000 mon.a (mon.0) 996 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-04-06T03:25:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:20 smithi050 bash[29221]: audit 2024-04-06T03:25:19.151571+0000 mgr.y (mgr.24368) 166 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:20 smithi050 bash[29221]: audit 2024-04-06T03:25:19.534139+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-06T03:25:20.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:20 smithi050 bash[29221]: cluster 2024-04-06T03:25:19.537143+0000 mon.a (mon.0) 996 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-04-06T03:25:20.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:20.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:20 smithi153 bash[23852]: audit 2024-04-06T03:25:19.151571+0000 mgr.y (mgr.24368) 166 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:20.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:20 smithi153 bash[23852]: audit 2024-04-06T03:25:19.534139+0000 mon.a (mon.0) 995 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-04-06T03:25:20.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:20 smithi153 bash[23852]: cluster 2024-04-06T03:25:19.537143+0000 mon.a (mon.0) 996 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2024-04-06T03:25:20.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:20.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:20.910 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:20.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:21.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:21.042 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:21.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:21.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:21.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:21.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:21.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:21.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:21.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:21.566 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:21.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:21.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:21.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-06T03:25:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:21 smithi153 bash[23852]: cluster 2024-04-06T03:25:20.039670+0000 mgr.y (mgr.24368) 167 : cluster 0 pgmap v138: 148 pgs: 16 unknown, 128 active+clean, 4 peering; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:21 smithi153 bash[23852]: cluster 2024-04-06T03:25:20.534322+0000 mon.a (mon.0) 997 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-04-06T03:25:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:21 smithi153 bash[23852]: cluster 2024-04-06T03:25:20.534393+0000 mon.a (mon.0) 998 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:25:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:21 smithi153 bash[23852]: cluster 2024-04-06T03:25:20.553297+0000 mon.a (mon.0) 999 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-04-06T03:25:21.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:21.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[21467]: cluster 2024-04-06T03:25:20.039670+0000 mgr.y (mgr.24368) 167 : cluster 0 pgmap v138: 148 pgs: 16 unknown, 128 active+clean, 4 peering; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:21.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[21467]: cluster 2024-04-06T03:25:20.534322+0000 mon.a (mon.0) 997 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-04-06T03:25:21.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[21467]: cluster 2024-04-06T03:25:20.534393+0000 mon.a (mon.0) 998 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:25:21.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[21467]: cluster 2024-04-06T03:25:20.553297+0000 mon.a (mon.0) 999 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-04-06T03:25:21.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[29221]: cluster 2024-04-06T03:25:20.039670+0000 mgr.y (mgr.24368) 167 : cluster 0 pgmap v138: 148 pgs: 16 unknown, 128 active+clean, 4 peering; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:25:21.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[29221]: cluster 2024-04-06T03:25:20.534322+0000 mon.a (mon.0) 997 : cluster 3 Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-04-06T03:25:21.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[29221]: cluster 2024-04-06T03:25:20.534393+0000 mon.a (mon.0) 998 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:25:21.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:21 smithi050 bash[29221]: cluster 2024-04-06T03:25:20.553297+0000 mon.a (mon.0) 999 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2024-04-06T03:25:21.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:21.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:21.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:21.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:22.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:22.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:22.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:22.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:22.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:22.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:22.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:22.498 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:22.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:22.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:22.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:22.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:22 smithi153 bash[23852]: cluster 2024-04-06T03:25:21.557562+0000 mon.a (mon.0) 1000 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-04-06T03:25:22.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:22.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:22 smithi050 bash[21467]: cluster 2024-04-06T03:25:21.557562+0000 mon.a (mon.0) 1000 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-04-06T03:25:22.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:22 smithi050 bash[29221]: cluster 2024-04-06T03:25:21.557562+0000 mon.a (mon.0) 1000 : cluster 0 osdmap e129: 8 total, 8 up, 8 in 2024-04-06T03:25:22.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:22.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:22.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:23.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:23.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:23.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:23.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:23.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:23.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:23.542 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:23.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:25:23.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:25:23.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:23.657 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:23 smithi153 bash[23852]: cluster 2024-04-06T03:25:22.040195+0000 mgr.y (mgr.24368) 168 : cluster 0 pgmap v141: 148 pgs: 16 unknown, 128 active+clean, 4 peering; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:23.674 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:23.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:23.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:23.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:23.911 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:23 smithi050 bash[21467]: cluster 2024-04-06T03:25:22.040195+0000 mgr.y (mgr.24368) 168 : cluster 0 pgmap v141: 148 pgs: 16 unknown, 128 active+clean, 4 peering; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:23.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:23 smithi050 bash[29221]: cluster 2024-04-06T03:25:22.040195+0000 mgr.y (mgr.24368) 168 : cluster 0 pgmap v141: 148 pgs: 16 unknown, 128 active+clean, 4 peering; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:25:23.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:23.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:24.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:24.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:24.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:24.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:24.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:24.318 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:24.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:24.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:24.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:24.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:24.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:24 smithi050 bash[29221]: cluster 2024-04-06T03:25:24.041677+0000 mgr.y (mgr.24368) 169 : cluster 0 pgmap v142: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-04-06T03:25:24.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:24 smithi050 bash[29221]: cluster 2024-04-06T03:25:24.565142+0000 mon.a (mon.0) 1001 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-04-06T03:25:24.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:24 smithi050 bash[21467]: cluster 2024-04-06T03:25:24.041677+0000 mgr.y (mgr.24368) 169 : cluster 0 pgmap v142: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-04-06T03:25:24.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:24 smithi050 bash[21467]: cluster 2024-04-06T03:25:24.565142+0000 mon.a (mon.0) 1001 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-04-06T03:25:24.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:24.822 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:24 smithi153 bash[23852]: cluster 2024-04-06T03:25:24.041677+0000 mgr.y (mgr.24368) 169 : cluster 0 pgmap v142: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-04-06T03:25:24.822 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:24 smithi153 bash[23852]: cluster 2024-04-06T03:25:24.565142+0000 mon.a (mon.0) 1001 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-04-06T03:25:24.838 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:24.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:24.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:24.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:25.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:25.099 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:25.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:25.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:25.282 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:25.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:25.359 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:25.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:25.490 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:25.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:25.622 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:25.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:25.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:25.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:25.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:25.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:26.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:26.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:26.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:26.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:26.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:26.302 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:26.330 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:25:26.330 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:25:26.330 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-06T03:25:26.330 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-04-06T03:25:26.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 5 0 2024-04-06T03:25:26.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:26.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:26.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:26.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:26.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:26.667 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:26.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:26.795 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:26.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:26.927 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:26.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:27.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:27.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:27.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:27 smithi050 bash[21467]: cluster 2024-04-06T03:25:26.042445+0000 mgr.y (mgr.24368) 170 : cluster 0 pgmap v143: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-04-06T03:25:27.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.175 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:27 smithi153 bash[23852]: cluster 2024-04-06T03:25:26.042445+0000 mgr.y (mgr.24368) 170 : cluster 0 pgmap v143: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-04-06T03:25:27.191 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:27.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.322 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:27.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:27 smithi050 bash[29221]: cluster 2024-04-06T03:25:26.042445+0000 mgr.y (mgr.24368) 170 : cluster 0 pgmap v143: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-04-06T03:25:27.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.454 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:27.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:27.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:27.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.714 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:27.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:27.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:27.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:27.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:28.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:28.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:28.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:28.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:28.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:28.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:28.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:28.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:28.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:28.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:28.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:28.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:28.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:28.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:28.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:28.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:29.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:29.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:29 smithi050 bash[21467]: cluster 2024-04-06T03:25:28.042969+0000 mgr.y (mgr.24368) 171 : cluster 0 pgmap v144: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:25:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:29 smithi050 bash[29221]: cluster 2024-04-06T03:25:28.042969+0000 mgr.y (mgr.24368) 171 : cluster 0 pgmap v144: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:25:29.144 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:29.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:29.274 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:29.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:29.387 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:25:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:25:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:29 smithi153 bash[23852]: cluster 2024-04-06T03:25:28.042969+0000 mgr.y (mgr.24368) 171 : cluster 0 pgmap v144: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:25:29.401 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:29.482 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:25:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:25:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:25:29.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:29.533 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:29.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:29.663 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:29.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:29.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:29.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:29.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:29.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:30.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:30.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:30.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:30.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:30.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:30.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:30 smithi050 bash[21467]: audit 2024-04-06T03:25:29.153581+0000 mgr.y (mgr.24368) 172 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:30.302 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:30 smithi050 bash[29221]: audit 2024-04-06T03:25:29.153581+0000 mgr.y (mgr.24368) 172 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:30.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:30.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:30.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:30 smithi153 bash[23852]: audit 2024-04-06T03:25:29.153581+0000 mgr.y (mgr.24368) 172 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:30.444 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:30.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:30.574 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:30.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:30.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:30.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:30.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:30.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:30.966 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:30.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:31.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:31.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:31.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:31.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:31.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:31 smithi153 bash[23852]: cluster 2024-04-06T03:25:30.044547+0000 mgr.y (mgr.24368) 173 : cluster 0 pgmap v145: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:25:31.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:31.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:31 smithi050 bash[21467]: cluster 2024-04-06T03:25:30.044547+0000 mgr.y (mgr.24368) 173 : cluster 0 pgmap v145: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:25:31.346 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:31 smithi050 bash[29221]: cluster 2024-04-06T03:25:30.044547+0000 mgr.y (mgr.24368) 173 : cluster 0 pgmap v145: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:25:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:31.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:31.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:31.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:31.622 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:31.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:31.749 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:31.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:31.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:31.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:31.887 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:25:31.887 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:25:31.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:31.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:32.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:32.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:32.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:32.258 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:32.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:32.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:32.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:32.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:32.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:32.647 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:32.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:32.775 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:32.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:32.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:32.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:33.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:33.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:33.125 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-06T03:25:33.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:33.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:33 smithi050 bash[29221]: cluster 2024-04-06T03:25:32.044993+0000 mgr.y (mgr.24368) 174 : cluster 0 pgmap v146: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-04-06T03:25:33.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:33 smithi050 bash[29221]: audit 2024-04-06T03:25:32.564867+0000 mon.c (mon.2) 35 : audit 1 from='client.? 172.21.15.50:0/298216272' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:25:33.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:33 smithi050 bash[29221]: audit 2024-04-06T03:25:32.565481+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:25:33.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:33 smithi050 bash[21467]: cluster 2024-04-06T03:25:32.044993+0000 mgr.y (mgr.24368) 174 : cluster 0 pgmap v146: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-04-06T03:25:33.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:33 smithi050 bash[21467]: audit 2024-04-06T03:25:32.564867+0000 mon.c (mon.2) 35 : audit 1 from='client.? 172.21.15.50:0/298216272' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:25:33.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:33 smithi050 bash[21467]: audit 2024-04-06T03:25:32.565481+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:25:33.163 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:33.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:33.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:33.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:33.407 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:33 smithi153 bash[23852]: cluster 2024-04-06T03:25:32.044993+0000 mgr.y (mgr.24368) 174 : cluster 0 pgmap v146: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 0 op/s 2024-04-06T03:25:33.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:33 smithi153 bash[23852]: audit 2024-04-06T03:25:32.564867+0000 mon.c (mon.2) 35 : audit 1 from='client.? 172.21.15.50:0/298216272' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:25:33.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:33 smithi153 bash[23852]: audit 2024-04-06T03:25:32.565481+0000 mon.a (mon.0) 1002 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:25:33.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:33.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-06T03:25:33.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:33.559 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:33.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:33.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:33.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:33.813 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:33.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:33.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:33.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:34.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:34.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:34.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:34.200 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:34.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:34.332 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[21467]: audit 2024-04-06T03:25:33.124832+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[21467]: cluster 2024-04-06T03:25:33.131813+0000 mon.a (mon.0) 1004 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[21467]: audit 2024-04-06T03:25:33.483602+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[21467]: audit 2024-04-06T03:25:33.484849+0000 mon.a (mon.0) 1006 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[29221]: audit 2024-04-06T03:25:33.124832+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[29221]: cluster 2024-04-06T03:25:33.131813+0000 mon.a (mon.0) 1004 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[29221]: audit 2024-04-06T03:25:33.483602+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:34.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:34 smithi050 bash[29221]: audit 2024-04-06T03:25:33.484849+0000 mon.a (mon.0) 1006 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:34.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:34.466 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:34.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:34.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:34 smithi153 bash[23852]: audit 2024-04-06T03:25:33.124832+0000 mon.a (mon.0) 1003 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:25:34.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:34 smithi153 bash[23852]: cluster 2024-04-06T03:25:33.131813+0000 mon.a (mon.0) 1004 : cluster 0 osdmap e130: 8 total, 8 up, 8 in 2024-04-06T03:25:34.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:34 smithi153 bash[23852]: audit 2024-04-06T03:25:33.483602+0000 mon.a (mon.0) 1005 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:34.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:34 smithi153 bash[23852]: audit 2024-04-06T03:25:33.484849+0000 mon.a (mon.0) 1006 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:34.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:34.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:34.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:34.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:34.730 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:34.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:34.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:34.858 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:34.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:34.988 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:34.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:35.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:35.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:35.249 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:35.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:35 smithi153 bash[23852]: cluster 2024-04-06T03:25:34.046481+0000 mgr.y (mgr.24368) 175 : cluster 0 pgmap v148: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:35.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:35 smithi153 bash[23852]: cluster 2024-04-06T03:25:34.480761+0000 mon.a (mon.0) 1007 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:25:35.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:35.380 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:35 smithi050 bash[29221]: cluster 2024-04-06T03:25:34.046481+0000 mgr.y (mgr.24368) 175 : cluster 0 pgmap v148: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:35.482 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:35 smithi050 bash[29221]: cluster 2024-04-06T03:25:34.480761+0000 mon.a (mon.0) 1007 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:25:35.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:35 smithi050 bash[21467]: cluster 2024-04-06T03:25:34.046481+0000 mgr.y (mgr.24368) 175 : cluster 0 pgmap v148: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:35.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:35 smithi050 bash[21467]: cluster 2024-04-06T03:25:34.480761+0000 mon.a (mon.0) 1007 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:25:35.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:35.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:35.614 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:35.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:35.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:35.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:35.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:35.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:35.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:35.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:35.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:36.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:36.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:36.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:36.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:36.206 DEBUG:teuthology.orchestra.run.smithi050:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-04-06T03:25:36.216 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-06T03:25:36.216 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-06T03:25:36.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:hints = 1 2024-04-06T03:25:36.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-06T03:25:36.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object prefix: benchmark_data_smithi050_68188 2024-04-06T03:25:36.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:25:36.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 0 0 0 0 0 0 - 0 2024-04-06T03:25:36.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:36.302 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:36.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:36.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:36.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:36.566 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:36.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:36.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:36.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:36.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:36.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:36.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:37.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:37.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:37.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:37.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:37 smithi153 bash[23852]: cluster 2024-04-06T03:25:36.047079+0000 mgr.y (mgr.24368) 176 : cluster 0 pgmap v149: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:37.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:37.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 1 15 2676 2661 166.303 166.312 0.00418786 0.00598843 2024-04-06T03:25:37.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:37.327 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:37 smithi050 bash[21467]: cluster 2024-04-06T03:25:36.047079+0000 mgr.y (mgr.24368) 176 : cluster 0 pgmap v149: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:37.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:37 smithi050 bash[29221]: cluster 2024-04-06T03:25:36.047079+0000 mgr.y (mgr.24368) 176 : cluster 0 pgmap v149: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:37.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:37.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:37.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:37.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:37.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:37.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:37.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:37.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:37.845 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:37.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:37.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:37.984 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:38.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:38.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:38.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:38.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:38.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 2 16 4361 4345 135.771 105.25 0.00622644 0.00734214 2024-04-06T03:25:38.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:38.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:38.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:38.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:38.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:38.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:38.701 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:38.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:38.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:38.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:38.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:38.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:39.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:39.047 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:39.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:39.177 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:39.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 3 16 6360 6344 132.156 124.938 0.00448469 0.00755655 2024-04-06T03:25:39.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:39.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:39.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:39.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:39.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:39 smithi153 bash[23852]: cluster 2024-04-06T03:25:38.047560+0000 mgr.y (mgr.24368) 177 : cluster 0 pgmap v150: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:39.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:25:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:25:39.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:39.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:39 smithi050 bash[21467]: cluster 2024-04-06T03:25:38.047560+0000 mgr.y (mgr.24368) 177 : cluster 0 pgmap v150: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:39.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:39 smithi050 bash[29221]: cluster 2024-04-06T03:25:38.047560+0000 mgr.y (mgr.24368) 177 : cluster 0 pgmap v150: 148 pgs: 148 active+clean; 582 KiB data, 251 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:25:39.556 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:25:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:25:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:25:39.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:39.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:39.708 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:39.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:39.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:39.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:39.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:39.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:40.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:40.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:40.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:40.228 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:40.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 4 16 8362 8346 130.396 125.125 0.00906951 0.00765726 2024-04-06T03:25:40.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:40.356 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:40.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:40.465 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:40 smithi153 bash[23852]: audit 2024-04-06T03:25:39.163641+0000 mgr.y (mgr.24368) 178 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:40.477 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:40.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:40.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:40.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:40.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:40.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:40.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:40.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:40 smithi050 bash[21467]: audit 2024-04-06T03:25:39.163641+0000 mgr.y (mgr.24368) 178 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:40.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:40 smithi050 bash[29221]: audit 2024-04-06T03:25:39.163641+0000 mgr.y (mgr.24368) 178 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:40.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:40.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:40.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:41.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:41.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:41.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:41.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:41.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 5 16 9965 9949 124.352 100.188 0.00255189 0.00803265 2024-04-06T03:25:41.350 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:41.371 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:41.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:41.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:41.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:41.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:41.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[21467]: cluster 2024-04-06T03:25:39.626641+0000 osd.1 (osd.1) 111 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:41.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[21467]: cluster 2024-04-06T03:25:39.626710+0000 osd.1 (osd.1) 112 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[21467]: cluster 2024-04-06T03:25:39.945867+0000 osd.5 (osd.5) 119 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[21467]: cluster 2024-04-06T03:25:39.945996+0000 osd.5 (osd.5) 120 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[21467]: cluster 2024-04-06T03:25:40.048666+0000 mgr.y (mgr.24368) 179 : cluster 0 pgmap v151: 148 pgs: 148 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 35 MiB/s wr, 559 op/s 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[21467]: cluster 2024-04-06T03:25:40.438229+0000 mon.a (mon.0) 1008 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[29221]: cluster 2024-04-06T03:25:39.626641+0000 osd.1 (osd.1) 111 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[29221]: cluster 2024-04-06T03:25:39.626710+0000 osd.1 (osd.1) 112 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[29221]: cluster 2024-04-06T03:25:39.945867+0000 osd.5 (osd.5) 119 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[29221]: cluster 2024-04-06T03:25:39.945996+0000 osd.5 (osd.5) 120 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[29221]: cluster 2024-04-06T03:25:40.048666+0000 mgr.y (mgr.24368) 179 : cluster 0 pgmap v151: 148 pgs: 148 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 35 MiB/s wr, 559 op/s 2024-04-06T03:25:41.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:41 smithi050 bash[29221]: cluster 2024-04-06T03:25:40.438229+0000 mon.a (mon.0) 1008 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-04-06T03:25:41.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:41.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:41.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:41 smithi153 bash[23852]: cluster 2024-04-06T03:25:39.626641+0000 osd.1 (osd.1) 111 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:41.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:41 smithi153 bash[23852]: cluster 2024-04-06T03:25:39.626710+0000 osd.1 (osd.1) 112 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:41.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:41 smithi153 bash[23852]: cluster 2024-04-06T03:25:39.945867+0000 osd.5 (osd.5) 119 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:41.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:41 smithi153 bash[23852]: cluster 2024-04-06T03:25:39.945996+0000 osd.5 (osd.5) 120 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:41.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:41 smithi153 bash[23852]: cluster 2024-04-06T03:25:40.048666+0000 mgr.y (mgr.24368) 179 : cluster 0 pgmap v151: 148 pgs: 148 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 35 MiB/s wr, 559 op/s 2024-04-06T03:25:41.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:41 smithi153 bash[23852]: cluster 2024-04-06T03:25:40.438229+0000 mon.a (mon.0) 1008 : cluster 0 osdmap e131: 8 total, 8 up, 8 in 2024-04-06T03:25:41.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:41.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:41.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:41.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:42.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:42.123 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:42.144 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:42.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:42.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 6 16 11631 11615 120.98 104.125 0.00345464 0.00825275 2024-04-06T03:25:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:42.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:42.400 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:42.426 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:25:42.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:42.532 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:42.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:42.645 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:42 smithi153 bash[23852]: audit 2024-04-06T03:25:41.775186+0000 mon.c (mon.2) 36 : audit 1 from='client.? 172.21.15.50:0/1398099279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:25:42.645 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:42 smithi153 bash[23852]: audit 2024-04-06T03:25:41.775636+0000 mon.a (mon.0) 1009 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:25:42.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:42.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:42 smithi050 bash[21467]: audit 2024-04-06T03:25:41.775186+0000 mon.c (mon.2) 36 : audit 1 from='client.? 172.21.15.50:0/1398099279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:25:42.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:42 smithi050 bash[21467]: audit 2024-04-06T03:25:41.775636+0000 mon.a (mon.0) 1009 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:25:42.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:42 smithi050 bash[29221]: audit 2024-04-06T03:25:41.775186+0000 mon.c (mon.2) 36 : audit 1 from='client.? 172.21.15.50:0/1398099279' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:25:42.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:42 smithi050 bash[29221]: audit 2024-04-06T03:25:41.775636+0000 mon.a (mon.0) 1009 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:25:42.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:42.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:42.926 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:42.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:43.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:43.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:43.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:43.175 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:43.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 7 16 13616 13600 121.418 124.062 0.00355794 0.00812029 2024-04-06T03:25:43.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:43.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:43.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:43.430 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:25:43.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:43.493 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:25:43.530 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:25:43.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:43.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:43.579 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:25:43.608 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:25:43.629 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:25:43.651 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:25:43.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:43.674 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:25:43.675 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:25:43.675 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:25:43.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:43 smithi153 bash[23852]: cluster 2024-04-06T03:25:42.049146+0000 mgr.y (mgr.24368) 180 : cluster 0 pgmap v153: 148 pgs: 148 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 627 op/s 2024-04-06T03:25:43.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:43 smithi153 bash[23852]: cluster 2024-04-06T03:25:42.413957+0000 mon.a (mon.0) 1010 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:25:43.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:43 smithi153 bash[23852]: audit 2024-04-06T03:25:42.421869+0000 mon.a (mon.0) 1011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:25:43.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:43 smithi153 bash[23852]: cluster 2024-04-06T03:25:42.446708+0000 mon.a (mon.0) 1012 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-04-06T03:25:43.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:43 smithi153 bash[23852]: audit 2024-04-06T03:25:43.164298+0000 mon.c (mon.2) 37 : audit 0 from='client.? 172.21.15.50:0/2982831128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:25:43.688 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:43.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:43.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[29221]: cluster 2024-04-06T03:25:42.049146+0000 mgr.y (mgr.24368) 180 : cluster 0 pgmap v153: 148 pgs: 148 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 627 op/s 2024-04-06T03:25:43.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[29221]: cluster 2024-04-06T03:25:42.413957+0000 mon.a (mon.0) 1010 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:25:43.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[29221]: audit 2024-04-06T03:25:42.421869+0000 mon.a (mon.0) 1011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[29221]: cluster 2024-04-06T03:25:42.446708+0000 mon.a (mon.0) 1012 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[29221]: audit 2024-04-06T03:25:43.164298+0000 mon.c (mon.2) 37 : audit 0 from='client.? 172.21.15.50:0/2982831128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[21467]: cluster 2024-04-06T03:25:42.049146+0000 mgr.y (mgr.24368) 180 : cluster 0 pgmap v153: 148 pgs: 148 active+clean; 350 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 39 MiB/s wr, 627 op/s 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[21467]: cluster 2024-04-06T03:25:42.413957+0000 mon.a (mon.0) 1010 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[21467]: audit 2024-04-06T03:25:42.421869+0000 mon.a (mon.0) 1011 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[21467]: cluster 2024-04-06T03:25:42.446708+0000 mon.a (mon.0) 1012 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2024-04-06T03:25:43.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:43 smithi050 bash[21467]: audit 2024-04-06T03:25:43.164298+0000 mon.c (mon.2) 37 : audit 0 from='client.? 172.21.15.50:0/2982831128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:25:43.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:43.820 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:43.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:43.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:43.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:44.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:44.075 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:44.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:44.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:44.197 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:44.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 8 16 14929 14913 116.497 82.0625 0.00716192 0.00857618 2024-04-06T03:25:44.300 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:44.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:44.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:44.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:44.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:44.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:44.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:44.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:44.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:44.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:44.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:44.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:44.954 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:44.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:45.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:45.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:45.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:45.218 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:45.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 9 16 16690 16674 115.781 110.062 0.00340571 0.00862642 2024-04-06T03:25:45.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:45.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:45.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:45.481 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:45.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:45 smithi153 bash[23852]: cluster 2024-04-06T03:25:44.050134+0000 mgr.y (mgr.24368) 181 : cluster 0 pgmap v155: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-04-06T03:25:45.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:45 smithi050 bash[21467]: cluster 2024-04-06T03:25:44.050134+0000 mgr.y (mgr.24368) 181 : cluster 0 pgmap v155: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-04-06T03:25:45.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:45 smithi050 bash[29221]: cluster 2024-04-06T03:25:44.050134+0000 mgr.y (mgr.24368) 181 : cluster 0 pgmap v155: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-04-06T03:25:45.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:45.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:45.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:45.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:45.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:45.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:45.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:46.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:46.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:46.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 10 16 18188 18172 113.565 93.625 0.00756962 0.00879902 2024-04-06T03:25:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:46.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:46.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:46.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:46.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:46.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:46.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:46.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:46.772 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:46.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:46.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:47.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:47.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:47.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:47.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:47.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 11 15 19400 19385 110.132 75.8125 0.00596648 0.00907372 2024-04-06T03:25:47.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:47.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:47.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:47.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:47.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:47.556 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:47.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:47.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:47 smithi153 bash[23852]: cluster 2024-04-06T03:25:46.050678+0000 mgr.y (mgr.24368) 182 : cluster 0 pgmap v156: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-04-06T03:25:47.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:47.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:47 smithi050 bash[21467]: cluster 2024-04-06T03:25:46.050678+0000 mgr.y (mgr.24368) 182 : cluster 0 pgmap v156: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-04-06T03:25:47.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:47 smithi050 bash[29221]: cluster 2024-04-06T03:25:46.050678+0000 mgr.y (mgr.24368) 182 : cluster 0 pgmap v156: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.80k op/s 2024-04-06T03:25:47.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:47.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:47.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:47.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:48.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:48.082 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:48.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:48.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:48.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 12 16 20670 20654 107.563 79.3125 0.0338256 0.0092813 2024-04-06T03:25:48.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:48.351 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:48.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:48.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:48.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:48.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:48.617 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:48.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:25:48.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:48.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:48.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:48.874 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:48.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:48.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:49.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:49.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:49.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:49.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 13 16 22545 22529 108.303 117.188 0.00726062 0.00922755 2024-04-06T03:25:49.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:49.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:49.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:25:49.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[23852]: cluster 2024-04-06T03:25:47.775163+0000 osd.6 (osd.6) 73 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[23852]: cluster 2024-04-06T03:25:47.775268+0000 osd.6 (osd.6) 74 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[23852]: cluster 2024-04-06T03:25:48.051113+0000 mgr.y (mgr.24368) 183 : cluster 0 pgmap v157: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:25:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[23852]: cluster 2024-04-06T03:25:48.443393+0000 mon.a (mon.0) 1013 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-04-06T03:25:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[23852]: audit 2024-04-06T03:25:48.509544+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:49.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:49 smithi153 bash[23852]: audit 2024-04-06T03:25:48.510981+0000 mon.a (mon.0) 1015 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:49.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21467]: cluster 2024-04-06T03:25:47.775163+0000 osd.6 (osd.6) 73 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21467]: cluster 2024-04-06T03:25:47.775268+0000 osd.6 (osd.6) 74 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21467]: cluster 2024-04-06T03:25:48.051113+0000 mgr.y (mgr.24368) 183 : cluster 0 pgmap v157: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21467]: cluster 2024-04-06T03:25:48.443393+0000 mon.a (mon.0) 1013 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21467]: audit 2024-04-06T03:25:48.509544+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21467]: audit 2024-04-06T03:25:48.510981+0000 mon.a (mon.0) 1015 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:49.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[29221]: cluster 2024-04-06T03:25:47.775163+0000 osd.6 (osd.6) 73 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:49.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[29221]: cluster 2024-04-06T03:25:47.775268+0000 osd.6 (osd.6) 74 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:49.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[29221]: cluster 2024-04-06T03:25:48.051113+0000 mgr.y (mgr.24368) 183 : cluster 0 pgmap v157: 148 pgs: 148 active+clean; 902 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:25:49.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[29221]: cluster 2024-04-06T03:25:48.443393+0000 mon.a (mon.0) 1013 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2024-04-06T03:25:49.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[29221]: audit 2024-04-06T03:25:48.509544+0000 mon.a (mon.0) 1014 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:25:49.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[29221]: audit 2024-04-06T03:25:48.510981+0000 mon.a (mon.0) 1015 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:25:49.562 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:25:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:25:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:25:49.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:49.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:49.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:49.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:49.843 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:49.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:49.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:49.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:49.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:50.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:50.112 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:50.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:50.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:50.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 14 16 24299 24283 108.397 109.625 0.00989568 0.00921906 2024-04-06T03:25:50.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:50.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:50.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:50.504 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:50.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:25:50.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:50.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:50 smithi050 bash[21467]: audit 2024-04-06T03:25:49.171701+0000 mgr.y (mgr.24368) 184 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:50.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:50 smithi050 bash[29221]: audit 2024-04-06T03:25:49.171701+0000 mgr.y (mgr.24368) 184 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:50.630 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:50.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:50.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:50 smithi153 bash[23852]: audit 2024-04-06T03:25:49.171701+0000 mgr.y (mgr.24368) 184 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:25:50.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:50.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:50.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:50.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:50.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:51.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:51.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:51.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:51.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:51.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 15 16 25739 25723 107.17 90 0.00557088 0.00932511 2024-04-06T03:25:51.279 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:51.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:51.406 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:51.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:51.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:51 smithi153 bash[23852]: cluster 2024-04-06T03:25:50.052100+0000 mgr.y (mgr.24368) 185 : cluster 0 pgmap v159: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-04-06T03:25:51.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:51.677 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:51.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:51 smithi050 bash[21467]: cluster 2024-04-06T03:25:50.052100+0000 mgr.y (mgr.24368) 185 : cluster 0 pgmap v159: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-04-06T03:25:51.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:51 smithi050 bash[29221]: cluster 2024-04-06T03:25:50.052100+0000 mgr.y (mgr.24368) 185 : cluster 0 pgmap v159: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.05k op/s 2024-04-06T03:25:51.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:51.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:51.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:51.942 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:51.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:52.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:52.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:52.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:52.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:52.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 16 16 27409 27393 106.994 104.375 0.00441978 0.00933785 2024-04-06T03:25:52.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:17585 7.7 e132/127 7:e0372a13:::benchmark_data_smithi050_68188_object17584:head v 132'1119, mlcod=132'1117)", 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.693151+0000", 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6380647509999999, 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27492272499999998, 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693151+0000", 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693151+0000", 2024-04-06T03:25:52.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693159+0000", 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2120000000000003e-06 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693239+0000", 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9997e-05 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693249+0000", 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5240000000000006e-06 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693255+0000", 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7660000000000001e-06 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693796+0000", 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054027800000000005 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693816+0000", 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0769000000000001e-05 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.967707+0000", 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27389054400000001 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.968074+0000", 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036663499999999998 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.331 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:17588 7.6 7:66d777fc:::benchmark_data_smithi050_68188_object17587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.693928+0000", 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6372875239999996, 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27796642999999999, 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 17588 2024-04-06T03:25:52.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693928+0000", 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693928+0000", 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693930+0000", 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.452e-06 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693963+0000", 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3905999999999997e-05 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693974+0000", 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1039999999999999e-05 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.693982+0000", 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7090000000000002e-06 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.694040+0000", 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7432999999999997e-05 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.694153+0000", 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011374000000000001 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.694385+0000", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023139100000000001 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696332+0000", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001946782 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965237+0000", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26890551200000001 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.971532+0000", 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006294844 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.971621+0000", 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9579000000000004e-05 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.971895+0000", 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027304199999999998 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:17589 7.e e132/127 7:758ed126:::benchmark_data_smithi050_68188_object17588:head v 132'1102, mlcod=132'1100)", 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.695022+0000", 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6361940229999998, 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27029654199999997, 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695022+0000", 2024-04-06T03:25:52.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695022+0000", 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695022+0000", 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2399999999999998e-07 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695045+0000", 2024-04-06T03:25:52.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2708999999999999e-05 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695053+0000", 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4249999999999998e-06 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695060+0000", 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5899999999999996e-06 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695149+0000", 2024-04-06T03:25:52.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9170000000000002e-05 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695167+0000", 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8263000000000001e-05 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965241+0000", 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27007356500000002 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965318+0000", 2024-04-06T03:25:52.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7195999999999997e-05 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:17591 7.f 7:fcbaa6ef:::benchmark_data_smithi050_68188_object17590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.695886+0000", 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6353293149999999, 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.30062645599999999, 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.342 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 17591 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695886+0000", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695886+0000", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695887+0000", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.145e-06 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695910+0000", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2561e-05 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695920+0000", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9850000000000001e-06 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.695927+0000", 2024-04-06T03:25:52.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2389999999999996e-06 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696363+0000", 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043611099999999999 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696501+0000", 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137396 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696600+0000", 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9248999999999998e-05 2024-04-06T03:25:52.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.970224+0000", 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27362409900000001 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.971019+0000", 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079524999999999995 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.996383+0000", 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025363694999999999 2024-04-06T03:25:52.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.996463+0000", 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0389999999999994e-05 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.996513+0000", 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9336000000000003e-05 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.347 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:17592 7.7 e132/127 7:e8487811:::benchmark_data_smithi050_68188_object17591:head v 132'1120, mlcod=132'1117)", 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.696480+0000", 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6347358989999998, 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27372379499999999, 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696480+0000", 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696480+0000", 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696481+0000", 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0500000000000002e-07 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696510+0000", 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9247e-05 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696520+0000", 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.028e-05 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696528+0000", 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4619999999999999e-06 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.964564+0000", 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26803608299999998 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.964916+0000", 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035230400000000001 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.970140+0000", 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052238880000000003 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.970204+0000", 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3726000000000006e-05 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:17595 7.6 7:6d517add:::benchmark_data_smithi050_68188_object17594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.696573+0000", 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6346422450000002, 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.296601911, 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 17595 2024-04-06T03:25:52.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696573+0000", 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696573+0000", 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696574+0000", 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7899999999999997e-07 2024-04-06T03:25:52.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.696597+0000", 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2478999999999999e-05 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.964784+0000", 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26818746100000002 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.964819+0000", 2024-04-06T03:25:52.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4421999999999997e-05 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965399+0000", 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058085899999999998 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965774+0000", 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037485900000000001 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:25:52.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.966169+0000", 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039438199999999998 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.970452+0000", 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042834279999999997 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.972002+0000", 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015494930000000001 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.993028+0000", 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021026627999999999 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.993110+0000", 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1285999999999994e-05 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.993175+0000", 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5735000000000007e-05 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:17602 7.5 e132/127 7:ab1704fd:::benchmark_data_smithi050_68188_object17601:head v 132'1167, mlcod=132'1165)", 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:45.699542+0000", 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.6316731759999996, 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26986217600000001, 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.699542+0000", 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.699542+0000", 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.699543+0000", 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2600000000000002e-07 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.699622+0000", 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8850000000000006e-05 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965148+0000", 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26552584200000001 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965163+0000", 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5065999999999999e-05 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965462+0000", 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029884900000000001 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.965496+0000", 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4521999999999999e-05 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.969335+0000", 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003838766 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:45.969405+0000", 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9554999999999994e-05 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18826 7.9 7:96cc1459:::benchmark_data_smithi050_68188_object18825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.737672+0000", 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5935439220000003, 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.24112668200000001, 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.362 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18826 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737672+0000", 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737672+0000", 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737673+0000", 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.356e-06 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737697+0000", 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3920000000000001e-05 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737708+0000", 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0900999999999999e-05 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737715+0000", 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3300000000000001e-06 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737996+0000", 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028105600000000001 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738099+0000", 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010244899999999999 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739767+0000", 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001667848 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740723+0000", 2024-04-06T03:25:52.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095677900000000005 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.743238+0000", 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025145440000000001 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978561+0000", 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.235322632 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978695+0000", 2024-04-06T03:25:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013479200000000001 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978798+0000", 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103075 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18830 7.4 7:2efffc2a:::benchmark_data_smithi050_68188_object18829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.738521+0000", 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.592694925, 2024-04-06T03:25:52.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.24011475299999999, 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18830 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738521+0000", 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738521+0000", 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738521+0000", 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7599999999999997e-07 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738546+0000", 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4298999999999999e-05 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738556+0000", 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0363999999999999e-05 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738563+0000", 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0180000000000004e-06 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740706+0000", 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021429690000000002 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740823+0000", 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117254 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740978+0000", 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015479299999999999 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742667+0000", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016889030000000001 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742914+0000", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024705499999999999 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978424+0000", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.235509989 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978583+0000", 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015894600000000001 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978635+0000", 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2587000000000003e-05 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18833 7.9 7:959dffa7:::benchmark_data_smithi050_68188_object18832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.741355+0000", 2024-04-06T03:25:52.374 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5898606380000002, 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23804914499999999, 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18833 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741355+0000", 2024-04-06T03:25:52.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741355+0000", 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741356+0000", 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.285e-06 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741384+0000", 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7515999999999999e-05 2024-04-06T03:25:52.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741439+0000", 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4781000000000003e-05 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741447+0000", 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8310000000000007e-06 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742070+0000", 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062262000000000005 2024-04-06T03:25:52.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742196+0000", 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000125761 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742308+0000", 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011186199999999999 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744561+0000", 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002253311 2024-04-06T03:25:52.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744583+0000", 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2260999999999999e-05 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.979106+0000", 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.234522965 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.979304+0000", 2024-04-06T03:25:52.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019739599999999999 2024-04-06T03:25:52.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.979404+0000", 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100556 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18839 7.9 7:9ec9e496:::benchmark_data_smithi050_68188_object18838:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.742472+0000", 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5887434339999995, 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23948657600000001, 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.382 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18839 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742472+0000", 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742472+0000", 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742473+0000", 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5000000000000001e-07 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742497+0000", 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4139999999999999e-05 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742507+0000", 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0176e-05 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.742516+0000", 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2390000000000002e-06 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.743300+0000", 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00078388199999999996 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.743453+0000", 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000153826 2024-04-06T03:25:52.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.743913+0000", 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045944300000000003 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744672+0000", 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075960899999999996 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.745004+0000", 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033127800000000001 2024-04-06T03:25:52.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.981903+0000", 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.236899156 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.981933+0000", 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0026e-05 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.981959+0000", 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5851e-05 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.387 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18842 7.4 7:242cc55d:::benchmark_data_smithi050_68188_object18841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.744049+0000", 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5871670550000001, 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23488188199999999, 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18842 2024-04-06T03:25:52.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744049+0000", 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744049+0000", 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744050+0000", 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2550000000000001e-06 2024-04-06T03:25:52.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744130+0000", 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9699999999999999e-05 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744141+0000", 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1389000000000001e-05 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744149+0000", 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2500000000000006e-06 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744219+0000", 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9838000000000005e-05 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744358+0000", 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013891899999999999 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744471+0000", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112762 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.746191+0000", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017208010000000001 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.746250+0000", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8922000000000003e-05 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978694+0000", 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23244364100000001 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978753+0000", 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8502999999999999e-05 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978930+0000", 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017790199999999999 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18845 7.4 7:2d2838ee:::benchmark_data_smithi050_68188_object18844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.744361+0000", 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5868548340000004, 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.234831343, 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18845 2024-04-06T03:25:52.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744361+0000", 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744361+0000", 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744362+0000", 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9299999999999996e-07 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744385+0000", 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3532000000000001e-05 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744399+0000", 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3711e-05 2024-04-06T03:25:52.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.744469+0000", 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9931000000000005e-05 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.745333+0000", 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00086428099999999995 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.745449+0000", 2024-04-06T03:25:52.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115901 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.745586+0000", 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013721499999999999 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.746355+0000", 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000768529 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.746804+0000", 2024-04-06T03:25:52.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000449071 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.979092+0000", 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23228771200000001 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.979148+0000", 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6294e-05 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.979192+0000", 2024-04-06T03:25:52.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4273000000000001e-05 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:25102 7.f 7:f16c9a8c:::benchmark_data_smithi050_68188_object25101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.722464+0000", 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.608751947, 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.22043396000000001, 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.401 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 25102 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.722464+0000", 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.722464+0000", 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.722464+0000", 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.09e-07 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.722485+0000", 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0998000000000001e-05 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.722496+0000", 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0271999999999999e-05 2024-04-06T03:25:52.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.722503+0000", 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4359999999999996e-06 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.740549+0000", 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018046092 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.740726+0000", 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017687700000000001 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.740827+0000", 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010107600000000001 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.743735+0000", 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029075199999999998 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.744638+0000", 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090327599999999995 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.942735+0000", 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19809685099999999 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.942829+0000", 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4116000000000005e-05 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.942898+0000", 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8637e-05 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:25122 7.f 7:fa028d46:::benchmark_data_smithi050_68188_object25121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.737215+0000", 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.594001038, 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20699399700000001, 2024-04-06T03:25:52.407 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 25122 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.737215+0000", 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.737215+0000", 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.737216+0000", 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6300000000000001e-06 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.737239+0000", 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3204e-05 2024-04-06T03:25:52.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.737252+0000", 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2566e-05 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.737260+0000", 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4209999999999995e-06 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.741758+0000", 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044975450000000004 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.741873+0000", 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011529199999999999 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.741979+0000", 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010529100000000001 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.744449+0000", 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002470164 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.744671+0000", 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022206400000000001 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.944002+0000", 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19933140999999999 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.944124+0000", 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121981 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.944209+0000", 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4429000000000001e-05 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:25133 7.7 e133/127 7:ea91ff98:::benchmark_data_smithi050_68188_object25132:head v 133'1616, mlcod=133'1608)", 2024-04-06T03:25:52.413 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.744236+0000", 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.586979704, 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20203833199999999, 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.744236+0000", 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.744236+0000", 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.744237+0000", 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4099999999999998e-07 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745636+0000", 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001399017 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745645+0000", 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6509999999999996e-06 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745653+0000", 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9549999999999992e-06 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745701+0000", 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7902999999999999e-05 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745721+0000", 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9346000000000001e-05 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.946195+0000", 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20047485100000001 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.946274+0000", 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8868000000000005e-05 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:25134 7.7 e133/127 7:ee4ca489:::benchmark_data_smithi050_68188_object25133:head v 133'1617, mlcod=133'1608)", 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.745673+0000", 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.5855429380000001, 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.200666288, 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745673+0000", 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745673+0000", 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745673+0000", 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6700000000000003e-07 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745707+0000", 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3689000000000002e-05 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745714+0000", 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.515e-06 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.745720+0000", 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.75e-06 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938456+0000", 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19273594999999999 2024-04-06T03:25:52.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938493+0000", 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6872000000000002e-05 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.946285+0000", 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077917860000000002 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.946339+0000", 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4159000000000001e-05 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.421 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:25140 7.9 7:988a9a3b:::benchmark_data_smithi050_68188_object25139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.746428+0000", 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.584787688, 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.198807136, 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.422 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 25140 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.746428+0000", 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.746428+0000", 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.746429+0000", 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0669999999999999e-06 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.746471+0000", 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1851000000000003e-05 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747380+0000", 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090952399999999999 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747395+0000", 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5048000000000001e-05 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747680+0000", 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028436299999999999 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747791+0000", 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111309 2024-04-06T03:25:52.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747919+0000", 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000127692 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938361+0000", 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19044202900000001 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938416+0000", 2024-04-06T03:25:52.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5438e-05 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945060+0000", 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006643258 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945158+0000", 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8957999999999997e-05 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945235+0000", 2024-04-06T03:25:52.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6599000000000003e-05 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:25142 7.4 7:24aee5e3:::benchmark_data_smithi050_68188_object25141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.747493+0000", 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.5837227110000001, 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19821976599999999, 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 25142 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747493+0000", 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747493+0000", 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747493+0000", 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8299999999999997e-07 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747516+0000", 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2316000000000001e-05 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747525+0000", 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9299999999999992e-06 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.747532+0000", 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1470000000000004e-06 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938368+0000", 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19083610500000001 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938708+0000", 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033968099999999999 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.939192+0000", 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000484802 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945343+0000", 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061508049999999996 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945409+0000", 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5807000000000002e-05 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945553+0000", 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014429500000000001 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945648+0000", 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4969000000000005e-05 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.945713+0000", 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4325999999999994e-05 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:25143 7.f 7:f6cc1f12:::benchmark_data_smithi050_68188_object25142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:50.749008+0000", 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.582207125, 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20183869400000001, 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:25:52.434 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 25143 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.749008+0000", 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.749008+0000", 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.749009+0000", 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4200000000000004e-07 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.749039+0000", 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9706e-05 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938310+0000", 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18927117500000001 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938322+0000", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1234e-05 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938476+0000", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154611 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.938608+0000", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013214699999999999 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:25:52.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.941255+0000", 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026465719999999998 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.946113+0000", 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048578909999999996 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.946353+0000", 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024016700000000001 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.950671+0000", 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043182339999999998 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.950770+0000", 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9152000000000002e-05 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:50.950847+0000", 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6862999999999999e-05 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:25:52.440 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:25:52.440 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:25:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:52.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:52.542 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:52.565 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:52.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:52.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:52.702 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:52.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:52.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:52.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:52.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:52.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:53.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:53.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:53.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:53.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:53.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 17 16 28949 28933 106.362 96.25 0.00670088 0.00939618 2024-04-06T03:25:53.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:53.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:53.469 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.5 primary-affinity to 0 (802) 2024-04-06T03:25:53.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:53.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:53.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:53.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:53.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:53 smithi153 bash[23852]: cluster 2024-04-06T03:25:51.998979+0000 osd.2 (osd.2) 81 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:53 smithi153 bash[23852]: cluster 2024-04-06T03:25:51.999047+0000 osd.2 (osd.2) 82 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:53 smithi153 bash[23852]: cluster 2024-04-06T03:25:52.052587+0000 mgr.y (mgr.24368) 186 : cluster 0 pgmap v160: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:25:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:53 smithi153 bash[23852]: cluster 2024-04-06T03:25:52.466321+0000 mon.a (mon.0) 1016 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-04-06T03:25:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:53 smithi153 bash[23852]: audit 2024-04-06T03:25:52.778635+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.50:0/1229877856' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-06T03:25:53.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:53 smithi153 bash[23852]: audit 2024-04-06T03:25:52.783650+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-06T03:25:53.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:53.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:53.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[21467]: cluster 2024-04-06T03:25:51.998979+0000 osd.2 (osd.2) 81 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:53.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[21467]: cluster 2024-04-06T03:25:51.999047+0000 osd.2 (osd.2) 82 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:53.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[21467]: cluster 2024-04-06T03:25:52.052587+0000 mgr.y (mgr.24368) 186 : cluster 0 pgmap v160: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:25:53.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[21467]: cluster 2024-04-06T03:25:52.466321+0000 mon.a (mon.0) 1016 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-04-06T03:25:53.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[21467]: audit 2024-04-06T03:25:52.778635+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.50:0/1229877856' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[21467]: audit 2024-04-06T03:25:52.783650+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[29221]: cluster 2024-04-06T03:25:51.998979+0000 osd.2 (osd.2) 81 : cluster 0 purged_snaps scrub starts 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[29221]: cluster 2024-04-06T03:25:51.999047+0000 osd.2 (osd.2) 82 : cluster 0 purged_snaps scrub ok 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[29221]: cluster 2024-04-06T03:25:52.052587+0000 mgr.y (mgr.24368) 186 : cluster 0 pgmap v160: 148 pgs: 148 active+clean; 1.3 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[29221]: cluster 2024-04-06T03:25:52.466321+0000 mon.a (mon.0) 1016 : cluster 0 osdmap e134: 8 total, 8 up, 8 in 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[29221]: audit 2024-04-06T03:25:52.778635+0000 mon.b (mon.1) 34 : audit 1 from='client.? 172.21.15.50:0/1229877856' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-06T03:25:53.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:53 smithi050 bash[29221]: audit 2024-04-06T03:25:52.783650+0000 mon.a (mon.0) 1017 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]: dispatch 2024-04-06T03:25:53.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:53.992 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:53.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:54.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:54.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:54.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:54.239 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:54.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 18 16 29351 29335 101.848 25.125 0.00317982 0.00957789 2024-04-06T03:25:54.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-06T03:25:54.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:54.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:54.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:54.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:54 smithi050 bash[21467]: audit 2024-04-06T03:25:53.457249+0000 mon.a (mon.0) 1018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-06T03:25:54.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:54 smithi050 bash[21467]: cluster 2024-04-06T03:25:53.460334+0000 mon.a (mon.0) 1019 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-04-06T03:25:54.483 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:54 smithi050 bash[21467]: cluster 2024-04-06T03:25:54.462494+0000 mon.a (mon.0) 1020 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-04-06T03:25:54.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:54 smithi050 bash[29221]: audit 2024-04-06T03:25:53.457249+0000 mon.a (mon.0) 1018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-06T03:25:54.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:54 smithi050 bash[29221]: cluster 2024-04-06T03:25:53.460334+0000 mon.a (mon.0) 1019 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-04-06T03:25:54.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:54 smithi050 bash[29221]: cluster 2024-04-06T03:25:54.462494+0000 mon.a (mon.0) 1020 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-04-06T03:25:54.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:54.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:54.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:54.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:54 smithi153 bash[23852]: audit 2024-04-06T03:25:53.457249+0000 mon.a (mon.0) 1018 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-04-06T03:25:54.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:54 smithi153 bash[23852]: cluster 2024-04-06T03:25:53.460334+0000 mon.a (mon.0) 1019 : cluster 0 osdmap e135: 8 total, 8 up, 8 in 2024-04-06T03:25:54.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:54 smithi153 bash[23852]: cluster 2024-04-06T03:25:54.462494+0000 mon.a (mon.0) 1020 : cluster 0 osdmap e136: 8 total, 8 up, 8 in 2024-04-06T03:25:54.777 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:54.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:54.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:54.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:55.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:55.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:55.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:55.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:55.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 19 16 30578 30562 100.524 76.6875 0.0168868 0.00993888 2024-04-06T03:25:55.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:55.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:55.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:55.508 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:55.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:55.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:55.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:55.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:55 smithi050 bash[21467]: cluster 2024-04-06T03:25:54.054258+0000 mgr.y (mgr.24368) 187 : cluster 0 pgmap v163: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-04-06T03:25:55.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:55 smithi050 bash[21467]: cluster 2024-04-06T03:25:54.472879+0000 mon.a (mon.0) 1021 : cluster 3 Health check failed: Reduced data availability: 15 pgs inactive, 20 pgs peering (PG_AVAILABILITY) 2024-04-06T03:25:55.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:55 smithi050 bash[29221]: cluster 2024-04-06T03:25:54.054258+0000 mgr.y (mgr.24368) 187 : cluster 0 pgmap v163: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-04-06T03:25:55.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:55 smithi050 bash[29221]: cluster 2024-04-06T03:25:54.472879+0000 mon.a (mon.0) 1021 : cluster 3 Health check failed: Reduced data availability: 15 pgs inactive, 20 pgs peering (PG_AVAILABILITY) 2024-04-06T03:25:55.720 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:55 smithi153 bash[23852]: cluster 2024-04-06T03:25:54.054258+0000 mgr.y (mgr.24368) 187 : cluster 0 pgmap v163: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-04-06T03:25:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:55 smithi153 bash[23852]: cluster 2024-04-06T03:25:54.472879+0000 mon.a (mon.0) 1021 : cluster 3 Health check failed: Reduced data availability: 15 pgs inactive, 20 pgs peering (PG_AVAILABILITY) 2024-04-06T03:25:55.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:55.847 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:55.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:55.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:55.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:56.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:56.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:56.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:56.235 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:56.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:25:56.267773+0000 min lat: 0.00125923 max lat: 0.834388 avg lat: 0.00991964 2024-04-06T03:25:56.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:25:56.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 20 16 31584 31568 98.6406 62.875 0.00179872 0.00991964 2024-04-06T03:25:56.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:56.365 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:56.468 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:56.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:56.520 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:25:56.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:56.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:56.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:56.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:56.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:56.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:56.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:56.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:57.009 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:57.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:57.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:57.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:57.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 21 16 32818 32802 97.6156 77.125 0.00404792 0.0102353 2024-04-06T03:25:57.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:57.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:57.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:57.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:57.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:57.641 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:57 smithi050 bash[21467]: cluster 2024-04-06T03:25:56.054985+0000 mgr.y (mgr.24368) 188 : cluster 0 pgmap v165: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-06T03:25:57.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:57 smithi050 bash[29221]: cluster 2024-04-06T03:25:56.054985+0000 mgr.y (mgr.24368) 188 : cluster 0 pgmap v165: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-06T03:25:57.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:57.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:57.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:57 smithi153 bash[23852]: cluster 2024-04-06T03:25:56.054985+0000 mgr.y (mgr.24368) 188 : cluster 0 pgmap v165: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-06T03:25:57.788 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:57.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:57.920 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:57.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:58.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:58.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:25:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:25:58.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:58.181 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:25:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:25:58.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 22 16 34088 34072 96.7861 79.375 0.00357716 0.0103258 2024-04-06T03:25:58.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:58.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:58.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:58.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:25:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:25:58.664 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:58.688 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:58.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:58.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:58.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:58.970 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:25:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:58.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:25:59.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:59.095 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:59.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:59.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:59.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:59.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 23 16 35396 35380 96.1321 81.75 0.00824021 0.0103966 2024-04-06T03:25:59.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:59.334 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:25:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:25:59.349 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:59.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:59.477 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:25:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:25:59.482 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:25:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:25:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:25:59.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:25:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:25:59.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:59.735 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:25:59 smithi153 bash[23852]: cluster 2024-04-06T03:25:58.055657+0000 mgr.y (mgr.24368) 189 : cluster 0 pgmap v166: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-06T03:25:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:25:59.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:25:59.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:59.878 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:25:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:25:59.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:25:59.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:25:59 smithi050 bash[21467]: cluster 2024-04-06T03:25:58.055657+0000 mgr.y (mgr.24368) 189 : cluster 0 pgmap v166: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-06T03:25:59.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:25:59 smithi050 bash[29221]: cluster 2024-04-06T03:25:58.055657+0000 mgr.y (mgr.24368) 189 : cluster 0 pgmap v166: 148 pgs: 22 peering, 126 active+clean; 1.8 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-04-06T03:25:59.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:00.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:00.046 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:26:00.046 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:26:00.046 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-06T03:26:00.046 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-04-06T03:26:00.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 6 0 2024-04-06T03:26:00.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:00.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:00.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 24 16 36814 36798 95.8188 88.625 0.00209999 0.0103892 2024-04-06T03:26:00.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:00.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:00.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:00.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:00.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:00.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:00.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:00.649 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:00 smithi153 bash[23852]: audit 2024-04-06T03:25:59.179794+0000 mgr.y (mgr.24368) 190 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:00.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:00.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:00.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:00.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:00.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:00 smithi050 bash[21467]: audit 2024-04-06T03:25:59.179794+0000 mgr.y (mgr.24368) 190 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:00.907 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:00 smithi050 bash[29221]: audit 2024-04-06T03:25:59.179794+0000 mgr.y (mgr.24368) 190 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:00.926 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:00.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:01.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:01.052 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:01.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:01.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:01.258 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 25 16 38221 38205 95.5032 87.9375 0.00729965 0.0104644 2024-04-06T03:26:01.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:01.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:01.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:01.449 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:01.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:01.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:01.690 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:01.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:01 smithi153 bash[23852]: cluster 2024-04-06T03:26:00.056443+0000 mgr.y (mgr.24368) 191 : cluster 0 pgmap v167: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:26:01.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:01 smithi153 bash[23852]: cluster 2024-04-06T03:26:00.492326+0000 mon.a (mon.0) 1022 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 15 pgs inactive, 20 pgs peering) 2024-04-06T03:26:01.714 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:01.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:01 smithi050 bash[21467]: cluster 2024-04-06T03:26:00.056443+0000 mgr.y (mgr.24368) 191 : cluster 0 pgmap v167: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:26:01.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:01 smithi050 bash[21467]: cluster 2024-04-06T03:26:00.492326+0000 mon.a (mon.0) 1022 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 15 pgs inactive, 20 pgs peering) 2024-04-06T03:26:01.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:01 smithi050 bash[29221]: cluster 2024-04-06T03:26:00.056443+0000 mgr.y (mgr.24368) 191 : cluster 0 pgmap v167: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:26:01.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:01 smithi050 bash[29221]: cluster 2024-04-06T03:26:00.492326+0000 mon.a (mon.0) 1022 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 15 pgs inactive, 20 pgs peering) 2024-04-06T03:26:01.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:01.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:01.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:01.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:02.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:02.099 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:02.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:02.228 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:02.231 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:02.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 26 16 39740 39724 95.4731 94.9375 0.013337 0.0104683 2024-04-06T03:26:02.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:02.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:02.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:02.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:02.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:02.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:02.772 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:02.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:02.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:02.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:03.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:03.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:03.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:03.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 27 16 41079 41063 95.0364 83.6875 0.00805406 0.0105165 2024-04-06T03:26:03.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:03.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:03.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:03.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:03.559 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:03.579 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:03.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:03.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:03.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:03 smithi153 bash[23852]: cluster 2024-04-06T03:26:02.056890+0000 mgr.y (mgr.24368) 192 : cluster 0 pgmap v168: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 952 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-06T03:26:03.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:03 smithi050 bash[21467]: cluster 2024-04-06T03:26:02.056890+0000 mgr.y (mgr.24368) 192 : cluster 0 pgmap v168: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 952 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-06T03:26:03.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:03 smithi050 bash[29221]: cluster 2024-04-06T03:26:02.056890+0000 mgr.y (mgr.24368) 192 : cluster 0 pgmap v168: 148 pgs: 148 active+clean; 2.1 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 952 B/s rd, 94 MiB/s wr, 1.51k op/s 2024-04-06T03:26:03.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:03.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:03.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:03.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:03.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:04.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:04.092 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:04.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:04.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:04.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 28 16 42324 42308 94.421 77.8125 0.0108341 0.0105849 2024-04-06T03:26:04.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:04.361 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:04.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:04.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:04.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:04.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:04 smithi050 bash[21467]: audit 2024-04-06T03:26:03.532160+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:04.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:04 smithi050 bash[29221]: audit 2024-04-06T03:26:03.532160+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:04.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:04.770 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:04.771 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:04 smithi153 bash[23852]: audit 2024-04-06T03:26:03.532160+0000 mon.a (mon.0) 1023 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:04.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:04.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:04.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:04.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:05.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:05.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:05.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:05.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:05.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 29 16 43628 43612 93.9752 81.5 0.00814446 0.0106358 2024-04-06T03:26:05.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:05.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:05.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:05.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:05.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:05.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:05.680 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:05.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:05 smithi050 bash[21467]: cluster 2024-04-06T03:26:04.057872+0000 mgr.y (mgr.24368) 193 : cluster 0 pgmap v169: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-06T03:26:05.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:05 smithi050 bash[29221]: cluster 2024-04-06T03:26:04.057872+0000 mgr.y (mgr.24368) 193 : cluster 0 pgmap v169: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-06T03:26:05.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:05.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:05 smithi153 bash[23852]: cluster 2024-04-06T03:26:04.057872+0000 mgr.y (mgr.24368) 193 : cluster 0 pgmap v169: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-06T03:26:05.809 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:05.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:05.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:05.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:06.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:06.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:06.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:06.190 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:06.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 30 15 45034 45019 93.7738 87.9375 0.00897544 0.0106582 2024-04-06T03:26:06.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:06.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:06.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:06.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:06.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:06.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:06.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:06.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:06.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:06.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:06.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:06.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:06.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:07.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:07.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:07.235 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:07.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 31 15 46257 46242 93.2144 76.4375 0.00510497 0.010723 2024-04-06T03:26:07.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:07.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:07.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:07.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:07.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:07.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:07.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:07 smithi050 bash[21467]: cluster 2024-04-06T03:26:06.058339+0000 mgr.y (mgr.24368) 194 : cluster 0 pgmap v170: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 882 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:26:07.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:07 smithi050 bash[29221]: cluster 2024-04-06T03:26:06.058339+0000 mgr.y (mgr.24368) 194 : cluster 0 pgmap v170: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 882 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:26:07.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:07.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:07 smithi153 bash[23852]: cluster 2024-04-06T03:26:06.058339+0000 mgr.y (mgr.24368) 194 : cluster 0 pgmap v170: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 882 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:26:07.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:07.897 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:07.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:08.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:08.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:08.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:08.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:08.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:08.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 32 16 47533 47517 92.7915 79.6875 0.0037161 0.0107693 2024-04-06T03:26:08.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:08.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:08.418 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:08.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:08.536 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:26:08.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:08 smithi050 bash[21467]: audit 2024-04-06T03:26:08.391736+0000 mon.c (mon.2) 38 : audit 1 from='client.? 172.21.15.50:0/3017490454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:26:08.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:08 smithi050 bash[21467]: audit 2024-04-06T03:26:08.393504+0000 mon.a (mon.0) 1024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:26:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:08.604 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:26:08.646 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:26:08.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:08.690 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:08.710 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:26:08.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:08 smithi153 bash[23852]: audit 2024-04-06T03:26:08.391736+0000 mon.c (mon.2) 38 : audit 1 from='client.? 172.21.15.50:0/3017490454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:26:08.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:08 smithi153 bash[23852]: audit 2024-04-06T03:26:08.393504+0000 mon.a (mon.0) 1024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:26:08.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:08.819 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:08.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:08.946 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:08.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:09.025 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:08 smithi050 bash[29221]: audit 2024-04-06T03:26:08.391736+0000 mon.c (mon.2) 38 : audit 1 from='client.? 172.21.15.50:0/3017490454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:26:09.025 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:08 smithi050 bash[29221]: audit 2024-04-06T03:26:08.393504+0000 mon.a (mon.0) 1024 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:26:09.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:09.087 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:26:09.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:09.141 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:26:09.178 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:26:09.200 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:09.210 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:26:09.213 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:26:09.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:09.248 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:26:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:26:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:26:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:26:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:26:09.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:26:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:26:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:26:09.255 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:26:09.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 33 15 48311 48296 91.455 48.6875 0.0178659 0.0109286 2024-04-06T03:26:09.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:09.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:09.353 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:26:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:26:09.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:09.481 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:09.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:09.608 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:09.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.058896+0000 mgr.y (mgr.24368) 195 : cluster 0 pgmap v171: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.528015+0000 mon.a (mon.0) 1025 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: audit 2024-04-06T03:26:08.529922+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.535956+0000 mon.a (mon.0) 1027 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: audit 2024-04-06T03:26:08.550444+0000 mon.c (mon.2) 39 : audit 0 from='client.? 172.21.15.50:0/2756270975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.605961+0000 osd.3 (osd.3) 127 : cluster 0 3.5 scrub starts 2024-04-06T03:26:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.616284+0000 osd.3 (osd.3) 128 : cluster 0 3.5 scrub ok 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.058896+0000 mgr.y (mgr.24368) 195 : cluster 0 pgmap v171: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.528015+0000 mon.a (mon.0) 1025 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: audit 2024-04-06T03:26:08.529922+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.535956+0000 mon.a (mon.0) 1027 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: audit 2024-04-06T03:26:08.550444+0000 mon.c (mon.2) 39 : audit 0 from='client.? 172.21.15.50:0/2756270975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.605961+0000 osd.3 (osd.3) 127 : cluster 0 3.5 scrub starts 2024-04-06T03:26:09.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:09 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.616284+0000 osd.3 (osd.3) 128 : cluster 0 3.5 scrub ok 2024-04-06T03:26:09.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:09.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:09.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:09.861 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:09.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:09.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:09.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:10.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.058896+0000 mgr.y (mgr.24368) 195 : cluster 0 pgmap v171: 148 pgs: 148 active+clean; 2.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-04-06T03:26:10.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.528015+0000 mon.a (mon.0) 1025 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:26:10.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: audit 2024-04-06T03:26:08.529922+0000 mon.a (mon.0) 1026 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:26:10.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.535956+0000 mon.a (mon.0) 1027 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2024-04-06T03:26:10.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: audit 2024-04-06T03:26:08.550444+0000 mon.c (mon.2) 39 : audit 0 from='client.? 172.21.15.50:0/2756270975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:26:10.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.605961+0000 osd.3 (osd.3) 127 : cluster 0 3.5 scrub starts 2024-04-06T03:26:10.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:09 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.616284+0000 osd.3 (osd.3) 128 : cluster 0 3.5 scrub ok 2024-04-06T03:26:10.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:10.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:10.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:10.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:10.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 34 16 49353 49337 90.6786 65.0625 0.0137903 0.0110216 2024-04-06T03:26:10.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:10.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:10.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:10.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:10.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:10.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:10.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:10.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:10 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.701447+0000 osd.4 (osd.4) 97 : cluster 0 6.1a scrub starts 2024-04-06T03:26:10.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:10 smithi153 bash[23852]: cluster 2024-04-06T03:26:08.704589+0000 osd.4 (osd.4) 98 : cluster 0 6.1a scrub ok 2024-04-06T03:26:10.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:10 smithi153 bash[23852]: audit 2024-04-06T03:26:09.187905+0000 mgr.y (mgr.24368) 196 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:10.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:10 smithi153 bash[23852]: cluster 2024-04-06T03:26:09.521105+0000 osd.2 (osd.2) 83 : cluster 0 2.c scrub starts 2024-04-06T03:26:10.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:10 smithi153 bash[23852]: cluster 2024-04-06T03:26:09.523803+0000 osd.2 (osd.2) 84 : cluster 0 2.c scrub ok 2024-04-06T03:26:10.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:10.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:10.901 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:10.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:11.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.701447+0000 osd.4 (osd.4) 97 : cluster 0 6.1a scrub starts 2024-04-06T03:26:11.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[21467]: cluster 2024-04-06T03:26:08.704589+0000 osd.4 (osd.4) 98 : cluster 0 6.1a scrub ok 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[21467]: audit 2024-04-06T03:26:09.187905+0000 mgr.y (mgr.24368) 196 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[21467]: cluster 2024-04-06T03:26:09.521105+0000 osd.2 (osd.2) 83 : cluster 0 2.c scrub starts 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[21467]: cluster 2024-04-06T03:26:09.523803+0000 osd.2 (osd.2) 84 : cluster 0 2.c scrub ok 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.701447+0000 osd.4 (osd.4) 97 : cluster 0 6.1a scrub starts 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[29221]: cluster 2024-04-06T03:26:08.704589+0000 osd.4 (osd.4) 98 : cluster 0 6.1a scrub ok 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[29221]: audit 2024-04-06T03:26:09.187905+0000 mgr.y (mgr.24368) 196 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[29221]: cluster 2024-04-06T03:26:09.521105+0000 osd.2 (osd.2) 83 : cluster 0 2.c scrub starts 2024-04-06T03:26:11.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:10 smithi050 bash[29221]: cluster 2024-04-06T03:26:09.523803+0000 osd.2 (osd.2) 84 : cluster 0 2.c scrub ok 2024-04-06T03:26:11.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:11.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:11.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:11.159 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:11.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 35 16 50425 50409 90.0019 67 0.00572605 0.0111047 2024-04-06T03:26:11.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:11.288 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:11.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:11.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:11.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:11.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:11.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:10.059625+0000 mgr.y (mgr.24368) 197 : cluster 0 pgmap v173: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:10.479557+0000 osd.2 (osd.2) 85 : cluster 0 6.14 scrub starts 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:10.482086+0000 osd.2 (osd.2) 86 : cluster 0 6.14 scrub ok 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:10.666759+0000 osd.4 (osd.4) 99 : cluster 0 6.1e scrub starts 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:10.667996+0000 osd.4 (osd.4) 100 : cluster 0 6.1e scrub ok 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:11.430239+0000 osd.0 (osd.0) 105 : cluster 0 5.1b scrub starts 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[21467]: cluster 2024-04-06T03:26:11.434668+0000 osd.0 (osd.0) 106 : cluster 0 5.1b scrub ok 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:10.059625+0000 mgr.y (mgr.24368) 197 : cluster 0 pgmap v173: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:10.479557+0000 osd.2 (osd.2) 85 : cluster 0 6.14 scrub starts 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:10.482086+0000 osd.2 (osd.2) 86 : cluster 0 6.14 scrub ok 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:10.666759+0000 osd.4 (osd.4) 99 : cluster 0 6.1e scrub starts 2024-04-06T03:26:11.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:10.667996+0000 osd.4 (osd.4) 100 : cluster 0 6.1e scrub ok 2024-04-06T03:26:11.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:11.430239+0000 osd.0 (osd.0) 105 : cluster 0 5.1b scrub starts 2024-04-06T03:26:11.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:11 smithi050 bash[29221]: cluster 2024-04-06T03:26:11.434668+0000 osd.0 (osd.0) 106 : cluster 0 5.1b scrub ok 2024-04-06T03:26:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:11.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:11.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:11.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:11.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:11.935 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:10.059625+0000 mgr.y (mgr.24368) 197 : cluster 0 pgmap v173: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-06T03:26:11.935 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:10.479557+0000 osd.2 (osd.2) 85 : cluster 0 6.14 scrub starts 2024-04-06T03:26:11.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:10.482086+0000 osd.2 (osd.2) 86 : cluster 0 6.14 scrub ok 2024-04-06T03:26:11.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:10.666759+0000 osd.4 (osd.4) 99 : cluster 0 6.1e scrub starts 2024-04-06T03:26:11.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:10.667996+0000 osd.4 (osd.4) 100 : cluster 0 6.1e scrub ok 2024-04-06T03:26:11.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:11.430239+0000 osd.0 (osd.0) 105 : cluster 0 5.1b scrub starts 2024-04-06T03:26:11.936 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:11 smithi153 bash[23852]: cluster 2024-04-06T03:26:11.434668+0000 osd.0 (osd.0) 106 : cluster 0 5.1b scrub ok 2024-04-06T03:26:11.948 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:11.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:12.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:12.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:12.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:12.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:12.261 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 36 16 51238 51222 88.9132 50.8125 0.0196919 0.0112417 2024-04-06T03:26:12.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:12.332 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:12.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:12.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:12.585 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:12 smithi153 bash[23852]: cluster 2024-04-06T03:26:11.496809+0000 osd.2 (osd.2) 87 : cluster 0 2.e scrub starts 2024-04-06T03:26:12.585 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:12 smithi153 bash[23852]: cluster 2024-04-06T03:26:11.501627+0000 osd.2 (osd.2) 88 : cluster 0 2.e scrub ok 2024-04-06T03:26:12.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:12.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:12.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:12 smithi050 bash[21467]: cluster 2024-04-06T03:26:11.496809+0000 osd.2 (osd.2) 87 : cluster 0 2.e scrub starts 2024-04-06T03:26:12.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:12 smithi050 bash[21467]: cluster 2024-04-06T03:26:11.501627+0000 osd.2 (osd.2) 88 : cluster 0 2.e scrub ok 2024-04-06T03:26:12.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:12 smithi050 bash[29221]: cluster 2024-04-06T03:26:11.496809+0000 osd.2 (osd.2) 87 : cluster 0 2.e scrub starts 2024-04-06T03:26:12.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:12 smithi050 bash[29221]: cluster 2024-04-06T03:26:11.501627+0000 osd.2 (osd.2) 88 : cluster 0 2.e scrub ok 2024-04-06T03:26:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:12.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:12.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:12.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:12.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:12.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:13.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:13.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:13.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:13.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 37 16 52349 52333 88.3864 69.4375 0.0105107 0.0113067 2024-04-06T03:26:13.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:13.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:13.404 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:13.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:13.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:13.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:13.657 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:13.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:13.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:13 smithi153 bash[23852]: cluster 2024-04-06T03:26:12.060164+0000 mgr.y (mgr.24368) 198 : cluster 0 pgmap v174: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-06T03:26:13.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:13 smithi153 bash[23852]: cluster 2024-04-06T03:26:12.559115+0000 osd.3 (osd.3) 129 : cluster 0 2.1b scrub starts 2024-04-06T03:26:13.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:13 smithi153 bash[23852]: cluster 2024-04-06T03:26:12.560826+0000 osd.3 (osd.3) 130 : cluster 0 2.1b scrub ok 2024-04-06T03:26:13.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:13.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:13.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:13 smithi050 bash[21467]: cluster 2024-04-06T03:26:12.060164+0000 mgr.y (mgr.24368) 198 : cluster 0 pgmap v174: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-06T03:26:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:13 smithi050 bash[21467]: cluster 2024-04-06T03:26:12.559115+0000 osd.3 (osd.3) 129 : cluster 0 2.1b scrub starts 2024-04-06T03:26:13.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:13 smithi050 bash[21467]: cluster 2024-04-06T03:26:12.560826+0000 osd.3 (osd.3) 130 : cluster 0 2.1b scrub ok 2024-04-06T03:26:13.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:13 smithi050 bash[29221]: cluster 2024-04-06T03:26:12.060164+0000 mgr.y (mgr.24368) 198 : cluster 0 pgmap v174: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-04-06T03:26:13.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:13 smithi050 bash[29221]: cluster 2024-04-06T03:26:12.559115+0000 osd.3 (osd.3) 129 : cluster 0 2.1b scrub starts 2024-04-06T03:26:13.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:13 smithi050 bash[29221]: cluster 2024-04-06T03:26:12.560826+0000 osd.3 (osd.3) 130 : cluster 0 2.1b scrub ok 2024-04-06T03:26:14.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:14.055 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:14.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:14.183 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:14.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 38 16 53475 53459 87.9123 70.375 0.00737442 0.0113691 2024-04-06T03:26:14.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:14.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:14.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:14.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:14.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:26:14.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:14.591 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:14.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:14.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:14.817 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[29221]: cluster 2024-04-06T03:26:12.718304+0000 osd.4 (osd.4) 101 : cluster 0 6.1d scrub starts 2024-04-06T03:26:14.817 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[29221]: cluster 2024-04-06T03:26:12.724887+0000 osd.4 (osd.4) 102 : cluster 0 6.1d scrub ok 2024-04-06T03:26:14.817 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[29221]: cluster 2024-04-06T03:26:13.991046+0000 osd.6 (osd.6) 75 : cluster 0 6.a scrub starts 2024-04-06T03:26:14.817 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[29221]: cluster 2024-04-06T03:26:13.993378+0000 osd.6 (osd.6) 76 : cluster 0 6.a scrub ok 2024-04-06T03:26:14.818 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[29221]: cluster 2024-04-06T03:26:14.445667+0000 osd.0 (osd.0) 107 : cluster 0 6.10 scrub starts 2024-04-06T03:26:14.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:14.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:14 smithi153 bash[23852]: cluster 2024-04-06T03:26:12.718304+0000 osd.4 (osd.4) 101 : cluster 0 6.1d scrub starts 2024-04-06T03:26:14.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:14 smithi153 bash[23852]: cluster 2024-04-06T03:26:12.724887+0000 osd.4 (osd.4) 102 : cluster 0 6.1d scrub ok 2024-04-06T03:26:14.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:14 smithi153 bash[23852]: cluster 2024-04-06T03:26:13.991046+0000 osd.6 (osd.6) 75 : cluster 0 6.a scrub starts 2024-04-06T03:26:14.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:14 smithi153 bash[23852]: cluster 2024-04-06T03:26:13.993378+0000 osd.6 (osd.6) 76 : cluster 0 6.a scrub ok 2024-04-06T03:26:14.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:14 smithi153 bash[23852]: cluster 2024-04-06T03:26:14.445667+0000 osd.0 (osd.0) 107 : cluster 0 6.10 scrub starts 2024-04-06T03:26:14.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:14.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:14.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:14.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:14.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:15.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:15.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:15.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:15.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[21467]: cluster 2024-04-06T03:26:12.718304+0000 osd.4 (osd.4) 101 : cluster 0 6.1d scrub starts 2024-04-06T03:26:15.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[21467]: cluster 2024-04-06T03:26:12.724887+0000 osd.4 (osd.4) 102 : cluster 0 6.1d scrub ok 2024-04-06T03:26:15.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[21467]: cluster 2024-04-06T03:26:13.991046+0000 osd.6 (osd.6) 75 : cluster 0 6.a scrub starts 2024-04-06T03:26:15.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[21467]: cluster 2024-04-06T03:26:13.993378+0000 osd.6 (osd.6) 76 : cluster 0 6.a scrub ok 2024-04-06T03:26:15.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:14 smithi050 bash[21467]: cluster 2024-04-06T03:26:14.445667+0000 osd.0 (osd.0) 107 : cluster 0 6.10 scrub starts 2024-04-06T03:26:15.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:15.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 39 16 54377 54361 87.1035 56.375 0.00199222 0.0114055 2024-04-06T03:26:15.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:15.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:15.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:15.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:15.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:15.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:15 smithi153 bash[23852]: cluster 2024-04-06T03:26:13.723526+0000 osd.4 (osd.4) 103 : cluster 0 3.3 scrub starts 2024-04-06T03:26:15.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:15 smithi153 bash[23852]: cluster 2024-04-06T03:26:13.731033+0000 osd.4 (osd.4) 104 : cluster 0 3.3 scrub ok 2024-04-06T03:26:15.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:15 smithi153 bash[23852]: cluster 2024-04-06T03:26:14.061120+0000 mgr.y (mgr.24368) 199 : cluster 0 pgmap v175: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:26:15.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:15 smithi153 bash[23852]: cluster 2024-04-06T03:26:14.452977+0000 osd.0 (osd.0) 108 : cluster 0 6.10 scrub ok 2024-04-06T03:26:15.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:15 smithi153 bash[23852]: cluster 2024-04-06T03:26:14.534953+0000 osd.2 (osd.2) 89 : cluster 0 2.3 scrub starts 2024-04-06T03:26:15.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:15 smithi153 bash[23852]: cluster 2024-04-06T03:26:14.536174+0000 osd.2 (osd.2) 90 : cluster 0 2.3 scrub ok 2024-04-06T03:26:15.635 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:15.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:15.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:15.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:15.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:15.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:15.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:15.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[21467]: cluster 2024-04-06T03:26:13.723526+0000 osd.4 (osd.4) 103 : cluster 0 3.3 scrub starts 2024-04-06T03:26:15.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[21467]: cluster 2024-04-06T03:26:13.731033+0000 osd.4 (osd.4) 104 : cluster 0 3.3 scrub ok 2024-04-06T03:26:15.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[21467]: cluster 2024-04-06T03:26:14.061120+0000 mgr.y (mgr.24368) 199 : cluster 0 pgmap v175: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:26:15.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[21467]: cluster 2024-04-06T03:26:14.452977+0000 osd.0 (osd.0) 108 : cluster 0 6.10 scrub ok 2024-04-06T03:26:15.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[21467]: cluster 2024-04-06T03:26:14.534953+0000 osd.2 (osd.2) 89 : cluster 0 2.3 scrub starts 2024-04-06T03:26:15.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[21467]: cluster 2024-04-06T03:26:14.536174+0000 osd.2 (osd.2) 90 : cluster 0 2.3 scrub ok 2024-04-06T03:26:15.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[29221]: cluster 2024-04-06T03:26:13.723526+0000 osd.4 (osd.4) 103 : cluster 0 3.3 scrub starts 2024-04-06T03:26:15.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[29221]: cluster 2024-04-06T03:26:13.731033+0000 osd.4 (osd.4) 104 : cluster 0 3.3 scrub ok 2024-04-06T03:26:15.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[29221]: cluster 2024-04-06T03:26:14.061120+0000 mgr.y (mgr.24368) 199 : cluster 0 pgmap v175: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:26:15.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[29221]: cluster 2024-04-06T03:26:14.452977+0000 osd.0 (osd.0) 108 : cluster 0 6.10 scrub ok 2024-04-06T03:26:15.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[29221]: cluster 2024-04-06T03:26:14.534953+0000 osd.2 (osd.2) 89 : cluster 0 2.3 scrub starts 2024-04-06T03:26:15.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:15 smithi050 bash[29221]: cluster 2024-04-06T03:26:14.536174+0000 osd.2 (osd.2) 90 : cluster 0 2.3 scrub ok 2024-04-06T03:26:15.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:16.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:16.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:16.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:16.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:16.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:16.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:26:16.271998+0000 min lat: 0.00125923 max lat: 0.834388 avg lat: 0.0116322 2024-04-06T03:26:16.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:26:16.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 40 15 55014 54999 85.9228 39.875 0.00415191 0.0116322 2024-04-06T03:26:16.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:16.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:16.422 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:16.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:26:16.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:16.552 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:16.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.336709+0000 osd.1 (osd.1) 113 : cluster 0 6.4 scrub starts 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.338739+0000 osd.1 (osd.1) 114 : cluster 0 6.4 scrub ok 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.494412+0000 osd.0 (osd.0) 109 : cluster 0 5.5 scrub starts 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.494998+0000 osd.3 (osd.3) 131 : cluster 0 6.13 scrub starts 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.497098+0000 osd.0 (osd.0) 110 : cluster 0 5.5 scrub ok 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.582566+0000 osd.3 (osd.3) 132 : cluster 0 6.13 scrub ok 2024-04-06T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.583197+0000 osd.2 (osd.2) 91 : cluster 0 2.10 scrub starts 2024-04-06T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:15.587661+0000 osd.2 (osd.2) 92 : cluster 0 2.10 scrub ok 2024-04-06T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.006091+0000 osd.6 (osd.6) 77 : cluster 0 3.10 scrub starts 2024-04-06T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.011025+0000 osd.6 (osd.6) 78 : cluster 0 3.10 scrub ok 2024-04-06T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:16 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.061469+0000 mgr.y (mgr.24368) 200 : cluster 0 pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:26:16.677 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:16.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:16.803 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:16.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:16.916 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:26:16.916 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:26:16.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:16.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:16.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.336709+0000 osd.1 (osd.1) 113 : cluster 0 6.4 scrub starts 2024-04-06T03:26:16.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.338739+0000 osd.1 (osd.1) 114 : cluster 0 6.4 scrub ok 2024-04-06T03:26:16.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.494412+0000 osd.0 (osd.0) 109 : cluster 0 5.5 scrub starts 2024-04-06T03:26:16.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.494998+0000 osd.3 (osd.3) 131 : cluster 0 6.13 scrub starts 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.497098+0000 osd.0 (osd.0) 110 : cluster 0 5.5 scrub ok 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.582566+0000 osd.3 (osd.3) 132 : cluster 0 6.13 scrub ok 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.583197+0000 osd.2 (osd.2) 91 : cluster 0 2.10 scrub starts 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:15.587661+0000 osd.2 (osd.2) 92 : cluster 0 2.10 scrub ok 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.006091+0000 osd.6 (osd.6) 77 : cluster 0 3.10 scrub starts 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.011025+0000 osd.6 (osd.6) 78 : cluster 0 3.10 scrub ok 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.061469+0000 mgr.y (mgr.24368) 200 : cluster 0 pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.336709+0000 osd.1 (osd.1) 113 : cluster 0 6.4 scrub starts 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.338739+0000 osd.1 (osd.1) 114 : cluster 0 6.4 scrub ok 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.494412+0000 osd.0 (osd.0) 109 : cluster 0 5.5 scrub starts 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.494998+0000 osd.3 (osd.3) 131 : cluster 0 6.13 scrub starts 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.497098+0000 osd.0 (osd.0) 110 : cluster 0 5.5 scrub ok 2024-04-06T03:26:16.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.582566+0000 osd.3 (osd.3) 132 : cluster 0 6.13 scrub ok 2024-04-06T03:26:16.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.583197+0000 osd.2 (osd.2) 91 : cluster 0 2.10 scrub starts 2024-04-06T03:26:16.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:15.587661+0000 osd.2 (osd.2) 92 : cluster 0 2.10 scrub ok 2024-04-06T03:26:16.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.006091+0000 osd.6 (osd.6) 77 : cluster 0 3.10 scrub starts 2024-04-06T03:26:16.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.011025+0000 osd.6 (osd.6) 78 : cluster 0 3.10 scrub ok 2024-04-06T03:26:16.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:16 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.061469+0000 mgr.y (mgr.24368) 200 : cluster 0 pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:26:17.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:17.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:17.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:17.184 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:17.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 41 16 56136 56120 85.5358 70.0625 0.00957809 0.0116848 2024-04-06T03:26:17.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:17.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:17.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:17.460 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:17.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:17.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:17.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:17.604 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.6 primary-affinity to 0 (802) 2024-04-06T03:26:17.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:17.709 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:17.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:17 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.330675+0000 osd.1 (osd.1) 115 : cluster 0 6.19 scrub starts 2024-04-06T03:26:17.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:17 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.331794+0000 osd.1 (osd.1) 116 : cluster 0 6.19 scrub ok 2024-04-06T03:26:17.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:17 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.637686+0000 osd.4 (osd.4) 105 : cluster 0 5.18 scrub starts 2024-04-06T03:26:17.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:17 smithi153 bash[23852]: cluster 2024-04-06T03:26:16.639557+0000 osd.4 (osd.4) 106 : cluster 0 5.18 scrub ok 2024-04-06T03:26:17.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:17 smithi153 bash[23852]: audit 2024-04-06T03:26:17.550540+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.50:0/1180747402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-04-06T03:26:17.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:17 smithi153 bash[23852]: audit 2024-04-06T03:26:17.551064+0000 mon.a (mon.0) 1028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-04-06T03:26:17.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:17.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:17.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:17.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:17.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:18.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:18.090 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:18.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:18.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.330675+0000 osd.1 (osd.1) 115 : cluster 0 6.19 scrub starts 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.331794+0000 osd.1 (osd.1) 116 : cluster 0 6.19 scrub ok 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.637686+0000 osd.4 (osd.4) 105 : cluster 0 5.18 scrub starts 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[21467]: cluster 2024-04-06T03:26:16.639557+0000 osd.4 (osd.4) 106 : cluster 0 5.18 scrub ok 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[21467]: audit 2024-04-06T03:26:17.550540+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.50:0/1180747402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[21467]: audit 2024-04-06T03:26:17.551064+0000 mon.a (mon.0) 1028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-04-06T03:26:18.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.330675+0000 osd.1 (osd.1) 115 : cluster 0 6.19 scrub starts 2024-04-06T03:26:18.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.331794+0000 osd.1 (osd.1) 116 : cluster 0 6.19 scrub ok 2024-04-06T03:26:18.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.637686+0000 osd.4 (osd.4) 105 : cluster 0 5.18 scrub starts 2024-04-06T03:26:18.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[29221]: cluster 2024-04-06T03:26:16.639557+0000 osd.4 (osd.4) 106 : cluster 0 5.18 scrub ok 2024-04-06T03:26:18.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[29221]: audit 2024-04-06T03:26:17.550540+0000 mon.c (mon.2) 40 : audit 1 from='client.? 172.21.15.50:0/1180747402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-04-06T03:26:18.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:17 smithi050 bash[29221]: audit 2024-04-06T03:26:17.551064+0000 mon.a (mon.0) 1028 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]: dispatch 2024-04-06T03:26:18.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 42 16 56469 56453 83.9947 20.8125 0.00335978 0.0117509 2024-04-06T03:26:18.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:18.350 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:18.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:18.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-06T03:26:18.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:18.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:18.615 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:18.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:18.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:18.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.452362+0000 osd.0 (osd.0) 111 : cluster 0 5.3 scrub starts 2024-04-06T03:26:18.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.460138+0000 osd.0 (osd.0) 112 : cluster 0 5.3 scrub ok 2024-04-06T03:26:18.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: audit 2024-04-06T03:26:17.607116+0000 mon.a (mon.0) 1029 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-04-06T03:26:18.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.610059+0000 mon.a (mon.0) 1030 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-04-06T03:26:18.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.729906+0000 osd.3 (osd.3) 133 : cluster 0 3.18 scrub starts 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.739659+0000 osd.3 (osd.3) 134 : cluster 0 3.18 scrub ok 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:18.062095+0000 mgr.y (mgr.24368) 201 : cluster 0 pgmap v178: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:18.426259+0000 osd.0 (osd.0) 113 : cluster 0 6.0 scrub starts 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: cluster 2024-04-06T03:26:18.429775+0000 osd.0 (osd.0) 114 : cluster 0 6.0 scrub ok 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[21467]: audit 2024-04-06T03:26:18.552669+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.452362+0000 osd.0 (osd.0) 111 : cluster 0 5.3 scrub starts 2024-04-06T03:26:18.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.460138+0000 osd.0 (osd.0) 112 : cluster 0 5.3 scrub ok 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: audit 2024-04-06T03:26:17.607116+0000 mon.a (mon.0) 1029 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.610059+0000 mon.a (mon.0) 1030 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.729906+0000 osd.3 (osd.3) 133 : cluster 0 3.18 scrub starts 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.739659+0000 osd.3 (osd.3) 134 : cluster 0 3.18 scrub ok 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:18.062095+0000 mgr.y (mgr.24368) 201 : cluster 0 pgmap v178: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:18.426259+0000 osd.0 (osd.0) 113 : cluster 0 6.0 scrub starts 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: cluster 2024-04-06T03:26:18.429775+0000 osd.0 (osd.0) 114 : cluster 0 6.0 scrub ok 2024-04-06T03:26:18.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:18 smithi050 bash[29221]: audit 2024-04-06T03:26:18.552669+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:18.742 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:18.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:18.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:18.880 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:18.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:18.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:19.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.452362+0000 osd.0 (osd.0) 111 : cluster 0 5.3 scrub starts 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.460138+0000 osd.0 (osd.0) 112 : cluster 0 5.3 scrub ok 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: audit 2024-04-06T03:26:17.607116+0000 mon.a (mon.0) 1029 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.610059+0000 mon.a (mon.0) 1030 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.729906+0000 osd.3 (osd.3) 133 : cluster 0 3.18 scrub starts 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.739659+0000 osd.3 (osd.3) 134 : cluster 0 3.18 scrub ok 2024-04-06T03:26:19.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:18.062095+0000 mgr.y (mgr.24368) 201 : cluster 0 pgmap v178: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-06T03:26:19.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:18.426259+0000 osd.0 (osd.0) 113 : cluster 0 6.0 scrub starts 2024-04-06T03:26:19.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: cluster 2024-04-06T03:26:18.429775+0000 osd.0 (osd.0) 114 : cluster 0 6.0 scrub ok 2024-04-06T03:26:19.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:18 smithi153 bash[23852]: audit 2024-04-06T03:26:18.552669+0000 mon.a (mon.0) 1031 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:19.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:19.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:19.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:19.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 43 16 57162 57146 83.0485 43.3125 0.0104277 0.0120353 2024-04-06T03:26:19.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:19.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:19.411 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:26:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:26:19.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:19.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:19.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:26:19.569 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:19.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:19.696 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.722790+0000 osd.4 (osd.4) 107 : cluster 0 2.f scrub starts 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:17.726750+0000 osd.4 (osd.4) 108 : cluster 0 2.f scrub ok 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:18.598576+0000 osd.3 (osd.3) 135 : cluster 0 3.7 scrub starts 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:18.602429+0000 osd.3 (osd.3) 136 : cluster 0 3.7 scrub ok 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:18.623071+0000 mon.a (mon.0) 1032 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: audit 2024-04-06T03:26:19.196021+0000 mgr.y (mgr.24368) 202 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:19.350780+0000 osd.1 (osd.1) 117 : cluster 0 2.d scrub starts 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: cluster 2024-04-06T03:26:19.356765+0000 osd.1 (osd.1) 118 : cluster 0 2.d scrub ok 2024-04-06T03:26:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[21467]: audit 2024-04-06T03:26:19.488812+0000 mon.a (mon.0) 1033 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.722790+0000 osd.4 (osd.4) 107 : cluster 0 2.f scrub starts 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:17.726750+0000 osd.4 (osd.4) 108 : cluster 0 2.f scrub ok 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:18.598576+0000 osd.3 (osd.3) 135 : cluster 0 3.7 scrub starts 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:18.602429+0000 osd.3 (osd.3) 136 : cluster 0 3.7 scrub ok 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:18.623071+0000 mon.a (mon.0) 1032 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: audit 2024-04-06T03:26:19.196021+0000 mgr.y (mgr.24368) 202 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:19.350780+0000 osd.1 (osd.1) 117 : cluster 0 2.d scrub starts 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: cluster 2024-04-06T03:26:19.356765+0000 osd.1 (osd.1) 118 : cluster 0 2.d scrub ok 2024-04-06T03:26:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:19 smithi050 bash[29221]: audit 2024-04-06T03:26:19.488812+0000 mon.a (mon.0) 1033 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:19.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:19.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:19.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:19.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.722790+0000 osd.4 (osd.4) 107 : cluster 0 2.f scrub starts 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:17.726750+0000 osd.4 (osd.4) 108 : cluster 0 2.f scrub ok 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:18.598576+0000 osd.3 (osd.3) 135 : cluster 0 3.7 scrub starts 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:18.602429+0000 osd.3 (osd.3) 136 : cluster 0 3.7 scrub ok 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:18.623071+0000 mon.a (mon.0) 1032 : cluster 0 osdmap e139: 8 total, 8 up, 8 in 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: audit 2024-04-06T03:26:19.196021+0000 mgr.y (mgr.24368) 202 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:19.350780+0000 osd.1 (osd.1) 117 : cluster 0 2.d scrub starts 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: cluster 2024-04-06T03:26:19.356765+0000 osd.1 (osd.1) 118 : cluster 0 2.d scrub ok 2024-04-06T03:26:19.939 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:19 smithi153 bash[23852]: audit 2024-04-06T03:26:19.488812+0000 mon.a (mon.0) 1033 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:26:19.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:19.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:20.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:20.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:20.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:20.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:20.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 44 16 58239 58223 82.6908 67.3125 0.00395389 0.0120872 2024-04-06T03:26:20.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:20.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:20.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:20.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:20.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:20.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:20.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:20.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:20.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: cluster 2024-04-06T03:26:19.519801+0000 osd.2 (osd.2) 93 : cluster 0 2.16 scrub starts 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: cluster 2024-04-06T03:26:19.521709+0000 osd.2 (osd.2) 94 : cluster 0 2.16 scrub ok 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: cluster 2024-04-06T03:26:19.947840+0000 osd.7 (osd.7) 117 : cluster 0 6.15 scrub starts 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: cluster 2024-04-06T03:26:19.955936+0000 osd.7 (osd.7) 118 : cluster 0 6.15 scrub ok 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: cluster 2024-04-06T03:26:20.063494+0000 mgr.y (mgr.24368) 203 : cluster 0 pgmap v180: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: audit 2024-04-06T03:26:20.379977+0000 mon.a (mon.0) 1034 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:20.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: audit 2024-04-06T03:26:20.383307+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:26:20.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:20 smithi153 bash[23852]: audit 2024-04-06T03:26:20.391898+0000 mon.a (mon.0) 1036 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:26:20.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:20.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:20.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:20.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: cluster 2024-04-06T03:26:19.519801+0000 osd.2 (osd.2) 93 : cluster 0 2.16 scrub starts 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: cluster 2024-04-06T03:26:19.521709+0000 osd.2 (osd.2) 94 : cluster 0 2.16 scrub ok 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: cluster 2024-04-06T03:26:19.947840+0000 osd.7 (osd.7) 117 : cluster 0 6.15 scrub starts 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: cluster 2024-04-06T03:26:19.955936+0000 osd.7 (osd.7) 118 : cluster 0 6.15 scrub ok 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: cluster 2024-04-06T03:26:20.063494+0000 mgr.y (mgr.24368) 203 : cluster 0 pgmap v180: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: audit 2024-04-06T03:26:20.379977+0000 mon.a (mon.0) 1034 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: audit 2024-04-06T03:26:20.383307+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:26:20.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[21467]: audit 2024-04-06T03:26:20.391898+0000 mon.a (mon.0) 1036 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: cluster 2024-04-06T03:26:19.519801+0000 osd.2 (osd.2) 93 : cluster 0 2.16 scrub starts 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: cluster 2024-04-06T03:26:19.521709+0000 osd.2 (osd.2) 94 : cluster 0 2.16 scrub ok 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: cluster 2024-04-06T03:26:19.947840+0000 osd.7 (osd.7) 117 : cluster 0 6.15 scrub starts 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: cluster 2024-04-06T03:26:19.955936+0000 osd.7 (osd.7) 118 : cluster 0 6.15 scrub ok 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: cluster 2024-04-06T03:26:20.063494+0000 mgr.y (mgr.24368) 203 : cluster 0 pgmap v180: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: audit 2024-04-06T03:26:20.379977+0000 mon.a (mon.0) 1034 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: audit 2024-04-06T03:26:20.383307+0000 mon.a (mon.0) 1035 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:26:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:20 smithi050 bash[29221]: audit 2024-04-06T03:26:20.391898+0000 mon.a (mon.0) 1036 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:26:20.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:21.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:21.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:21.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:21.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 45 15 59414 59399 82.4865 73.5 0.0294047 0.0121168 2024-04-06T03:26:21.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:21.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:21.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:21.512 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:21.514 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:21.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:21.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:21.644 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:21.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:21.777 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:21 smithi153 bash[23852]: cluster 2024-04-06T03:26:20.555800+0000 osd.2 (osd.2) 95 : cluster 0 5.d scrub starts 2024-04-06T03:26:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:21 smithi153 bash[23852]: cluster 2024-04-06T03:26:20.562543+0000 osd.2 (osd.2) 96 : cluster 0 5.d scrub ok 2024-04-06T03:26:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:21 smithi153 bash[23852]: cluster 2024-04-06T03:26:20.673176+0000 osd.4 (osd.4) 109 : cluster 0 3.a scrub starts 2024-04-06T03:26:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:21 smithi153 bash[23852]: cluster 2024-04-06T03:26:20.677269+0000 osd.4 (osd.4) 110 : cluster 0 3.a scrub ok 2024-04-06T03:26:21.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:21.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:21.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[21467]: cluster 2024-04-06T03:26:20.555800+0000 osd.2 (osd.2) 95 : cluster 0 5.d scrub starts 2024-04-06T03:26:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[21467]: cluster 2024-04-06T03:26:20.562543+0000 osd.2 (osd.2) 96 : cluster 0 5.d scrub ok 2024-04-06T03:26:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[21467]: cluster 2024-04-06T03:26:20.673176+0000 osd.4 (osd.4) 109 : cluster 0 3.a scrub starts 2024-04-06T03:26:21.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[21467]: cluster 2024-04-06T03:26:20.677269+0000 osd.4 (osd.4) 110 : cluster 0 3.a scrub ok 2024-04-06T03:26:21.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[29221]: cluster 2024-04-06T03:26:20.555800+0000 osd.2 (osd.2) 95 : cluster 0 5.d scrub starts 2024-04-06T03:26:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[29221]: cluster 2024-04-06T03:26:20.562543+0000 osd.2 (osd.2) 96 : cluster 0 5.d scrub ok 2024-04-06T03:26:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[29221]: cluster 2024-04-06T03:26:20.673176+0000 osd.4 (osd.4) 109 : cluster 0 3.a scrub starts 2024-04-06T03:26:21.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:21 smithi050 bash[29221]: cluster 2024-04-06T03:26:20.677269+0000 osd.4 (osd.4) 110 : cluster 0 3.a scrub ok 2024-04-06T03:26:22.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:22.130 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.151 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:22.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:22.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 46 16 60622 60606 82.3331 75.4375 0.0218833 0.0121403 2024-04-06T03:26:22.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:22.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:22.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:22.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:22.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.656 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:22.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:22.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:22 smithi153 bash[23852]: cluster 2024-04-06T03:26:21.630229+0000 osd.3 (osd.3) 137 : cluster 0 5.0 scrub starts 2024-04-06T03:26:22.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:22 smithi153 bash[23852]: cluster 2024-04-06T03:26:21.633966+0000 osd.3 (osd.3) 138 : cluster 0 5.0 scrub ok 2024-04-06T03:26:22.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:22 smithi153 bash[23852]: cluster 2024-04-06T03:26:21.979374+0000 osd.7 (osd.7) 119 : cluster 0 2.5 scrub starts 2024-04-06T03:26:22.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:22 smithi153 bash[23852]: cluster 2024-04-06T03:26:21.980343+0000 osd.7 (osd.7) 120 : cluster 0 2.5 scrub ok 2024-04-06T03:26:22.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:22 smithi153 bash[23852]: cluster 2024-04-06T03:26:22.063930+0000 mgr.y (mgr.24368) 204 : cluster 0 pgmap v181: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 29 MiB/s wr, 466 op/s 2024-04-06T03:26:22.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:22.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:23.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:23.037 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[21467]: cluster 2024-04-06T03:26:21.630229+0000 osd.3 (osd.3) 137 : cluster 0 5.0 scrub starts 2024-04-06T03:26:23.037 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[21467]: cluster 2024-04-06T03:26:21.633966+0000 osd.3 (osd.3) 138 : cluster 0 5.0 scrub ok 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[21467]: cluster 2024-04-06T03:26:21.979374+0000 osd.7 (osd.7) 119 : cluster 0 2.5 scrub starts 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[21467]: cluster 2024-04-06T03:26:21.980343+0000 osd.7 (osd.7) 120 : cluster 0 2.5 scrub ok 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[21467]: cluster 2024-04-06T03:26:22.063930+0000 mgr.y (mgr.24368) 204 : cluster 0 pgmap v181: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 29 MiB/s wr, 466 op/s 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[29221]: cluster 2024-04-06T03:26:21.630229+0000 osd.3 (osd.3) 137 : cluster 0 5.0 scrub starts 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[29221]: cluster 2024-04-06T03:26:21.633966+0000 osd.3 (osd.3) 138 : cluster 0 5.0 scrub ok 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[29221]: cluster 2024-04-06T03:26:21.979374+0000 osd.7 (osd.7) 119 : cluster 0 2.5 scrub starts 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[29221]: cluster 2024-04-06T03:26:21.980343+0000 osd.7 (osd.7) 120 : cluster 0 2.5 scrub ok 2024-04-06T03:26:23.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:22 smithi050 bash[29221]: cluster 2024-04-06T03:26:22.063930+0000 mgr.y (mgr.24368) 204 : cluster 0 pgmap v181: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 29 MiB/s wr, 466 op/s 2024-04-06T03:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:23.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:23.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:23.262 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 47 15 61310 61295 81.4975 43.0625 0.00436829 0.0122646 2024-04-06T03:26:23.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:23.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:23.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:23.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:23.562 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:23.563 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:23.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:23.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:23.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:23.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:23.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:23.948 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:23.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:23 smithi050 bash[21467]: cluster 2024-04-06T03:26:23.732314+0000 osd.2 (osd.2) 97 : cluster 0 2.17 scrub starts 2024-04-06T03:26:23.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:23 smithi050 bash[21467]: cluster 2024-04-06T03:26:23.734257+0000 osd.2 (osd.2) 98 : cluster 0 2.17 scrub ok 2024-04-06T03:26:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:23 smithi050 bash[29221]: cluster 2024-04-06T03:26:23.732314+0000 osd.2 (osd.2) 97 : cluster 0 2.17 scrub starts 2024-04-06T03:26:23.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:23 smithi050 bash[29221]: cluster 2024-04-06T03:26:23.734257+0000 osd.2 (osd.2) 98 : cluster 0 2.17 scrub ok 2024-04-06T03:26:24.016 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:26:24.016 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:26:24.016 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-06T03:26:24.016 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-04-06T03:26:24.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 2 1 2024-04-06T03:26:24.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:24.075 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:24.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:24.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:24.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 48 16 62195 62179 80.9507 55.25 0.0183809 0.0123474 2024-04-06T03:26:24.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:23 smithi153 bash[23852]: cluster 2024-04-06T03:26:23.732314+0000 osd.2 (osd.2) 97 : cluster 0 2.17 scrub starts 2024-04-06T03:26:24.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:23 smithi153 bash[23852]: cluster 2024-04-06T03:26:23.734257+0000 osd.2 (osd.2) 98 : cluster 0 2.17 scrub ok 2024-04-06T03:26:24.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:24.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:24.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:24.474 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:24.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:24.602 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:24.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:24.738 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:24.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:24.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:24.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:23.725014+0000 osd.4 (osd.4) 111 : cluster 0 3.1a scrub starts 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:23.735350+0000 osd.4 (osd.4) 112 : cluster 0 3.1a scrub ok 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.064735+0000 mgr.y (mgr.24368) 205 : cluster 0 pgmap v182: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 56 MiB/s wr, 903 op/s 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.419594+0000 osd.0 (osd.0) 115 : cluster 0 2.1f scrub starts 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.426329+0000 osd.0 (osd.0) 116 : cluster 0 2.1f scrub ok 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.770071+0000 osd.2 (osd.2) 99 : cluster 0 3.4 scrub starts 2024-04-06T03:26:24.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.776523+0000 osd.2 (osd.2) 100 : cluster 0 3.4 scrub ok 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:23.725014+0000 osd.4 (osd.4) 111 : cluster 0 3.1a scrub starts 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:23.735350+0000 osd.4 (osd.4) 112 : cluster 0 3.1a scrub ok 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.064735+0000 mgr.y (mgr.24368) 205 : cluster 0 pgmap v182: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 56 MiB/s wr, 903 op/s 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.419594+0000 osd.0 (osd.0) 115 : cluster 0 2.1f scrub starts 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.426329+0000 osd.0 (osd.0) 116 : cluster 0 2.1f scrub ok 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.770071+0000 osd.2 (osd.2) 99 : cluster 0 3.4 scrub starts 2024-04-06T03:26:24.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:24 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.776523+0000 osd.2 (osd.2) 100 : cluster 0 3.4 scrub ok 2024-04-06T03:26:24.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:25.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:25.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:25.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:25.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 49 16 63132 63116 80.4936 58.5625 0.0123228 0.0124185 2024-04-06T03:26:25.274 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:25.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:23.725014+0000 osd.4 (osd.4) 111 : cluster 0 3.1a scrub starts 2024-04-06T03:26:25.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:23.735350+0000 osd.4 (osd.4) 112 : cluster 0 3.1a scrub ok 2024-04-06T03:26:25.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.064735+0000 mgr.y (mgr.24368) 205 : cluster 0 pgmap v182: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 56 MiB/s wr, 903 op/s 2024-04-06T03:26:25.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.419594+0000 osd.0 (osd.0) 115 : cluster 0 2.1f scrub starts 2024-04-06T03:26:25.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.426329+0000 osd.0 (osd.0) 116 : cluster 0 2.1f scrub ok 2024-04-06T03:26:25.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.770071+0000 osd.2 (osd.2) 99 : cluster 0 3.4 scrub starts 2024-04-06T03:26:25.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:24 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.776523+0000 osd.2 (osd.2) 100 : cluster 0 3.4 scrub ok 2024-04-06T03:26:25.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:25.402 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:25.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:25.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:25.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:25.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:25.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:25.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:25.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:25.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:25.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:26.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:26.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:26.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:26.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.664028+0000 osd.3 (osd.3) 139 : cluster 0 2.14 scrub starts 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[21467]: cluster 2024-04-06T03:26:24.684230+0000 osd.3 (osd.3) 140 : cluster 0 2.14 scrub ok 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[21467]: cluster 2024-04-06T03:26:25.684416+0000 osd.4 (osd.4) 113 : cluster 0 3.14 scrub starts 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[21467]: cluster 2024-04-06T03:26:25.778150+0000 osd.2 (osd.2) 101 : cluster 0 5.b scrub starts 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[21467]: cluster 2024-04-06T03:26:25.781678+0000 osd.2 (osd.2) 102 : cluster 0 5.b scrub ok 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.664028+0000 osd.3 (osd.3) 139 : cluster 0 2.14 scrub starts 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[29221]: cluster 2024-04-06T03:26:24.684230+0000 osd.3 (osd.3) 140 : cluster 0 2.14 scrub ok 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[29221]: cluster 2024-04-06T03:26:25.684416+0000 osd.4 (osd.4) 113 : cluster 0 3.14 scrub starts 2024-04-06T03:26:26.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[29221]: cluster 2024-04-06T03:26:25.778150+0000 osd.2 (osd.2) 101 : cluster 0 5.b scrub starts 2024-04-06T03:26:26.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:25 smithi050 bash[29221]: cluster 2024-04-06T03:26:25.781678+0000 osd.2 (osd.2) 102 : cluster 0 5.b scrub ok 2024-04-06T03:26:26.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 50 16 64145 64129 80.1499 63.3125 0.00320257 0.0124698 2024-04-06T03:26:26.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:26.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:25 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.664028+0000 osd.3 (osd.3) 139 : cluster 0 2.14 scrub starts 2024-04-06T03:26:26.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:25 smithi153 bash[23852]: cluster 2024-04-06T03:26:24.684230+0000 osd.3 (osd.3) 140 : cluster 0 2.14 scrub ok 2024-04-06T03:26:26.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:25 smithi153 bash[23852]: cluster 2024-04-06T03:26:25.684416+0000 osd.4 (osd.4) 113 : cluster 0 3.14 scrub starts 2024-04-06T03:26:26.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:25 smithi153 bash[23852]: cluster 2024-04-06T03:26:25.778150+0000 osd.2 (osd.2) 101 : cluster 0 5.b scrub starts 2024-04-06T03:26:26.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:25 smithi153 bash[23852]: cluster 2024-04-06T03:26:25.781678+0000 osd.2 (osd.2) 102 : cluster 0 5.b scrub ok 2024-04-06T03:26:26.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:26.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:26.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:26.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:26.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:26.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:26.709 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:26.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:26.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:26.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:26.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:27.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:27.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:25.630274+0000 osd.3 (osd.3) 141 : cluster 0 5.17 scrub starts 2024-04-06T03:26:27.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:25.636661+0000 osd.3 (osd.3) 142 : cluster 0 5.17 scrub ok 2024-04-06T03:26:27.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:25.686123+0000 osd.4 (osd.4) 114 : cluster 0 3.14 scrub ok 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.064653+0000 osd.7 (osd.7) 121 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.064764+0000 osd.7 (osd.7) 122 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.065390+0000 mgr.y (mgr.24368) 206 : cluster 0 pgmap v183: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.678834+0000 osd.4 (osd.4) 115 : cluster 0 6.17 scrub starts 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.680008+0000 osd.4 (osd.4) 116 : cluster 0 6.17 scrub ok 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.750023+0000 osd.2 (osd.2) 103 : cluster 0 6.f scrub starts 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.759884+0000 osd.2 (osd.2) 104 : cluster 0 6.f scrub ok 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:25.630274+0000 osd.3 (osd.3) 141 : cluster 0 5.17 scrub starts 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:25.636661+0000 osd.3 (osd.3) 142 : cluster 0 5.17 scrub ok 2024-04-06T03:26:27.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:25.686123+0000 osd.4 (osd.4) 114 : cluster 0 3.14 scrub ok 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.064653+0000 osd.7 (osd.7) 121 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.064764+0000 osd.7 (osd.7) 122 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.065390+0000 mgr.y (mgr.24368) 206 : cluster 0 pgmap v183: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.678834+0000 osd.4 (osd.4) 115 : cluster 0 6.17 scrub starts 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.680008+0000 osd.4 (osd.4) 116 : cluster 0 6.17 scrub ok 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.750023+0000 osd.2 (osd.2) 103 : cluster 0 6.f scrub starts 2024-04-06T03:26:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:26 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.759884+0000 osd.2 (osd.2) 104 : cluster 0 6.f scrub ok 2024-04-06T03:26:27.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:27.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:27.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 51 16 65206 65190 79.8785 66.3125 0.00396787 0.0125107 2024-04-06T03:26:27.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:25.630274+0000 osd.3 (osd.3) 141 : cluster 0 5.17 scrub starts 2024-04-06T03:26:27.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:25.636661+0000 osd.3 (osd.3) 142 : cluster 0 5.17 scrub ok 2024-04-06T03:26:27.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:25.686123+0000 osd.4 (osd.4) 114 : cluster 0 3.14 scrub ok 2024-04-06T03:26:27.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.064653+0000 osd.7 (osd.7) 121 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:27.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.064764+0000 osd.7 (osd.7) 122 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:27.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.065390+0000 mgr.y (mgr.24368) 206 : cluster 0 pgmap v183: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-06T03:26:27.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.678834+0000 osd.4 (osd.4) 115 : cluster 0 6.17 scrub starts 2024-04-06T03:26:27.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.680008+0000 osd.4 (osd.4) 116 : cluster 0 6.17 scrub ok 2024-04-06T03:26:27.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.750023+0000 osd.2 (osd.2) 103 : cluster 0 6.f scrub starts 2024-04-06T03:26:27.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:26 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.759884+0000 osd.2 (osd.2) 104 : cluster 0 6.f scrub ok 2024-04-06T03:26:27.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:27.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:27.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:27.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:27.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:27.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:27.886 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:27.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:27.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:28.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:28.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:28.127 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:27 smithi153 bash[23852]: cluster 2024-04-06T03:26:26.937141+0000 mon.a (mon.0) 1037 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-04-06T03:26:28.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:27 smithi153 bash[23852]: cluster 2024-04-06T03:26:27.265876+0000 osd.1 (osd.1) 119 : cluster 0 3.19 scrub starts 2024-04-06T03:26:28.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:27 smithi153 bash[23852]: cluster 2024-04-06T03:26:27.269108+0000 osd.1 (osd.1) 120 : cluster 0 3.19 scrub ok 2024-04-06T03:26:28.141 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:28.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:27 smithi050 bash[21467]: cluster 2024-04-06T03:26:26.937141+0000 mon.a (mon.0) 1037 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-04-06T03:26:28.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:27 smithi050 bash[21467]: cluster 2024-04-06T03:26:27.265876+0000 osd.1 (osd.1) 119 : cluster 0 3.19 scrub starts 2024-04-06T03:26:28.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:27 smithi050 bash[21467]: cluster 2024-04-06T03:26:27.269108+0000 osd.1 (osd.1) 120 : cluster 0 3.19 scrub ok 2024-04-06T03:26:28.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:27 smithi050 bash[29221]: cluster 2024-04-06T03:26:26.937141+0000 mon.a (mon.0) 1037 : cluster 0 osdmap e140: 8 total, 8 up, 8 in 2024-04-06T03:26:28.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:27 smithi050 bash[29221]: cluster 2024-04-06T03:26:27.265876+0000 osd.1 (osd.1) 119 : cluster 0 3.19 scrub starts 2024-04-06T03:26:28.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:27 smithi050 bash[29221]: cluster 2024-04-06T03:26:27.269108+0000 osd.1 (osd.1) 120 : cluster 0 3.19 scrub ok 2024-04-06T03:26:28.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:28.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 52 16 66319 66303 79.68 69.5625 0.0131093 0.0125446 2024-04-06T03:26:28.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:28.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:28.396 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:28.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:28.523 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:28.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:28.656 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:28.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:28.780 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:28.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:28.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:28.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:29.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:29.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:29.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:29.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:29.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 53 16 67349 67333 79.3911 64.375 0.00433285 0.0125897 2024-04-06T03:26:29.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:26:29.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:29.409 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:26:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:26:29.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.446 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:29.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:27.963232+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:29.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:27.963285+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:29.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:28.065754+0000 mgr.y (mgr.24368) 207 : cluster 0 pgmap v185: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 974 B/s rd, 58 MiB/s wr, 933 op/s 2024-04-06T03:26:29.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:28.220517+0000 osd.1 (osd.1) 121 : cluster 0 2.2 scrub starts 2024-04-06T03:26:29.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:28.226180+0000 osd.1 (osd.1) 122 : cluster 0 2.2 scrub ok 2024-04-06T03:26:29.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:28.420097+0000 mon.a (mon.0) 1038 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-04-06T03:26:29.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.047094+0000 osd.7 (osd.7) 123 : cluster 0 3.13 scrub starts 2024-04-06T03:26:29.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:29 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.051898+0000 osd.7 (osd.7) 124 : cluster 0 3.13 scrub ok 2024-04-06T03:26:29.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:29.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:29.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:29.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:29.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:27.963232+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:27.963285+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:28.065754+0000 mgr.y (mgr.24368) 207 : cluster 0 pgmap v185: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 974 B/s rd, 58 MiB/s wr, 933 op/s 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:28.220517+0000 osd.1 (osd.1) 121 : cluster 0 2.2 scrub starts 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:28.226180+0000 osd.1 (osd.1) 122 : cluster 0 2.2 scrub ok 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:28.420097+0000 mon.a (mon.0) 1038 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.047094+0000 osd.7 (osd.7) 123 : cluster 0 3.13 scrub starts 2024-04-06T03:26:29.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.051898+0000 osd.7 (osd.7) 124 : cluster 0 3.13 scrub ok 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:27.963232+0000 osd.4 (osd.4) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:27.963285+0000 osd.4 (osd.4) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:28.065754+0000 mgr.y (mgr.24368) 207 : cluster 0 pgmap v185: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 974 B/s rd, 58 MiB/s wr, 933 op/s 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:28.220517+0000 osd.1 (osd.1) 121 : cluster 0 2.2 scrub starts 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:28.226180+0000 osd.1 (osd.1) 122 : cluster 0 2.2 scrub ok 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:28.420097+0000 mon.a (mon.0) 1038 : cluster 0 osdmap e141: 8 total, 8 up, 8 in 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.047094+0000 osd.7 (osd.7) 123 : cluster 0 3.13 scrub starts 2024-04-06T03:26:29.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:29 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.051898+0000 osd.7 (osd.7) 124 : cluster 0 3.13 scrub ok 2024-04-06T03:26:29.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:29.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:29.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:30.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:30.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:30.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:30.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:30.263 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 54 16 67990 67974 78.6628 40.0625 0.0115114 0.0127068 2024-04-06T03:26:30.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:30.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:30.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: audit 2024-04-06T03:26:29.204114+0000 mgr.y (mgr.24368) 208 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.660750+0000 osd.4 (osd.4) 119 : cluster 0 5.e scrub starts 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.662112+0000 osd.4 (osd.4) 120 : cluster 0 5.e scrub ok 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.726646+0000 osd.2 (osd.2) 105 : cluster 0 5.6 scrub starts 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.727406+0000 osd.2 (osd.2) 106 : cluster 0 5.6 scrub ok 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: cluster 2024-04-06T03:26:30.068261+0000 osd.7 (osd.7) 125 : cluster 0 5.1a scrub starts 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[21467]: cluster 2024-04-06T03:26:30.070176+0000 osd.7 (osd.7) 126 : cluster 0 5.1a scrub ok 2024-04-06T03:26:30.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: audit 2024-04-06T03:26:29.204114+0000 mgr.y (mgr.24368) 208 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:30.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.660750+0000 osd.4 (osd.4) 119 : cluster 0 5.e scrub starts 2024-04-06T03:26:30.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.662112+0000 osd.4 (osd.4) 120 : cluster 0 5.e scrub ok 2024-04-06T03:26:30.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.726646+0000 osd.2 (osd.2) 105 : cluster 0 5.6 scrub starts 2024-04-06T03:26:30.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.727406+0000 osd.2 (osd.2) 106 : cluster 0 5.6 scrub ok 2024-04-06T03:26:30.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: cluster 2024-04-06T03:26:30.068261+0000 osd.7 (osd.7) 125 : cluster 0 5.1a scrub starts 2024-04-06T03:26:30.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:30 smithi050 bash[29221]: cluster 2024-04-06T03:26:30.070176+0000 osd.7 (osd.7) 126 : cluster 0 5.1a scrub ok 2024-04-06T03:26:30.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:30.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:30.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: audit 2024-04-06T03:26:29.204114+0000 mgr.y (mgr.24368) 208 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:30.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.660750+0000 osd.4 (osd.4) 119 : cluster 0 5.e scrub starts 2024-04-06T03:26:30.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.662112+0000 osd.4 (osd.4) 120 : cluster 0 5.e scrub ok 2024-04-06T03:26:30.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.726646+0000 osd.2 (osd.2) 105 : cluster 0 5.6 scrub starts 2024-04-06T03:26:30.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.727406+0000 osd.2 (osd.2) 106 : cluster 0 5.6 scrub ok 2024-04-06T03:26:30.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: cluster 2024-04-06T03:26:30.068261+0000 osd.7 (osd.7) 125 : cluster 0 5.1a scrub starts 2024-04-06T03:26:30.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:30 smithi153 bash[23852]: cluster 2024-04-06T03:26:30.070176+0000 osd.7 (osd.7) 126 : cluster 0 5.1a scrub ok 2024-04-06T03:26:30.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:30.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:30.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:30.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:30.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:30.987 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:30.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:31.092 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:31.127 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:31.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:31.262 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:31.265 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 55 16 69064 69048 78.45 67.125 0.0275138 0.0127422 2024-04-06T03:26:31.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:31.396 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:31.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:31.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.646772+0000 osd.3 (osd.3) 143 : cluster 0 6.11 scrub starts 2024-04-06T03:26:31.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[21467]: cluster 2024-04-06T03:26:29.654188+0000 osd.3 (osd.3) 144 : cluster 0 6.11 scrub ok 2024-04-06T03:26:31.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[21467]: cluster 2024-04-06T03:26:30.067164+0000 mgr.y (mgr.24368) 209 : cluster 0 pgmap v187: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[21467]: cluster 2024-04-06T03:26:31.089502+0000 osd.7 (osd.7) 127 : cluster 0 6.5 scrub starts 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[21467]: cluster 2024-04-06T03:26:31.095718+0000 osd.7 (osd.7) 128 : cluster 0 6.5 scrub ok 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.646772+0000 osd.3 (osd.3) 143 : cluster 0 6.11 scrub starts 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[29221]: cluster 2024-04-06T03:26:29.654188+0000 osd.3 (osd.3) 144 : cluster 0 6.11 scrub ok 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[29221]: cluster 2024-04-06T03:26:30.067164+0000 mgr.y (mgr.24368) 209 : cluster 0 pgmap v187: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[29221]: cluster 2024-04-06T03:26:31.089502+0000 osd.7 (osd.7) 127 : cluster 0 6.5 scrub starts 2024-04-06T03:26:31.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:31 smithi050 bash[29221]: cluster 2024-04-06T03:26:31.095718+0000 osd.7 (osd.7) 128 : cluster 0 6.5 scrub ok 2024-04-06T03:26:31.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:31.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:31.665 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:31.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:31.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:31 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.646772+0000 osd.3 (osd.3) 143 : cluster 0 6.11 scrub starts 2024-04-06T03:26:31.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:31 smithi153 bash[23852]: cluster 2024-04-06T03:26:29.654188+0000 osd.3 (osd.3) 144 : cluster 0 6.11 scrub ok 2024-04-06T03:26:31.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:31 smithi153 bash[23852]: cluster 2024-04-06T03:26:30.067164+0000 mgr.y (mgr.24368) 209 : cluster 0 pgmap v187: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:26:31.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:31 smithi153 bash[23852]: cluster 2024-04-06T03:26:31.089502+0000 osd.7 (osd.7) 127 : cluster 0 6.5 scrub starts 2024-04-06T03:26:31.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:31 smithi153 bash[23852]: cluster 2024-04-06T03:26:31.095718+0000 osd.7 (osd.7) 128 : cluster 0 6.5 scrub ok 2024-04-06T03:26:31.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:31.807 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:31.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:31.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:31.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:32.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:32.048 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:32.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:32.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:32.265 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 56 16 69764 69748 77.8303 43.75 0.00276979 0.0128178 2024-04-06T03:26:32.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:32.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:32.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:32.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:32 smithi153 bash[23852]: cluster 2024-04-06T03:26:31.247115+0000 osd.1 (osd.1) 123 : cluster 0 5.c scrub starts 2024-04-06T03:26:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:32 smithi153 bash[23852]: cluster 2024-04-06T03:26:31.256715+0000 osd.1 (osd.1) 124 : cluster 0 5.c scrub ok 2024-04-06T03:26:32.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:32.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:32.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:32.704 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:32 smithi050 bash[21467]: cluster 2024-04-06T03:26:31.247115+0000 osd.1 (osd.1) 123 : cluster 0 5.c scrub starts 2024-04-06T03:26:32.704 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:32 smithi050 bash[21467]: cluster 2024-04-06T03:26:31.256715+0000 osd.1 (osd.1) 124 : cluster 0 5.c scrub ok 2024-04-06T03:26:32.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:32 smithi050 bash[29221]: cluster 2024-04-06T03:26:31.247115+0000 osd.1 (osd.1) 123 : cluster 0 5.c scrub starts 2024-04-06T03:26:32.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:32 smithi050 bash[29221]: cluster 2024-04-06T03:26:31.256715+0000 osd.1 (osd.1) 124 : cluster 0 5.c scrub ok 2024-04-06T03:26:32.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:32.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:32.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:32.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:32.979 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:32.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:33.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:33.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:33.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:33.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:33.265 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 57 16 70566 70550 77.3442 50.125 0.0255774 0.0129225 2024-04-06T03:26:33.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:33.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:33.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.033382+0000 osd.0 (osd.0) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.033480+0000 osd.0 (osd.0) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.067851+0000 mgr.y (mgr.24368) 210 : cluster 0 pgmap v188: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.240890+0000 osd.1 (osd.1) 125 : cluster 0 2.15 scrub starts 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.242193+0000 osd.1 (osd.1) 126 : cluster 0 2.15 scrub ok 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.433032+0000 mon.a (mon.0) 1039 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.764887+0000 osd.2 (osd.2) 107 : cluster 0 7.e scrub starts 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.072331+0000 osd.7 (osd.7) 129 : cluster 0 6.12 scrub starts 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.073502+0000 osd.7 (osd.7) 130 : cluster 0 6.12 scrub ok 2024-04-06T03:26:33.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:33 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.421417+0000 mon.a (mon.0) 1040 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-04-06T03:26:33.511 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:33.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:33.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.033382+0000 osd.0 (osd.0) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.033480+0000 osd.0 (osd.0) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.067851+0000 mgr.y (mgr.24368) 210 : cluster 0 pgmap v188: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.240890+0000 osd.1 (osd.1) 125 : cluster 0 2.15 scrub starts 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.242193+0000 osd.1 (osd.1) 126 : cluster 0 2.15 scrub ok 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.433032+0000 mon.a (mon.0) 1039 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.764887+0000 osd.2 (osd.2) 107 : cluster 0 7.e scrub starts 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.072331+0000 osd.7 (osd.7) 129 : cluster 0 6.12 scrub starts 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.073502+0000 osd.7 (osd.7) 130 : cluster 0 6.12 scrub ok 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.421417+0000 mon.a (mon.0) 1040 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.033382+0000 osd.0 (osd.0) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:33.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.033480+0000 osd.0 (osd.0) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.067851+0000 mgr.y (mgr.24368) 210 : cluster 0 pgmap v188: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 840 op/s 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.240890+0000 osd.1 (osd.1) 125 : cluster 0 2.15 scrub starts 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.242193+0000 osd.1 (osd.1) 126 : cluster 0 2.15 scrub ok 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.433032+0000 mon.a (mon.0) 1039 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.764887+0000 osd.2 (osd.2) 107 : cluster 0 7.e scrub starts 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.072331+0000 osd.7 (osd.7) 129 : cluster 0 6.12 scrub starts 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.073502+0000 osd.7 (osd.7) 130 : cluster 0 6.12 scrub ok 2024-04-06T03:26:33.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:33 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.421417+0000 mon.a (mon.0) 1040 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2024-04-06T03:26:33.647 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:33.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:33.775 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:33.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:33.901 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:33.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:34.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:34.026 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:34.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:34.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:34.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:34.265 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 58 16 71446 71430 76.9589 55 0.00676796 0.0129883 2024-04-06T03:26:34.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:34.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:34.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:34.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:34.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:34.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:34.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.705340+0000 osd.4 (osd.4) 121 : cluster 0 5.f scrub starts 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:32.706346+0000 osd.4 (osd.4) 122 : cluster 0 5.f scrub ok 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.131824+0000 osd.3 (osd.3) 145 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.131888+0000 osd.3 (osd.3) 146 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.217169+0000 osd.1 (osd.1) 127 : cluster 0 5.12 scrub starts 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.222904+0000 osd.1 (osd.1) 128 : cluster 0 5.12 scrub ok 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: audit 2024-04-06T03:26:33.581424+0000 mon.a (mon.0) 1041 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.737886+0000 osd.2 (osd.2) 108 : cluster 0 5.8 scrub starts 2024-04-06T03:26:34.665 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:34 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.742198+0000 osd.2 (osd.2) 109 : cluster 0 5.8 scrub ok 2024-04-06T03:26:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:34.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.705340+0000 osd.4 (osd.4) 121 : cluster 0 5.f scrub starts 2024-04-06T03:26:34.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:32.706346+0000 osd.4 (osd.4) 122 : cluster 0 5.f scrub ok 2024-04-06T03:26:34.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.131824+0000 osd.3 (osd.3) 145 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:34.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.131888+0000 osd.3 (osd.3) 146 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:34.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.217169+0000 osd.1 (osd.1) 127 : cluster 0 5.12 scrub starts 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.222904+0000 osd.1 (osd.1) 128 : cluster 0 5.12 scrub ok 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: audit 2024-04-06T03:26:33.581424+0000 mon.a (mon.0) 1041 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.737886+0000 osd.2 (osd.2) 108 : cluster 0 5.8 scrub starts 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.742198+0000 osd.2 (osd.2) 109 : cluster 0 5.8 scrub ok 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.705340+0000 osd.4 (osd.4) 121 : cluster 0 5.f scrub starts 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:32.706346+0000 osd.4 (osd.4) 122 : cluster 0 5.f scrub ok 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.131824+0000 osd.3 (osd.3) 145 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.131888+0000 osd.3 (osd.3) 146 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.217169+0000 osd.1 (osd.1) 127 : cluster 0 5.12 scrub starts 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.222904+0000 osd.1 (osd.1) 128 : cluster 0 5.12 scrub ok 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: audit 2024-04-06T03:26:33.581424+0000 mon.a (mon.0) 1041 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:34.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.737886+0000 osd.2 (osd.2) 108 : cluster 0 5.8 scrub starts 2024-04-06T03:26:34.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:34 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.742198+0000 osd.2 (osd.2) 109 : cluster 0 5.8 scrub ok 2024-04-06T03:26:34.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:34.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:34.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:34.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:34.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:35.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:35.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:35.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 59 16 72061 72045 76.306 38.4375 0.0110548 0.0131005 2024-04-06T03:26:35.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:35.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:35.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:35.477 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:35.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.608432+0000 osd.3 (osd.3) 147 : cluster 0 2.a scrub starts 2024-04-06T03:26:35.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[21467]: cluster 2024-04-06T03:26:33.611179+0000 osd.3 (osd.3) 148 : cluster 0 2.a scrub ok 2024-04-06T03:26:35.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[21467]: cluster 2024-04-06T03:26:34.068600+0000 mgr.y (mgr.24368) 211 : cluster 0 pgmap v191: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-06T03:26:35.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[21467]: cluster 2024-04-06T03:26:34.491845+0000 osd.2 (osd.2) 110 : cluster 0 7.e scrub ok 2024-04-06T03:26:35.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.608432+0000 osd.3 (osd.3) 147 : cluster 0 2.a scrub starts 2024-04-06T03:26:35.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[29221]: cluster 2024-04-06T03:26:33.611179+0000 osd.3 (osd.3) 148 : cluster 0 2.a scrub ok 2024-04-06T03:26:35.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[29221]: cluster 2024-04-06T03:26:34.068600+0000 mgr.y (mgr.24368) 211 : cluster 0 pgmap v191: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-06T03:26:35.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:35 smithi050 bash[29221]: cluster 2024-04-06T03:26:34.491845+0000 osd.2 (osd.2) 110 : cluster 0 7.e scrub ok 2024-04-06T03:26:35.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:35.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.743 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:35.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:35 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.608432+0000 osd.3 (osd.3) 147 : cluster 0 2.a scrub starts 2024-04-06T03:26:35.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:35 smithi153 bash[23852]: cluster 2024-04-06T03:26:33.611179+0000 osd.3 (osd.3) 148 : cluster 0 2.a scrub ok 2024-04-06T03:26:35.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:35 smithi153 bash[23852]: cluster 2024-04-06T03:26:34.068600+0000 mgr.y (mgr.24368) 211 : cluster 0 pgmap v191: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-04-06T03:26:35.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:35 smithi153 bash[23852]: cluster 2024-04-06T03:26:34.491845+0000 osd.2 (osd.2) 110 : cluster 0 7.e scrub ok 2024-04-06T03:26:35.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:35.872 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:35.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:35.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:36.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:36.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:36.131 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:36.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:36.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:36.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:26:36.275907+0000 min lat: 0.00125923 max lat: 1.17752 avg lat: 0.0131384 2024-04-06T03:26:36.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:26:36.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 60 16 73066 73050 76.081 62.8125 0.0136238 0.0131384 2024-04-06T03:26:36.322 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:26:36.322 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:26:36.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:36.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:36.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:36.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[29221]: cluster 2024-04-06T03:26:34.641251+0000 osd.3 (osd.3) 149 : cluster 0 3.1e scrub starts 2024-04-06T03:26:36.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[29221]: cluster 2024-04-06T03:26:34.645326+0000 osd.3 (osd.3) 150 : cluster 0 3.1e scrub ok 2024-04-06T03:26:36.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[29221]: cluster 2024-04-06T03:26:35.276314+0000 osd.0 (osd.0) 119 : cluster 0 6.18 scrub starts 2024-04-06T03:26:36.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[29221]: cluster 2024-04-06T03:26:35.279811+0000 osd.0 (osd.0) 120 : cluster 0 6.18 scrub ok 2024-04-06T03:26:36.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.086026+0000 osd.7 (osd.7) 131 : cluster 0 5.10 scrub starts 2024-04-06T03:26:36.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.089563+0000 osd.7 (osd.7) 132 : cluster 0 5.10 scrub ok 2024-04-06T03:26:36.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[21467]: cluster 2024-04-06T03:26:34.641251+0000 osd.3 (osd.3) 149 : cluster 0 3.1e scrub starts 2024-04-06T03:26:36.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[21467]: cluster 2024-04-06T03:26:34.645326+0000 osd.3 (osd.3) 150 : cluster 0 3.1e scrub ok 2024-04-06T03:26:36.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[21467]: cluster 2024-04-06T03:26:35.276314+0000 osd.0 (osd.0) 119 : cluster 0 6.18 scrub starts 2024-04-06T03:26:36.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[21467]: cluster 2024-04-06T03:26:35.279811+0000 osd.0 (osd.0) 120 : cluster 0 6.18 scrub ok 2024-04-06T03:26:36.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.086026+0000 osd.7 (osd.7) 131 : cluster 0 5.10 scrub starts 2024-04-06T03:26:36.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:36 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.089563+0000 osd.7 (osd.7) 132 : cluster 0 5.10 scrub ok 2024-04-06T03:26:36.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:36.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:36.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:36.658 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:26:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:36.691 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:26:36.733 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:26:36.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:36.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:36 smithi153 bash[23852]: cluster 2024-04-06T03:26:34.641251+0000 osd.3 (osd.3) 149 : cluster 0 3.1e scrub starts 2024-04-06T03:26:36.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:36 smithi153 bash[23852]: cluster 2024-04-06T03:26:34.645326+0000 osd.3 (osd.3) 150 : cluster 0 3.1e scrub ok 2024-04-06T03:26:36.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:36 smithi153 bash[23852]: cluster 2024-04-06T03:26:35.276314+0000 osd.0 (osd.0) 119 : cluster 0 6.18 scrub starts 2024-04-06T03:26:36.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:36 smithi153 bash[23852]: cluster 2024-04-06T03:26:35.279811+0000 osd.0 (osd.0) 120 : cluster 0 6.18 scrub ok 2024-04-06T03:26:36.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:36 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.086026+0000 osd.7 (osd.7) 131 : cluster 0 5.10 scrub starts 2024-04-06T03:26:36.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:36 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.089563+0000 osd.7 (osd.7) 132 : cluster 0 5.10 scrub ok 2024-04-06T03:26:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:36.799 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:26:36.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:26:36.859 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:26:36.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:36.905 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:26:36.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:36.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:36.944 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:26:36.954 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:26:36.954 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:26:36.954 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:26:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:26:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:26:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:26:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:26:36.955 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:26:37.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:37.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:37.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:37.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 61 16 73947 73931 75.7364 55.0625 0.00648737 0.0131978 2024-04-06T03:26:37.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:37.302 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:37.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:37.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:37.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:37.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.069109+0000 mgr.y (mgr.24368) 212 : cluster 0 pgmap v192: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-04-06T03:26:37.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[21467]: audit 2024-04-06T03:26:36.601034+0000 mon.c (mon.2) 41 : audit 0 from='client.? 172.21.15.50:0/3936892460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:26:37.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.783667+0000 osd.2 (osd.2) 111 : cluster 0 5.a scrub starts 2024-04-06T03:26:37.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.792214+0000 osd.2 (osd.2) 112 : cluster 0 5.a scrub ok 2024-04-06T03:26:37.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:37.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.069109+0000 mgr.y (mgr.24368) 212 : cluster 0 pgmap v192: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-04-06T03:26:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[29221]: audit 2024-04-06T03:26:36.601034+0000 mon.c (mon.2) 41 : audit 0 from='client.? 172.21.15.50:0/3936892460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:26:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.783667+0000 osd.2 (osd.2) 111 : cluster 0 5.a scrub starts 2024-04-06T03:26:37.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:37 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.792214+0000 osd.2 (osd.2) 112 : cluster 0 5.a scrub ok 2024-04-06T03:26:37.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:37 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.069109+0000 mgr.y (mgr.24368) 212 : cluster 0 pgmap v192: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 76 MiB/s wr, 1.23k op/s 2024-04-06T03:26:37.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:37 smithi153 bash[23852]: audit 2024-04-06T03:26:36.601034+0000 mon.c (mon.2) 41 : audit 0 from='client.? 172.21.15.50:0/3936892460' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:26:37.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:37 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.783667+0000 osd.2 (osd.2) 111 : cluster 0 5.a scrub starts 2024-04-06T03:26:37.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:37 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.792214+0000 osd.2 (osd.2) 112 : cluster 0 5.a scrub ok 2024-04-06T03:26:37.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:37.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:37.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:37.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:37.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:38.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:38.085 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:38.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:38.217 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:38.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 62 16 74661 74645 75.2346 44.625 0.00675133 0.0132865 2024-04-06T03:26:38.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:38.340 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:38.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:38.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:38.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:38.591 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:38.592 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:38 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.680620+0000 osd.3 (osd.3) 151 : cluster 0 4.0 scrub starts 2024-04-06T03:26:38.592 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:38 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.696244+0000 osd.3 (osd.3) 152 : cluster 0 4.0 scrub ok 2024-04-06T03:26:38.592 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:38 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.813884+0000 osd.4 (osd.4) 123 : cluster 0 2.11 scrub starts 2024-04-06T03:26:38.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:38 smithi153 bash[23852]: cluster 2024-04-06T03:26:36.814728+0000 osd.4 (osd.4) 124 : cluster 0 2.11 scrub ok 2024-04-06T03:26:38.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:38 smithi153 bash[23852]: cluster 2024-04-06T03:26:38.426573+0000 mon.a (mon.0) 1042 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-04-06T03:26:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:38.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:38.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:38.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.680620+0000 osd.3 (osd.3) 151 : cluster 0 4.0 scrub starts 2024-04-06T03:26:38.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.696244+0000 osd.3 (osd.3) 152 : cluster 0 4.0 scrub ok 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.813884+0000 osd.4 (osd.4) 123 : cluster 0 2.11 scrub starts 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[21467]: cluster 2024-04-06T03:26:36.814728+0000 osd.4 (osd.4) 124 : cluster 0 2.11 scrub ok 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[21467]: cluster 2024-04-06T03:26:38.426573+0000 mon.a (mon.0) 1042 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.680620+0000 osd.3 (osd.3) 151 : cluster 0 4.0 scrub starts 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.696244+0000 osd.3 (osd.3) 152 : cluster 0 4.0 scrub ok 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.813884+0000 osd.4 (osd.4) 123 : cluster 0 2.11 scrub starts 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[29221]: cluster 2024-04-06T03:26:36.814728+0000 osd.4 (osd.4) 124 : cluster 0 2.11 scrub ok 2024-04-06T03:26:38.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:38 smithi050 bash[29221]: cluster 2024-04-06T03:26:38.426573+0000 mon.a (mon.0) 1042 : cluster 0 osdmap e144: 8 total, 8 up, 8 in 2024-04-06T03:26:38.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:38.844 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:38.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:38.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:39.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:39.101 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:39.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:39.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:39.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 63 16 75288 75272 74.6624 39.1875 0.197333 0.0133534 2024-04-06T03:26:39.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:39.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:39.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:39.480 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:26:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:26:39.490 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:26:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:39.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[21467]: cluster 2024-04-06T03:26:38.069581+0000 mgr.y (mgr.24368) 213 : cluster 0 pgmap v193: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 529 op/s 2024-04-06T03:26:39.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[21467]: audit 2024-04-06T03:26:39.081956+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.50:0/3722503647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:26:39.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[21467]: audit 2024-04-06T03:26:39.082350+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:26:39.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[29221]: cluster 2024-04-06T03:26:38.069581+0000 mgr.y (mgr.24368) 213 : cluster 0 pgmap v193: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 529 op/s 2024-04-06T03:26:39.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[29221]: audit 2024-04-06T03:26:39.081956+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.50:0/3722503647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:26:39.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[29221]: audit 2024-04-06T03:26:39.082350+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:26:39.538 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:26:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:26:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:26:39.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:39.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:39.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:39 smithi153 bash[23852]: cluster 2024-04-06T03:26:38.069581+0000 mgr.y (mgr.24368) 213 : cluster 0 pgmap v193: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 529 op/s 2024-04-06T03:26:39.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:39 smithi153 bash[23852]: audit 2024-04-06T03:26:39.081956+0000 mon.c (mon.2) 42 : audit 1 from='client.? 172.21.15.50:0/3722503647' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:26:39.733 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:39 smithi153 bash[23852]: audit 2024-04-06T03:26:39.082350+0000 mon.a (mon.0) 1043 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:26:39.747 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:39.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:39.850 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:39.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-06T03:26:39.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:39.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:39.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:40.001 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:40.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:40.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:40.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:40.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:40.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:40.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 64 16 76114 76098 74.3024 51.625 0.0109609 0.0134525 2024-04-06T03:26:40.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:40.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:40.512 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:40.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[21467]: audit 2024-04-06T03:26:39.212490+0000 mgr.y (mgr.24368) 214 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[21467]: cluster 2024-04-06T03:26:39.482609+0000 mon.a (mon.0) 1044 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[21467]: cluster 2024-04-06T03:26:39.482672+0000 mon.a (mon.0) 1045 : cluster 1 Cluster is now healthy 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[21467]: audit 2024-04-06T03:26:39.483985+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[21467]: cluster 2024-04-06T03:26:39.486990+0000 mon.a (mon.0) 1047 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[29221]: audit 2024-04-06T03:26:39.212490+0000 mgr.y (mgr.24368) 214 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[29221]: cluster 2024-04-06T03:26:39.482609+0000 mon.a (mon.0) 1044 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[29221]: cluster 2024-04-06T03:26:39.482672+0000 mon.a (mon.0) 1045 : cluster 1 Cluster is now healthy 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[29221]: audit 2024-04-06T03:26:39.483985+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:26:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:40 smithi050 bash[29221]: cluster 2024-04-06T03:26:39.486990+0000 mon.a (mon.0) 1047 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-04-06T03:26:40.653 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:40.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:40.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:40 smithi153 bash[23852]: audit 2024-04-06T03:26:39.212490+0000 mgr.y (mgr.24368) 214 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:40.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:40 smithi153 bash[23852]: cluster 2024-04-06T03:26:39.482609+0000 mon.a (mon.0) 1044 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:26:40.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:40 smithi153 bash[23852]: cluster 2024-04-06T03:26:39.482672+0000 mon.a (mon.0) 1045 : cluster 1 Cluster is now healthy 2024-04-06T03:26:40.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:40 smithi153 bash[23852]: audit 2024-04-06T03:26:39.483985+0000 mon.a (mon.0) 1046 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:26:40.767 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:40 smithi153 bash[23852]: cluster 2024-04-06T03:26:39.486990+0000 mon.a (mon.0) 1047 : cluster 0 osdmap e145: 8 total, 8 up, 8 in 2024-04-06T03:26:40.782 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:40.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:40.915 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:40.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:41.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:41.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:41.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:41.266 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 65 16 77020 77004 74.0303 56.625 0.00831324 0.0135026 2024-04-06T03:26:41.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:41.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:41.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:41.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:41.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:41.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:41 smithi153 bash[23852]: cluster 2024-04-06T03:26:40.070074+0000 mgr.y (mgr.24368) 215 : cluster 0 pgmap v196: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-06T03:26:41.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:41 smithi153 bash[23852]: cluster 2024-04-06T03:26:40.747386+0000 osd.3 (osd.3) 153 : cluster 0 6.1f scrub starts 2024-04-06T03:26:41.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:41 smithi153 bash[23852]: cluster 2024-04-06T03:26:40.749056+0000 osd.3 (osd.3) 154 : cluster 0 6.1f scrub ok 2024-04-06T03:26:41.570 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:41.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:41.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:41.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:41 smithi050 bash[21467]: cluster 2024-04-06T03:26:40.070074+0000 mgr.y (mgr.24368) 215 : cluster 0 pgmap v196: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-06T03:26:41.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:41 smithi050 bash[21467]: cluster 2024-04-06T03:26:40.747386+0000 osd.3 (osd.3) 153 : cluster 0 6.1f scrub starts 2024-04-06T03:26:41.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:41 smithi050 bash[21467]: cluster 2024-04-06T03:26:40.749056+0000 osd.3 (osd.3) 154 : cluster 0 6.1f scrub ok 2024-04-06T03:26:41.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:41 smithi050 bash[29221]: cluster 2024-04-06T03:26:40.070074+0000 mgr.y (mgr.24368) 215 : cluster 0 pgmap v196: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-04-06T03:26:41.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:41 smithi050 bash[29221]: cluster 2024-04-06T03:26:40.747386+0000 osd.3 (osd.3) 153 : cluster 0 6.1f scrub starts 2024-04-06T03:26:41.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:41 smithi050 bash[29221]: cluster 2024-04-06T03:26:40.749056+0000 osd.3 (osd.3) 154 : cluster 0 6.1f scrub ok 2024-04-06T03:26:41.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:41.833 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:41.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:41.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:41.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:42.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:26:42.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:42.098 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:42.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:42.222 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:42.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 66 16 77949 77933 73.7876 58.0625 0.0141954 0.0135469 2024-04-06T03:26:42.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:42.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:42.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:42.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:42.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:42.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[21467]: cluster 2024-04-06T03:26:40.862036+0000 osd.4 (osd.4) 125 : cluster 0 3.1c deep-scrub starts 2024-04-06T03:26:42.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[21467]: cluster 2024-04-06T03:26:40.864824+0000 osd.4 (osd.4) 126 : cluster 0 3.1c deep-scrub ok 2024-04-06T03:26:42.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[21467]: cluster 2024-04-06T03:26:41.781083+0000 osd.3 (osd.3) 155 : cluster 0 5.14 scrub starts 2024-04-06T03:26:42.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[21467]: cluster 2024-04-06T03:26:41.790113+0000 osd.3 (osd.3) 156 : cluster 0 5.14 scrub ok 2024-04-06T03:26:42.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[21467]: cluster 2024-04-06T03:26:42.320549+0000 osd.0 (osd.0) 121 : cluster 0 6.16 scrub starts 2024-04-06T03:26:42.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[21467]: cluster 2024-04-06T03:26:42.329891+0000 osd.0 (osd.0) 122 : cluster 0 6.16 scrub ok 2024-04-06T03:26:42.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[29221]: cluster 2024-04-06T03:26:40.862036+0000 osd.4 (osd.4) 125 : cluster 0 3.1c deep-scrub starts 2024-04-06T03:26:42.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[29221]: cluster 2024-04-06T03:26:40.864824+0000 osd.4 (osd.4) 126 : cluster 0 3.1c deep-scrub ok 2024-04-06T03:26:42.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[29221]: cluster 2024-04-06T03:26:41.781083+0000 osd.3 (osd.3) 155 : cluster 0 5.14 scrub starts 2024-04-06T03:26:42.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[29221]: cluster 2024-04-06T03:26:41.790113+0000 osd.3 (osd.3) 156 : cluster 0 5.14 scrub ok 2024-04-06T03:26:42.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[29221]: cluster 2024-04-06T03:26:42.320549+0000 osd.0 (osd.0) 121 : cluster 0 6.16 scrub starts 2024-04-06T03:26:42.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:42 smithi050 bash[29221]: cluster 2024-04-06T03:26:42.329891+0000 osd.0 (osd.0) 122 : cluster 0 6.16 scrub ok 2024-04-06T03:26:42.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:42.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:42.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:42.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:42 smithi153 bash[23852]: cluster 2024-04-06T03:26:40.862036+0000 osd.4 (osd.4) 125 : cluster 0 3.1c deep-scrub starts 2024-04-06T03:26:42.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:42 smithi153 bash[23852]: cluster 2024-04-06T03:26:40.864824+0000 osd.4 (osd.4) 126 : cluster 0 3.1c deep-scrub ok 2024-04-06T03:26:42.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:42 smithi153 bash[23852]: cluster 2024-04-06T03:26:41.781083+0000 osd.3 (osd.3) 155 : cluster 0 5.14 scrub starts 2024-04-06T03:26:42.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:42 smithi153 bash[23852]: cluster 2024-04-06T03:26:41.790113+0000 osd.3 (osd.3) 156 : cluster 0 5.14 scrub ok 2024-04-06T03:26:42.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:42 smithi153 bash[23852]: cluster 2024-04-06T03:26:42.320549+0000 osd.0 (osd.0) 121 : cluster 0 6.16 scrub starts 2024-04-06T03:26:42.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:42 smithi153 bash[23852]: cluster 2024-04-06T03:26:42.329891+0000 osd.0 (osd.0) 122 : cluster 0 6.16 scrub ok 2024-04-06T03:26:42.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:42.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:43.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:43.045 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:43.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:43.170 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:43.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:43.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 67 16 78937 78921 73.6079 61.75 0.030458 0.0135802 2024-04-06T03:26:43.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:43.295 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:43.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:43.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:43.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:43.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:43.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:43 smithi050 bash[21467]: cluster 2024-04-06T03:26:42.070567+0000 mgr.y (mgr.24368) 216 : cluster 0 pgmap v197: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 856 op/s 2024-04-06T03:26:43.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:43 smithi050 bash[21467]: cluster 2024-04-06T03:26:43.430359+0000 mon.a (mon.0) 1048 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-04-06T03:26:43.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:43 smithi050 bash[29221]: cluster 2024-04-06T03:26:42.070567+0000 mgr.y (mgr.24368) 216 : cluster 0 pgmap v197: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 856 op/s 2024-04-06T03:26:43.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:43 smithi050 bash[29221]: cluster 2024-04-06T03:26:43.430359+0000 mon.a (mon.0) 1048 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-04-06T03:26:43.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:43.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:26:43.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:43 smithi153 bash[23852]: cluster 2024-04-06T03:26:42.070567+0000 mgr.y (mgr.24368) 216 : cluster 0 pgmap v197: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 856 op/s 2024-04-06T03:26:43.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:43 smithi153 bash[23852]: cluster 2024-04-06T03:26:43.430359+0000 mon.a (mon.0) 1048 : cluster 0 osdmap e146: 8 total, 8 up, 8 in 2024-04-06T03:26:43.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:43.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:43.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:43.950 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:43.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:44.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:44.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:44.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:44.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:44.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 68 16 79825 79809 73.3415 55.5 0.0174437 0.0136286 2024-04-06T03:26:44.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:44.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:44.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:44.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-06T03:26:44.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[21467]: audit 2024-04-06T03:26:44.081144+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.50:0/3107183722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:26:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[21467]: audit 2024-04-06T03:26:44.082341+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:26:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[21467]: cluster 2024-04-06T03:26:44.336483+0000 osd.0 (osd.0) 123 : cluster 0 3.6 scrub starts 2024-04-06T03:26:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[21467]: cluster 2024-04-06T03:26:44.345745+0000 osd.0 (osd.0) 124 : cluster 0 3.6 scrub ok 2024-04-06T03:26:44.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[29221]: audit 2024-04-06T03:26:44.081144+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.50:0/3107183722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:26:44.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[29221]: audit 2024-04-06T03:26:44.082341+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:26:44.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[29221]: cluster 2024-04-06T03:26:44.336483+0000 osd.0 (osd.0) 123 : cluster 0 3.6 scrub starts 2024-04-06T03:26:44.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:44 smithi050 bash[29221]: cluster 2024-04-06T03:26:44.345745+0000 osd.0 (osd.0) 124 : cluster 0 3.6 scrub ok 2024-04-06T03:26:44.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:44.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:44.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:44 smithi153 bash[23852]: audit 2024-04-06T03:26:44.081144+0000 mon.c (mon.2) 43 : audit 1 from='client.? 172.21.15.50:0/3107183722' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:26:44.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:44 smithi153 bash[23852]: audit 2024-04-06T03:26:44.082341+0000 mon.a (mon.0) 1049 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:26:44.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:44 smithi153 bash[23852]: cluster 2024-04-06T03:26:44.336483+0000 osd.0 (osd.0) 123 : cluster 0 3.6 scrub starts 2024-04-06T03:26:44.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:44 smithi153 bash[23852]: cluster 2024-04-06T03:26:44.345745+0000 osd.0 (osd.0) 124 : cluster 0 3.6 scrub ok 2024-04-06T03:26:44.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:44.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:44.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:44.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:44.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:45.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:45.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:45.135 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:45.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:45.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:45.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 69 16 80810 80794 73.1707 61.5625 0.00650964 0.0136602 2024-04-06T03:26:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:45.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:45.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:45.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:45.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:45.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:45.660 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:45.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:45.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:45.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:45 smithi153 bash[23852]: cluster 2024-04-06T03:26:43.819178+0000 osd.4 (osd.4) 127 : cluster 0 2.18 scrub starts 2024-04-06T03:26:45.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:45 smithi153 bash[23852]: cluster 2024-04-06T03:26:43.821090+0000 osd.4 (osd.4) 128 : cluster 0 2.18 scrub ok 2024-04-06T03:26:45.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:45 smithi153 bash[23852]: cluster 2024-04-06T03:26:44.071360+0000 mgr.y (mgr.24368) 217 : cluster 0 pgmap v199: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 889 op/s 2024-04-06T03:26:45.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:45 smithi153 bash[23852]: audit 2024-04-06T03:26:44.521405+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:26:45.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:45 smithi153 bash[23852]: cluster 2024-04-06T03:26:44.525517+0000 mon.a (mon.0) 1051 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-04-06T03:26:45.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:45.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:45.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[21467]: cluster 2024-04-06T03:26:43.819178+0000 osd.4 (osd.4) 127 : cluster 0 2.18 scrub starts 2024-04-06T03:26:45.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[21467]: cluster 2024-04-06T03:26:43.821090+0000 osd.4 (osd.4) 128 : cluster 0 2.18 scrub ok 2024-04-06T03:26:45.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[21467]: cluster 2024-04-06T03:26:44.071360+0000 mgr.y (mgr.24368) 217 : cluster 0 pgmap v199: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 889 op/s 2024-04-06T03:26:45.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[21467]: audit 2024-04-06T03:26:44.521405+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:26:45.935 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[21467]: cluster 2024-04-06T03:26:44.525517+0000 mon.a (mon.0) 1051 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-04-06T03:26:45.936 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[29221]: cluster 2024-04-06T03:26:43.819178+0000 osd.4 (osd.4) 127 : cluster 0 2.18 scrub starts 2024-04-06T03:26:45.936 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[29221]: cluster 2024-04-06T03:26:43.821090+0000 osd.4 (osd.4) 128 : cluster 0 2.18 scrub ok 2024-04-06T03:26:45.936 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[29221]: cluster 2024-04-06T03:26:44.071360+0000 mgr.y (mgr.24368) 217 : cluster 0 pgmap v199: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 889 op/s 2024-04-06T03:26:45.936 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[29221]: audit 2024-04-06T03:26:44.521405+0000 mon.a (mon.0) 1050 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:26:45.936 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:45 smithi050 bash[29221]: cluster 2024-04-06T03:26:44.525517+0000 mon.a (mon.0) 1051 : cluster 0 osdmap e147: 8 total, 8 up, 8 in 2024-04-06T03:26:46.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:46.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:46.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:46.193 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:46.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 70 16 81514 81498 72.754 44 0.00933523 0.013739 2024-04-06T03:26:46.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:46.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:46.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:46.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:46.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:46.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:46.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:46.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:46 smithi050 bash[21467]: cluster 2024-04-06T03:26:46.052285+0000 osd.2 (osd.2) 113 : cluster 0 7.8 scrub starts 2024-04-06T03:26:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:46 smithi050 bash[21467]: cluster 2024-04-06T03:26:46.086123+0000 osd.7 (osd.7) 133 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:26:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:46 smithi050 bash[21467]: cluster 2024-04-06T03:26:46.087123+0000 osd.7 (osd.7) 134 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:26:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:46 smithi050 bash[29221]: cluster 2024-04-06T03:26:46.052285+0000 osd.2 (osd.2) 113 : cluster 0 7.8 scrub starts 2024-04-06T03:26:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:46 smithi050 bash[29221]: cluster 2024-04-06T03:26:46.086123+0000 osd.7 (osd.7) 133 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:26:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:46 smithi050 bash[29221]: cluster 2024-04-06T03:26:46.087123+0000 osd.7 (osd.7) 134 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:26:46.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:46 smithi153 bash[23852]: cluster 2024-04-06T03:26:46.052285+0000 osd.2 (osd.2) 113 : cluster 0 7.8 scrub starts 2024-04-06T03:26:46.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:46 smithi153 bash[23852]: cluster 2024-04-06T03:26:46.086123+0000 osd.7 (osd.7) 133 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:26:46.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:46 smithi153 bash[23852]: cluster 2024-04-06T03:26:46.087123+0000 osd.7 (osd.7) 134 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:26:46.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:46.822 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:46.940 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:46.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:46.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:46.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:47.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:47.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:47.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:47.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:47.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 71 16 82368 82352 72.481 53.375 0.0580777 0.0137841 2024-04-06T03:26:47.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:47.355 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:47.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:47.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:47.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:47.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:47.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:47.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:47 smithi050 bash[21467]: cluster 2024-04-06T03:26:46.073414+0000 mgr.y (mgr.24368) 218 : cluster 0 pgmap v201: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 777 B/s rd, 42 MiB/s wr, 680 op/s 2024-04-06T03:26:47.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:47 smithi050 bash[29221]: cluster 2024-04-06T03:26:46.073414+0000 mgr.y (mgr.24368) 218 : cluster 0 pgmap v201: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 777 B/s rd, 42 MiB/s wr, 680 op/s 2024-04-06T03:26:47.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:47.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:47.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:47 smithi153 bash[23852]: cluster 2024-04-06T03:26:46.073414+0000 mgr.y (mgr.24368) 218 : cluster 0 pgmap v201: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 777 B/s rd, 42 MiB/s wr, 680 op/s 2024-04-06T03:26:47.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:47.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:47.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:47.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:48.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:48.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:48.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:48.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:48.267 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 72 16 82986 82970 72.0107 38.625 0.00797419 0.0138816 2024-04-06T03:26:48.285 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:48.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:48.412 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:48.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:48.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:48.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:48.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:48.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[21467]: cluster 2024-04-06T03:26:47.354332+0000 osd.2 (osd.2) 114 : cluster 0 7.8 scrub ok 2024-04-06T03:26:48.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[21467]: cluster 2024-04-06T03:26:47.753229+0000 osd.3 (osd.3) 157 : cluster 0 3.b scrub starts 2024-04-06T03:26:48.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[21467]: cluster 2024-04-06T03:26:47.762164+0000 osd.3 (osd.3) 158 : cluster 0 3.b scrub ok 2024-04-06T03:26:48.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[21467]: cluster 2024-04-06T03:26:48.432818+0000 mon.a (mon.0) 1052 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-04-06T03:26:48.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[29221]: cluster 2024-04-06T03:26:47.354332+0000 osd.2 (osd.2) 114 : cluster 0 7.8 scrub ok 2024-04-06T03:26:48.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[29221]: cluster 2024-04-06T03:26:47.753229+0000 osd.3 (osd.3) 157 : cluster 0 3.b scrub starts 2024-04-06T03:26:48.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[29221]: cluster 2024-04-06T03:26:47.762164+0000 osd.3 (osd.3) 158 : cluster 0 3.b scrub ok 2024-04-06T03:26:48.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:48 smithi050 bash[29221]: cluster 2024-04-06T03:26:48.432818+0000 mon.a (mon.0) 1052 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-04-06T03:26:48.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:48.899 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:26:48.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:48.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:48.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:49.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:48 smithi153 bash[23852]: cluster 2024-04-06T03:26:47.354332+0000 osd.2 (osd.2) 114 : cluster 0 7.8 scrub ok 2024-04-06T03:26:49.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:48 smithi153 bash[23852]: cluster 2024-04-06T03:26:47.753229+0000 osd.3 (osd.3) 157 : cluster 0 3.b scrub starts 2024-04-06T03:26:49.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:48 smithi153 bash[23852]: cluster 2024-04-06T03:26:47.762164+0000 osd.3 (osd.3) 158 : cluster 0 3.b scrub ok 2024-04-06T03:26:49.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:48 smithi153 bash[23852]: cluster 2024-04-06T03:26:48.432818+0000 mon.a (mon.0) 1052 : cluster 0 osdmap e148: 8 total, 8 up, 8 in 2024-04-06T03:26:49.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:49.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:49.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:49.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 73 15 83703 83688 71.639 44.875 0.0133746 0.013954 2024-04-06T03:26:49.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:49.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:49.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:49.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:26:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:26:49.482 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:26:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:26:49.483 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:49.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:49.612 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:49.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:49.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:49.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:49 smithi153 bash[23852]: cluster 2024-04-06T03:26:48.073785+0000 mgr.y (mgr.24368) 219 : cluster 0 pgmap v202: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 35 MiB/s wr, 559 op/s 2024-04-06T03:26:49.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:49 smithi153 bash[23852]: audit 2024-04-06T03:26:48.605357+0000 mon.a (mon.0) 1053 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:26:49.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:49 smithi153 bash[23852]: audit 2024-04-06T03:26:48.606721+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:49.850 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:49.876 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:49.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:49.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[21467]: cluster 2024-04-06T03:26:48.073785+0000 mgr.y (mgr.24368) 219 : cluster 0 pgmap v202: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 35 MiB/s wr, 559 op/s 2024-04-06T03:26:49.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[21467]: audit 2024-04-06T03:26:48.605357+0000 mon.a (mon.0) 1053 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:26:49.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[21467]: audit 2024-04-06T03:26:48.606721+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:49.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[29221]: cluster 2024-04-06T03:26:48.073785+0000 mgr.y (mgr.24368) 219 : cluster 0 pgmap v202: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 35 MiB/s wr, 559 op/s 2024-04-06T03:26:49.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[29221]: audit 2024-04-06T03:26:48.605357+0000 mon.a (mon.0) 1053 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:26:49.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:49 smithi050 bash[29221]: audit 2024-04-06T03:26:48.606721+0000 mon.a (mon.0) 1054 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:26:49.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:49.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:50.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:50.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:50.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:50.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 74 16 84584 84568 71.4141 55 0.00850465 0.0139974 2024-04-06T03:26:50.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:50.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:50.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:50.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:50.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:50.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:50.650 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[21467]: cluster 2024-04-06T03:26:48.913546+0000 osd.5 (osd.5) 121 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:50.650 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[21467]: cluster 2024-04-06T03:26:48.913612+0000 osd.5 (osd.5) 122 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:50.650 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[21467]: cluster 2024-04-06T03:26:49.207550+0000 osd.2 (osd.2) 115 : cluster 0 2.1 scrub starts 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[21467]: cluster 2024-04-06T03:26:49.215700+0000 osd.2 (osd.2) 116 : cluster 0 2.1 scrub ok 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[21467]: audit 2024-04-06T03:26:49.222066+0000 mgr.y (mgr.24368) 220 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[21467]: cluster 2024-04-06T03:26:49.608902+0000 mon.a (mon.0) 1055 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[29221]: cluster 2024-04-06T03:26:48.913546+0000 osd.5 (osd.5) 121 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[29221]: cluster 2024-04-06T03:26:48.913612+0000 osd.5 (osd.5) 122 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[29221]: cluster 2024-04-06T03:26:49.207550+0000 osd.2 (osd.2) 115 : cluster 0 2.1 scrub starts 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[29221]: cluster 2024-04-06T03:26:49.215700+0000 osd.2 (osd.2) 116 : cluster 0 2.1 scrub ok 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[29221]: audit 2024-04-06T03:26:49.222066+0000 mgr.y (mgr.24368) 220 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:50.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:50 smithi050 bash[29221]: cluster 2024-04-06T03:26:49.608902+0000 mon.a (mon.0) 1055 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-04-06T03:26:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:50.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:50.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:50 smithi153 bash[23852]: cluster 2024-04-06T03:26:48.913546+0000 osd.5 (osd.5) 121 : cluster 0 purged_snaps scrub starts 2024-04-06T03:26:50.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:50 smithi153 bash[23852]: cluster 2024-04-06T03:26:48.913612+0000 osd.5 (osd.5) 122 : cluster 0 purged_snaps scrub ok 2024-04-06T03:26:50.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:50 smithi153 bash[23852]: cluster 2024-04-06T03:26:49.207550+0000 osd.2 (osd.2) 115 : cluster 0 2.1 scrub starts 2024-04-06T03:26:50.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:50 smithi153 bash[23852]: cluster 2024-04-06T03:26:49.215700+0000 osd.2 (osd.2) 116 : cluster 0 2.1 scrub ok 2024-04-06T03:26:50.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:50 smithi153 bash[23852]: audit 2024-04-06T03:26:49.222066+0000 mgr.y (mgr.24368) 220 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:26:50.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:50 smithi153 bash[23852]: cluster 2024-04-06T03:26:49.608902+0000 mon.a (mon.0) 1055 : cluster 0 osdmap e149: 8 total, 8 up, 8 in 2024-04-06T03:26:50.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:50.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:50.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:50.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:51.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:51.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:51.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:51.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:51.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:51.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 75 16 85224 85208 70.9952 40 0.0116473 0.0140798 2024-04-06T03:26:51.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:51.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:51.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:26:51.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:26:51.400 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-06T03:26:51.400 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.561337 2024-04-06T03:26:51.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 1 0.5613365618271599 2024-04-06T03:26:51.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:51.447 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:51.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:51.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:51.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:49.773367+0000 osd.4 (osd.4) 129 : cluster 0 5.11 scrub starts 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:49.774634+0000 osd.4 (osd.4) 130 : cluster 0 5.11 scrub ok 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.074213+0000 mgr.y (mgr.24368) 221 : cluster 0 pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.149040+0000 osd.1 (osd.1) 129 : cluster 0 6.d scrub starts 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.150650+0000 osd.1 (osd.1) 130 : cluster 0 6.d scrub ok 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.659968+0000 osd.3 (osd.3) 159 : cluster 0 2.1e scrub starts 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.666967+0000 osd.3 (osd.3) 160 : cluster 0 2.1e scrub ok 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.104762+0000 osd.7 (osd.7) 135 : cluster 0 5.19 scrub starts 2024-04-06T03:26:51.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:51 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.105790+0000 osd.7 (osd.7) 136 : cluster 0 5.19 scrub ok 2024-04-06T03:26:51.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:51.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:49.773367+0000 osd.4 (osd.4) 129 : cluster 0 5.11 scrub starts 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:49.774634+0000 osd.4 (osd.4) 130 : cluster 0 5.11 scrub ok 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.074213+0000 mgr.y (mgr.24368) 221 : cluster 0 pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.149040+0000 osd.1 (osd.1) 129 : cluster 0 6.d scrub starts 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.150650+0000 osd.1 (osd.1) 130 : cluster 0 6.d scrub ok 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.659968+0000 osd.3 (osd.3) 159 : cluster 0 2.1e scrub starts 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.666967+0000 osd.3 (osd.3) 160 : cluster 0 2.1e scrub ok 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.104762+0000 osd.7 (osd.7) 135 : cluster 0 5.19 scrub starts 2024-04-06T03:26:51.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.105790+0000 osd.7 (osd.7) 136 : cluster 0 5.19 scrub ok 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:49.773367+0000 osd.4 (osd.4) 129 : cluster 0 5.11 scrub starts 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:49.774634+0000 osd.4 (osd.4) 130 : cluster 0 5.11 scrub ok 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.074213+0000 mgr.y (mgr.24368) 221 : cluster 0 pgmap v205: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.149040+0000 osd.1 (osd.1) 129 : cluster 0 6.d scrub starts 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.150650+0000 osd.1 (osd.1) 130 : cluster 0 6.d scrub ok 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.659968+0000 osd.3 (osd.3) 159 : cluster 0 2.1e scrub starts 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.666967+0000 osd.3 (osd.3) 160 : cluster 0 2.1e scrub ok 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.104762+0000 osd.7 (osd.7) 135 : cluster 0 5.19 scrub starts 2024-04-06T03:26:51.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:51 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.105790+0000 osd.7 (osd.7) 136 : cluster 0 5.19 scrub ok 2024-04-06T03:26:51.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:51.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:51.984 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:51.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:52.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:52.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:52.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:52.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:52.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 76 16 85910 85894 70.6252 42.875 0.0133344 0.0141535 2024-04-06T03:26:52.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:52.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:52.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:52.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:52.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:52.645 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:52.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:52.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:52.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:52.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:52 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.679482+0000 osd.0 (osd.0) 125 : cluster 0 3.1b scrub starts 2024-04-06T03:26:52.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:52 smithi153 bash[23852]: cluster 2024-04-06T03:26:50.681508+0000 osd.0 (osd.0) 126 : cluster 0 3.1b scrub ok 2024-04-06T03:26:52.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:52 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.700533+0000 osd.3 (osd.3) 161 : cluster 0 2.12 scrub starts 2024-04-06T03:26:52.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:52 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.701781+0000 osd.3 (osd.3) 162 : cluster 0 2.12 scrub ok 2024-04-06T03:26:52.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:52.908 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:52.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.679482+0000 osd.0 (osd.0) 125 : cluster 0 3.1b scrub starts 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[21467]: cluster 2024-04-06T03:26:50.681508+0000 osd.0 (osd.0) 126 : cluster 0 3.1b scrub ok 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.700533+0000 osd.3 (osd.3) 161 : cluster 0 2.12 scrub starts 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.701781+0000 osd.3 (osd.3) 162 : cluster 0 2.12 scrub ok 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.679482+0000 osd.0 (osd.0) 125 : cluster 0 3.1b scrub starts 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[29221]: cluster 2024-04-06T03:26:50.681508+0000 osd.0 (osd.0) 126 : cluster 0 3.1b scrub ok 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.700533+0000 osd.3 (osd.3) 161 : cluster 0 2.12 scrub starts 2024-04-06T03:26:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:52 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.701781+0000 osd.3 (osd.3) 162 : cluster 0 2.12 scrub ok 2024-04-06T03:26:53.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:53.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:53.162 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:53.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:53.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 77 16 86591 86575 70.2607 42.5625 0.0106904 0.0142269 2024-04-06T03:26:53.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:53.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:53.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:53.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:53.568 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:53.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:53.695 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.665537+0000 osd.0 (osd.0) 127 : cluster 0 5.1e scrub starts 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.666999+0000 osd.0 (osd.0) 128 : cluster 0 5.1e scrub ok 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.779745+0000 osd.4 (osd.4) 131 : cluster 0 2.1a scrub starts 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[21467]: cluster 2024-04-06T03:26:51.780757+0000 osd.4 (osd.4) 132 : cluster 0 2.1a scrub ok 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[21467]: cluster 2024-04-06T03:26:52.074831+0000 mgr.y (mgr.24368) 222 : cluster 0 pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 677 B/s rd, 33 MiB/s wr, 521 op/s 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[21467]: cluster 2024-04-06T03:26:53.436841+0000 mon.a (mon.0) 1056 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-04-06T03:26:53.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.665537+0000 osd.0 (osd.0) 127 : cluster 0 5.1e scrub starts 2024-04-06T03:26:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.666999+0000 osd.0 (osd.0) 128 : cluster 0 5.1e scrub ok 2024-04-06T03:26:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.779745+0000 osd.4 (osd.4) 131 : cluster 0 2.1a scrub starts 2024-04-06T03:26:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[29221]: cluster 2024-04-06T03:26:51.780757+0000 osd.4 (osd.4) 132 : cluster 0 2.1a scrub ok 2024-04-06T03:26:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[29221]: cluster 2024-04-06T03:26:52.074831+0000 mgr.y (mgr.24368) 222 : cluster 0 pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 677 B/s rd, 33 MiB/s wr, 521 op/s 2024-04-06T03:26:53.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:53 smithi050 bash[29221]: cluster 2024-04-06T03:26:53.436841+0000 mon.a (mon.0) 1056 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-04-06T03:26:53.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:53.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:53.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:53.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:53 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.665537+0000 osd.0 (osd.0) 127 : cluster 0 5.1e scrub starts 2024-04-06T03:26:53.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:53 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.666999+0000 osd.0 (osd.0) 128 : cluster 0 5.1e scrub ok 2024-04-06T03:26:53.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:53 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.779745+0000 osd.4 (osd.4) 131 : cluster 0 2.1a scrub starts 2024-04-06T03:26:53.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:53 smithi153 bash[23852]: cluster 2024-04-06T03:26:51.780757+0000 osd.4 (osd.4) 132 : cluster 0 2.1a scrub ok 2024-04-06T03:26:53.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:53 smithi153 bash[23852]: cluster 2024-04-06T03:26:52.074831+0000 mgr.y (mgr.24368) 222 : cluster 0 pgmap v206: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 677 B/s rd, 33 MiB/s wr, 521 op/s 2024-04-06T03:26:53.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:53 smithi153 bash[23852]: cluster 2024-04-06T03:26:53.436841+0000 mon.a (mon.0) 1056 : cluster 0 osdmap e150: 8 total, 8 up, 8 in 2024-04-06T03:26:53.964 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:53.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:53.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:54.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:54.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:54.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:54.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:54.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 78 16 87463 87447 70.0586 54.5 0.00272017 0.0142528 2024-04-06T03:26:54.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:54.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:54.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:54.466 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:54.570 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:54.598 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:54.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:54.724 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:54.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:54.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:54.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:54.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:54 smithi050 bash[21467]: cluster 2024-04-06T03:26:52.645769+0000 osd.0 (osd.0) 129 : cluster 0 3.1 scrub starts 2024-04-06T03:26:54.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:54 smithi050 bash[21467]: cluster 2024-04-06T03:26:52.651889+0000 osd.0 (osd.0) 130 : cluster 0 3.1 scrub ok 2024-04-06T03:26:54.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:54 smithi050 bash[29221]: cluster 2024-04-06T03:26:52.645769+0000 osd.0 (osd.0) 129 : cluster 0 3.1 scrub starts 2024-04-06T03:26:54.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:54 smithi050 bash[29221]: cluster 2024-04-06T03:26:52.651889+0000 osd.0 (osd.0) 130 : cluster 0 3.1 scrub ok 2024-04-06T03:26:54.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:54.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:55.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:54 smithi153 bash[23852]: cluster 2024-04-06T03:26:52.645769+0000 osd.0 (osd.0) 129 : cluster 0 3.1 scrub starts 2024-04-06T03:26:55.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:54 smithi153 bash[23852]: cluster 2024-04-06T03:26:52.651889+0000 osd.0 (osd.0) 130 : cluster 0 3.1 scrub ok 2024-04-06T03:26:55.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:55.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:55.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:55.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 79 16 88222 88206 69.7722 47.4375 0.0164841 0.0143274 2024-04-06T03:26:55.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:55.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:55.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:55.525 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:55.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:55.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:55 smithi050 bash[21467]: cluster 2024-04-06T03:26:53.819413+0000 osd.4 (osd.4) 133 : cluster 0 5.1f scrub starts 2024-04-06T03:26:55.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:55 smithi050 bash[21467]: cluster 2024-04-06T03:26:53.820551+0000 osd.4 (osd.4) 134 : cluster 0 5.1f scrub ok 2024-04-06T03:26:55.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:55 smithi050 bash[21467]: cluster 2024-04-06T03:26:54.075605+0000 mgr.y (mgr.24368) 223 : cluster 0 pgmap v208: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 840 op/s 2024-04-06T03:26:55.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:55 smithi050 bash[29221]: cluster 2024-04-06T03:26:53.819413+0000 osd.4 (osd.4) 133 : cluster 0 5.1f scrub starts 2024-04-06T03:26:55.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:55 smithi050 bash[29221]: cluster 2024-04-06T03:26:53.820551+0000 osd.4 (osd.4) 134 : cluster 0 5.1f scrub ok 2024-04-06T03:26:55.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:55 smithi050 bash[29221]: cluster 2024-04-06T03:26:54.075605+0000 mgr.y (mgr.24368) 223 : cluster 0 pgmap v208: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 840 op/s 2024-04-06T03:26:55.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:55.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:55.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:55 smithi153 bash[23852]: cluster 2024-04-06T03:26:53.819413+0000 osd.4 (osd.4) 133 : cluster 0 5.1f scrub starts 2024-04-06T03:26:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:55 smithi153 bash[23852]: cluster 2024-04-06T03:26:53.820551+0000 osd.4 (osd.4) 134 : cluster 0 5.1f scrub ok 2024-04-06T03:26:55.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:55 smithi153 bash[23852]: cluster 2024-04-06T03:26:54.075605+0000 mgr.y (mgr.24368) 223 : cluster 0 pgmap v208: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 840 op/s 2024-04-06T03:26:55.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:55.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:55.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:55.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:56.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:56.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:56.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:56.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:56.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:26:56.278463+0000 min lat: 0.00125923 max lat: 1.17752 avg lat: 0.014363 2024-04-06T03:26:56.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:26:56.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 80 16 89110 89094 69.5937 55.5 0.00799536 0.014363 2024-04-06T03:26:56.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:56.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:56.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:56.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:56.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:56.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:56.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:56.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:56.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:56.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:56.957 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[29221]: cluster 2024-04-06T03:26:55.621067+0000 osd.0 (osd.0) 131 : cluster 0 3.17 scrub starts 2024-04-06T03:26:56.957 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[29221]: cluster 2024-04-06T03:26:55.622817+0000 osd.0 (osd.0) 132 : cluster 0 3.17 scrub ok 2024-04-06T03:26:56.957 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[29221]: cluster 2024-04-06T03:26:55.635329+0000 osd.3 (osd.3) 163 : cluster 0 5.16 scrub starts 2024-04-06T03:26:56.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[29221]: cluster 2024-04-06T03:26:55.636333+0000 osd.3 (osd.3) 164 : cluster 0 5.16 scrub ok 2024-04-06T03:26:56.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[21467]: cluster 2024-04-06T03:26:55.621067+0000 osd.0 (osd.0) 131 : cluster 0 3.17 scrub starts 2024-04-06T03:26:56.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[21467]: cluster 2024-04-06T03:26:55.622817+0000 osd.0 (osd.0) 132 : cluster 0 3.17 scrub ok 2024-04-06T03:26:56.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[21467]: cluster 2024-04-06T03:26:55.635329+0000 osd.3 (osd.3) 163 : cluster 0 5.16 scrub starts 2024-04-06T03:26:56.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:56 smithi050 bash[21467]: cluster 2024-04-06T03:26:55.636333+0000 osd.3 (osd.3) 164 : cluster 0 5.16 scrub ok 2024-04-06T03:26:56.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:56.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:57.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:56 smithi153 bash[23852]: cluster 2024-04-06T03:26:55.621067+0000 osd.0 (osd.0) 131 : cluster 0 3.17 scrub starts 2024-04-06T03:26:57.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:56 smithi153 bash[23852]: cluster 2024-04-06T03:26:55.622817+0000 osd.0 (osd.0) 132 : cluster 0 3.17 scrub ok 2024-04-06T03:26:57.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:56 smithi153 bash[23852]: cluster 2024-04-06T03:26:55.635329+0000 osd.3 (osd.3) 163 : cluster 0 5.16 scrub starts 2024-04-06T03:26:57.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:56 smithi153 bash[23852]: cluster 2024-04-06T03:26:55.636333+0000 osd.3 (osd.3) 164 : cluster 0 5.16 scrub ok 2024-04-06T03:26:57.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:57.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:57.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:57.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:57.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 81 16 89861 89845 69.314 46.9375 0.0238155 0.0144216 2024-04-06T03:26:57.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:57.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:57.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:57.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:57.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:57.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:57.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:57.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:26:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:15172 7.b 7:da5f8542:::benchmark_data_smithi050_68188_object15171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:44.396252+0000", 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.420211003000006, 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.191445749, 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.811 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 15172 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396252+0000", 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396252+0000", 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396253+0000", 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.299e-06 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396277+0000", 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3626999999999999e-05 2024-04-06T03:26:57.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396288+0000", 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1368e-05 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396296+0000", 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8730000000000007e-06 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396350+0000", 2024-04-06T03:26:57.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4630999999999999e-05 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396495+0000", 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014473000000000001 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.396783+0000", 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028804500000000001 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:26:57.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398338+0000", 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015543969999999999 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398760+0000", 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042224199999999997 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587585+0000", 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18882529200000001 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587650+0000", 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4508e-05 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587697+0000", 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7737000000000001e-05 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:15175 7.b 7:d1b3d3b2:::benchmark_data_smithi050_68188_object15174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:26:57.817 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:44.397518+0000", 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.418945133999998, 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19204671300000001, 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 15175 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397518+0000", 2024-04-06T03:26:57.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397518+0000", 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397519+0000", 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.128e-06 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397558+0000", 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9097000000000002e-05 2024-04-06T03:26:57.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397570+0000", 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.259e-05 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397579+0000", 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1810000000000007e-06 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397640+0000", 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0412999999999999e-05 2024-04-06T03:26:57.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397771+0000", 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013095500000000001 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.397898+0000", 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012761900000000001 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398671+0000", 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077272400000000002 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398801+0000", 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013017700000000001 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589453+0000", 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19065132500000001 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589513+0000", 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0300999999999997e-05 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589564+0000", 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1202999999999997e-05 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14745.0:15172 7.b e132/127 ondisk, result = 0)", 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:44.398925+0000", 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.417537663000004, 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18878578800000001, 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.823 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398925+0000", 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398925+0000", 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398926+0000", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7199999999999998e-07 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398942+0000", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6418000000000001e-05 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398960+0000", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7805999999999999e-05 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.398965+0000", 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9359999999999998e-06 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587568+0000", 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18860329000000001 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587578+0000", 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4679999999999998e-06 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587711+0000", 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013309800000000001 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14745.0:15175 7.b e132/127 ondisk, result = 0)", 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:44.399044+0000", 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.417418502000004, 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19053339599999999, 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:26:57.827 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399044+0000", 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399044+0000", 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399045+0000", 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8100000000000002e-07 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399049+0000", 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4399999999999998e-06 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587383+0000", 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18833372000000001 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587401+0000", 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8193999999999999e-05 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589438+0000", 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002036646 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589447+0000", 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6889999999999993e-06 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589578+0000", 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013102599999999999 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:15177 7.4 e132/127 7:24aa1f51:::benchmark_data_smithi050_68188_object15176:head v 132'902, mlcod=132'900)", 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:44.399525+0000", 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.416937623999999, 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.189984658, 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399525+0000", 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399525+0000", 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399526+0000", 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3099999999999997e-07 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399548+0000", 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1789000000000001e-05 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587445+0000", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18789788700000001 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587455+0000", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2159999999999995e-06 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587526+0000", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0922999999999998e-05 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587557+0000", 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1257000000000002e-05 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589469+0000", 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019123899999999999 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.589510+0000", 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0465e-05 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.834 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:15176 7.2 e132/127 7:472191df:::benchmark_data_smithi050_68188_object15175:head v 132'946, mlcod=132'944)", 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:44.399979+0000", 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 73.416483764000006, 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.190161987, 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399979+0000", 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399979+0000", 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.399980+0000", 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9299999999999997e-07 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.400004+0000", 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4015999999999998e-05 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587502+0000", 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.18749850700000001 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587509+0000", 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4459999999999998e-06 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587765+0000", 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025612399999999999 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.587795+0000", 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9997000000000001e-05 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.590062+0000", 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022676369999999999 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:44.590141+0000", 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8467000000000006e-05 2024-04-06T03:26:57.838 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18827 7.b 7:d76c64db:::benchmark_data_smithi050_68188_object18826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.736972+0000", 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.079490159000002, 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.24041083999999999, 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:26:57.839 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18827 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.736972+0000", 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.736972+0000", 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.736974+0000", 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4169999999999999e-06 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737011+0000", 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6661e-05 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737021+0000", 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0852999999999999e-05 2024-04-06T03:26:57.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737029+0000", 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7360000000000001e-06 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737310+0000", 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028101500000000002 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737426+0000", 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116261 2024-04-06T03:26:57.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.737572+0000", 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014574499999999999 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738934+0000", 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013617779999999999 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739979+0000", 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001045175 2024-04-06T03:26:57.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977284+0000", 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.237305285 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977336+0000", 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1369000000000002e-05 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977383+0000", 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7545e-05 2024-04-06T03:26:57.844 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18831 7.b 7:d331bc92:::benchmark_data_smithi050_68188_object18830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.738626+0000", 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.077836898000001, 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23885795100000001, 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:26:57.845 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18831 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738626+0000", 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738626+0000", 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738627+0000", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1319999999999999e-06 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738653+0000", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6078e-05 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738665+0000", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2357999999999999e-05 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738675+0000", 2024-04-06T03:26:57.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4890000000000001e-06 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.738964+0000", 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000289597 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739079+0000", 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114436 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739205+0000", 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000125852 2024-04-06T03:26:57.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976073+0000", 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.236868519 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976117+0000", 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3680000000000002e-05 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977394+0000", 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001277271 2024-04-06T03:26:57.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977446+0000", 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2058999999999997e-05 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977484+0000", 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.748e-05 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:18836 7.b 7:dcbe557d:::benchmark_data_smithi050_68188_object18835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.739762+0000", 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.076700609, 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23816216100000001, 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 18836 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.851 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739762+0000", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739762+0000", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739763+0000", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1200000000000001e-06 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739786+0000", 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2424000000000001e-05 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739798+0000", 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2274e-05 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.739805+0000", 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6120000000000001e-06 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740042+0000", 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000236276 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740157+0000", 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115339 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740284+0000", 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012668099999999999 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976156+0000", 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23587178 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976195+0000", 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9567000000000003e-05 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977837+0000", 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016415309999999999 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977884+0000", 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7364000000000003e-05 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977924+0000", 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0193e-05 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:18825 7.2 e132/127 7:489ffb91:::benchmark_data_smithi050_68188_object18824:head v 132'1151, mlcod=132'1149)", 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.740095+0000", 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.076367856000005, 2024-04-06T03:26:57.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23481653199999999, 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740095+0000", 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740095+0000", 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740096+0000", 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3e-07 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740117+0000", 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1903e-05 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740128+0000", 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0607e-05 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740135+0000", 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4329999999999998e-06 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740204+0000", 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8298000000000002e-05 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740221+0000", 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7167e-05 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974836+0000", 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23461557199999999 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974911+0000", 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4821999999999996e-05 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14745.0:18831 7.b e132/127 ondisk, result = 0)", 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.740475+0000", 2024-04-06T03:26:57.860 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.075987888, 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23560752700000001, 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740475+0000", 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740475+0000", 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740475+0000", 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5000000000000002e-07 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.740480+0000", 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3109999999999999e-06 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974844+0000", 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23436468399999999 2024-04-06T03:26:57.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974875+0000", 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0317000000000001e-05 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976056+0000", 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001181762 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976065+0000", 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-06 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.976082+0000", 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6903000000000001e-05 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:18826 7.9 e132/127 7:96cc1459:::benchmark_data_smithi050_68188_object18825:head v 132'1157, mlcod=132'1155)", 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.741201+0000", 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.075261558999998, 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23706258899999999, 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.864 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741201+0000", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741201+0000", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741202+0000", 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5499999999999997e-07 2024-04-06T03:26:57.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741261+0000", 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9629000000000001e-05 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.975054+0000", 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23379272700000001 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.975064+0000", 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.679e-06 2024-04-06T03:26:57.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.975253+0000", 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000189202 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.975274+0000", 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1423000000000001e-05 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978149+0000", 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028740580000000001 2024-04-06T03:26:57.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.978264+0000", 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115116 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:18832 7.2 e132/127 7:42a9dd66:::benchmark_data_smithi050_68188_object18831:head v 132'1152, mlcod=132'1149)", 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:46.741257+0000", 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 71.075205259000001, 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23600743599999999, 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741257+0000", 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741257+0000", 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741258+0000", 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6700000000000003e-07 2024-04-06T03:26:57.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.741283+0000", 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4926000000000002e-05 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974843+0000", 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23355959000000001 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974876+0000", 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3285999999999999e-05 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974953+0000", 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7636e-05 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.974981+0000", 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7732000000000001e-05 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977222+0000", 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022407159999999998 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:46.977265+0000", 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2883000000000003e-05 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:32745 7.b 7:d7c1c26e:::benchmark_data_smithi050_68188_object32744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:57.060603+0000", 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 60.755859923999999, 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.179319968, 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.872 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 32745 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060603+0000", 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060603+0000", 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060604+0000", 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3999999999999999e-06 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060629+0000", 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4686e-05 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060641+0000", 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2102e-05 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060649+0000", 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4880000000000007e-06 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060703+0000", 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3946999999999997e-05 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060841+0000", 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013798 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.060989+0000", 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148071 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.063092+0000", 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021030229999999999 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.063608+0000", 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051594100000000001 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.239724+0000", 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17611543299999999 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.239817+0000", 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3059000000000005e-05 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:57.239923+0000", 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105838 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.877 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:81085 7.2 e147/127 7:4e46e5f0:::benchmark_data_smithi050_68188_object81084:head v 147'4980, mlcod=147'4978)", 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:45.568981+0000", 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.247481291, 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27616159000000001, 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.568981+0000", 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.568981+0000", 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.568982+0000", 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7900000000000007e-07 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.569005+0000", 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2931000000000001e-05 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.569016+0000", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0859999999999999e-05 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.569023+0000", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.199e-06 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.569076+0000", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2376999999999997e-05 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.569096+0000", 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0205000000000001e-05 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.575923+0000", 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068270500000000003 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.845143+0000", 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26921998899999999 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.881 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:81082 7.d e147/138 7:b700aabf:::benchmark_data_smithi050_68188_object81081:head v 147'5108, mlcod=147'5106)", 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:45.572455+0000", 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.244007848000001, 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.28401637299999999, 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572455+0000", 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572455+0000", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572456+0000", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0359999999999999e-06 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572484+0000", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8615e-05 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572496+0000", 2024-04-06T03:26:57.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1063999999999999e-05 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572503+0000", 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5989999999999996e-06 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572554+0000", 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0844999999999999e-05 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572576+0000", 2024-04-06T03:26:57.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1593000000000001e-05 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.856386+0000", 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.28381072600000001 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.856471+0000", 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4895000000000006e-05 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.885 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:81083 7.0 e147/135 7:01561b7e:::benchmark_data_smithi050_68188_object81082:head v 147'5010, mlcod=147'5004)", 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:45.572464+0000", 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.243998597999999, 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.28162515900000001, 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572464+0000", 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572464+0000", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572465+0000", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8299999999999996e-07 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572495+0000", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.01e-05 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572506+0000", 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0933e-05 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572523+0000", 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7116e-05 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.844904+0000", 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.272380968 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.844941+0000", 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7506999999999999e-05 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.853993+0000", 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.009051712 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.854089+0000", 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6039999999999995e-05 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.889 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:81088 7.4 e147/127 7:2c7b5dc1:::benchmark_data_smithi050_68188_object81087:head v 147'5027, mlcod=147'5025)", 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:45.572525+0000", 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.243937486, 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.28094026900000002, 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572525+0000", 2024-04-06T03:26:57.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572525+0000", 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572526+0000", 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0900000000000002e-07 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572548+0000", 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2357999999999998e-05 2024-04-06T03:26:57.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572556+0000", 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.371e-06 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.572562+0000", 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9599999999999997e-06 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.844924+0000", 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27236158199999999 2024-04-06T03:26:57.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.844943+0000", 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9465000000000001e-05 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.852333+0000", 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073897279999999999 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.853465+0000", 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011323959999999999 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.893 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:81087 7.0 e147/135 7:0b29cd13:::benchmark_data_smithi050_68188_object81086:head v 147'5011, mlcod=147'5008)", 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:45.576447+0000", 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.240015644, 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.27768575099999998, 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.576447+0000", 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.576447+0000", 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.576448+0000", 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1999999999999998e-07 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.576477+0000", 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9141000000000001e-05 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.845472+0000", 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26899503400000002 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.845481+0000", 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6230000000000003e-06 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.847973+0000", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024922360000000001 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.847997+0000", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4457e-05 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.854099+0000", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061017349999999996 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.854133+0000", 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3605000000000001e-05 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:81103 7.d e147/138 7:b66db90d:::benchmark_data_smithi050_68188_object81102:head v 147'5109, mlcod=147'5106)", 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:45.588492+0000", 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.227970403, 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.26802183099999999, 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:26:57.898 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.588492+0000", 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.588492+0000", 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.588493+0000", 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.029e-06 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.588517+0000", 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3427000000000001e-05 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.845225+0000", 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.256708086 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.845235+0000", 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0703000000000001e-05 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.848872+0000", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036368469999999999 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.848898+0000", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5734000000000001e-05 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.856478+0000", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075799689999999998 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:26:57.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:45.856514+0000", 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6035999999999999e-05 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:26:57.902 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:26:57.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:57.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[21467]: cluster 2024-04-06T03:26:56.076455+0000 mgr.y (mgr.24368) 224 : cluster 0 pgmap v209: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-06T03:26:57.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[21467]: cluster 2024-04-06T03:26:56.219201+0000 osd.1 (osd.1) 131 : cluster 0 5.7 scrub starts 2024-04-06T03:26:57.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[21467]: cluster 2024-04-06T03:26:56.220452+0000 osd.1 (osd.1) 132 : cluster 0 5.7 scrub ok 2024-04-06T03:26:57.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[21467]: cluster 2024-04-06T03:26:56.650227+0000 osd.3 (osd.3) 165 : cluster 0 3.2 scrub starts 2024-04-06T03:26:57.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[21467]: cluster 2024-04-06T03:26:56.656556+0000 osd.3 (osd.3) 166 : cluster 0 3.2 scrub ok 2024-04-06T03:26:57.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[29221]: cluster 2024-04-06T03:26:56.076455+0000 mgr.y (mgr.24368) 224 : cluster 0 pgmap v209: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-06T03:26:57.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[29221]: cluster 2024-04-06T03:26:56.219201+0000 osd.1 (osd.1) 131 : cluster 0 5.7 scrub starts 2024-04-06T03:26:57.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[29221]: cluster 2024-04-06T03:26:56.220452+0000 osd.1 (osd.1) 132 : cluster 0 5.7 scrub ok 2024-04-06T03:26:57.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[29221]: cluster 2024-04-06T03:26:56.650227+0000 osd.3 (osd.3) 165 : cluster 0 3.2 scrub starts 2024-04-06T03:26:57.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:57 smithi050 bash[29221]: cluster 2024-04-06T03:26:56.656556+0000 osd.3 (osd.3) 166 : cluster 0 3.2 scrub ok 2024-04-06T03:26:57.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:57.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:58.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:57 smithi153 bash[23852]: cluster 2024-04-06T03:26:56.076455+0000 mgr.y (mgr.24368) 224 : cluster 0 pgmap v209: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-06T03:26:58.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:57 smithi153 bash[23852]: cluster 2024-04-06T03:26:56.219201+0000 osd.1 (osd.1) 131 : cluster 0 5.7 scrub starts 2024-04-06T03:26:58.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:57 smithi153 bash[23852]: cluster 2024-04-06T03:26:56.220452+0000 osd.1 (osd.1) 132 : cluster 0 5.7 scrub ok 2024-04-06T03:26:58.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:57 smithi153 bash[23852]: cluster 2024-04-06T03:26:56.650227+0000 osd.3 (osd.3) 165 : cluster 0 3.2 scrub starts 2024-04-06T03:26:58.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:57 smithi153 bash[23852]: cluster 2024-04-06T03:26:56.656556+0000 osd.3 (osd.3) 166 : cluster 0 3.2 scrub ok 2024-04-06T03:26:58.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:58.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.195 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:58.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 82 16 90726 90710 69.1279 54.0625 0.0139785 0.0144605 2024-04-06T03:26:58.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:58.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:26:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:26:58.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:58.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:26:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:26:58.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:58.581 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:26:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:26:58.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:58.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:58.822 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.846 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:58.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.971 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:58.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:59.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:59.098 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:59.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:59.222 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:26:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:26:59.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 83 16 91661 91645 68.9991 58.4375 0.0281951 0.0144877 2024-04-06T03:26:59.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:59.349 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:59.350 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:26:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:26:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:59.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:59.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:59.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:26:59 smithi153 bash[23852]: cluster 2024-04-06T03:26:58.076986+0000 mgr.y (mgr.24368) 225 : cluster 0 pgmap v210: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 604 B/s rd, 27 MiB/s wr, 438 op/s 2024-04-06T03:26:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:59.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:59.601 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:26:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:26:59.705 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:59.727 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:26:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:26:59.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:26:59 smithi050 bash[21467]: cluster 2024-04-06T03:26:58.076986+0000 mgr.y (mgr.24368) 225 : cluster 0 pgmap v210: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 604 B/s rd, 27 MiB/s wr, 438 op/s 2024-04-06T03:26:59.732 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:26:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:26:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:26:59.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:26:59 smithi050 bash[29221]: cluster 2024-04-06T03:26:58.076986+0000 mgr.y (mgr.24368) 225 : cluster 0 pgmap v210: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 604 B/s rd, 27 MiB/s wr, 438 op/s 2024-04-06T03:26:59.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:59.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:26:59.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:26:59.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:26:59.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:00.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.116 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:00.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.241 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:00.268 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 84 15 92554 92539 68.8428 55.875 0.0137325 0.0145212 2024-04-06T03:27:00.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.378 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:00.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:00.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:26:58.761507+0000 osd.4 (osd.4) 135 : cluster 0 5.1c deep-scrub starts 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:26:58.762465+0000 osd.4 (osd.4) 136 : cluster 0 5.1c deep-scrub ok 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: audit 2024-04-06T03:26:59.232566+0000 mgr.y (mgr.24368) 226 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:26:59.328861+0000 osd.1 (osd.1) 133 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:26:59.328923+0000 osd.1 (osd.1) 134 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:26:59.623161+0000 mon.a (mon.0) 1057 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:00.073952+0000 osd.7 (osd.7) 137 : cluster 0 6.3 scrub starts 2024-04-06T03:27:00.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:00.076958+0000 osd.7 (osd.7) 138 : cluster 0 6.3 scrub ok 2024-04-06T03:27:00.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:00.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:26:58.761507+0000 osd.4 (osd.4) 135 : cluster 0 5.1c deep-scrub starts 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:26:58.762465+0000 osd.4 (osd.4) 136 : cluster 0 5.1c deep-scrub ok 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: audit 2024-04-06T03:26:59.232566+0000 mgr.y (mgr.24368) 226 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:26:59.328861+0000 osd.1 (osd.1) 133 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:26:59.328923+0000 osd.1 (osd.1) 134 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:26:59.623161+0000 mon.a (mon.0) 1057 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-04-06T03:27:00.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:00.073952+0000 osd.7 (osd.7) 137 : cluster 0 6.3 scrub starts 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:00.076958+0000 osd.7 (osd.7) 138 : cluster 0 6.3 scrub ok 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:26:58.761507+0000 osd.4 (osd.4) 135 : cluster 0 5.1c deep-scrub starts 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:26:58.762465+0000 osd.4 (osd.4) 136 : cluster 0 5.1c deep-scrub ok 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: audit 2024-04-06T03:26:59.232566+0000 mgr.y (mgr.24368) 226 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:26:59.328861+0000 osd.1 (osd.1) 133 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:26:59.328923+0000 osd.1 (osd.1) 134 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:26:59.623161+0000 mon.a (mon.0) 1057 : cluster 0 osdmap e151: 8 total, 8 up, 8 in 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:00.073952+0000 osd.7 (osd.7) 137 : cluster 0 6.3 scrub starts 2024-04-06T03:27:00.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:00.076958+0000 osd.7 (osd.7) 138 : cluster 0 6.3 scrub ok 2024-04-06T03:27:00.753 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:00.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-06T03:27:00.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:00.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:00.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:01.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:01.050 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:01.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:01.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:01.214 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:01.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 85 16 93296 93280 68.5777 46.3125 0.00758144 0.014577 2024-04-06T03:27:01.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:01.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:01.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:01.447 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:01.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:01.565 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:01 smithi153 bash[23852]: cluster 2024-04-06T03:26:59.799949+0000 osd.4 (osd.4) 137 : cluster 0 6.2 scrub starts 2024-04-06T03:27:01.565 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:01 smithi153 bash[23852]: cluster 2024-04-06T03:26:59.829269+0000 osd.4 (osd.4) 138 : cluster 0 6.2 scrub ok 2024-04-06T03:27:01.565 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:01 smithi153 bash[23852]: cluster 2024-04-06T03:27:00.077418+0000 mgr.y (mgr.24368) 227 : cluster 0 pgmap v212: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 991 op/s 2024-04-06T03:27:01.584 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:01.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:01.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:01 smithi050 bash[21467]: cluster 2024-04-06T03:26:59.799949+0000 osd.4 (osd.4) 137 : cluster 0 6.2 scrub starts 2024-04-06T03:27:01.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:01 smithi050 bash[21467]: cluster 2024-04-06T03:26:59.829269+0000 osd.4 (osd.4) 138 : cluster 0 6.2 scrub ok 2024-04-06T03:27:01.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:01 smithi050 bash[21467]: cluster 2024-04-06T03:27:00.077418+0000 mgr.y (mgr.24368) 227 : cluster 0 pgmap v212: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 991 op/s 2024-04-06T03:27:01.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:01 smithi050 bash[29221]: cluster 2024-04-06T03:26:59.799949+0000 osd.4 (osd.4) 137 : cluster 0 6.2 scrub starts 2024-04-06T03:27:01.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:01 smithi050 bash[29221]: cluster 2024-04-06T03:26:59.829269+0000 osd.4 (osd.4) 138 : cluster 0 6.2 scrub ok 2024-04-06T03:27:01.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:01 smithi050 bash[29221]: cluster 2024-04-06T03:27:00.077418+0000 mgr.y (mgr.24368) 227 : cluster 0 pgmap v212: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 991 op/s 2024-04-06T03:27:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:01.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:01.853 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:01.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:01.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:01.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:02.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:02.108 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:02.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:02.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:02.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 86 16 94096 94080 68.3616 50 0.00334026 0.0146206 2024-04-06T03:27:02.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:02.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:02.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:02.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:02.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:02.728 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:02 smithi153 bash[23852]: cluster 2024-04-06T03:27:01.573817+0000 osd.0 (osd.0) 133 : cluster 0 5.2 scrub starts 2024-04-06T03:27:02.728 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:02 smithi153 bash[23852]: cluster 2024-04-06T03:27:01.575493+0000 osd.0 (osd.0) 134 : cluster 0 5.2 scrub ok 2024-04-06T03:27:02.746 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:02.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:02.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:02.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:02 smithi050 bash[21467]: cluster 2024-04-06T03:27:01.573817+0000 osd.0 (osd.0) 133 : cluster 0 5.2 scrub starts 2024-04-06T03:27:02.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:02 smithi050 bash[21467]: cluster 2024-04-06T03:27:01.575493+0000 osd.0 (osd.0) 134 : cluster 0 5.2 scrub ok 2024-04-06T03:27:02.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:02 smithi050 bash[29221]: cluster 2024-04-06T03:27:01.573817+0000 osd.0 (osd.0) 133 : cluster 0 5.2 scrub starts 2024-04-06T03:27:02.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:02 smithi050 bash[29221]: cluster 2024-04-06T03:27:01.575493+0000 osd.0 (osd.0) 134 : cluster 0 5.2 scrub ok 2024-04-06T03:27:02.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:02.992 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:02.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:03.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:03.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:03.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:03.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:03.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 87 16 94729 94713 68.0306 39.5625 0.0138821 0.0146943 2024-04-06T03:27:03.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:03.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:03.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:03.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:03.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:03 smithi153 bash[23852]: cluster 2024-04-06T03:27:01.901977+0000 osd.3 (osd.3) 167 : cluster 0 6.b scrub starts 2024-04-06T03:27:03.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:03 smithi153 bash[23852]: cluster 2024-04-06T03:27:01.917806+0000 osd.3 (osd.3) 168 : cluster 0 6.b scrub ok 2024-04-06T03:27:03.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:03 smithi153 bash[23852]: cluster 2024-04-06T03:27:02.077878+0000 mgr.y (mgr.24368) 228 : cluster 0 pgmap v213: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 918 op/s 2024-04-06T03:27:03.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:03.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:03 smithi050 bash[21467]: cluster 2024-04-06T03:27:01.901977+0000 osd.3 (osd.3) 167 : cluster 0 6.b scrub starts 2024-04-06T03:27:03.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:03 smithi050 bash[21467]: cluster 2024-04-06T03:27:01.917806+0000 osd.3 (osd.3) 168 : cluster 0 6.b scrub ok 2024-04-06T03:27:03.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:03 smithi050 bash[21467]: cluster 2024-04-06T03:27:02.077878+0000 mgr.y (mgr.24368) 228 : cluster 0 pgmap v213: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 918 op/s 2024-04-06T03:27:03.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:03 smithi050 bash[29221]: cluster 2024-04-06T03:27:01.901977+0000 osd.3 (osd.3) 167 : cluster 0 6.b scrub starts 2024-04-06T03:27:03.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:03 smithi050 bash[29221]: cluster 2024-04-06T03:27:01.917806+0000 osd.3 (osd.3) 168 : cluster 0 6.b scrub ok 2024-04-06T03:27:03.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:03 smithi050 bash[29221]: cluster 2024-04-06T03:27:02.077878+0000 mgr.y (mgr.24368) 228 : cluster 0 pgmap v213: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 918 op/s 2024-04-06T03:27:03.653 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:03.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:03.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:03.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:03.906 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:03.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:04.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:04.010 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:27:04.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:04.064 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:27:04.124 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:27:04.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:04.180 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:27:04.181 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:04.221 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:27:04.263 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:27:04.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 88 16 95492 95476 67.7994 47.6875 0.00910455 0.0147428 2024-04-06T03:27:04.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:04.305 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:27:04.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:04.340 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:27:04.350 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:27:04.351 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:27:04.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:04.468 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:04.471 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:27:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:04.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:04.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:04.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: cluster 2024-04-06T03:27:03.207684+0000 osd.1 (osd.1) 135 : cluster 0 5.1d scrub starts 2024-04-06T03:27:04.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: cluster 2024-04-06T03:27:03.210848+0000 osd.1 (osd.1) 136 : cluster 0 5.1d scrub ok 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: cluster 2024-04-06T03:27:03.615695+0000 osd.0 (osd.0) 135 : cluster 0 7.f scrub starts 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: audit 2024-04-06T03:27:03.642499+0000 mon.a (mon.0) 1058 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: audit 2024-04-06T03:27:03.958899+0000 mon.c (mon.2) 44 : audit 0 from='client.? 172.21.15.50:0/2052736765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: cluster 2024-04-06T03:27:04.092468+0000 osd.7 (osd.7) 139 : cluster 0 3.15 scrub starts 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: cluster 2024-04-06T03:27:04.102303+0000 osd.7 (osd.7) 140 : cluster 0 3.15 scrub ok 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: audit 2024-04-06T03:27:04.431071+0000 mon.b (mon.1) 35 : audit 1 from='client.? 172.21.15.50:0/1440070228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:27:04.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[21467]: audit 2024-04-06T03:27:04.436666+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: cluster 2024-04-06T03:27:03.207684+0000 osd.1 (osd.1) 135 : cluster 0 5.1d scrub starts 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: cluster 2024-04-06T03:27:03.210848+0000 osd.1 (osd.1) 136 : cluster 0 5.1d scrub ok 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: cluster 2024-04-06T03:27:03.615695+0000 osd.0 (osd.0) 135 : cluster 0 7.f scrub starts 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: audit 2024-04-06T03:27:03.642499+0000 mon.a (mon.0) 1058 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: audit 2024-04-06T03:27:03.958899+0000 mon.c (mon.2) 44 : audit 0 from='client.? 172.21.15.50:0/2052736765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: cluster 2024-04-06T03:27:04.092468+0000 osd.7 (osd.7) 139 : cluster 0 3.15 scrub starts 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: cluster 2024-04-06T03:27:04.102303+0000 osd.7 (osd.7) 140 : cluster 0 3.15 scrub ok 2024-04-06T03:27:04.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: audit 2024-04-06T03:27:04.431071+0000 mon.b (mon.1) 35 : audit 1 from='client.? 172.21.15.50:0/1440070228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:27:04.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:04 smithi050 bash[29221]: audit 2024-04-06T03:27:04.436666+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:27:04.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: cluster 2024-04-06T03:27:03.207684+0000 osd.1 (osd.1) 135 : cluster 0 5.1d scrub starts 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: cluster 2024-04-06T03:27:03.210848+0000 osd.1 (osd.1) 136 : cluster 0 5.1d scrub ok 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: cluster 2024-04-06T03:27:03.615695+0000 osd.0 (osd.0) 135 : cluster 0 7.f scrub starts 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: audit 2024-04-06T03:27:03.642499+0000 mon.a (mon.0) 1058 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: audit 2024-04-06T03:27:03.958899+0000 mon.c (mon.2) 44 : audit 0 from='client.? 172.21.15.50:0/2052736765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: cluster 2024-04-06T03:27:04.092468+0000 osd.7 (osd.7) 139 : cluster 0 3.15 scrub starts 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: cluster 2024-04-06T03:27:04.102303+0000 osd.7 (osd.7) 140 : cluster 0 3.15 scrub ok 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: audit 2024-04-06T03:27:04.431071+0000 mon.b (mon.1) 35 : audit 1 from='client.? 172.21.15.50:0/1440070228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:27:04.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:04 smithi153 bash[23852]: audit 2024-04-06T03:27:04.436666+0000 mon.a (mon.0) 1059 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:27:04.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:04.878 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:04.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:04.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:05.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:05.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:05.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 89 16 96297 96281 67.6028 50.3125 0.00381642 0.0147697 2024-04-06T03:27:05.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:05.314 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:05.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:05.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:05.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:05.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:05.704 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:05 smithi153 bash[23852]: cluster 2024-04-06T03:27:04.078634+0000 mgr.y (mgr.24368) 229 : cluster 0 pgmap v214: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 804 op/s 2024-04-06T03:27:05.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:05 smithi153 bash[23852]: cluster 2024-04-06T03:27:04.448727+0000 mon.a (mon.0) 1060 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:05.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:05 smithi153 bash[23852]: audit 2024-04-06T03:27:04.461756+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:27:05.705 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:05 smithi153 bash[23852]: cluster 2024-04-06T03:27:04.467369+0000 mon.a (mon.0) 1062 : cluster 0 osdmap e152: 8 total, 8 up, 8 in 2024-04-06T03:27:05.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:05.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[21467]: cluster 2024-04-06T03:27:04.078634+0000 mgr.y (mgr.24368) 229 : cluster 0 pgmap v214: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 804 op/s 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[21467]: cluster 2024-04-06T03:27:04.448727+0000 mon.a (mon.0) 1060 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[21467]: audit 2024-04-06T03:27:04.461756+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[21467]: cluster 2024-04-06T03:27:04.467369+0000 mon.a (mon.0) 1062 : cluster 0 osdmap e152: 8 total, 8 up, 8 in 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[29221]: cluster 2024-04-06T03:27:04.078634+0000 mgr.y (mgr.24368) 229 : cluster 0 pgmap v214: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 804 op/s 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[29221]: cluster 2024-04-06T03:27:04.448727+0000 mon.a (mon.0) 1060 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[29221]: audit 2024-04-06T03:27:04.461756+0000 mon.a (mon.0) 1061 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:27:05.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:05 smithi050 bash[29221]: cluster 2024-04-06T03:27:04.467369+0000 mon.a (mon.0) 1062 : cluster 0 osdmap e152: 8 total, 8 up, 8 in 2024-04-06T03:27:05.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.851 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:05.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:05.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:05.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:05.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:06.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:06.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:06.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:06.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 90 3 96747 96744 67.1732 28.9375 0.260081 0.0148739 2024-04-06T03:27:06.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:06.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:06.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:06.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:06.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:06.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:06.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:06.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:06.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:06.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:06.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:07.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:07.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:07.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:07.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:07.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total time run: 90.0144 2024-04-06T03:27:07.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total writes made: 96747 2024-04-06T03:27:07.269 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Write size: 65536 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object size: 65536 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Bandwidth (MB/sec): 67.1747 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Bandwidth: 25.5217 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max bandwidth (MB/sec): 166.312 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min bandwidth (MB/sec): 20.8125 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average IOPS: 1074 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev IOPS: 408.348 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max IOPS: 2661 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min IOPS: 333 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average Latency(s): 0.0148818 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Latency(s): 0.0284624 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max latency(s): 1.17752 2024-04-06T03:27:07.270 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min latency(s): 0.00125923 2024-04-06T03:27:07.276 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Cleaning up (deleting benchmark objects) 2024-04-06T03:27:07.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:07.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:07.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:07.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:07.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:07.687 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:07 smithi050 bash[21467]: cluster 2024-04-06T03:27:06.079298+0000 mgr.y (mgr.24368) 230 : cluster 0 pgmap v216: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-06T03:27:07.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:07 smithi050 bash[29221]: cluster 2024-04-06T03:27:06.079298+0000 mgr.y (mgr.24368) 230 : cluster 0 pgmap v216: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-06T03:27:07.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:07.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:07 smithi153 bash[23852]: cluster 2024-04-06T03:27:06.079298+0000 mgr.y (mgr.24368) 230 : cluster 0 pgmap v216: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-04-06T03:27:07.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:07.833 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:07.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:07.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:07.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:08.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:08.085 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:08.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:08.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:08.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:08.355 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:08.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:08.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:08.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:08.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:08.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:08.748 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:08.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:08.883 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:08.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:08.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:09.008 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:09.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:09.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:09.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:09.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:27:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:27:09.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:09.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:09.477 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.1 primary-affinity to 0.561337 (8367872) 2024-04-06T03:27:09.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:09.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[21467]: cluster 2024-04-06T03:27:08.079769+0000 mgr.y (mgr.24368) 231 : cluster 0 pgmap v217: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 605 B/s rd, 29 MiB/s wr, 457 op/s 2024-04-06T03:27:09.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[21467]: audit 2024-04-06T03:27:09.135836+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.50:0/3388669601' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]: dispatch 2024-04-06T03:27:09.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[21467]: audit 2024-04-06T03:27:09.136416+0000 mon.a (mon.0) 1063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]: dispatch 2024-04-06T03:27:09.515 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:27:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:27:09.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[29221]: cluster 2024-04-06T03:27:08.079769+0000 mgr.y (mgr.24368) 231 : cluster 0 pgmap v217: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 605 B/s rd, 29 MiB/s wr, 457 op/s 2024-04-06T03:27:09.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[29221]: audit 2024-04-06T03:27:09.135836+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.50:0/3388669601' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]: dispatch 2024-04-06T03:27:09.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:09 smithi050 bash[29221]: audit 2024-04-06T03:27:09.136416+0000 mon.a (mon.0) 1063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]: dispatch 2024-04-06T03:27:09.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:09 smithi153 bash[23852]: cluster 2024-04-06T03:27:08.079769+0000 mgr.y (mgr.24368) 231 : cluster 0 pgmap v217: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 605 B/s rd, 29 MiB/s wr, 457 op/s 2024-04-06T03:27:09.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:09 smithi153 bash[23852]: audit 2024-04-06T03:27:09.135836+0000 mon.c (mon.2) 45 : audit 1 from='client.? 172.21.15.50:0/3388669601' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]: dispatch 2024-04-06T03:27:09.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:09 smithi153 bash[23852]: audit 2024-04-06T03:27:09.136416+0000 mon.a (mon.0) 1063 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]: dispatch 2024-04-06T03:27:09.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:09.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:09.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:09.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:09.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:09.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:09.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:09.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:10.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:10.082 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:10.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:10.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:27:10.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:27:10.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:10.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:10.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:10.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:10.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:10.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:10.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:10.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:10.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:10.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:10.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:10 smithi153 bash[23852]: audit 2024-04-06T03:27:09.243903+0000 mgr.y (mgr.24368) 232 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:10.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:10 smithi153 bash[23852]: audit 2024-04-06T03:27:09.470284+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]': finished 2024-04-06T03:27:10.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:10 smithi153 bash[23852]: cluster 2024-04-06T03:27:09.474091+0000 mon.a (mon.0) 1065 : cluster 0 osdmap e153: 8 total, 8 up, 8 in 2024-04-06T03:27:10.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:10.897 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:10.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:10 smithi050 bash[21467]: audit 2024-04-06T03:27:09.243903+0000 mgr.y (mgr.24368) 232 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:10 smithi050 bash[21467]: audit 2024-04-06T03:27:09.470284+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]': finished 2024-04-06T03:27:10.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:10 smithi050 bash[21467]: cluster 2024-04-06T03:27:09.474091+0000 mon.a (mon.0) 1065 : cluster 0 osdmap e153: 8 total, 8 up, 8 in 2024-04-06T03:27:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:10 smithi050 bash[29221]: audit 2024-04-06T03:27:09.243903+0000 mgr.y (mgr.24368) 232 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:10 smithi050 bash[29221]: audit 2024-04-06T03:27:09.470284+0000 mon.a (mon.0) 1064 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5613365618271599}]': finished 2024-04-06T03:27:10.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:10 smithi050 bash[29221]: cluster 2024-04-06T03:27:09.474091+0000 mon.a (mon.0) 1065 : cluster 0 osdmap e153: 8 total, 8 up, 8 in 2024-04-06T03:27:11.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:11.030 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:11.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:11.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:11.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:11.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:11.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:11.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:11.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:11.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:11 smithi050 bash[29221]: cluster 2024-04-06T03:27:10.080205+0000 mgr.y (mgr.24368) 233 : cluster 0 pgmap v219: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 40 MiB/s wr, 840 op/s 2024-04-06T03:27:11.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:11 smithi050 bash[29221]: cluster 2024-04-06T03:27:10.485201+0000 mon.a (mon.0) 1066 : cluster 0 osdmap e154: 8 total, 8 up, 8 in 2024-04-06T03:27:11.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:11 smithi050 bash[21467]: cluster 2024-04-06T03:27:10.080205+0000 mgr.y (mgr.24368) 233 : cluster 0 pgmap v219: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 40 MiB/s wr, 840 op/s 2024-04-06T03:27:11.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:11 smithi050 bash[21467]: cluster 2024-04-06T03:27:10.485201+0000 mon.a (mon.0) 1066 : cluster 0 osdmap e154: 8 total, 8 up, 8 in 2024-04-06T03:27:11.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:11.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:11.659 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:11 smithi153 bash[23852]: cluster 2024-04-06T03:27:10.080205+0000 mgr.y (mgr.24368) 233 : cluster 0 pgmap v219: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 40 MiB/s wr, 840 op/s 2024-04-06T03:27:11.659 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:11 smithi153 bash[23852]: cluster 2024-04-06T03:27:10.485201+0000 mon.a (mon.0) 1066 : cluster 0 osdmap e154: 8 total, 8 up, 8 in 2024-04-06T03:27:11.683 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:11.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:11.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:11.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:11.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:11.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:12.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:12.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:12.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:12.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:12.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:12.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:12.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:12.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:12.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:12.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:12.736 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:12.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:12.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:12.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:12.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:12.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:13.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:13.124 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:13.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:13.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:13.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:13.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:13.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:13.516 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:13.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:13 smithi050 bash[21467]: cluster 2024-04-06T03:27:12.080606+0000 mgr.y (mgr.24368) 234 : cluster 0 pgmap v221: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 806 B/s rd, 10 MiB/s wr, 375 op/s 2024-04-06T03:27:13.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:13 smithi050 bash[29221]: cluster 2024-04-06T03:27:12.080606+0000 mgr.y (mgr.24368) 234 : cluster 0 pgmap v221: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 806 B/s rd, 10 MiB/s wr, 375 op/s 2024-04-06T03:27:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:13.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:13.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:13.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:13.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:13 smithi153 bash[23852]: cluster 2024-04-06T03:27:12.080606+0000 mgr.y (mgr.24368) 234 : cluster 0 pgmap v221: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 806 B/s rd, 10 MiB/s wr, 375 op/s 2024-04-06T03:27:13.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:13.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:13.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:14.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:14.055 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:14.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:14.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:14.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:14.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:14.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:14.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:14.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:14.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:14.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:14.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:14.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:14.814 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:27:14.815 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:27:14.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:14.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:14.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:14.995 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:14.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:15.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:15.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:15.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:15.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:15.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:15.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:15.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:15.520 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:15.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:15.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:15.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:15.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:15.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:15 smithi153 bash[23852]: cluster 2024-04-06T03:27:14.081268+0000 mgr.y (mgr.24368) 235 : cluster 0 pgmap v222: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 9.9 MiB/s wr, 507 op/s 2024-04-06T03:27:15.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:15.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:16.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:16.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:16.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:16.166 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:15 smithi050 bash[21467]: cluster 2024-04-06T03:27:14.081268+0000 mgr.y (mgr.24368) 235 : cluster 0 pgmap v222: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 9.9 MiB/s wr, 507 op/s 2024-04-06T03:27:16.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:15 smithi050 bash[29221]: cluster 2024-04-06T03:27:14.081268+0000 mgr.y (mgr.24368) 235 : cluster 0 pgmap v222: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 9.9 MiB/s wr, 507 op/s 2024-04-06T03:27:16.196 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:16.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-06T03:27:16.300 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:16.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:16.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:16.447 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:16.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:16.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:16.637 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:16.637 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:16.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:16.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:16 smithi153 bash[23852]: cluster 2024-04-06T03:27:15.855675+0000 osd.6 (osd.6) 79 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:16.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:16 smithi153 bash[23852]: cluster 2024-04-06T03:27:15.855746+0000 osd.6 (osd.6) 80 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:16.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:16.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:16.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:16.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:27:16.886 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:27:16.886 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-06T03:27:16.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-06T03:27:16.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:16 smithi050 bash[21467]: cluster 2024-04-06T03:27:15.855675+0000 osd.6 (osd.6) 79 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:16.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:16 smithi050 bash[21467]: cluster 2024-04-06T03:27:15.855746+0000 osd.6 (osd.6) 80 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:16.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:16 smithi050 bash[29221]: cluster 2024-04-06T03:27:15.855675+0000 osd.6 (osd.6) 79 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:16.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:16 smithi050 bash[29221]: cluster 2024-04-06T03:27:15.855746+0000 osd.6 (osd.6) 80 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:16.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:16.980 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:16.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:17.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:17.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:17.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:17.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:17.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:17.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:17.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:17.488 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:17.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:17.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:17 smithi050 bash[21467]: cluster 2024-04-06T03:27:16.081914+0000 mgr.y (mgr.24368) 236 : cluster 0 pgmap v223: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.9 MiB/s wr, 850 op/s 2024-04-06T03:27:17.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:17 smithi050 bash[21467]: cluster 2024-04-06T03:27:16.522845+0000 mon.a (mon.0) 1067 : cluster 0 osdmap e155: 8 total, 8 up, 8 in 2024-04-06T03:27:17.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:17 smithi050 bash[29221]: cluster 2024-04-06T03:27:16.081914+0000 mgr.y (mgr.24368) 236 : cluster 0 pgmap v223: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.9 MiB/s wr, 850 op/s 2024-04-06T03:27:17.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:17 smithi050 bash[29221]: cluster 2024-04-06T03:27:16.522845+0000 mon.a (mon.0) 1067 : cluster 0 osdmap e155: 8 total, 8 up, 8 in 2024-04-06T03:27:17.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:17.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:17.730 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:17 smithi153 bash[23852]: cluster 2024-04-06T03:27:16.081914+0000 mgr.y (mgr.24368) 236 : cluster 0 pgmap v223: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.9 MiB/s wr, 850 op/s 2024-04-06T03:27:17.730 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:17 smithi153 bash[23852]: cluster 2024-04-06T03:27:16.522845+0000 mon.a (mon.0) 1067 : cluster 0 osdmap e155: 8 total, 8 up, 8 in 2024-04-06T03:27:17.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:17.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:17.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:17.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:17.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:17.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:18.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:18.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:18.135 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:18.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:18.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:18.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:18.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:18.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:18.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:18.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:18.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:18.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:18.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:18.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:18.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:18.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:18.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:19.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:19.052 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:19.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:19.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:19.307 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:19.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:19.434 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:19.479 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:27:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:27:19.527 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:27:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:27:19.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:19.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:19.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:19.700 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[21467]: cluster 2024-04-06T03:27:18.082356+0000 mgr.y (mgr.24368) 237 : cluster 0 pgmap v225: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 493 op/s 2024-04-06T03:27:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[21467]: audit 2024-04-06T03:27:18.666606+0000 mon.a (mon.0) 1068 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:27:19.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[21467]: audit 2024-04-06T03:27:18.667945+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:19.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[29221]: cluster 2024-04-06T03:27:18.082356+0000 mgr.y (mgr.24368) 237 : cluster 0 pgmap v225: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 493 op/s 2024-04-06T03:27:19.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[29221]: audit 2024-04-06T03:27:18.666606+0000 mon.a (mon.0) 1068 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:27:19.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:19 smithi050 bash[29221]: audit 2024-04-06T03:27:18.667945+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:19.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:19 smithi153 bash[23852]: cluster 2024-04-06T03:27:18.082356+0000 mgr.y (mgr.24368) 237 : cluster 0 pgmap v225: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 493 op/s 2024-04-06T03:27:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:19 smithi153 bash[23852]: audit 2024-04-06T03:27:18.666606+0000 mon.a (mon.0) 1068 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:27:19.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:19 smithi153 bash[23852]: audit 2024-04-06T03:27:18.667945+0000 mon.a (mon.0) 1069 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:19.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:19.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:19.957 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:19.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:20.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:20.082 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:20.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:20.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:20.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:20.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:20.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:20.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:20.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[21467]: audit 2024-04-06T03:27:19.255945+0000 mgr.y (mgr.24368) 238 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[21467]: cluster 2024-04-06T03:27:20.087599+0000 osd.2 (osd.2) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[21467]: cluster 2024-04-06T03:27:20.087676+0000 osd.2 (osd.2) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[21467]: audit 2024-04-06T03:27:20.487564+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[29221]: audit 2024-04-06T03:27:19.255945+0000 mgr.y (mgr.24368) 238 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[29221]: cluster 2024-04-06T03:27:20.087599+0000 osd.2 (osd.2) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[29221]: cluster 2024-04-06T03:27:20.087676+0000 osd.2 (osd.2) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:20.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:20 smithi050 bash[29221]: audit 2024-04-06T03:27:20.487564+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:20.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:20.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:20.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:20.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:20 smithi153 bash[23852]: audit 2024-04-06T03:27:19.255945+0000 mgr.y (mgr.24368) 238 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:20.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:20 smithi153 bash[23852]: cluster 2024-04-06T03:27:20.087599+0000 osd.2 (osd.2) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:20.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:20 smithi153 bash[23852]: cluster 2024-04-06T03:27:20.087676+0000 osd.2 (osd.2) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:20.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:20 smithi153 bash[23852]: audit 2024-04-06T03:27:20.487564+0000 mon.a (mon.0) 1070 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:27:20.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:20.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:20.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:21.002 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:21.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:21.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:21.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:21.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:21.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:21.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:21.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:21.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:21.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:21.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:21.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:21.637 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[21467]: cluster 2024-04-06T03:27:20.083047+0000 mgr.y (mgr.24368) 239 : cluster 0 pgmap v226: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:27:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[21467]: cluster 2024-04-06T03:27:20.545458+0000 mon.a (mon.0) 1071 : cluster 0 osdmap e156: 8 total, 8 up, 8 in 2024-04-06T03:27:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[21467]: audit 2024-04-06T03:27:21.336446+0000 mon.a (mon.0) 1072 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[21467]: audit 2024-04-06T03:27:21.337815+0000 mon.a (mon.0) 1073 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:21.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[21467]: audit 2024-04-06T03:27:21.347811+0000 mon.a (mon.0) 1074 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:27:21.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[29221]: cluster 2024-04-06T03:27:20.083047+0000 mgr.y (mgr.24368) 239 : cluster 0 pgmap v226: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:27:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[29221]: cluster 2024-04-06T03:27:20.545458+0000 mon.a (mon.0) 1071 : cluster 0 osdmap e156: 8 total, 8 up, 8 in 2024-04-06T03:27:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[29221]: audit 2024-04-06T03:27:21.336446+0000 mon.a (mon.0) 1072 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[29221]: audit 2024-04-06T03:27:21.337815+0000 mon.a (mon.0) 1073 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:21 smithi050 bash[29221]: audit 2024-04-06T03:27:21.347811+0000 mon.a (mon.0) 1074 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:27:21.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:21.775 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:21 smithi153 bash[23852]: cluster 2024-04-06T03:27:20.083047+0000 mgr.y (mgr.24368) 239 : cluster 0 pgmap v226: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 959 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:27:21.793 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:21 smithi153 bash[23852]: cluster 2024-04-06T03:27:20.545458+0000 mon.a (mon.0) 1071 : cluster 0 osdmap e156: 8 total, 8 up, 8 in 2024-04-06T03:27:21.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:21 smithi153 bash[23852]: audit 2024-04-06T03:27:21.336446+0000 mon.a (mon.0) 1072 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:27:21.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:21 smithi153 bash[23852]: audit 2024-04-06T03:27:21.337815+0000 mon.a (mon.0) 1073 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:27:21.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:21 smithi153 bash[23852]: audit 2024-04-06T03:27:21.347811+0000 mon.a (mon.0) 1074 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:27:21.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:21.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:21.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:22.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:22.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:22.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:22.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:22.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:22.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:22.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:22.422 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:22.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:22.547 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:22.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:22.675 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:22.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:22.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:22.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:22.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:22.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:23.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:23.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:23.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:23.200 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:23.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:23.336 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:23.336 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:23.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:23.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:23.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:23.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:23.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:23 smithi050 bash[21467]: cluster 2024-04-06T03:27:22.083650+0000 mgr.y (mgr.24368) 240 : cluster 0 pgmap v228: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s 2024-04-06T03:27:23.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:23 smithi050 bash[29221]: cluster 2024-04-06T03:27:22.083650+0000 mgr.y (mgr.24368) 240 : cluster 0 pgmap v228: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s 2024-04-06T03:27:23.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:23.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:23.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:23.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:23 smithi153 bash[23852]: cluster 2024-04-06T03:27:22.083650+0000 mgr.y (mgr.24368) 240 : cluster 0 pgmap v228: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s 2024-04-06T03:27:23.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:23.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:23.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:23.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:23.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:24.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:24.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:24.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:24.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:24.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:24.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:24.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:24.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:24.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:24.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:24.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:24.780 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:24.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:24.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:24.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:25.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:25.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:25.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:25.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:25.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:25.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:25.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:25.440 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:25.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:25.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:25 smithi153 bash[23852]: cluster 2024-04-06T03:27:24.084673+0000 mgr.y (mgr.24368) 241 : cluster 0 pgmap v229: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 765 op/s 2024-04-06T03:27:25.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:25.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:25.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:25.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:25.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:25.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:25.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:25.938 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:25 smithi050 bash[21467]: cluster 2024-04-06T03:27:24.084673+0000 mgr.y (mgr.24368) 241 : cluster 0 pgmap v229: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 765 op/s 2024-04-06T03:27:25.938 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:25 smithi050 bash[29221]: cluster 2024-04-06T03:27:24.084673+0000 mgr.y (mgr.24368) 241 : cluster 0 pgmap v229: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 765 op/s 2024-04-06T03:27:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:25.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:26.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:26.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:26.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:26.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:26.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:26.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:26.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:26.480 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:26.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:26.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:26.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:26.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:26.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:26.761 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:27:26.802 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:27:26.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:26.850 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:27:26.854 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:26.912 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:27:26.952 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:27:26.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:26.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:26.996 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:27:27.038 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:27:27.080 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:27:27.083 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:27:27.083 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:27:27.083 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:27:27.084 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:27:27.084 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:27:27.084 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:27:27.084 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:27:27.084 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:27:27.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:27.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:27.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:27.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:27.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:27.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:27.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:27.558 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:27:27.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:27.631 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:27 smithi153 bash[23852]: cluster 2024-04-06T03:27:26.085308+0000 mgr.y (mgr.24368) 242 : cluster 0 pgmap v230: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 944 op/s 2024-04-06T03:27:27.631 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:27 smithi153 bash[23852]: audit 2024-04-06T03:27:26.650568+0000 mon.a (mon.0) 1075 : audit 1 from='client.? 172.21.15.50:0/545372535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:27:27.631 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:27 smithi153 bash[23852]: audit 2024-04-06T03:27:26.726885+0000 mon.a (mon.0) 1076 : audit 0 from='client.? 172.21.15.50:0/923530771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:27.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:27 smithi050 bash[21467]: cluster 2024-04-06T03:27:26.085308+0000 mgr.y (mgr.24368) 242 : cluster 0 pgmap v230: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 944 op/s 2024-04-06T03:27:27.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:27 smithi050 bash[21467]: audit 2024-04-06T03:27:26.650568+0000 mon.a (mon.0) 1075 : audit 1 from='client.? 172.21.15.50:0/545372535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:27:27.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:27 smithi050 bash[21467]: audit 2024-04-06T03:27:26.726885+0000 mon.a (mon.0) 1076 : audit 0 from='client.? 172.21.15.50:0/923530771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:27 smithi050 bash[29221]: cluster 2024-04-06T03:27:26.085308+0000 mgr.y (mgr.24368) 242 : cluster 0 pgmap v230: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 944 op/s 2024-04-06T03:27:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:27 smithi050 bash[29221]: audit 2024-04-06T03:27:26.650568+0000 mon.a (mon.0) 1075 : audit 1 from='client.? 172.21.15.50:0/545372535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:27:27.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:27 smithi050 bash[29221]: audit 2024-04-06T03:27:26.726885+0000 mon.a (mon.0) 1076 : audit 0 from='client.? 172.21.15.50:0/923530771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:27.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:27.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:27.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:27.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:27.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:28.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:28.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:28.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:28.197 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:28.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:28.300 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:28.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:28.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:28.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:28.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:28.594 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:28.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:28.724 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:28.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:28.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:28 smithi153 bash[23852]: cluster 2024-04-06T03:27:27.555042+0000 mon.a (mon.0) 1077 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:28.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:28 smithi153 bash[23852]: audit 2024-04-06T03:27:27.556815+0000 mon.a (mon.0) 1078 : audit 1 from='client.? 172.21.15.50:0/545372535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:27:28.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:28 smithi153 bash[23852]: cluster 2024-04-06T03:27:27.562319+0000 mon.a (mon.0) 1079 : cluster 0 osdmap e157: 8 total, 8 up, 8 in 2024-04-06T03:27:28.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:28.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:28 smithi050 bash[21467]: cluster 2024-04-06T03:27:27.555042+0000 mon.a (mon.0) 1077 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:28.968 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:28 smithi050 bash[21467]: audit 2024-04-06T03:27:27.556815+0000 mon.a (mon.0) 1078 : audit 1 from='client.? 172.21.15.50:0/545372535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:27:28.968 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:28 smithi050 bash[21467]: cluster 2024-04-06T03:27:27.562319+0000 mon.a (mon.0) 1079 : cluster 0 osdmap e157: 8 total, 8 up, 8 in 2024-04-06T03:27:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:28 smithi050 bash[29221]: cluster 2024-04-06T03:27:27.555042+0000 mon.a (mon.0) 1077 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:28 smithi050 bash[29221]: audit 2024-04-06T03:27:27.556815+0000 mon.a (mon.0) 1078 : audit 1 from='client.? 172.21.15.50:0/545372535' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:27:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:28 smithi050 bash[29221]: cluster 2024-04-06T03:27:27.562319+0000 mon.a (mon.0) 1079 : cluster 0 osdmap e157: 8 total, 8 up, 8 in 2024-04-06T03:27:28.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:28.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:29.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:29.110 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:29.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:29.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:29.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:29.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:29.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:29.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:27:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:27:29.570 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:27:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:27:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:27:29.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:29.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:29 smithi050 bash[21467]: cluster 2024-04-06T03:27:28.085735+0000 mgr.y (mgr.24368) 243 : cluster 0 pgmap v232: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 531 op/s 2024-04-06T03:27:29.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:29 smithi050 bash[29221]: cluster 2024-04-06T03:27:28.085735+0000 mgr.y (mgr.24368) 243 : cluster 0 pgmap v232: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 531 op/s 2024-04-06T03:27:29.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:29.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:29.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:29.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:29.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:29.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:29.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:29 smithi153 bash[23852]: cluster 2024-04-06T03:27:28.085735+0000 mgr.y (mgr.24368) 243 : cluster 0 pgmap v232: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 531 op/s 2024-04-06T03:27:29.897 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:29.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:30.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:30.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:30.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:30.265 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.291 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:30.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:30.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:30.539 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:30.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.667 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:30.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:30.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:30 smithi153 bash[23852]: audit 2024-04-06T03:27:29.267630+0000 mgr.y (mgr.24368) 244 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:30.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:30.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:30.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:30.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:30 smithi050 bash[21467]: audit 2024-04-06T03:27:29.267630+0000 mgr.y (mgr.24368) 244 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:30.907 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:30 smithi050 bash[29221]: audit 2024-04-06T03:27:29.267630+0000 mgr.y (mgr.24368) 244 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:30.942 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:30.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:31.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:31.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:31.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:31.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:31.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:31.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:31.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:31.466 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:31.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:31.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:31.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:31.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:31.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:31 smithi153 bash[23852]: cluster 2024-04-06T03:27:30.088726+0000 mgr.y (mgr.24368) 245 : cluster 0 pgmap v233: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:27:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:31.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:31.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:31 smithi050 bash[21467]: cluster 2024-04-06T03:27:30.088726+0000 mgr.y (mgr.24368) 245 : cluster 0 pgmap v233: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:27:31.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:31 smithi050 bash[29221]: cluster 2024-04-06T03:27:30.088726+0000 mgr.y (mgr.24368) 245 : cluster 0 pgmap v233: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:27:31.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:31.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:31.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:31.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:31.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:32.092 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:32.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:32.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:32.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:32.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:32.388 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:32.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.525 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:32.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:32.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:32.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:32.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:32.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:32.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:32.905 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:32.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:33.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:33.034 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:33.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:33.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:33.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:33.297 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:33.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:33.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:33.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:33.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:33.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:33.687 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:33.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:33.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:33.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:33.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:33 smithi050 bash[21467]: cluster 2024-04-06T03:27:32.089173+0000 mgr.y (mgr.24368) 246 : cluster 0 pgmap v234: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:27:33.931 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:33 smithi050 bash[29221]: cluster 2024-04-06T03:27:32.089173+0000 mgr.y (mgr.24368) 246 : cluster 0 pgmap v234: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:27:33.945 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:33.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:34.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:33 smithi153 bash[23852]: cluster 2024-04-06T03:27:32.089173+0000 mgr.y (mgr.24368) 246 : cluster 0 pgmap v234: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:27:34.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.069 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:34.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:34.214 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:27:34.214 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:27:34.214 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:27:34.214 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:27:34.214 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:27:34.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:34.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:34.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:34.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:34.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:34.609 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 158, 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-06T03:10:26.969128+0000", 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-06T03:27:34.593755+0000", 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-06T03:20:09.072730+0000", 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-06T03:19:42.545394+0000", 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5806080, 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-06T03:27:34.614 INFO:teuthology.orchestra.run.smithi050.stdout: "nodeep-scrub", 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "noscrub", 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 18, 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 7, 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-06T03:27:34.615 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:15:48.096900+0000", 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.616 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.617 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:27:34.618 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.619 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.620 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 6, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 6, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.5, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:27:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:50.011930+0000", 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.622 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.623 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "79", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.624 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.626 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:27:34.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.627 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.627 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.627 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.3200000524520874, 2024-04-06T03:27:34.629 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.3200000524520874, 2024-04-06T03:27:34.629 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:27:34.629 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-06T03:27:34.629 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.630 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:51.687715+0000", 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.631 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:27:34.632 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.633 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "81", 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.634 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.635 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.636 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.638 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.639 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:27:34.640 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.301921+0000", 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.641 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-06T03:27:34.642 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.643 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "88", 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 88, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.644 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.645 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.646 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.1699999570846558, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.810316+0000", 2024-04-06T03:27:34.647 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:27:34.648 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "83", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:27:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.650 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.651 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:27:34.652 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:56.321502+0000", 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.653 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.654 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "85", 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:27:34.655 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.656 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.657 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:27:34.658 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 7, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "unique_pool_0", 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:25:19.407695+0000", 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:27:34.659 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 16, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 16, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 16, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 16, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 16, 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:27:34.660 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "130", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:27:34.661 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:27:34.662 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:27:34.663 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 2.2599999904632568, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 2.2599999904632568, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:27:34.664 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "e607f527-4592-4acd-a47f-76575615ae1b", 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.665 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-06T03:27:34.666 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.669 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/3531754646", 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/3531754646", 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/3531754646", 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/3531754646", 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10", 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.670 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0.5613250732421875, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.672 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-06T03:27:34.673 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-06T03:27:34.674 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/4128410069", 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/4128410069", 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/4128410069", 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/4128410069", 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:27:34.675 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b", 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 19, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.676 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/1307247904", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/1307247904", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/1307247904", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/1307247904", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.680 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240", 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 138, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.681 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.682 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.683 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.684 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/1397888508", 2024-04-06T03:27:34.685 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/1397888508", 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/1397888508", 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/1397888508", 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48", 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.686 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6800", 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6801", 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6802", 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6803", 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6806", 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6807", 2024-04-06T03:27:34.689 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6804", 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6805", 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:27:34.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6801/3377683473", 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6803/3377683473", 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6807/3377683473", 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6805/3377683473", 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241", 2024-04-06T03:27:34.691 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 46, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 127, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6808", 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6809", 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6810", 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6811", 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6814", 2024-04-06T03:27:34.694 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6815", 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6812", 2024-04-06T03:27:34.695 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6813", 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6809/357179117", 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6811/357179117", 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6815/357179117", 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6813/357179117", 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.696 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "98027d53-4324-44d1-89ce-8da85be58efb", 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:27:34.697 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6816", 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6817", 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.698 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6818", 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6819", 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6822", 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6823", 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.700 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6820", 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6821", 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6817/1552538208", 2024-04-06T03:27:34.701 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6819/1552538208", 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6823/1552538208", 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6821/1552538208", 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd", 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:27:34.702 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 138, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6824", 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6825", 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6826", 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.704 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6827", 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6830", 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.705 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6831", 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6828", 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6829", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6825/2479193924", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6827/2479193924", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6831/2479193924", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6829/2479193924", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.707 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:27:33.837843+0000", 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:26:59.328936+0000", 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.709 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:27:20.087680+0000", 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:26:33.131891+0000", 2024-04-06T03:27:34.710 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:26:27.963287+0000", 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:27:34.711 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:26:48.913615+0000", 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.712 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:27:15.855749+0000", 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:26:26.064769+0000", 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-04-06T03:27:34.713 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.1f", 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-06T03:27:34.714 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/283275195": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2738019433": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1268273764": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/567103210": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2891059051": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2170219983": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/4082841962": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/570623774": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1379780741": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:27:34.715 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2880368026": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2044358308": "2024-04-07T03:21:47.915758+0000" 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-04-06T03:27:34.716 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:27:34.717 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:27:34.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:34.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:34.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:27:34.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.860 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:34 smithi153 bash[23852]: audit 2024-04-06T03:27:33.694039+0000 mon.a (mon.0) 1080 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:34.860 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:34 smithi153 bash[23852]: cluster 2024-04-06T03:27:33.837774+0000 osd.0 (osd.0) 136 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:34.860 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:34 smithi153 bash[23852]: cluster 2024-04-06T03:27:33.837840+0000 osd.0 (osd.0) 137 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:34.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:34.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:34.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:34 smithi050 bash[21467]: audit 2024-04-06T03:27:33.694039+0000 mon.a (mon.0) 1080 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:34 smithi050 bash[21467]: cluster 2024-04-06T03:27:33.837774+0000 osd.0 (osd.0) 136 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:34.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:34 smithi050 bash[21467]: cluster 2024-04-06T03:27:33.837840+0000 osd.0 (osd.0) 137 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:34.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:34 smithi050 bash[29221]: audit 2024-04-06T03:27:33.694039+0000 mon.a (mon.0) 1080 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:34.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:34 smithi050 bash[29221]: cluster 2024-04-06T03:27:33.837774+0000 osd.0 (osd.0) 136 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:34.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:34 smithi050 bash[29221]: cluster 2024-04-06T03:27:33.837840+0000 osd.0 (osd.0) 137 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:35.004 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:35.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:35.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:27:35.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:35.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:35.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:35.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:35.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:35.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:35.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:35.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:35.653 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:35.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:35 smithi050 bash[21467]: cluster 2024-04-06T03:27:34.089868+0000 mgr.y (mgr.24368) 247 : cluster 0 pgmap v235: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s 2024-04-06T03:27:35.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:35 smithi050 bash[21467]: cluster 2024-04-06T03:27:34.598839+0000 mon.a (mon.0) 1081 : cluster 0 osdmap e158: 8 total, 8 up, 8 in 2024-04-06T03:27:35.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:35 smithi050 bash[21467]: audit 2024-04-06T03:27:34.611773+0000 mon.a (mon.0) 1082 : audit 0 from='client.? 172.21.15.50:0/208845801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:27:35.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:35 smithi050 bash[29221]: cluster 2024-04-06T03:27:34.089868+0000 mgr.y (mgr.24368) 247 : cluster 0 pgmap v235: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s 2024-04-06T03:27:35.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:35 smithi050 bash[29221]: cluster 2024-04-06T03:27:34.598839+0000 mon.a (mon.0) 1081 : cluster 0 osdmap e158: 8 total, 8 up, 8 in 2024-04-06T03:27:35.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:35 smithi050 bash[29221]: audit 2024-04-06T03:27:34.611773+0000 mon.a (mon.0) 1082 : audit 0 from='client.? 172.21.15.50:0/208845801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:27:35.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:35.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:35 smithi153 bash[23852]: cluster 2024-04-06T03:27:34.089868+0000 mgr.y (mgr.24368) 247 : cluster 0 pgmap v235: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s 2024-04-06T03:27:35.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:35 smithi153 bash[23852]: cluster 2024-04-06T03:27:34.598839+0000 mon.a (mon.0) 1081 : cluster 0 osdmap e158: 8 total, 8 up, 8 in 2024-04-06T03:27:35.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:35 smithi153 bash[23852]: audit 2024-04-06T03:27:34.611773+0000 mon.a (mon.0) 1082 : audit 0 from='client.? 172.21.15.50:0/208845801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:27:35.787 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:35.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:35.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:35.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:36.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:36.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:36.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:36.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:36.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:36.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:36.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:36.434 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:36.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:36.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:36.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:36.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:36.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:36.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:36.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:36.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-06T03:27:36.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:37.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:37.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:37.108 INFO:tasks.thrashosds.thrasher:j is {'epoch': 158, 'fsid': '0313e520-f3c3-11ee-b649-cb9ed24678a4', 'created': '2024-04-06T03:10:26.969128+0000', 'modified': '2024-04-06T03:27:34.593755+0000', 'last_up_change': '2024-04-06T03:20:09.072730+0000', 'last_in_change': '2024-04-06T03:19:42.545394+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:15:48.096900+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-06T03:20:50.011930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-06T03:20:51.687715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-06T03:20:53.301921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-06T03:20:53.810316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-06T03:20:56.321502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-06T03:25:19.407695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.259999990463257, 'score_stable': 2.259999990463257, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'e607f527-4592-4acd-a47f-76575615ae1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 3531754646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 3531754646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3531754646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 3531754646}]}, 'public_addr': '172.21.15.50:6803/3531754646', 'cluster_addr': '172.21.15.50:6805/3531754646', 'heartbeat_back_addr': '172.21.15.50:6809/3531754646', 'heartbeat_front_addr': '172.21.15.50:6807/3531754646', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2ae8f784-d36a-4633-8630-2aed04e4ff10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5613250732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 4128410069}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 4128410069}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 4128410069}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 4128410069}]}, 'public_addr': '172.21.15.50:6811/4128410069', 'cluster_addr': '172.21.15.50:6813/4128410069', 'heartbeat_back_addr': '172.21.15.50:6817/4128410069', 'heartbeat_front_addr': '172.21.15.50:6815/4128410069', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '43f31d09-5c16-4aea-8822-e0d877cc931b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1307247904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1307247904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 1307247904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1307247904}]}, 'public_addr': '172.21.15.50:6819/1307247904', 'cluster_addr': '172.21.15.50:6821/1307247904', 'heartbeat_back_addr': '172.21.15.50:6825/1307247904', 'heartbeat_front_addr': '172.21.15.50:6823/1307247904', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4c913325-442e-4ee5-94f1-8b5bca0c3240', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 1397888508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 1397888508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 1397888508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 1397888508}]}, 'public_addr': '172.21.15.50:6827/1397888508', 'cluster_addr': '172.21.15.50:6829/1397888508', 'heartbeat_back_addr': '172.21.15.50:6833/1397888508', 'heartbeat_front_addr': '172.21.15.50:6831/1397888508', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50c02bfc-c2f2-4aae-b1d5-c38a86726b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6800', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6801', 'nonce': 3377683473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6802', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6803', 'nonce': 3377683473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6806', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6807', 'nonce': 3377683473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6804', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6805', 'nonce': 3377683473}]}, 'public_addr': '172.21.15.153:6801/3377683473', 'cluster_addr': '172.21.15.153:6803/3377683473', 'heartbeat_back_addr': '172.21.15.153:6807/3377683473', 'heartbeat_front_addr': '172.21.15.153:6805/3377683473', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'c3e8194b-2c20-4ffb-859d-00c33902b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6808', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6809', 'nonce': 357179117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6810', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6811', 'nonce': 357179117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6814', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6815', 'nonce': 357179117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6812', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6813', 'nonce': 357179117}]}, 'public_addr': '172.21.15.153:6809/357179117', 'cluster_addr': '172.21.15.153:6811/357179117', 'heartbeat_back_addr': '172.21.15.153:6815/357179117', 'heartbeat_front_addr': '172.21.15.153:6813/357179117', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98027d53-4324-44d1-89ce-8da85be58efb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6816', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6817', 'nonce': 1552538208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6818', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6819', 'nonce': 1552538208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6822', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6823', 'nonce': 1552538208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6820', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6821', 'nonce': 1552538208}]}, 'public_addr': '172.21.15.153:6817/1552538208', 'cluster_addr': '172.21.15.153:6819/1552538208', 'heartbeat_back_addr': '172.21.15.153:6823/1552538208', 'heartbeat_front_addr': '172.21.15.153:6821/1552538208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6b47d95a-1228-4fb1-a315-5371d9b42bbd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6824', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6825', 'nonce': 2479193924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6826', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6827', 'nonce': 2479193924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6830', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6831', 'nonce': 2479193924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6828', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6829', 'nonce': 2479193924}]}, 'public_addr': '172.21.15.153:6825/2479193924', 'cluster_addr': '172.21.15.153:6827/2479193924', 'heartbeat_back_addr': '172.21.15.153:6831/2479193924', 'heartbeat_front_addr': '172.21.15.153:6829/2479193924', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:27:33.837843+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:26:59.328936+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:27:20.087680+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:26:33.131891+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:26:27.963287+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:26:48.913615+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:27:15.855749+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:26:26.064769+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1f', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/283275195': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2738019433': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1268273764': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/567103210': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/2891059051': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2170219983': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/4082841962': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6800/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/570623774': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6801/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1379780741': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2880368026': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2044358308': '2024-04-07T03:21:47.915758+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-06T03:27:37.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:27:37.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:37.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:37.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:37.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:37.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:37.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:37.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:37.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:37.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:37.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:37 smithi050 bash[21467]: cluster 2024-04-06T03:27:36.090346+0000 mgr.y (mgr.24368) 248 : cluster 0 pgmap v237: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:27:37.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:37 smithi050 bash[29221]: cluster 2024-04-06T03:27:36.090346+0000 mgr.y (mgr.24368) 248 : cluster 0 pgmap v237: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:27:37.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:37.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:37.840 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:37.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:37.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:37.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:38.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:37 smithi153 bash[23852]: cluster 2024-04-06T03:27:36.090346+0000 mgr.y (mgr.24368) 248 : cluster 0 pgmap v237: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:27:38.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:38.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:38.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:38.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:38.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:38.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:38.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:38.533 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:38.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:38.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:38.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:38.796 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:38.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:38.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:39.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:39.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:39.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:39.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:39.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:39.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:39.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:39.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:39.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:27:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:27:39.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:39.578 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:39.608 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:27:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:27:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:27:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:39.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:39.702 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:39.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:39.837 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:39.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:39.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:39 smithi050 bash[21467]: cluster 2024-04-06T03:27:38.090859+0000 mgr.y (mgr.24368) 249 : cluster 0 pgmap v238: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-04-06T03:27:39.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:39 smithi050 bash[29221]: cluster 2024-04-06T03:27:38.090859+0000 mgr.y (mgr.24368) 249 : cluster 0 pgmap v238: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-04-06T03:27:39.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:39.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:40.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:39 smithi153 bash[23852]: cluster 2024-04-06T03:27:38.090859+0000 mgr.y (mgr.24368) 249 : cluster 0 pgmap v238: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-04-06T03:27:40.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:40.097 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:40.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:40.232 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:40.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:40.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:40.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:40.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:40.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:40.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:40.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:40.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:40.745 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:40 smithi153 bash[23852]: audit 2024-04-06T03:27:39.276797+0000 mgr.y (mgr.24368) 250 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:40.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:40.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:40.885 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:40.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:40.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:40 smithi050 bash[21467]: audit 2024-04-06T03:27:39.276797+0000 mgr.y (mgr.24368) 250 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:40.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:40 smithi050 bash[29221]: audit 2024-04-06T03:27:39.276797+0000 mgr.y (mgr.24368) 250 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:40.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:41.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:41.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:41.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:41.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:41.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:41.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:41.414 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:41.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:41.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:41.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:41.672 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:27:41.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:41.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:41.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:41 smithi050 bash[21467]: cluster 2024-04-06T03:27:40.091574+0000 mgr.y (mgr.24368) 251 : cluster 0 pgmap v239: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:27:41.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:41 smithi050 bash[29221]: cluster 2024-04-06T03:27:40.091574+0000 mgr.y (mgr.24368) 251 : cluster 0 pgmap v239: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:27:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:41.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:41.927 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:41 smithi153 bash[23852]: cluster 2024-04-06T03:27:40.091574+0000 mgr.y (mgr.24368) 251 : cluster 0 pgmap v239: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:27:41.941 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:41.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:42.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:42.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:42.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:42.301 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:42.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:42.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:42.561 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:42.582 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:42.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:42.713 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:42.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:42.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:42 smithi153 bash[23852]: cluster 2024-04-06T03:27:42.092580+0000 mgr.y (mgr.24368) 252 : cluster 0 pgmap v240: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:27:42.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:42.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:42.971 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:42.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:42.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:42 smithi050 bash[21467]: cluster 2024-04-06T03:27:42.092580+0000 mgr.y (mgr.24368) 252 : cluster 0 pgmap v240: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:27:42.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:42 smithi050 bash[29221]: cluster 2024-04-06T03:27:42.092580+0000 mgr.y (mgr.24368) 252 : cluster 0 pgmap v240: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:27:43.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:43.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:43.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:43.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:43.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:43.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:43.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:43.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:43.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:43.623 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:43.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:43.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:43.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:43.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:43.873 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:43.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:43.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.004 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:44.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:44.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:44.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:44.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:44.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:44.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:44.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:44.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:44.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:44.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:44.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:44.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:45.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:45.038 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:45.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:45.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:45.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:45.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:45 smithi153 bash[23852]: cluster 2024-04-06T03:27:44.041998+0000 osd.7 (osd.7) 141 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:45.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:45 smithi153 bash[23852]: cluster 2024-04-06T03:27:44.042055+0000 osd.7 (osd.7) 142 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:45.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:45 smithi153 bash[23852]: cluster 2024-04-06T03:27:44.093338+0000 mgr.y (mgr.24368) 253 : cluster 0 pgmap v241: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 890 op/s 2024-04-06T03:27:45.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:45 smithi153 bash[23852]: cluster 2024-04-06T03:27:44.115784+0000 mon.a (mon.0) 1083 : cluster 0 osdmap e159: 8 total, 8 up, 8 in 2024-04-06T03:27:45.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:45 smithi153 bash[23852]: cluster 2024-04-06T03:27:44.207965+0000 osd.4 (osd.4) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:45.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:45 smithi153 bash[23852]: cluster 2024-04-06T03:27:44.208045+0000 osd.4 (osd.4) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:45.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:45.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:45.430 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[21467]: cluster 2024-04-06T03:27:44.041998+0000 osd.7 (osd.7) 141 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[21467]: cluster 2024-04-06T03:27:44.042055+0000 osd.7 (osd.7) 142 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[21467]: cluster 2024-04-06T03:27:44.093338+0000 mgr.y (mgr.24368) 253 : cluster 0 pgmap v241: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 890 op/s 2024-04-06T03:27:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[21467]: cluster 2024-04-06T03:27:44.115784+0000 mon.a (mon.0) 1083 : cluster 0 osdmap e159: 8 total, 8 up, 8 in 2024-04-06T03:27:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[21467]: cluster 2024-04-06T03:27:44.207965+0000 osd.4 (osd.4) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:45.482 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[21467]: cluster 2024-04-06T03:27:44.208045+0000 osd.4 (osd.4) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[29221]: cluster 2024-04-06T03:27:44.041998+0000 osd.7 (osd.7) 141 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[29221]: cluster 2024-04-06T03:27:44.042055+0000 osd.7 (osd.7) 142 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[29221]: cluster 2024-04-06T03:27:44.093338+0000 mgr.y (mgr.24368) 253 : cluster 0 pgmap v241: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 890 op/s 2024-04-06T03:27:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[29221]: cluster 2024-04-06T03:27:44.115784+0000 mon.a (mon.0) 1083 : cluster 0 osdmap e159: 8 total, 8 up, 8 in 2024-04-06T03:27:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[29221]: cluster 2024-04-06T03:27:44.207965+0000 osd.4 (osd.4) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:27:45.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:45 smithi050 bash[29221]: cluster 2024-04-06T03:27:44.208045+0000 osd.4 (osd.4) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:27:45.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:45.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:45.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:45.702 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:45.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:45.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:45.831 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:45.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:45.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:45.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:46.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:46.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:46.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:46.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:46.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:46 smithi050 bash[21467]: cluster 2024-04-06T03:27:45.152524+0000 mon.a (mon.0) 1084 : cluster 0 osdmap e160: 8 total, 8 up, 8 in 2024-04-06T03:27:46.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:46 smithi050 bash[29221]: cluster 2024-04-06T03:27:45.152524+0000 mon.a (mon.0) 1084 : cluster 0 osdmap e160: 8 total, 8 up, 8 in 2024-04-06T03:27:46.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:46.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:46.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:46.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:46 smithi153 bash[23852]: cluster 2024-04-06T03:27:45.152524+0000 mon.a (mon.0) 1084 : cluster 0 osdmap e160: 8 total, 8 up, 8 in 2024-04-06T03:27:46.464 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:46.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:46.591 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:46.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:46.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:46.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:46.850 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:46.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:46.977 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:46.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:47.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:47.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:47.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:47.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:47.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:47.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:47.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:47 smithi050 bash[29221]: cluster 2024-04-06T03:27:46.093782+0000 mgr.y (mgr.24368) 254 : cluster 0 pgmap v244: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-04-06T03:27:47.370 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:47 smithi050 bash[21467]: cluster 2024-04-06T03:27:46.093782+0000 mgr.y (mgr.24368) 254 : cluster 0 pgmap v244: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-04-06T03:27:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:47.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:47.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:47 smithi153 bash[23852]: cluster 2024-04-06T03:27:46.093782+0000 mgr.y (mgr.24368) 254 : cluster 0 pgmap v244: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-04-06T03:27:47.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:47.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:47.630 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:47.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:47.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:47.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:47.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:48.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:48.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:48.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:48.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:48.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:48.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:48.418 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:48.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:48.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:48.674 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:48.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:48.802 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:48.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:48.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:48.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:49.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:49.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:49.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:49.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:49.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:49.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:49.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:49.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:27:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:27:49.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:49.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:49 smithi153 bash[23852]: cluster 2024-04-06T03:27:48.094094+0000 mgr.y (mgr.24368) 255 : cluster 0 pgmap v245: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 576 op/s 2024-04-06T03:27:49.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:49 smithi153 bash[23852]: audit 2024-04-06T03:27:48.718212+0000 mon.a (mon.0) 1085 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:49.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:49.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:49 smithi050 bash[21467]: cluster 2024-04-06T03:27:48.094094+0000 mgr.y (mgr.24368) 255 : cluster 0 pgmap v245: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 576 op/s 2024-04-06T03:27:49.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:49 smithi050 bash[21467]: audit 2024-04-06T03:27:48.718212+0000 mon.a (mon.0) 1085 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:49.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:49 smithi050 bash[29221]: cluster 2024-04-06T03:27:48.094094+0000 mgr.y (mgr.24368) 255 : cluster 0 pgmap v245: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 576 op/s 2024-04-06T03:27:49.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:49 smithi050 bash[29221]: audit 2024-04-06T03:27:48.718212+0000 mon.a (mon.0) 1085 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:27:49.576 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:27:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:27:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:27:49.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:49.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:49.720 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:49.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:49.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:49.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:49.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:49.985 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:49.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:50.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:50.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:50.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:50.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:50.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:50.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:50.497 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:50 smithi153 bash[23852]: audit 2024-04-06T03:27:49.284969+0000 mgr.y (mgr.24368) 256 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:50.510 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:50.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:50.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:50.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:50 smithi050 bash[21467]: audit 2024-04-06T03:27:49.284969+0000 mgr.y (mgr.24368) 256 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:50.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:50 smithi050 bash[29221]: audit 2024-04-06T03:27:49.284969+0000 mgr.y (mgr.24368) 256 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:27:50.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:50.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:50.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:50.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:50.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:51.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:51.029 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:51.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:51.154 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:51.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:51.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:51.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:51.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:51.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:51.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:51.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:51 smithi153 bash[23852]: cluster 2024-04-06T03:27:50.095233+0000 mgr.y (mgr.24368) 257 : cluster 0 pgmap v246: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-04-06T03:27:51.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:51.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:51 smithi050 bash[21467]: cluster 2024-04-06T03:27:50.095233+0000 mgr.y (mgr.24368) 257 : cluster 0 pgmap v246: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-04-06T03:27:51.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:51 smithi050 bash[29221]: cluster 2024-04-06T03:27:50.095233+0000 mgr.y (mgr.24368) 257 : cluster 0 pgmap v246: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-04-06T03:27:51.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:51.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:51.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:51.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:51.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:52.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:52.069 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:52.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:52.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:52.318 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:52.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:52.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:52.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:52.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:52.589 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:52.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:52.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:52.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:52.846 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:52.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:52.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:52.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:53.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:53.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:53.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:53.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:53.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:53.354 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:53.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:53.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:53.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:53.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:53.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:53.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:53.726 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:53 smithi153 bash[23852]: cluster 2024-04-06T03:27:52.095751+0000 mgr.y (mgr.24368) 258 : cluster 0 pgmap v247: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 895 op/s 2024-04-06T03:27:53.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:53 smithi050 bash[21467]: cluster 2024-04-06T03:27:52.095751+0000 mgr.y (mgr.24368) 258 : cluster 0 pgmap v247: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 895 op/s 2024-04-06T03:27:53.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:53 smithi050 bash[29221]: cluster 2024-04-06T03:27:52.095751+0000 mgr.y (mgr.24368) 258 : cluster 0 pgmap v247: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 895 op/s 2024-04-06T03:27:53.740 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:53.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:53.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:53.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:53.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:54.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:54.128 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:27:54.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:54.162 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:54.182 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:27:54.248 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:27:54.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:54.287 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:54.316 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:27:54.355 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:27:54.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:54.397 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:27:54.410 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:54.435 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:27:54.436 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:27:54.439 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:27:54.483 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:27:54.499 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:27:54.504 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:27:54.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:54.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:54.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:54 smithi153 bash[23852]: audit 2024-04-06T03:27:54.078229+0000 mon.c (mon.2) 46 : audit 0 from='client.? 172.21.15.50:0/3483983414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:54.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:54 smithi153 bash[23852]: audit 2024-04-06T03:27:54.163550+0000 mon.c (mon.2) 47 : audit 1 from='client.? 172.21.15.50:0/1747187000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:27:54.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:54 smithi153 bash[23852]: audit 2024-04-06T03:27:54.164141+0000 mon.a (mon.0) 1086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:27:54.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:54 smithi050 bash[21467]: audit 2024-04-06T03:27:54.078229+0000 mon.c (mon.2) 46 : audit 0 from='client.? 172.21.15.50:0/3483983414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:54 smithi050 bash[21467]: audit 2024-04-06T03:27:54.163550+0000 mon.c (mon.2) 47 : audit 1 from='client.? 172.21.15.50:0/1747187000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:27:54.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:54 smithi050 bash[21467]: audit 2024-04-06T03:27:54.164141+0000 mon.a (mon.0) 1086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:27:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:54 smithi050 bash[29221]: audit 2024-04-06T03:27:54.078229+0000 mon.c (mon.2) 46 : audit 0 from='client.? 172.21.15.50:0/3483983414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:27:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:54 smithi050 bash[29221]: audit 2024-04-06T03:27:54.163550+0000 mon.c (mon.2) 47 : audit 1 from='client.? 172.21.15.50:0/1747187000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:27:54.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:54 smithi050 bash[29221]: audit 2024-04-06T03:27:54.164141+0000 mon.a (mon.0) 1086 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:27:54.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:54.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:54.875 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:27:54.889 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:27:54.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:54.961 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:54.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:55.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:55.085 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:55.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:55.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:55.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:55.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:55.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:55.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:55.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:55.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:55.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:55.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:55.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[21467]: cluster 2024-04-06T03:27:54.096413+0000 mgr.y (mgr.24368) 259 : cluster 0 pgmap v248: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 922 B/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:27:55.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[21467]: cluster 2024-04-06T03:27:54.484312+0000 mon.a (mon.0) 1087 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:55.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[21467]: audit 2024-04-06T03:27:54.491275+0000 mon.a (mon.0) 1088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:27:55.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[21467]: cluster 2024-04-06T03:27:54.497188+0000 mon.a (mon.0) 1089 : cluster 0 osdmap e161: 8 total, 8 up, 8 in 2024-04-06T03:27:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[29221]: cluster 2024-04-06T03:27:54.096413+0000 mgr.y (mgr.24368) 259 : cluster 0 pgmap v248: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 922 B/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:27:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[29221]: cluster 2024-04-06T03:27:54.484312+0000 mon.a (mon.0) 1087 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[29221]: audit 2024-04-06T03:27:54.491275+0000 mon.a (mon.0) 1088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:27:55.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:55 smithi050 bash[29221]: cluster 2024-04-06T03:27:54.497188+0000 mon.a (mon.0) 1089 : cluster 0 osdmap e161: 8 total, 8 up, 8 in 2024-04-06T03:27:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:55.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:55 smithi153 bash[23852]: cluster 2024-04-06T03:27:54.096413+0000 mgr.y (mgr.24368) 259 : cluster 0 pgmap v248: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 922 B/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:27:55.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:55 smithi153 bash[23852]: cluster 2024-04-06T03:27:54.484312+0000 mon.a (mon.0) 1087 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:27:55.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:55 smithi153 bash[23852]: audit 2024-04-06T03:27:54.491275+0000 mon.a (mon.0) 1088 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:27:55.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:55 smithi153 bash[23852]: cluster 2024-04-06T03:27:54.497188+0000 mon.a (mon.0) 1089 : cluster 0 osdmap e161: 8 total, 8 up, 8 in 2024-04-06T03:27:55.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:55.857 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:55.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:55.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:56.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:56.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:56.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:56.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:56.266 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:56.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:56.394 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:56.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:56.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:56 smithi153 bash[23852]: audit 2024-04-06T03:27:54.857452+0000 mgr.y (mgr.24368) 260 : audit 0 from='client.24607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:27:56.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:56 smithi153 bash[23852]: cluster 2024-04-06T03:27:55.731179+0000 osd.7 (osd.7) 143 : cluster 0 2.0 scrub starts 2024-04-06T03:27:56.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:56 smithi153 bash[23852]: cluster 2024-04-06T03:27:55.738389+0000 osd.7 (osd.7) 144 : cluster 0 2.0 scrub ok 2024-04-06T03:27:56.522 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:56.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:56.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:56.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:56.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:56 smithi050 bash[21467]: audit 2024-04-06T03:27:54.857452+0000 mgr.y (mgr.24368) 260 : audit 0 from='client.24607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:27:56.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:56 smithi050 bash[21467]: cluster 2024-04-06T03:27:55.731179+0000 osd.7 (osd.7) 143 : cluster 0 2.0 scrub starts 2024-04-06T03:27:56.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:56 smithi050 bash[21467]: cluster 2024-04-06T03:27:55.738389+0000 osd.7 (osd.7) 144 : cluster 0 2.0 scrub ok 2024-04-06T03:27:56.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:56 smithi050 bash[29221]: audit 2024-04-06T03:27:54.857452+0000 mgr.y (mgr.24368) 260 : audit 0 from='client.24607 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:27:56.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:56 smithi050 bash[29221]: cluster 2024-04-06T03:27:55.731179+0000 osd.7 (osd.7) 143 : cluster 0 2.0 scrub starts 2024-04-06T03:27:56.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:56 smithi050 bash[29221]: cluster 2024-04-06T03:27:55.738389+0000 osd.7 (osd.7) 144 : cluster 0 2.0 scrub ok 2024-04-06T03:27:56.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:56.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:56.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:57.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:57.026 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:27:57.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:57.149 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:27:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:27:57.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:57.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:57.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:57.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-06T03:27:57.405 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:57.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:57.530 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:27:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:27:57.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.097073+0000 mgr.y (mgr.24368) 261 : cluster 0 pgmap v250: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 856 op/s 2024-04-06T03:27:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.708649+0000 osd.7 (osd.7) 145 : cluster 0 6.1c scrub starts 2024-04-06T03:27:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.711516+0000 osd.7 (osd.7) 146 : cluster 0 6.1c scrub ok 2024-04-06T03:27:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.736512+0000 osd.2 (osd.2) 119 : cluster 0 3.0 scrub starts 2024-04-06T03:27:57.641 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.747917+0000 osd.2 (osd.2) 120 : cluster 0 3.0 scrub ok 2024-04-06T03:27:57.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.097073+0000 mgr.y (mgr.24368) 261 : cluster 0 pgmap v250: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 856 op/s 2024-04-06T03:27:57.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.708649+0000 osd.7 (osd.7) 145 : cluster 0 6.1c scrub starts 2024-04-06T03:27:57.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.711516+0000 osd.7 (osd.7) 146 : cluster 0 6.1c scrub ok 2024-04-06T03:27:57.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.736512+0000 osd.2 (osd.2) 119 : cluster 0 3.0 scrub starts 2024-04-06T03:27:57.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:57 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.747917+0000 osd.2 (osd.2) 120 : cluster 0 3.0 scrub ok 2024-04-06T03:27:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:57.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:57 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.097073+0000 mgr.y (mgr.24368) 261 : cluster 0 pgmap v250: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 856 op/s 2024-04-06T03:27:57.764 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:57 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.708649+0000 osd.7 (osd.7) 145 : cluster 0 6.1c scrub starts 2024-04-06T03:27:57.764 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:57 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.711516+0000 osd.7 (osd.7) 146 : cluster 0 6.1c scrub ok 2024-04-06T03:27:57.764 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:57 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.736512+0000 osd.2 (osd.2) 119 : cluster 0 3.0 scrub starts 2024-04-06T03:27:57.764 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:57 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.747917+0000 osd.2 (osd.2) 120 : cluster 0 3.0 scrub ok 2024-04-06T03:27:57.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:57.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:57.905 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:57.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:58.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:58.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:58.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:58.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:27:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:27:58.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:58.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:58.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:27:58.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:27:58.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.811521+0000 osd.4 (osd.4) 141 : cluster 0 3.9 scrub starts 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:56.816394+0000 osd.4 (osd.4) 142 : cluster 0 3.9 scrub ok 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:57.060858+0000 osd.3 (osd.3) 169 : cluster 0 5.17 scrub starts 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:57.089903+0000 osd.3 (osd.3) 170 : cluster 0 5.17 scrub ok 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:57.791801+0000 osd.7 (osd.7) 147 : cluster 0 1.0 scrub starts 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:57.798000+0000 osd.7 (osd.7) 148 : cluster 0 1.0 scrub ok 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:57.811460+0000 osd.2 (osd.2) 121 : cluster 0 2.c scrub starts 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:57.819038+0000 osd.2 (osd.2) 122 : cluster 0 2.c scrub ok 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.044158+0000 osd.3 (osd.3) 171 : cluster 0 6.6 scrub starts 2024-04-06T03:27:58.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.045372+0000 osd.3 (osd.3) 172 : cluster 0 6.6 scrub ok 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.811521+0000 osd.4 (osd.4) 141 : cluster 0 3.9 scrub starts 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:56.816394+0000 osd.4 (osd.4) 142 : cluster 0 3.9 scrub ok 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:57.060858+0000 osd.3 (osd.3) 169 : cluster 0 5.17 scrub starts 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:57.089903+0000 osd.3 (osd.3) 170 : cluster 0 5.17 scrub ok 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:57.791801+0000 osd.7 (osd.7) 147 : cluster 0 1.0 scrub starts 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:57.798000+0000 osd.7 (osd.7) 148 : cluster 0 1.0 scrub ok 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:57.811460+0000 osd.2 (osd.2) 121 : cluster 0 2.c scrub starts 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:57.819038+0000 osd.2 (osd.2) 122 : cluster 0 2.c scrub ok 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.044158+0000 osd.3 (osd.3) 171 : cluster 0 6.6 scrub starts 2024-04-06T03:27:58.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:58 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.045372+0000 osd.3 (osd.3) 172 : cluster 0 6.6 scrub ok 2024-04-06T03:27:58.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:58.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:58.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:58.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.811521+0000 osd.4 (osd.4) 141 : cluster 0 3.9 scrub starts 2024-04-06T03:27:58.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:56.816394+0000 osd.4 (osd.4) 142 : cluster 0 3.9 scrub ok 2024-04-06T03:27:58.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:57.060858+0000 osd.3 (osd.3) 169 : cluster 0 5.17 scrub starts 2024-04-06T03:27:58.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:57.089903+0000 osd.3 (osd.3) 170 : cluster 0 5.17 scrub ok 2024-04-06T03:27:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:57.791801+0000 osd.7 (osd.7) 147 : cluster 0 1.0 scrub starts 2024-04-06T03:27:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:57.798000+0000 osd.7 (osd.7) 148 : cluster 0 1.0 scrub ok 2024-04-06T03:27:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:57.811460+0000 osd.2 (osd.2) 121 : cluster 0 2.c scrub starts 2024-04-06T03:27:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:57.819038+0000 osd.2 (osd.2) 122 : cluster 0 2.c scrub ok 2024-04-06T03:27:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.044158+0000 osd.3 (osd.3) 171 : cluster 0 6.6 scrub starts 2024-04-06T03:27:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:58 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.045372+0000 osd.3 (osd.3) 172 : cluster 0 6.6 scrub ok 2024-04-06T03:27:58.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:58.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:58.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:58.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:59.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:59.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:27:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:27:59.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:59.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:59.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:59.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:59.496 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:27:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:27:59.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:27:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:27:59.534 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-04-06T03:27:54.491926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3876723062,"num_objects":59376,"num_object_clones":0,"num_object_copies":178128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59376,"num_whiteouts":0,"num_read":1630,"num_read_kb":1444,"num_write":135157,"num_write_kb":6194270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86965,"ondisk_log_size":86965,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11818888,"kb_used_data":11187528,"kb_used_omap":12,"kb_used_meta":631091,"kb_avail":738142328,"statfs":{"total":767960285184,"available":755857743872,"internally_reserved":0,"allocated":11456028672,"data_stored":11453293474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":646237745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,7,3,2,0,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-356122624,"num_objects":-5434,"num_object_clones":0,"num_object_copies":-16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5434,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.340023"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.164801+0000","last_change":"2024-04-06T03:24:57.139640+0000","last_active":"2024-04-06T03:27:45.164801+0000","last_peered":"2024-04-06T03:27:45.164801+0000","last_clean":"2024-04-06T03:27:45.164801+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:27:45.164801+0000","last_undegraded":"2024-04-06T03:27:45.164801+0000","last_fullsized":"2024-04-06T03:27:45.164801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:57.139428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:57.139428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166212+0000","last_change":"2024-04-06T03:26:16.639704+0000","last_active":"2024-04-06T03:27:45.166212+0000","last_peered":"2024-04-06T03:27:45.166212+0000","last_clean":"2024-04-06T03:27:45.166212+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:27:45.166212+0000","last_undegraded":"2024-04-06T03:27:45.166212+0000","last_fullsized":"2024-04-06T03:27:45.166212+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:16.639561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:03.509185+0000","last_clean_scrub_stamp":"2024-04-06T03:26:16.639561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.260163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":206,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172338+0000","last_change":"2024-04-06T03:26:24.426453+0000","last_active":"2024-04-06T03:27:45.172338+0000","last_peered":"2024-04-06T03:27:45.172338+0000","last_clean":"2024-04-06T03:27:45.172338+0000","last_became_active":"2024-04-06T03:21:35.160225+0000","last_became_peered":"2024-04-06T03:21:35.160225+0000","last_unstale":"2024-04-06T03:27:45.172338+0000","last_undegraded":"2024-04-06T03:27:45.172338+0000","last_fullsized":"2024-04-06T03:27:45.172338+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:24.426335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:24.426335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846838+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"118'22","reported_seq":239,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161820+0000","last_change":"2024-04-06T03:26:34.645678+0000","last_active":"2024-04-06T03:27:45.161820+0000","last_peered":"2024-04-06T03:27:45.161820+0000","last_clean":"2024-04-06T03:27:45.161820+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:27:45.161820+0000","last_undegraded":"2024-04-06T03:27:45.161820+0000","last_fullsized":"2024-04-06T03:27:45.161820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-04-06T03:26:34.645348+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:26:34.645348+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166179+0000","last_change":"2024-04-06T03:26:08.704686+0000","last_active":"2024-04-06T03:27:45.166179+0000","last_peered":"2024-04-06T03:27:45.166179+0000","last_clean":"2024-04-06T03:27:45.166179+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:27:45.166179+0000","last_undegraded":"2024-04-06T03:27:45.166179+0000","last_fullsized":"2024-04-06T03:27:45.166179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:08.704593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:08.704593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":72,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160631+0000","last_change":"2024-04-06T03:26:51.105887+0000","last_active":"2024-04-06T03:27:45.160631+0000","last_peered":"2024-04-06T03:27:45.160631+0000","last_clean":"2024-04-06T03:27:45.160631+0000","last_became_active":"2024-04-06T03:25:54.480600+0000","last_became_peered":"2024-04-06T03:25:54.480600+0000","last_unstale":"2024-04-06T03:27:45.160631+0000","last_undegraded":"2024-04-06T03:27:45.160631+0000","last_fullsized":"2024-04-06T03:27:45.160631+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:51.105796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:51.105796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:54.746412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161596+0000","last_change":"2024-04-06T03:26:50.667097+0000","last_active":"2024-04-06T03:27:45.161596+0000","last_peered":"2024-04-06T03:27:45.161596+0000","last_clean":"2024-04-06T03:27:45.161596+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:27:45.161596+0000","last_undegraded":"2024-04-06T03:27:45.161596+0000","last_fullsized":"2024-04-06T03:27:45.161596+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:50.666973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:26:50.666973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:53.549925+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"125'23","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172195+0000","last_change":"2024-04-06T03:25:17.013955+0000","last_active":"2024-04-06T03:27:45.172195+0000","last_peered":"2024-04-06T03:27:45.172195+0000","last_clean":"2024-04-06T03:27:45.172195+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:27:45.172195+0000","last_undegraded":"2024-04-06T03:27:45.172195+0000","last_fullsized":"2024-04-06T03:27:45.172195+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'23","last_scrub_stamp":"2024-04-06T03:25:17.013823+0000","last_deep_scrub":"125'23","last_deep_scrub_stamp":"2024-04-06T03:25:17.013823+0000","last_clean_scrub_stamp":"2024-04-06T03:25:17.013823+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.834825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":69,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170561+0000","last_change":"2024-04-06T03:26:16.332192+0000","last_active":"2024-04-06T03:27:45.170561+0000","last_peered":"2024-04-06T03:27:45.170561+0000","last_clean":"2024-04-06T03:27:45.170561+0000","last_became_active":"2024-04-06T03:25:54.484120+0000","last_became_peered":"2024-04-06T03:25:54.484120+0000","last_unstale":"2024-04-06T03:27:45.170561+0000","last_undegraded":"2024-04-06T03:27:45.170561+0000","last_fullsized":"2024-04-06T03:27:45.170561+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:16.331800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:16.331800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":191,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160964+0000","last_change":"2024-04-06T03:26:30.070300+0000","last_active":"2024-04-06T03:27:45.160964+0000","last_peered":"2024-04-06T03:27:45.160964+0000","last_clean":"2024-04-06T03:27:45.160964+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:27:45.160964+0000","last_undegraded":"2024-04-06T03:27:45.160964+0000","last_fullsized":"2024-04-06T03:27:45.160964+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:30.070183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:26:30.070183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161595+0000","last_change":"2024-04-06T03:24:54.848117+0000","last_active":"2024-04-06T03:27:45.161595+0000","last_peered":"2024-04-06T03:27:45.161595+0000","last_clean":"2024-04-06T03:27:45.161595+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:27:45.161595+0000","last_undegraded":"2024-04-06T03:27:45.161595+0000","last_fullsized":"2024-04-06T03:27:45.161595+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:24:54.847895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:54.847895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"112'22","reported_seq":179,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166571+0000","last_change":"2024-04-06T03:26:40.864933+0000","last_active":"2024-04-06T03:27:45.166571+0000","last_peered":"2024-04-06T03:27:45.166571+0000","last_clean":"2024-04-06T03:27:45.166571+0000","last_became_active":"2024-04-06T03:25:54.489462+0000","last_became_peered":"2024-04-06T03:25:54.489462+0000","last_unstale":"2024-04-06T03:27:45.166571+0000","last_undegraded":"2024-04-06T03:27:45.166571+0000","last_fullsized":"2024-04-06T03:27:45.166571+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'22","last_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:26:40.864830+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:07.236955+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":187,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168690+0000","last_change":"2024-04-06T03:26:35.279895+0000","last_active":"2024-04-06T03:27:45.168690+0000","last_peered":"2024-04-06T03:27:45.168690+0000","last_clean":"2024-04-06T03:27:45.168690+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:27:45.168690+0000","last_undegraded":"2024-04-06T03:27:45.168690+0000","last_fullsized":"2024-04-06T03:27:45.168690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:35.279815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:35.279815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.622096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171524+0000","last_change":"2024-04-06T03:26:11.434816+0000","last_active":"2024-04-06T03:27:45.171524+0000","last_peered":"2024-04-06T03:27:45.171524+0000","last_clean":"2024-04-06T03:27:45.171524+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:27:45.171524+0000","last_undegraded":"2024-04-06T03:27:45.171524+0000","last_fullsized":"2024-04-06T03:27:45.171524+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:11.434676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:11.434676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846755+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":199,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.162078+0000","last_change":"2024-04-06T03:26:46.087205+0000","last_active":"2024-04-06T03:27:45.162078+0000","last_peered":"2024-04-06T03:27:45.162078+0000","last_clean":"2024-04-06T03:27:45.162078+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:27:45.162078+0000","last_undegraded":"2024-04-06T03:27:45.162078+0000","last_fullsized":"2024-04-06T03:27:45.162078+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:46.087127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:46.087127+0000","last_clean_scrub_stamp":"2024-04-06T03:26:46.087127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:10.342834+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"105'24","reported_seq":158,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166724+0000","last_change":"2024-04-06T03:25:54.485353+0000","last_active":"2024-04-06T03:27:45.166724+0000","last_peered":"2024-04-06T03:27:45.166724+0000","last_clean":"2024-04-06T03:27:45.166724+0000","last_became_active":"2024-04-06T03:25:54.483976+0000","last_became_peered":"2024-04-06T03:25:54.483976+0000","last_unstale":"2024-04-06T03:27:45.166724+0000","last_undegraded":"2024-04-06T03:27:45.166724+0000","last_fullsized":"2024-04-06T03:27:45.166724+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'24","last_scrub_stamp":"2024-04-06T03:25:10.015565+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:25:10.015565+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168032+0000","last_change":"2024-04-06T03:26:40.749202+0000","last_active":"2024-04-06T03:27:45.168032+0000","last_peered":"2024-04-06T03:27:45.168032+0000","last_clean":"2024-04-06T03:27:45.168032+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:27:45.168032+0000","last_undegraded":"2024-04-06T03:27:45.168032+0000","last_fullsized":"2024-04-06T03:27:45.168032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:40.749062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:40.749062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166400+0000","last_change":"2024-04-06T03:26:58.762554+0000","last_active":"2024-04-06T03:27:45.166400+0000","last_peered":"2024-04-06T03:27:45.166400+0000","last_clean":"2024-04-06T03:27:45.166400+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:27:45.166400+0000","last_undegraded":"2024-04-06T03:27:45.166400+0000","last_fullsized":"2024-04-06T03:27:45.166400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:26:58.762469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:59.121072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.164976+0000","last_change":"2024-04-06T03:26:12.560964+0000","last_active":"2024-04-06T03:27:45.164976+0000","last_peered":"2024-04-06T03:27:45.164976+0000","last_clean":"2024-04-06T03:27:45.164976+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:27:45.164976+0000","last_undegraded":"2024-04-06T03:27:45.164976+0000","last_fullsized":"2024-04-06T03:27:45.164976+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:12.560832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:12.560832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"111'27","reported_seq":251,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166291+0000","last_change":"2024-04-06T03:26:23.735580+0000","last_active":"2024-04-06T03:27:45.166291+0000","last_peered":"2024-04-06T03:27:45.166291+0000","last_clean":"2024-04-06T03:27:45.166291+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:27:45.166291+0000","last_undegraded":"2024-04-06T03:27:45.166291+0000","last_fullsized":"2024-04-06T03:27:45.166291+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'27","last_scrub_stamp":"2024-04-06T03:26:23.735363+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:23:18.287127+0000","last_clean_scrub_stamp":"2024-04-06T03:26:23.735363+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.831676+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":190,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166902+0000","last_change":"2024-04-06T03:26:10.668141+0000","last_active":"2024-04-06T03:27:45.166902+0000","last_peered":"2024-04-06T03:27:45.166902+0000","last_clean":"2024-04-06T03:27:45.166902+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:27:45.166902+0000","last_undegraded":"2024-04-06T03:27:45.166902+0000","last_fullsized":"2024-04-06T03:27:45.166902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:10.668004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:10.668004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":192,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171670+0000","last_change":"2024-04-06T03:27:03.211111+0000","last_active":"2024-04-06T03:27:45.171670+0000","last_peered":"2024-04-06T03:27:45.171670+0000","last_clean":"2024-04-06T03:27:45.171670+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:27:45.171670+0000","last_undegraded":"2024-04-06T03:27:45.171670+0000","last_fullsized":"2024-04-06T03:27:45.171670+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:27:03.210854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:01.686796+0000","last_clean_scrub_stamp":"2024-04-06T03:27:03.210854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:15.697641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":62,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166349+0000","last_change":"2024-04-06T03:26:51.780885+0000","last_active":"2024-04-06T03:27:45.166349+0000","last_peered":"2024-04-06T03:27:45.166349+0000","last_clean":"2024-04-06T03:27:45.166349+0000","last_became_active":"2024-04-06T03:26:18.617121+0000","last_became_peered":"2024-04-06T03:26:18.617121+0000","last_unstale":"2024-04-06T03:27:45.166349+0000","last_undegraded":"2024-04-06T03:27:45.166349+0000","last_fullsized":"2024-04-06T03:27:45.166349+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:51.780762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:51.780762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:07.013238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"85'11","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171002+0000","last_change":"2024-04-06T03:26:50.681662+0000","last_active":"2024-04-06T03:27:45.171002+0000","last_peered":"2024-04-06T03:27:45.171002+0000","last_clean":"2024-04-06T03:27:45.171002+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:27:45.171002+0000","last_undegraded":"2024-04-06T03:27:45.171002+0000","last_fullsized":"2024-04-06T03:27:45.171002+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:26:50.681513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:50.681513+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:02.463250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":71,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166360+0000","last_change":"2024-04-06T03:26:12.724981+0000","last_active":"2024-04-06T03:27:45.166360+0000","last_peered":"2024-04-06T03:27:45.166360+0000","last_clean":"2024-04-06T03:27:45.166360+0000","last_became_active":"2024-04-06T03:25:54.483894+0000","last_became_peered":"2024-04-06T03:25:54.483894+0000","last_unstale":"2024-04-06T03:27:45.166360+0000","last_undegraded":"2024-04-06T03:27:45.166360+0000","last_fullsized":"2024-04-06T03:27:45.166360+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:12.724893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:12.724893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703377+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":200,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171447+0000","last_change":"2024-04-06T03:26:51.667744+0000","last_active":"2024-04-06T03:27:45.171447+0000","last_peered":"2024-04-06T03:27:45.171447+0000","last_clean":"2024-04-06T03:27:45.171447+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:27:45.171447+0000","last_undegraded":"2024-04-06T03:27:45.171447+0000","last_fullsized":"2024-04-06T03:27:45.171447+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:51.667008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:26:51.667008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:07.644451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.162100+0000","last_change":"2024-04-06T03:25:11.208415+0000","last_active":"2024-04-06T03:27:45.162100+0000","last_peered":"2024-04-06T03:27:45.162100+0000","last_clean":"2024-04-06T03:27:45.162100+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:27:45.162100+0000","last_undegraded":"2024-04-06T03:27:45.162100+0000","last_fullsized":"2024-04-06T03:27:45.162100+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:25:11.208313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:25:11.208313+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"109'20","reported_seq":230,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161117+0000","last_change":"2024-04-06T03:26:17.739921+0000","last_active":"2024-04-06T03:27:45.161117+0000","last_peered":"2024-04-06T03:27:45.161117+0000","last_clean":"2024-04-06T03:27:45.161117+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:27:45.161117+0000","last_undegraded":"2024-04-06T03:27:45.161117+0000","last_fullsized":"2024-04-06T03:27:45.161117+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'20","last_scrub_stamp":"2024-04-06T03:26:17.739670+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:26:17.739670+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.779834+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"85'1","reported_seq":181,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.162319+0000","last_change":"2024-04-06T03:24:20.853706+0000","last_active":"2024-04-06T03:27:45.162319+0000","last_peered":"2024-04-06T03:27:45.162319+0000","last_clean":"2024-04-06T03:27:45.162319+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:27:45.162319+0000","last_undegraded":"2024-04-06T03:27:45.162319+0000","last_fullsized":"2024-04-06T03:27:45.162319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1","last_scrub_stamp":"2024-04-06T03:24:20.853465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:20.853465+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"139'13","reported_seq":164,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:46.531481+0000","last_change":"2024-04-06T03:26:53.820647+0000","last_active":"2024-04-06T03:27:46.531481+0000","last_peered":"2024-04-06T03:27:46.531481+0000","last_clean":"2024-04-06T03:27:46.531481+0000","last_became_active":"2024-04-06T03:26:18.615209+0000","last_became_peered":"2024-04-06T03:26:18.615209+0000","last_unstale":"2024-04-06T03:27:46.531481+0000","last_undegraded":"2024-04-06T03:27:46.531481+0000","last_fullsized":"2024-04-06T03:27:46.531481+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'13","last_scrub_stamp":"2024-04-06T03:26:53.820556+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:26:53.820556+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:09.750772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166635+0000","last_change":"2024-04-06T03:26:43.821216+0000","last_active":"2024-04-06T03:27:45.166635+0000","last_peered":"2024-04-06T03:27:45.166635+0000","last_clean":"2024-04-06T03:27:45.166635+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:27:45.166635+0000","last_undegraded":"2024-04-06T03:27:45.166635+0000","last_fullsized":"2024-04-06T03:27:45.166635+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:43.821096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:26:43.821096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"111'50","reported_seq":280,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170175+0000","last_change":"2024-04-06T03:26:27.269216+0000","last_active":"2024-04-06T03:27:45.170175+0000","last_peered":"2024-04-06T03:27:45.170175+0000","last_clean":"2024-04-06T03:27:45.170175+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:27:45.170175+0000","last_undegraded":"2024-04-06T03:27:45.170175+0000","last_fullsized":"2024-04-06T03:27:45.170175+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'50","last_scrub_stamp":"2024-04-06T03:26:27.269113+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:23:04.830548+0000","last_clean_scrub_stamp":"2024-04-06T03:26:27.269113+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.165437+0000","last_change":"2024-04-06T03:26:15.582707+0000","last_active":"2024-04-06T03:27:45.165437+0000","last_peered":"2024-04-06T03:27:45.165437+0000","last_clean":"2024-04-06T03:27:45.165437+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:27:45.165437+0000","last_undegraded":"2024-04-06T03:27:45.165437+0000","last_fullsized":"2024-04-06T03:27:45.165437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:15.582573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:26:15.582573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160717+0000","last_change":"2024-04-06T03:26:36.089681+0000","last_active":"2024-04-06T03:27:45.160717+0000","last_peered":"2024-04-06T03:27:45.160717+0000","last_clean":"2024-04-06T03:27:45.160717+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:27:45.160717+0000","last_undegraded":"2024-04-06T03:27:45.160717+0000","last_fullsized":"2024-04-06T03:27:45.160717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:36.089568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:53.517730+0000","last_clean_scrub_stamp":"2024-04-06T03:26:36.089568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:56.494004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":62,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.173553+0000","last_change":"2024-04-06T03:26:23.734415+0000","last_active":"2024-04-06T03:27:45.173553+0000","last_peered":"2024-04-06T03:27:45.173553+0000","last_clean":"2024-04-06T03:27:45.173553+0000","last_became_active":"2024-04-06T03:26:18.635349+0000","last_became_peered":"2024-04-06T03:26:18.635349+0000","last_unstale":"2024-04-06T03:27:45.173553+0000","last_undegraded":"2024-04-06T03:27:45.173553+0000","last_fullsized":"2024-04-06T03:27:45.173553+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:23.734265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:26:23.734265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.521794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"85'21","reported_seq":119,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.164302+0000","last_change":"2024-04-06T03:25:54.496821+0000","last_active":"2024-04-06T03:27:45.164302+0000","last_peered":"2024-04-06T03:27:45.164302+0000","last_clean":"2024-04-06T03:27:45.164302+0000","last_became_active":"2024-04-06T03:25:54.491804+0000","last_became_peered":"2024-04-06T03:25:54.491804+0000","last_unstale":"2024-04-06T03:27:45.164302+0000","last_undegraded":"2024-04-06T03:27:45.164302+0000","last_fullsized":"2024-04-06T03:27:45.164302+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:25:14.975603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:25:14.975603+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":187,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160246+0000","last_change":"2024-04-06T03:26:33.073600+0000","last_active":"2024-04-06T03:27:45.160246+0000","last_peered":"2024-04-06T03:27:45.160246+0000","last_clean":"2024-04-06T03:27:45.160246+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:27:45.160246+0000","last_undegraded":"2024-04-06T03:27:45.160246+0000","last_fullsized":"2024-04-06T03:27:45.160246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:33.073506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:29.395501+0000","last_clean_scrub_stamp":"2024-04-06T03:26:33.073506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:50.207767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":63,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167837+0000","last_change":"2024-04-06T03:26:49.774765+0000","last_active":"2024-04-06T03:27:45.167837+0000","last_peered":"2024-04-06T03:27:45.167837+0000","last_clean":"2024-04-06T03:27:45.167837+0000","last_became_active":"2024-04-06T03:26:18.618151+0000","last_became_peered":"2024-04-06T03:26:18.618151+0000","last_unstale":"2024-04-06T03:27:45.167837+0000","last_undegraded":"2024-04-06T03:27:45.167837+0000","last_fullsized":"2024-04-06T03:27:45.167837+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:49.774641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:49.774641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:03.364570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":65,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.173370+0000","last_change":"2024-04-06T03:26:19.521846+0000","last_active":"2024-04-06T03:27:45.173370+0000","last_peered":"2024-04-06T03:27:45.173370+0000","last_clean":"2024-04-06T03:27:45.173370+0000","last_became_active":"2024-04-06T03:26:18.637825+0000","last_became_peered":"2024-04-06T03:26:18.637825+0000","last_unstale":"2024-04-06T03:27:45.173370+0000","last_undegraded":"2024-04-06T03:27:45.173370+0000","last_fullsized":"2024-04-06T03:27:45.173370+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:19.521717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:19.521717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:45.529787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"121'16","reported_seq":228,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172102+0000","last_change":"2024-04-06T03:26:55.622903+0000","last_active":"2024-04-06T03:27:45.172102+0000","last_peered":"2024-04-06T03:27:45.172102+0000","last_clean":"2024-04-06T03:27:45.172102+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:27:45.172102+0000","last_undegraded":"2024-04-06T03:27:45.172102+0000","last_fullsized":"2024-04-06T03:27:45.172102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'16","last_scrub_stamp":"2024-04-06T03:26:55.622821+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:26:55.622821+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:03.992473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161994+0000","last_change":"2024-04-06T03:26:29.654310+0000","last_active":"2024-04-06T03:27:45.161994+0000","last_peered":"2024-04-06T03:27:45.161994+0000","last_clean":"2024-04-06T03:27:45.161994+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:27:45.161994+0000","last_undegraded":"2024-04-06T03:27:45.161994+0000","last_fullsized":"2024-04-06T03:27:45.161994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:29.654194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:29.654194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167827+0000","last_change":"2024-04-06T03:26:33.223021+0000","last_active":"2024-04-06T03:27:45.167827+0000","last_peered":"2024-04-06T03:27:45.167827+0000","last_clean":"2024-04-06T03:27:45.167827+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:27:45.167827+0000","last_undegraded":"2024-04-06T03:27:45.167827+0000","last_fullsized":"2024-04-06T03:27:45.167827+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:33.222911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:33.222911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:00.839201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171442+0000","last_change":"2024-04-06T03:26:32.242314+0000","last_active":"2024-04-06T03:27:45.171442+0000","last_peered":"2024-04-06T03:27:45.171442+0000","last_clean":"2024-04-06T03:27:45.171442+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:27:45.171442+0000","last_undegraded":"2024-04-06T03:27:45.171442+0000","last_fullsized":"2024-04-06T03:27:45.171442+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:32.242196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:12.824475+0000","last_clean_scrub_stamp":"2024-04-06T03:26:32.242196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"85'23","reported_seq":245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166492+0000","last_change":"2024-04-06T03:26:25.686241+0000","last_active":"2024-04-06T03:27:45.166492+0000","last_peered":"2024-04-06T03:27:45.166492+0000","last_clean":"2024-04-06T03:27:45.166492+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:27:45.166492+0000","last_undegraded":"2024-04-06T03:27:45.166492+0000","last_fullsized":"2024-04-06T03:27:45.166492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'23","last_scrub_stamp":"2024-04-06T03:26:25.686128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:25.686128+0000","objects_scrubbed":7,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":187,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168293+0000","last_change":"2024-04-06T03:26:14.453072+0000","last_active":"2024-04-06T03:27:45.168293+0000","last_peered":"2024-04-06T03:27:45.168293+0000","last_clean":"2024-04-06T03:27:45.168293+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:27:45.168293+0000","last_undegraded":"2024-04-06T03:27:45.168293+0000","last_fullsized":"2024-04-06T03:27:45.168293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:14.452981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:14.452981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846788+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.165155+0000","last_change":"2024-04-06T03:24:59.156076+0000","last_active":"2024-04-06T03:27:45.165155+0000","last_peered":"2024-04-06T03:27:45.165155+0000","last_clean":"2024-04-06T03:27:45.165155+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:27:45.165155+0000","last_undegraded":"2024-04-06T03:27:45.165155+0000","last_fullsized":"2024-04-06T03:27:45.165155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_clean_scrub_stamp":"2024-04-06T03:24:59.155815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":62,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163415+0000","last_change":"2024-04-06T03:26:24.685349+0000","last_active":"2024-04-06T03:27:45.163415+0000","last_peered":"2024-04-06T03:27:45.163415+0000","last_clean":"2024-04-06T03:27:45.163415+0000","last_became_active":"2024-04-06T03:26:18.643806+0000","last_became_peered":"2024-04-06T03:26:18.643806+0000","last_unstale":"2024-04-06T03:27:45.163415+0000","last_undegraded":"2024-04-06T03:27:45.163415+0000","last_fullsized":"2024-04-06T03:27:45.163415+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:24.684242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:26:24.684242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.611520+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"85'15","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161792+0000","last_change":"2024-04-06T03:27:04.102482+0000","last_active":"2024-04-06T03:27:45.161792+0000","last_peered":"2024-04-06T03:27:45.161792+0000","last_clean":"2024-04-06T03:27:45.161792+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:27:45.161792+0000","last_undegraded":"2024-04-06T03:27:45.161792+0000","last_fullsized":"2024-04-06T03:27:45.161792+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:27:04.102311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:27:04.102311+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:26.752130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167109+0000","last_change":"2024-04-06T03:26:26.680359+0000","last_active":"2024-04-06T03:27:45.167109+0000","last_peered":"2024-04-06T03:27:45.167109+0000","last_clean":"2024-04-06T03:27:45.167109+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:27:45.167109+0000","last_undegraded":"2024-04-06T03:27:45.167109+0000","last_fullsized":"2024-04-06T03:27:45.167109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:26.680017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:26.680017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:52.327169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":288,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163551+0000","last_change":"2024-04-06T03:26:41.790264+0000","last_active":"2024-04-06T03:27:45.163551+0000","last_peered":"2024-04-06T03:27:45.163551+0000","last_clean":"2024-04-06T03:27:45.163551+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:27:45.163551+0000","last_undegraded":"2024-04-06T03:27:45.163551+0000","last_fullsized":"2024-04-06T03:27:45.163551+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:26:41.790121+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:26:41.790121+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":199,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171208+0000","last_change":"2024-04-06T03:25:09.937411+0000","last_active":"2024-04-06T03:27:45.171208+0000","last_peered":"2024-04-06T03:27:45.171208+0000","last_clean":"2024-04-06T03:27:45.171208+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:27:45.171208+0000","last_undegraded":"2024-04-06T03:27:45.171208+0000","last_fullsized":"2024-04-06T03:27:45.171208+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:09.937198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:25:09.937198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.834852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":211,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170495+0000","last_change":"2024-04-06T03:25:17.979692+0000","last_active":"2024-04-06T03:27:45.170495+0000","last_peered":"2024-04-06T03:27:45.170495+0000","last_clean":"2024-04-06T03:27:45.170495+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:27:45.170495+0000","last_undegraded":"2024-04-06T03:27:45.170495+0000","last_fullsized":"2024-04-06T03:27:45.170495+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-04-06T03:25:17.979578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:25:17.979578+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.834912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171655+0000","last_change":"2024-04-06T03:26:42.330011+0000","last_active":"2024-04-06T03:27:45.171655+0000","last_peered":"2024-04-06T03:27:45.171655+0000","last_clean":"2024-04-06T03:27:45.171655+0000","last_became_active":"2024-04-06T03:20:57.843310+0000","last_became_peered":"2024-04-06T03:20:57.843310+0000","last_unstale":"2024-04-06T03:27:45.171655+0000","last_undegraded":"2024-04-06T03:27:45.171655+0000","last_fullsized":"2024-04-06T03:27:45.171655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:42.329898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:26:42.329898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:58.899928+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"154'13","reported_seq":59,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172269+0000","last_change":"2024-04-06T03:27:10.499161+0000","last_active":"2024-04-06T03:27:45.172269+0000","last_peered":"2024-04-06T03:27:45.172269+0000","last_clean":"2024-04-06T03:27:45.172269+0000","last_became_active":"2024-04-06T03:27:10.498254+0000","last_became_peered":"2024-04-06T03:27:10.498254+0000","last_unstale":"2024-04-06T03:27:45.172269+0000","last_undegraded":"2024-04-06T03:27:45.172269+0000","last_fullsized":"2024-04-06T03:27:45.172269+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:25:09.064599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:25:09.064599+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.834720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163910+0000","last_change":"2024-04-06T03:26:51.701904+0000","last_active":"2024-04-06T03:27:45.163910+0000","last_peered":"2024-04-06T03:27:45.163910+0000","last_clean":"2024-04-06T03:27:45.163910+0000","last_became_active":"2024-04-06T03:25:54.494855+0000","last_became_peered":"2024-04-06T03:25:54.494855+0000","last_unstale":"2024-04-06T03:27:45.163910+0000","last_undegraded":"2024-04-06T03:27:45.163910+0000","last_fullsized":"2024-04-06T03:27:45.163910+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:51.701787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:55.999187+0000","last_clean_scrub_stamp":"2024-04-06T03:26:51.701787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:58.697666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"85'11","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160716+0000","last_change":"2024-04-06T03:26:29.052079+0000","last_active":"2024-04-06T03:27:45.160716+0000","last_peered":"2024-04-06T03:27:45.160716+0000","last_clean":"2024-04-06T03:27:45.160716+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:27:45.160716+0000","last_undegraded":"2024-04-06T03:27:45.160716+0000","last_fullsized":"2024-04-06T03:27:45.160716+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:26:29.051916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:29.051916+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.672272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":187,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161147+0000","last_change":"2024-04-06T03:26:19.956659+0000","last_active":"2024-04-06T03:27:45.161147+0000","last_peered":"2024-04-06T03:27:45.161147+0000","last_clean":"2024-04-06T03:27:45.161147+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:27:45.161147+0000","last_undegraded":"2024-04-06T03:27:45.161147+0000","last_fullsized":"2024-04-06T03:27:45.161147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:19.955943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:19.955943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161578+0000","last_change":"2024-04-06T03:26:55.636515+0000","last_active":"2024-04-06T03:27:45.161578+0000","last_peered":"2024-04-06T03:27:45.161578+0000","last_clean":"2024-04-06T03:27:45.161578+0000","last_became_active":"2024-04-06T03:25:54.485733+0000","last_became_peered":"2024-04-06T03:25:54.485733+0000","last_unstale":"2024-04-06T03:27:45.161578+0000","last_undegraded":"2024-04-06T03:27:45.161578+0000","last_fullsized":"2024-04-06T03:27:45.161578+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:55.636340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:55.636340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:04.412485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":62,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168018+0000","last_change":"2024-04-06T03:26:36.814808+0000","last_active":"2024-04-06T03:27:45.168018+0000","last_peered":"2024-04-06T03:27:45.168018+0000","last_clean":"2024-04-06T03:27:45.168018+0000","last_became_active":"2024-04-06T03:26:18.615468+0000","last_became_peered":"2024-04-06T03:26:18.615468+0000","last_unstale":"2024-04-06T03:27:45.168018+0000","last_undegraded":"2024-04-06T03:27:45.168018+0000","last_fullsized":"2024-04-06T03:27:45.168018+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:36.814733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:36.814733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.922870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"111'6","reported_seq":143,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.169339+0000","last_change":"2024-04-06T03:26:18.640706+0000","last_active":"2024-04-06T03:27:45.169339+0000","last_peered":"2024-04-06T03:27:45.169339+0000","last_clean":"2024-04-06T03:27:45.169339+0000","last_became_active":"2024-04-06T03:26:18.639720+0000","last_became_peered":"2024-04-06T03:26:18.639720+0000","last_unstale":"2024-04-06T03:27:45.169339+0000","last_undegraded":"2024-04-06T03:27:45.169339+0000","last_fullsized":"2024-04-06T03:27:45.169339+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'6","last_scrub_stamp":"2024-04-06T03:26:16.011032+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:26:16.011032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":199,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172632+0000","last_change":"2024-04-06T03:26:10.482238+0000","last_active":"2024-04-06T03:27:45.172632+0000","last_peered":"2024-04-06T03:27:45.172632+0000","last_clean":"2024-04-06T03:27:45.172632+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:27:45.172632+0000","last_undegraded":"2024-04-06T03:27:45.172632+0000","last_fullsized":"2024-04-06T03:27:45.172632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:26:10.482095+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:26:10.482095+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.519267+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":193,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168108+0000","last_change":"2024-04-06T03:26:25.636780+0000","last_active":"2024-04-06T03:27:45.168108+0000","last_peered":"2024-04-06T03:27:45.168108+0000","last_clean":"2024-04-06T03:27:45.168108+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:27:45.168108+0000","last_undegraded":"2024-04-06T03:27:45.168108+0000","last_fullsized":"2024-04-06T03:27:45.168108+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:25.636666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:25.636666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.779774+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.175288+0000","last_change":"2024-04-06T03:26:15.588998+0000","last_active":"2024-04-06T03:27:45.175288+0000","last_peered":"2024-04-06T03:27:45.175288+0000","last_clean":"2024-04-06T03:27:45.175288+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:27:45.175288+0000","last_undegraded":"2024-04-06T03:27:45.175288+0000","last_fullsized":"2024-04-06T03:27:45.175288+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:15.587678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:15.587678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:53.732457+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"111'23","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160574+0000","last_change":"2024-04-06T03:24:40.808976+0000","last_active":"2024-04-06T03:27:45.160574+0000","last_peered":"2024-04-06T03:27:45.160574+0000","last_clean":"2024-04-06T03:27:45.160574+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:27:45.160574+0000","last_undegraded":"2024-04-06T03:27:45.160574+0000","last_fullsized":"2024-04-06T03:27:45.160574+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'23","last_scrub_stamp":"2024-04-06T03:24:40.808626+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:23:20.835514+0000","last_clean_scrub_stamp":"2024-04-06T03:24:40.808626+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.672051+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"160'8182","reported_seq":8259,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.476532+0000","last_change":"2024-04-06T03:25:20.582611+0000","last_active":"2024-04-06T03:27:49.476532+0000","last_peered":"2024-04-06T03:27:49.476532+0000","last_clean":"2024-04-06T03:27:49.476532+0000","last_became_active":"2024-04-06T03:25:20.576983+0000","last_became_peered":"2024-04-06T03:25:20.576983+0000","last_unstale":"2024-04-06T03:27:49.476532+0000","last_undegraded":"2024-04-06T03:27:49.476532+0000","last_fullsized":"2024-04-06T03:27:49.476532+0000","mapping_epoch":127,"log_start":"137'3000","ondisk_log_start":"137'3000","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5182,"log_dups_size":0,"ondisk_log_size":5182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.847194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245366784,"num_objects":3744,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8182,"num_write_kb":381632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.165236+0000","last_change":"2024-04-06T03:27:01.917940+0000","last_active":"2024-04-06T03:27:45.165236+0000","last_peered":"2024-04-06T03:27:45.165236+0000","last_clean":"2024-04-06T03:27:45.165236+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:27:45.165236+0000","last_undegraded":"2024-04-06T03:27:45.165236+0000","last_fullsized":"2024-04-06T03:27:45.165236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:27:01.917814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:27:01.917814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:02.525056+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":226,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166861+0000","last_change":"2024-04-06T03:26:17.727053+0000","last_active":"2024-04-06T03:27:45.166861+0000","last_peered":"2024-04-06T03:27:45.166861+0000","last_clean":"2024-04-06T03:27:45.166861+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:27:45.166861+0000","last_undegraded":"2024-04-06T03:27:45.166861+0000","last_fullsized":"2024-04-06T03:27:45.166861+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-04-06T03:26:17.726762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:17.726762+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703291+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"85'17","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.164223+0000","last_change":"2024-04-06T03:24:48.868773+0000","last_active":"2024-04-06T03:27:45.164223+0000","last_peered":"2024-04-06T03:27:45.164223+0000","last_clean":"2024-04-06T03:27:45.164223+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:27:45.164223+0000","last_undegraded":"2024-04-06T03:27:45.164223+0000","last_fullsized":"2024-04-06T03:27:45.164223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:24:48.868547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:48.868547+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.175175+0000","last_change":"2024-04-06T03:26:33.742325+0000","last_active":"2024-04-06T03:27:45.175175+0000","last_peered":"2024-04-06T03:27:45.175175+0000","last_clean":"2024-04-06T03:27:45.175175+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:27:45.175175+0000","last_undegraded":"2024-04-06T03:27:45.175175+0000","last_fullsized":"2024-04-06T03:27:45.175175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:33.742207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:33.742207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.381922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"160'8614","reported_seq":8691,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:53.964010+0000","last_change":"2024-04-06T03:25:20.562684+0000","last_active":"2024-04-06T03:27:53.964010+0000","last_peered":"2024-04-06T03:27:53.964010+0000","last_clean":"2024-04-06T03:27:53.964010+0000","last_became_active":"2024-04-06T03:25:20.562369+0000","last_became_peered":"2024-04-06T03:25:20.562369+0000","last_unstale":"2024-04-06T03:27:53.964010+0000","last_undegraded":"2024-04-06T03:27:53.964010+0000","last_fullsized":"2024-04-06T03:27:53.964010+0000","mapping_epoch":127,"log_start":"139'3600","ondisk_log_start":"139'3600","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230555648,"num_objects":3518,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8614,"num_write_kb":388224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":188,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168358+0000","last_change":"2024-04-06T03:26:18.430146+0000","last_active":"2024-04-06T03:27:45.168358+0000","last_peered":"2024-04-06T03:27:45.168358+0000","last_clean":"2024-04-06T03:27:45.168358+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:27:45.168358+0000","last_undegraded":"2024-04-06T03:27:45.168358+0000","last_fullsized":"2024-04-06T03:27:45.168358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:18.429785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:26:18.429785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":148,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170724+0000","last_change":"2024-04-06T03:27:10.500969+0000","last_active":"2024-04-06T03:27:45.170724+0000","last_peered":"2024-04-06T03:27:45.170724+0000","last_clean":"2024-04-06T03:27:45.170724+0000","last_became_active":"2024-04-06T03:27:10.500422+0000","last_became_peered":"2024-04-06T03:27:10.500422+0000","last_unstale":"2024-04-06T03:27:45.170724+0000","last_undegraded":"2024-04-06T03:27:45.170724+0000","last_fullsized":"2024-04-06T03:27:45.170724+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:25:04.056574+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:25:04.056574+0000","last_clean_scrub_stamp":"2024-04-06T03:25:04.056574+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.834556+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"136'17","reported_seq":134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161424+0000","last_change":"2024-04-06T03:26:08.616391+0000","last_active":"2024-04-06T03:27:45.161424+0000","last_peered":"2024-04-06T03:27:45.161424+0000","last_clean":"2024-04-06T03:27:45.161424+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:27:45.161424+0000","last_undegraded":"2024-04-06T03:27:45.161424+0000","last_fullsized":"2024-04-06T03:27:45.161424+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'17","last_scrub_stamp":"2024-04-06T03:26:08.616289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:08.616289+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.169580+0000","last_change":"2024-04-06T03:25:18.059281+0000","last_active":"2024-04-06T03:27:45.169580+0000","last_peered":"2024-04-06T03:27:45.169580+0000","last_clean":"2024-04-06T03:27:45.169580+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:27:45.169580+0000","last_undegraded":"2024-04-06T03:27:45.169580+0000","last_fullsized":"2024-04-06T03:27:45.169580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:25:18.059061+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:25:18.059061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":286,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170946+0000","last_change":"2024-04-06T03:26:17.460284+0000","last_active":"2024-04-06T03:27:45.170946+0000","last_peered":"2024-04-06T03:27:45.170946+0000","last_clean":"2024-04-06T03:27:45.170946+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:27:45.170946+0000","last_undegraded":"2024-04-06T03:27:45.170946+0000","last_fullsized":"2024-04-06T03:27:45.170946+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:26:17.460147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:17.460147+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846806+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"160'8328","reported_seq":8410,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.471723+0000","last_change":"2024-04-06T03:26:18.649948+0000","last_active":"2024-04-06T03:27:49.471723+0000","last_peered":"2024-04-06T03:27:49.471723+0000","last_clean":"2024-04-06T03:27:49.471723+0000","last_became_active":"2024-04-06T03:26:18.638004+0000","last_became_peered":"2024-04-06T03:26:18.638004+0000","last_unstale":"2024-04-06T03:27:49.471723+0000","last_undegraded":"2024-04-06T03:27:49.471723+0000","last_fullsized":"2024-04-06T03:27:49.471723+0000","mapping_epoch":138,"log_start":"137'3100","ondisk_log_start":"137'3100","created":127,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":385984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":190,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161714+0000","last_change":"2024-04-06T03:25:14.197862+0000","last_active":"2024-04-06T03:27:45.161714+0000","last_peered":"2024-04-06T03:27:45.161714+0000","last_clean":"2024-04-06T03:27:45.161714+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:27:45.161714+0000","last_undegraded":"2024-04-06T03:27:45.161714+0000","last_fullsized":"2024-04-06T03:27:45.161714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:14.197743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:25:14.197743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167466+0000","last_change":"2024-04-06T03:26:28.226435+0000","last_active":"2024-04-06T03:27:45.167466+0000","last_peered":"2024-04-06T03:27:45.167466+0000","last_clean":"2024-04-06T03:27:45.167466+0000","last_became_active":"2024-04-06T03:25:54.484235+0000","last_became_peered":"2024-04-06T03:25:54.484235+0000","last_unstale":"2024-04-06T03:27:45.167466+0000","last_undegraded":"2024-04-06T03:27:45.167466+0000","last_fullsized":"2024-04-06T03:27:45.167466+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:28.226189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:28.226189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"106'34","reported_seq":255,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166466+0000","last_change":"2024-04-06T03:26:13.731426+0000","last_active":"2024-04-06T03:27:45.166466+0000","last_peered":"2024-04-06T03:27:45.166466+0000","last_clean":"2024-04-06T03:27:45.166466+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:27:45.166466+0000","last_undegraded":"2024-04-06T03:27:45.166466+0000","last_fullsized":"2024-04-06T03:27:45.166466+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'34","last_scrub_stamp":"2024-04-06T03:26:13.731053+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:22:31.389226+0000","last_clean_scrub_stamp":"2024-04-06T03:26:13.731053+0000","objects_scrubbed":9,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703337+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":192,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171917+0000","last_change":"2024-04-06T03:26:15.497218+0000","last_active":"2024-04-06T03:27:45.171917+0000","last_peered":"2024-04-06T03:27:45.171917+0000","last_clean":"2024-04-06T03:27:45.171917+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:27:45.171917+0000","last_undegraded":"2024-04-06T03:27:45.171917+0000","last_fullsized":"2024-04-06T03:27:45.171917+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:15.497102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:15.497102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846724+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"160'8253","reported_seq":8329,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.504544+0000","last_change":"2024-04-06T03:25:20.581566+0000","last_active":"2024-04-06T03:27:49.504544+0000","last_peered":"2024-04-06T03:27:49.504544+0000","last_clean":"2024-04-06T03:27:49.504544+0000","last_became_active":"2024-04-06T03:25:20.580760+0000","last_became_peered":"2024-04-06T03:25:20.580760+0000","last_unstale":"2024-04-06T03:27:49.504544+0000","last_undegraded":"2024-04-06T03:27:49.504544+0000","last_fullsized":"2024-04-06T03:27:49.504544+0000","mapping_epoch":127,"log_start":"137'3000","ondisk_log_start":"137'3000","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248446976,"num_objects":3791,"num_object_clones":0,"num_object_copies":11373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8253,"num_write_kb":385408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":188,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161492+0000","last_change":"2024-04-06T03:26:31.095985+0000","last_active":"2024-04-06T03:27:45.161492+0000","last_peered":"2024-04-06T03:27:45.161492+0000","last_clean":"2024-04-06T03:27:45.161492+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:27:45.161492+0000","last_undegraded":"2024-04-06T03:27:45.161492+0000","last_fullsized":"2024-04-06T03:27:45.161492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:31.095732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:26:31.095732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'30","reported_seq":99,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.175335+0000","last_change":"2024-04-06T03:27:10.496286+0000","last_active":"2024-04-06T03:27:45.175335+0000","last_peered":"2024-04-06T03:27:45.175335+0000","last_clean":"2024-04-06T03:27:45.175335+0000","last_became_active":"2024-04-06T03:27:10.495663+0000","last_became_peered":"2024-04-06T03:27:10.495663+0000","last_unstale":"2024-04-06T03:27:45.175335+0000","last_undegraded":"2024-04-06T03:27:45.175335+0000","last_fullsized":"2024-04-06T03:27:45.175335+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'30","last_scrub_stamp":"2024-04-06T03:25:00.029144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:25:00.029144+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.519101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":37,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172109+0000","last_change":"2024-04-06T03:26:49.216077+0000","last_active":"2024-04-06T03:27:45.172109+0000","last_peered":"2024-04-06T03:27:45.172109+0000","last_clean":"2024-04-06T03:27:45.172109+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:27:45.172109+0000","last_undegraded":"2024-04-06T03:27:45.172109+0000","last_fullsized":"2024-04-06T03:27:45.172109+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:49.215712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:49.215712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:18.249779+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.173461+0000","last_change":"2024-04-06T03:26:29.727495+0000","last_active":"2024-04-06T03:27:45.173461+0000","last_peered":"2024-04-06T03:27:45.173461+0000","last_clean":"2024-04-06T03:27:45.173461+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:27:45.173461+0000","last_undegraded":"2024-04-06T03:27:45.173461+0000","last_fullsized":"2024-04-06T03:27:45.173461+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:29.727410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:29.727410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"160'8337","reported_seq":8414,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.495515+0000","last_change":"2024-04-06T03:25:20.588047+0000","last_active":"2024-04-06T03:27:49.495515+0000","last_peered":"2024-04-06T03:27:49.495515+0000","last_clean":"2024-04-06T03:27:49.495515+0000","last_became_active":"2024-04-06T03:25:20.587414+0000","last_became_peered":"2024-04-06T03:25:20.587414+0000","last_unstale":"2024-04-06T03:27:49.495515+0000","last_undegraded":"2024-04-06T03:27:49.495515+0000","last_fullsized":"2024-04-06T03:27:49.495515+0000","mapping_epoch":127,"log_start":"137'3100","ondisk_log_start":"137'3100","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251461632,"num_objects":3837,"num_object_clones":0,"num_object_copies":11511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8337,"num_write_kb":389568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":61,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168129+0000","last_change":"2024-04-06T03:25:54.488014+0000","last_active":"2024-04-06T03:27:45.168129+0000","last_peered":"2024-04-06T03:27:45.168129+0000","last_clean":"2024-04-06T03:27:45.168129+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:27:45.168129+0000","last_undegraded":"2024-04-06T03:27:45.168129+0000","last_fullsized":"2024-04-06T03:27:45.168129+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:05.061461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:25:05.061461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'16","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163748+0000","last_change":"2024-04-06T03:26:56.656837+0000","last_active":"2024-04-06T03:27:45.163748+0000","last_peered":"2024-04-06T03:27:45.163748+0000","last_clean":"2024-04-06T03:27:45.163748+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:27:45.163748+0000","last_undegraded":"2024-04-06T03:27:45.163748+0000","last_fullsized":"2024-04-06T03:27:45.163748+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'16","last_scrub_stamp":"2024-04-06T03:26:56.656563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:56.656563+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:13.115493+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172119+0000","last_change":"2024-04-06T03:26:14.536308+0000","last_active":"2024-04-06T03:27:45.172119+0000","last_peered":"2024-04-06T03:27:45.172119+0000","last_clean":"2024-04-06T03:27:45.172119+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:27:45.172119+0000","last_undegraded":"2024-04-06T03:27:45.172119+0000","last_fullsized":"2024-04-06T03:27:45.172119+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:14.536181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:14.536181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.519200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":451,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160403+0000","last_change":"2024-04-06T03:24:18.862939+0000","last_active":"2024-04-06T03:27:45.160403+0000","last_peered":"2024-04-06T03:27:45.160403+0000","last_clean":"2024-04-06T03:27:45.160403+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:27:45.160403+0000","last_undegraded":"2024-04-06T03:27:45.160403+0000","last_fullsized":"2024-04-06T03:27:45.160403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:24:18.862554+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:24:18.862554+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":184,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.162228+0000","last_change":"2024-04-06T03:24:14.817313+0000","last_active":"2024-04-06T03:27:45.162228+0000","last_peered":"2024-04-06T03:27:45.162228+0000","last_clean":"2024-04-06T03:27:45.162228+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:27:45.162228+0000","last_undegraded":"2024-04-06T03:27:45.162228+0000","last_fullsized":"2024-04-06T03:27:45.162228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:24:14.817089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"160'8679","reported_seq":8756,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:53.950118+0000","last_change":"2024-04-06T03:25:20.570118+0000","last_active":"2024-04-06T03:27:53.950118+0000","last_peered":"2024-04-06T03:27:53.950118+0000","last_clean":"2024-04-06T03:27:53.950118+0000","last_became_active":"2024-04-06T03:25:20.569489+0000","last_became_peered":"2024-04-06T03:25:20.569489+0000","last_unstale":"2024-04-06T03:27:53.950118+0000","last_undegraded":"2024-04-06T03:27:53.950118+0000","last_fullsized":"2024-04-06T03:27:53.950118+0000","mapping_epoch":127,"log_start":"139'3900","ondisk_log_start":"139'3900","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":4779,"log_dups_size":0,"ondisk_log_size":4779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.672156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233242624,"num_objects":3559,"num_object_clones":0,"num_object_copies":10677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8679,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167282+0000","last_change":"2024-04-06T03:26:15.338986+0000","last_active":"2024-04-06T03:27:45.167282+0000","last_peered":"2024-04-06T03:27:45.167282+0000","last_clean":"2024-04-06T03:27:45.167282+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:27:45.167282+0000","last_undegraded":"2024-04-06T03:27:45.167282+0000","last_fullsized":"2024-04-06T03:27:45.167282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:15.338748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:15.338748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"85'21","reported_seq":238,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.169887+0000","last_change":"2024-04-06T03:26:52.652087+0000","last_active":"2024-04-06T03:27:45.169887+0000","last_peered":"2024-04-06T03:27:45.169887+0000","last_clean":"2024-04-06T03:27:45.169887+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:27:45.169887+0000","last_undegraded":"2024-04-06T03:27:45.169887+0000","last_fullsized":"2024-04-06T03:27:45.169887+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'21","last_scrub_stamp":"2024-04-06T03:26:52.651897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:52.651897+0000","objects_scrubbed":8,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:08.080725+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":192,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160296+0000","last_change":"2024-04-06T03:24:36.795218+0000","last_active":"2024-04-06T03:27:45.160296+0000","last_peered":"2024-04-06T03:27:45.160296+0000","last_clean":"2024-04-06T03:27:45.160296+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:27:45.160296+0000","last_undegraded":"2024-04-06T03:27:45.160296+0000","last_fullsized":"2024-04-06T03:27:45.160296+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:36.794985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:36.794985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167763+0000","last_change":"2024-04-06T03:26:56.220579+0000","last_active":"2024-04-06T03:27:45.167763+0000","last_peered":"2024-04-06T03:27:45.167763+0000","last_clean":"2024-04-06T03:27:45.167763+0000","last_became_active":"2024-04-06T03:25:54.492757+0000","last_became_peered":"2024-04-06T03:25:54.492757+0000","last_unstale":"2024-04-06T03:27:45.167763+0000","last_undegraded":"2024-04-06T03:27:45.167763+0000","last_fullsized":"2024-04-06T03:27:45.167763+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:56.220458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:56.220458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:07.716933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"160'8162","reported_seq":8244,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.496193+0000","last_change":"2024-04-06T03:25:54.487149+0000","last_active":"2024-04-06T03:27:49.496193+0000","last_peered":"2024-04-06T03:27:49.496193+0000","last_clean":"2024-04-06T03:27:49.496193+0000","last_became_active":"2024-04-06T03:25:54.485255+0000","last_became_peered":"2024-04-06T03:25:54.485255+0000","last_unstale":"2024-04-06T03:27:49.496193+0000","last_undegraded":"2024-04-06T03:27:49.496193+0000","last_fullsized":"2024-04-06T03:27:49.496193+0000","mapping_epoch":135,"log_start":"137'3000","ondisk_log_start":"137'3000","created":127,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.846982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248381440,"num_objects":3790,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8162,"num_write_kb":382464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":189,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170409+0000","last_change":"2024-04-06T03:25:05.071816+0000","last_active":"2024-04-06T03:27:45.170409+0000","last_peered":"2024-04-06T03:27:45.170409+0000","last_clean":"2024-04-06T03:27:45.170409+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:27:45.170409+0000","last_undegraded":"2024-04-06T03:27:45.170409+0000","last_fullsized":"2024-04-06T03:27:45.170409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:05.071503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:25:05.071503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":200,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161952+0000","last_change":"2024-04-06T03:26:21.980424+0000","last_active":"2024-04-06T03:27:45.161952+0000","last_peered":"2024-04-06T03:27:45.161952+0000","last_clean":"2024-04-06T03:27:45.161952+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:27:45.161952+0000","last_undegraded":"2024-04-06T03:27:45.161952+0000","last_fullsized":"2024-04-06T03:27:45.161952+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:21.980348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:06.712077+0000","last_clean_scrub_stamp":"2024-04-06T03:26:21.980348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"128'38","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:46.538945+0000","last_change":"2024-04-06T03:26:24.776706+0000","last_active":"2024-04-06T03:27:46.538945+0000","last_peered":"2024-04-06T03:27:46.538945+0000","last_clean":"2024-04-06T03:27:46.538945+0000","last_became_active":"2024-04-06T03:25:19.551712+0000","last_became_peered":"2024-04-06T03:25:19.551712+0000","last_unstale":"2024-04-06T03:27:46.538945+0000","last_undegraded":"2024-04-06T03:27:46.538945+0000","last_fullsized":"2024-04-06T03:27:46.538945+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'38","last_scrub_stamp":"2024-04-06T03:26:24.776532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:24.776532+0000","objects_scrubbed":11,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.595340+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":63,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170446+0000","last_change":"2024-04-06T03:27:01.575604+0000","last_active":"2024-04-06T03:27:45.170446+0000","last_peered":"2024-04-06T03:27:45.170446+0000","last_clean":"2024-04-06T03:27:45.170446+0000","last_became_active":"2024-04-06T03:26:18.643845+0000","last_became_peered":"2024-04-06T03:26:18.643845+0000","last_unstale":"2024-04-06T03:27:45.170446+0000","last_undegraded":"2024-04-06T03:27:45.170446+0000","last_fullsized":"2024-04-06T03:27:45.170446+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:27:01.575499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:27:01.575499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:31.198818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"160'8440","reported_seq":8514,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.495629+0000","last_change":"2024-04-06T03:25:20.575042+0000","last_active":"2024-04-06T03:27:49.495629+0000","last_peered":"2024-04-06T03:27:49.495629+0000","last_clean":"2024-04-06T03:27:49.495629+0000","last_became_active":"2024-04-06T03:25:20.574121+0000","last_became_peered":"2024-04-06T03:25:20.574121+0000","last_unstale":"2024-04-06T03:27:49.495629+0000","last_undegraded":"2024-04-06T03:27:49.495629+0000","last_fullsized":"2024-04-06T03:27:49.495629+0000","mapping_epoch":127,"log_start":"137'3300","ondisk_log_start":"137'3300","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5140,"log_dups_size":0,"ondisk_log_size":5140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.847062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245760000,"num_objects":3750,"num_object_clones":0,"num_object_copies":11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":390080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166232+0000","last_change":"2024-04-06T03:26:59.829359+0000","last_active":"2024-04-06T03:27:45.166232+0000","last_peered":"2024-04-06T03:27:45.166232+0000","last_clean":"2024-04-06T03:27:45.166232+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:27:45.166232+0000","last_undegraded":"2024-04-06T03:27:45.166232+0000","last_fullsized":"2024-04-06T03:27:45.166232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:59.829275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:26:59.829275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:27.872161+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167663+0000","last_change":"2024-04-06T03:24:51.044590+0000","last_active":"2024-04-06T03:27:45.167663+0000","last_peered":"2024-04-06T03:27:45.167663+0000","last_clean":"2024-04-06T03:27:45.167663+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:27:45.167663+0000","last_undegraded":"2024-04-06T03:27:45.167663+0000","last_fullsized":"2024-04-06T03:27:45.167663+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:24:51.044301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:51.044301+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"85'19","reported_seq":237,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.162927+0000","last_change":"2024-04-06T03:26:18.602594+0000","last_active":"2024-04-06T03:27:45.162927+0000","last_peered":"2024-04-06T03:27:45.162927+0000","last_clean":"2024-04-06T03:27:45.162927+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:27:45.162927+0000","last_undegraded":"2024-04-06T03:27:45.162927+0000","last_fullsized":"2024-04-06T03:27:45.162927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'19","last_scrub_stamp":"2024-04-06T03:26:18.602438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:18.602438+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":609,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:47.538714+0000","last_change":"2024-04-06T03:26:36.696391+0000","last_active":"2024-04-06T03:27:47.538714+0000","last_peered":"2024-04-06T03:27:47.538714+0000","last_clean":"2024-04-06T03:27:47.538714+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:27:47.538714+0000","last_undegraded":"2024-04-06T03:27:47.538714+0000","last_fullsized":"2024-04-06T03:27:47.538714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:26:36.696252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:26:36.696252+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:50.065543+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":395,"num_read_kb":390,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167886+0000","last_change":"2024-04-06T03:25:17.737051+0000","last_active":"2024-04-06T03:27:45.167886+0000","last_peered":"2024-04-06T03:27:45.167886+0000","last_clean":"2024-04-06T03:27:45.167886+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:27:45.167886+0000","last_undegraded":"2024-04-06T03:27:45.167886+0000","last_fullsized":"2024-04-06T03:27:45.167886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:17.736852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:25:17.736852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"160'8297","reported_seq":8374,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.451056+0000","last_change":"2024-04-06T03:25:20.575881+0000","last_active":"2024-04-06T03:27:49.451056+0000","last_peered":"2024-04-06T03:27:49.451056+0000","last_clean":"2024-04-06T03:27:49.451056+0000","last_became_active":"2024-04-06T03:25:20.574748+0000","last_became_peered":"2024-04-06T03:25:20.574748+0000","last_unstale":"2024-04-06T03:27:49.451056+0000","last_undegraded":"2024-04-06T03:27:49.451056+0000","last_fullsized":"2024-04-06T03:27:49.451056+0000","mapping_epoch":127,"log_start":"137'3200","ondisk_log_start":"137'3200","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.847306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244908032,"num_objects":3737,"num_object_clones":0,"num_object_copies":11211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8297,"num_write_kb":385088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":188,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160449+0000","last_change":"2024-04-06T03:27:00.077042+0000","last_active":"2024-04-06T03:27:45.160449+0000","last_peered":"2024-04-06T03:27:45.160449+0000","last_clean":"2024-04-06T03:27:45.160449+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:27:45.160449+0000","last_undegraded":"2024-04-06T03:27:45.160449+0000","last_fullsized":"2024-04-06T03:27:45.160449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:27:00.076962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:27:00.076962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:02.512029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161864+0000","last_change":"2024-04-06T03:26:18.622133+0000","last_active":"2024-04-06T03:27:45.161864+0000","last_peered":"2024-04-06T03:27:45.161864+0000","last_clean":"2024-04-06T03:27:45.161864+0000","last_became_active":"2024-04-06T03:26:18.621727+0000","last_became_peered":"2024-04-06T03:26:18.621727+0000","last_unstale":"2024-04-06T03:27:45.161864+0000","last_undegraded":"2024-04-06T03:27:45.161864+0000","last_fullsized":"2024-04-06T03:27:45.161864+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:34.893734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:24:34.893734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"85'12","reported_seq":222,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171188+0000","last_change":"2024-04-06T03:26:44.345844+0000","last_active":"2024-04-06T03:27:45.171188+0000","last_peered":"2024-04-06T03:27:45.171188+0000","last_clean":"2024-04-06T03:27:45.171188+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:27:45.171188+0000","last_undegraded":"2024-04-06T03:27:45.171188+0000","last_fullsized":"2024-04-06T03:27:45.171188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12","last_scrub_stamp":"2024-04-06T03:26:44.345749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:44.345749+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:11.958446+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":203,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167947+0000","last_change":"2024-04-06T03:25:07.824846+0000","last_active":"2024-04-06T03:27:45.167947+0000","last_peered":"2024-04-06T03:27:45.167947+0000","last_clean":"2024-04-06T03:27:45.167947+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:27:45.167947+0000","last_undegraded":"2024-04-06T03:27:45.167947+0000","last_fullsized":"2024-04-06T03:27:45.167947+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:07.824556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:07.824556+0000","last_clean_scrub_stamp":"2024-04-06T03:25:07.824556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":289,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161571+0000","last_change":"2024-04-06T03:26:21.635250+0000","last_active":"2024-04-06T03:27:45.161571+0000","last_peered":"2024-04-06T03:27:45.161571+0000","last_clean":"2024-04-06T03:27:45.161571+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:27:45.161571+0000","last_undegraded":"2024-04-06T03:27:45.161571+0000","last_fullsized":"2024-04-06T03:27:45.161571+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:26:21.633973+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:26:21.633973+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"160'8333","reported_seq":8416,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.483763+0000","last_change":"2024-04-06T03:26:18.651403+0000","last_active":"2024-04-06T03:27:49.483763+0000","last_peered":"2024-04-06T03:27:49.483763+0000","last_clean":"2024-04-06T03:27:49.483763+0000","last_became_active":"2024-04-06T03:26:18.638299+0000","last_became_peered":"2024-04-06T03:26:18.638299+0000","last_unstale":"2024-04-06T03:27:49.483763+0000","last_undegraded":"2024-04-06T03:27:49.483763+0000","last_fullsized":"2024-04-06T03:27:49.483763+0000","mapping_epoch":138,"log_start":"137'3100","ondisk_log_start":"137'3100","created":127,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247726104,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8333,"num_write_kb":387585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":190,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163602+0000","last_change":"2024-04-06T03:24:54.158186+0000","last_active":"2024-04-06T03:27:45.163602+0000","last_peered":"2024-04-06T03:27:45.163602+0000","last_clean":"2024-04-06T03:27:45.163602+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:27:45.163602+0000","last_undegraded":"2024-04-06T03:27:45.163602+0000","last_fullsized":"2024-04-06T03:27:45.163602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:54.157976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:24:54.157976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167068+0000","last_change":"2024-04-06T03:26:32.706453+0000","last_active":"2024-04-06T03:27:45.167068+0000","last_peered":"2024-04-06T03:27:45.167068+0000","last_clean":"2024-04-06T03:27:45.167068+0000","last_became_active":"2024-04-06T03:25:54.491253+0000","last_became_peered":"2024-04-06T03:25:54.491253+0000","last_unstale":"2024-04-06T03:27:45.167068+0000","last_undegraded":"2024-04-06T03:27:45.167068+0000","last_fullsized":"2024-04-06T03:27:45.167068+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:32.706350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:32.706350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:02.041639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":192,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160516+0000","last_change":"2024-04-06T03:24:24.887165+0000","last_active":"2024-04-06T03:27:45.160516+0000","last_peered":"2024-04-06T03:27:45.160516+0000","last_clean":"2024-04-06T03:27:45.160516+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:27:45.160516+0000","last_undegraded":"2024-04-06T03:27:45.160516+0000","last_fullsized":"2024-04-06T03:27:45.160516+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:24.886928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:24.886928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"122'31","reported_seq":252,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.167908+0000","last_change":"2024-04-06T03:25:06.862557+0000","last_active":"2024-04-06T03:27:45.167908+0000","last_peered":"2024-04-06T03:27:45.167908+0000","last_clean":"2024-04-06T03:27:45.167908+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:27:45.167908+0000","last_undegraded":"2024-04-06T03:27:45.167908+0000","last_fullsized":"2024-04-06T03:27:45.167908+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'31","last_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:25:06.862413+0000","objects_scrubbed":7,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"160'8323","reported_seq":8404,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.502736+0000","last_change":"2024-04-06T03:26:18.647554+0000","last_active":"2024-04-06T03:27:49.502736+0000","last_peered":"2024-04-06T03:27:49.502736+0000","last_clean":"2024-04-06T03:27:49.502736+0000","last_became_active":"2024-04-06T03:26:18.625461+0000","last_became_peered":"2024-04-06T03:26:18.625461+0000","last_unstale":"2024-04-06T03:27:49.502736+0000","last_undegraded":"2024-04-06T03:27:49.502736+0000","last_fullsized":"2024-04-06T03:27:49.502736+0000","mapping_epoch":138,"log_start":"137'3200","ondisk_log_start":"137'3200","created":127,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5123,"log_dups_size":0,"ondisk_log_size":5123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.847171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248184832,"num_objects":3787,"num_object_clones":0,"num_object_copies":11361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":387520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":28,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171783+0000","last_change":"2024-04-06T03:27:10.498920+0000","last_active":"2024-04-06T03:27:45.171783+0000","last_peered":"2024-04-06T03:27:45.171783+0000","last_clean":"2024-04-06T03:27:45.171783+0000","last_became_active":"2024-04-06T03:27:10.497525+0000","last_became_peered":"2024-04-06T03:27:10.497525+0000","last_unstale":"2024-04-06T03:27:45.171783+0000","last_undegraded":"2024-04-06T03:27:45.171783+0000","last_fullsized":"2024-04-06T03:27:45.171783+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:50.150657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:26:50.150657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:14.519362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"85'11","reported_seq":297,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:46.531462+0000","last_change":"2024-04-06T03:26:29.662202+0000","last_active":"2024-04-06T03:27:46.531462+0000","last_peered":"2024-04-06T03:27:46.531462+0000","last_clean":"2024-04-06T03:27:46.531462+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:27:46.531462+0000","last_undegraded":"2024-04-06T03:27:46.531462+0000","last_fullsized":"2024-04-06T03:27:46.531462+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:26:29.662117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:29.662117+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:53.704775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":201,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170276+0000","last_change":"2024-04-06T03:25:08.176261+0000","last_active":"2024-04-06T03:27:45.170276+0000","last_peered":"2024-04-06T03:27:45.170276+0000","last_clean":"2024-04-06T03:27:45.170276+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:27:45.170276+0000","last_undegraded":"2024-04-06T03:27:45.170276+0000","last_fullsized":"2024-04-06T03:27:45.170276+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:08.175998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:25:08.175998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350341+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"85'27","reported_seq":247,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161327+0000","last_change":"2024-04-06T03:25:00.152299+0000","last_active":"2024-04-06T03:27:45.161327+0000","last_peered":"2024-04-06T03:27:45.161327+0000","last_clean":"2024-04-06T03:27:45.161327+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:27:45.161327+0000","last_undegraded":"2024-04-06T03:27:45.161327+0000","last_fullsized":"2024-04-06T03:27:45.161327+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'27","last_scrub_stamp":"2024-04-06T03:25:00.152150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:25:00.152150+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"160'8258","reported_seq":8352,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.500627+0000","last_change":"2024-04-06T03:27:04.528595+0000","last_active":"2024-04-06T03:27:49.500627+0000","last_peered":"2024-04-06T03:27:49.500627+0000","last_clean":"2024-04-06T03:27:49.500627+0000","last_became_active":"2024-04-06T03:25:20.589769+0000","last_became_peered":"2024-04-06T03:25:20.589769+0000","last_unstale":"2024-04-06T03:27:49.500627+0000","last_undegraded":"2024-04-06T03:27:49.500627+0000","last_fullsized":"2024-04-06T03:27:49.500627+0000","mapping_epoch":127,"log_start":"137'3000","ondisk_log_start":"137'3000","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":1349,"log_size":5258,"log_dups_size":0,"ondisk_log_size":5258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245891072,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":384320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":190,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166913+0000","last_change":"2024-04-06T03:25:15.710998+0000","last_active":"2024-04-06T03:27:45.166913+0000","last_peered":"2024-04-06T03:27:45.166913+0000","last_clean":"2024-04-06T03:27:45.166913+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:27:45.166913+0000","last_undegraded":"2024-04-06T03:27:45.166913+0000","last_fullsized":"2024-04-06T03:27:45.166913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:15.710852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:25:15.710852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:53.704536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":298,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:46.539192+0000","last_change":"2024-04-06T03:26:20.562822+0000","last_active":"2024-04-06T03:27:46.539192+0000","last_peered":"2024-04-06T03:27:46.539192+0000","last_clean":"2024-04-06T03:27:46.539192+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:27:46.539192+0000","last_undegraded":"2024-04-06T03:27:46.539192+0000","last_fullsized":"2024-04-06T03:27:46.539192+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:26:20.562552+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:26:20.562552+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:53.732261+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":87,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163651+0000","last_change":"2024-04-06T03:26:33.611315+0000","last_active":"2024-04-06T03:27:45.163651+0000","last_peered":"2024-04-06T03:27:45.163651+0000","last_clean":"2024-04-06T03:27:45.163651+0000","last_became_active":"2024-04-06T03:25:19.549690+0000","last_became_peered":"2024-04-06T03:25:19.549690+0000","last_unstale":"2024-04-06T03:27:45.163651+0000","last_undegraded":"2024-04-06T03:27:45.163651+0000","last_fullsized":"2024-04-06T03:27:45.163651+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:33.611187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:26:33.611187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.779803+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"85'15","reported_seq":234,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.165063+0000","last_change":"2024-04-06T03:26:47.762293+0000","last_active":"2024-04-06T03:27:45.165063+0000","last_peered":"2024-04-06T03:27:45.165063+0000","last_clean":"2024-04-06T03:27:45.165063+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:27:45.165063+0000","last_undegraded":"2024-04-06T03:27:45.165063+0000","last_fullsized":"2024-04-06T03:27:45.165063+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:26:47.762169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:47.762169+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:12.055208+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"160'8686","reported_seq":8814,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:54.093789+0000","last_change":"2024-04-06T03:26:34.492007+0000","last_active":"2024-04-06T03:27:54.093789+0000","last_peered":"2024-04-06T03:27:54.093789+0000","last_clean":"2024-04-06T03:27:54.093789+0000","last_became_active":"2024-04-06T03:25:20.570322+0000","last_became_peered":"2024-04-06T03:25:20.570322+0000","last_unstale":"2024-04-06T03:27:54.093789+0000","last_undegraded":"2024-04-06T03:27:54.093789+0000","last_fullsized":"2024-04-06T03:27:54.093789+0000","mapping_epoch":127,"log_start":"136'2000","ondisk_log_start":"136'2000","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'4525","last_scrub_stamp":"2024-04-06T03:26:34.491853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:26:34.491853+0000","objects_scrubbed":4470,"log_size":6686,"log_dups_size":0,"ondisk_log_size":6686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:00.240364+0000","scrub_duration":1727,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230686720,"num_objects":3520,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8686,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.173341+0000","last_change":"2024-04-06T03:26:26.759988+0000","last_active":"2024-04-06T03:27:45.173341+0000","last_peered":"2024-04-06T03:27:45.173341+0000","last_clean":"2024-04-06T03:27:45.173341+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:27:45.173341+0000","last_undegraded":"2024-04-06T03:27:45.173341+0000","last_fullsized":"2024-04-06T03:27:45.173341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:26.759892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:26:26.759892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.521820+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166444+0000","last_change":"2024-04-06T03:27:10.501729+0000","last_active":"2024-04-06T03:27:45.166444+0000","last_peered":"2024-04-06T03:27:45.166444+0000","last_clean":"2024-04-06T03:27:45.166444+0000","last_became_active":"2024-04-06T03:27:10.501454+0000","last_became_peered":"2024-04-06T03:27:10.501454+0000","last_unstale":"2024-04-06T03:27:45.166444+0000","last_undegraded":"2024-04-06T03:27:45.166444+0000","last_fullsized":"2024-04-06T03:27:45.166444+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:31.256722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:26:31.256722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:51.104847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":200,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160569+0000","last_change":"2024-04-06T03:24:45.912279+0000","last_active":"2024-04-06T03:27:45.160569+0000","last_peered":"2024-04-06T03:27:45.160569+0000","last_clean":"2024-04-06T03:27:45.160569+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:27:45.160569+0000","last_undegraded":"2024-04-06T03:27:45.160569+0000","last_fullsized":"2024-04-06T03:27:45.160569+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:45.911985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:24:45.911985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"85'25","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.166441+0000","last_change":"2024-04-06T03:26:20.677403+0000","last_active":"2024-04-06T03:27:45.166441+0000","last_peered":"2024-04-06T03:27:45.166441+0000","last_clean":"2024-04-06T03:27:45.166441+0000","last_became_active":"2024-04-06T03:26:18.616944+0000","last_became_peered":"2024-04-06T03:26:18.616944+0000","last_unstale":"2024-04-06T03:27:45.166441+0000","last_undegraded":"2024-04-06T03:27:45.166441+0000","last_fullsized":"2024-04-06T03:27:45.166441+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'25","last_scrub_stamp":"2024-04-06T03:26:20.677274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:26:20.677274+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:46.703315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"160'8379","reported_seq":8456,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:49.494015+0000","last_change":"2024-04-06T03:25:20.584328+0000","last_active":"2024-04-06T03:27:49.494015+0000","last_peered":"2024-04-06T03:27:49.494015+0000","last_clean":"2024-04-06T03:27:49.494015+0000","last_became_active":"2024-04-06T03:25:20.582828+0000","last_became_peered":"2024-04-06T03:25:20.582828+0000","last_unstale":"2024-04-06T03:27:49.494015+0000","last_undegraded":"2024-04-06T03:27:49.494015+0000","last_fullsized":"2024-04-06T03:27:49.494015+0000","mapping_epoch":127,"log_start":"137'3200","ondisk_log_start":"137'3200","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249888768,"num_objects":3813,"num_object_clones":0,"num_object_copies":11439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8379,"num_write_kb":390144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":180,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.160187+0000","last_change":"2024-04-06T03:24:35.830607+0000","last_active":"2024-04-06T03:27:45.160187+0000","last_peered":"2024-04-06T03:27:45.160187+0000","last_clean":"2024-04-06T03:27:45.160187+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:27:45.160187+0000","last_undegraded":"2024-04-06T03:27:45.160187+0000","last_fullsized":"2024-04-06T03:27:45.160187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:35.830390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:35.830390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172036+0000","last_change":"2024-04-06T03:26:25.781812+0000","last_active":"2024-04-06T03:27:45.172036+0000","last_peered":"2024-04-06T03:27:45.172036+0000","last_clean":"2024-04-06T03:27:45.172036+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:27:45.172036+0000","last_undegraded":"2024-04-06T03:27:45.172036+0000","last_fullsized":"2024-04-06T03:27:45.172036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:25.781686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:26:25.781686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.443099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.172429+0000","last_change":"2024-04-06T03:26:09.523926+0000","last_active":"2024-04-06T03:27:45.172429+0000","last_peered":"2024-04-06T03:27:45.172429+0000","last_clean":"2024-04-06T03:27:45.172429+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:27:45.172429+0000","last_undegraded":"2024-04-06T03:27:45.172429+0000","last_fullsized":"2024-04-06T03:27:45.172429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:09.523809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:26:09.523809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.519136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'17","reported_seq":216,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161692+0000","last_change":"2024-04-06T03:24:37.752966+0000","last_active":"2024-04-06T03:27:45.161692+0000","last_peered":"2024-04-06T03:27:45.161692+0000","last_clean":"2024-04-06T03:27:45.161692+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:27:45.161692+0000","last_undegraded":"2024-04-06T03:27:45.161692+0000","last_fullsized":"2024-04-06T03:27:45.161692+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-04-06T03:24:37.752713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:24:37.752713+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"160'8530","reported_seq":8666,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:54.056606+0000","last_change":"2024-04-06T03:26:47.354461+0000","last_active":"2024-04-06T03:27:54.056606+0000","last_peered":"2024-04-06T03:27:54.056606+0000","last_clean":"2024-04-06T03:27:54.056606+0000","last_became_active":"2024-04-06T03:25:20.562741+0000","last_became_peered":"2024-04-06T03:25:20.562741+0000","last_unstale":"2024-04-06T03:27:54.056606+0000","last_undegraded":"2024-04-06T03:27:54.056606+0000","last_fullsized":"2024-04-06T03:27:54.056606+0000","mapping_epoch":127,"log_start":"134'1800","ondisk_log_start":"134'1800","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5139","last_scrub_stamp":"2024-04-06T03:26:47.354337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:26:47.354337+0000","objects_scrubbed":5113,"log_size":6730,"log_dups_size":0,"ondisk_log_size":6730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:17.305941+0000","scrub_duration":1303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231866368,"num_objects":3538,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":386176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":188,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.168603+0000","last_change":"2024-04-06T03:24:58.076190+0000","last_active":"2024-04-06T03:27:45.168603+0000","last_peered":"2024-04-06T03:27:45.168603+0000","last_clean":"2024-04-06T03:27:45.168603+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:27:45.168603+0000","last_undegraded":"2024-04-06T03:27:45.168603+0000","last_fullsized":"2024-04-06T03:27:45.168603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:24:58.075944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:24:58.075944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.834645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":202,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.173839+0000","last_change":"2024-04-06T03:26:36.792347+0000","last_active":"2024-04-06T03:27:45.173839+0000","last_peered":"2024-04-06T03:27:45.173839+0000","last_clean":"2024-04-06T03:27:45.173839+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:27:45.173839+0000","last_undegraded":"2024-04-06T03:27:45.173839+0000","last_fullsized":"2024-04-06T03:27:45.173839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:36.792221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:44.525616+0000","last_clean_scrub_stamp":"2024-04-06T03:26:36.792221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:28:05.633808+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":209,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.171587+0000","last_change":"2024-04-06T03:26:19.356998+0000","last_active":"2024-04-06T03:27:45.171587+0000","last_peered":"2024-04-06T03:27:45.171587+0000","last_clean":"2024-04-06T03:27:45.171587+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:27:45.171587+0000","last_undegraded":"2024-04-06T03:27:45.171587+0000","last_fullsized":"2024-04-06T03:27:45.171587+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:19.356772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:02.771476+0000","last_clean_scrub_stamp":"2024-04-06T03:26:19.356772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:49.350510+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"125'15","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.163308+0000","last_change":"2024-04-06T03:25:54.496063+0000","last_active":"2024-04-06T03:27:45.163308+0000","last_peered":"2024-04-06T03:27:45.163308+0000","last_clean":"2024-04-06T03:27:45.163308+0000","last_became_active":"2024-04-06T03:25:54.495696+0000","last_became_peered":"2024-04-06T03:25:54.495696+0000","last_unstale":"2024-04-06T03:27:45.163308+0000","last_undegraded":"2024-04-06T03:27:45.163308+0000","last_fullsized":"2024-04-06T03:27:45.163308+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'15","last_scrub_stamp":"2024-04-06T03:25:15.970079+0000","last_deep_scrub":"125'15","last_deep_scrub_stamp":"2024-04-06T03:25:15.970079+0000","last_clean_scrub_stamp":"2024-04-06T03:25:15.970079+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"160'8549","reported_seq":8626,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:53.988100+0000","last_change":"2024-04-06T03:25:20.563660+0000","last_active":"2024-04-06T03:27:53.988100+0000","last_peered":"2024-04-06T03:27:53.988100+0000","last_clean":"2024-04-06T03:27:53.988100+0000","last_became_active":"2024-04-06T03:25:20.562659+0000","last_became_peered":"2024-04-06T03:25:20.562659+0000","last_unstale":"2024-04-06T03:27:53.988100+0000","last_undegraded":"2024-04-06T03:27:53.988100+0000","last_fullsized":"2024-04-06T03:27:53.988100+0000","mapping_epoch":127,"log_start":"136'2800","ondisk_log_start":"136'2800","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:25:19.530265+0000","last_clean_scrub_stamp":"2024-04-06T03:25:19.530265+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:57.202968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229048320,"num_objects":3495,"num_object_clones":0,"num_object_copies":10485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8549,"num_write_kb":385408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":58,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.170357+0000","last_change":"2024-04-06T03:26:18.642870+0000","last_active":"2024-04-06T03:27:45.170357+0000","last_peered":"2024-04-06T03:27:45.170357+0000","last_clean":"2024-04-06T03:27:45.170357+0000","last_became_active":"2024-04-06T03:26:18.641975+0000","last_became_peered":"2024-04-06T03:26:18.641975+0000","last_unstale":"2024-04-06T03:27:45.170357+0000","last_undegraded":"2024-04-06T03:27:45.170357+0000","last_fullsized":"2024-04-06T03:27:45.170357+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:13.993384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:26:13.993384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:48.846739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.173147+0000","last_change":"2024-04-06T03:26:11.501788+0000","last_active":"2024-04-06T03:27:45.173147+0000","last_peered":"2024-04-06T03:27:45.173147+0000","last_clean":"2024-04-06T03:27:45.173147+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:27:45.173147+0000","last_undegraded":"2024-04-06T03:27:45.173147+0000","last_fullsized":"2024-04-06T03:27:45.173147+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:26:11.501633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:26:11.501633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:47.521760+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"85'15","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:45.161870+0000","last_change":"2024-04-06T03:25:08.950725+0000","last_active":"2024-04-06T03:27:45.161870+0000","last_peered":"2024-04-06T03:27:45.161870+0000","last_clean":"2024-04-06T03:27:45.161870+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:27:45.161870+0000","last_undegraded":"2024-04-06T03:27:45.161870+0000","last_fullsized":"2024-04-06T03:27:45.161870+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'15","last_scrub_stamp":"2024-04-06T03:25:08.950492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:25:08.950492+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"85'11","reported_seq":287,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-04-06T03:27:46.531411+0000","last_change":"2024-04-06T03:24:53.861193+0000","last_active":"2024-04-06T03:27:46.531411+0000","last_peered":"2024-04-06T03:27:46.531411+0000","last_clean":"2024-04-06T03:27:46.531411+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:27:46.531411+0000","last_undegraded":"2024-04-06T03:27:46.531411+0000","last_fullsized":"2024-04-06T03:27:46.531411+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:24:53.860959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:24:53.860959+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:27:55.672015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3876126744,"num_objects":59146,"num_object_clones":0,"num_object_copies":177438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134350,"num_write_kb":6191809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11352748032,"data_stored":11352735816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86050,"ondisk_log_size":86050,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":672,"ondisk_log_size":672,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":395,"num_read_kb":390,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743519,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1214664,"kb_used_data":1139072,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92530488,"statfs":{"total":95995035648,"available":94751219712,"internally_reserved":0,"allocated":1166409728,"data_stored":1166046163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135977,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210792,"kb_used_data":1139296,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92534360,"statfs":{"total":95995035648,"available":94755184640,"internally_reserved":0,"allocated":1166639104,"data_stored":1166306611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1433204,"kb_used_data":1365932,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92311948,"statfs":{"total":95995035648,"available":94527434752,"internally_reserved":0,"allocated":1398714368,"data_stored":1398368897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738535,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2060028,"kb_used_data":1960820,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91685124,"statfs":{"total":95995035648,"available":93885566976,"internally_reserved":0,"allocated":2007879680,"data_stored":2007527760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378772,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1422820,"kb_used_data":1355548,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92322332,"statfs":{"total":95995035648,"available":94538067968,"internally_reserved":0,"allocated":1388081152,"data_stored":1387751785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084423,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2301596,"kb_used_data":2193940,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91443556,"statfs":{"total":95995035648,"available":93638201344,"internally_reserved":0,"allocated":2246594560,"data_stored":2246273299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855487,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1204104,"kb_used_data":1132672,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92541048,"statfs":{"total":95995035648,"available":94762033152,"internally_reserved":0,"allocated":1159856128,"data_stored":1159490382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495733,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":971680,"kb_used_data":900248,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92773472,"statfs":{"total":95995035648,"available":95000035328,"internally_reserved":0,"allocated":921853952,"data_stored":921528577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1985482752,"data_stored":1985478680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1380061184,"data_stored":1380057112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1379729408,"data_stored":1379729408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2224029696,"data_stored":2224029696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1156579328,"data_stored":1156579328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":918945792,"data_stored":918945792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1147998208,"data_stored":1147994136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1159921664,"data_stored":1159921664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:27:59.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:27:59.541 INFO:tasks.thrashosds.thrasher:Setting 6.16 to [3, 0, 6] 2024-04-06T03:27:59.541 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.16', '3', '0', '6'] 2024-04-06T03:27:59.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pg-upmap 6.16 3 0 6 2024-04-06T03:27:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:27:59.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:59 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.099150+0000 mgr.y (mgr.24368) 262 : cluster 0 pgmap v251: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 856 op/s 2024-04-06T03:27:59.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:59 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.803250+0000 osd.7 (osd.7) 149 : cluster 0 5.4 scrub starts 2024-04-06T03:27:59.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:59 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.807434+0000 osd.7 (osd.7) 150 : cluster 0 5.4 scrub ok 2024-04-06T03:27:59.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:59 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.812763+0000 osd.2 (osd.2) 123 : cluster 0 5.d scrub starts 2024-04-06T03:27:59.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:27:59 smithi153 bash[23852]: cluster 2024-04-06T03:27:58.815505+0000 osd.2 (osd.2) 124 : cluster 0 5.d scrub ok 2024-04-06T03:27:59.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.099150+0000 mgr.y (mgr.24368) 262 : cluster 0 pgmap v251: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 856 op/s 2024-04-06T03:27:59.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.803250+0000 osd.7 (osd.7) 149 : cluster 0 5.4 scrub starts 2024-04-06T03:27:59.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.807434+0000 osd.7 (osd.7) 150 : cluster 0 5.4 scrub ok 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.812763+0000 osd.2 (osd.2) 123 : cluster 0 5.d scrub starts 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[21467]: cluster 2024-04-06T03:27:58.815505+0000 osd.2 (osd.2) 124 : cluster 0 5.d scrub ok 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.099150+0000 mgr.y (mgr.24368) 262 : cluster 0 pgmap v251: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 856 op/s 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.803250+0000 osd.7 (osd.7) 149 : cluster 0 5.4 scrub starts 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.807434+0000 osd.7 (osd.7) 150 : cluster 0 5.4 scrub ok 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.812763+0000 osd.2 (osd.2) 123 : cluster 0 5.d scrub starts 2024-04-06T03:27:59.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:27:59 smithi050 bash[29221]: cluster 2024-04-06T03:27:58.815505+0000 osd.2 (osd.2) 124 : cluster 0 5.d scrub ok 2024-04-06T03:27:59.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:59.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:59.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:27:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:27:59.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:59.958 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:27:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:27:59.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:00.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:00.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:00.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:00.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:00.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:00.358 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:00.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:00.486 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:00.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.114234+0000 osd.0 (osd.0) 138 : cluster 0 2.4 scrub starts 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.123503+0000 osd.0 (osd.0) 139 : cluster 0 2.4 scrub ok 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: audit 2024-04-06T03:27:59.294408+0000 mgr.y (mgr.24368) 263 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.778582+0000 osd.2 (osd.2) 125 : cluster 0 2.3 scrub starts 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.787746+0000 osd.2 (osd.2) 126 : cluster 0 2.3 scrub ok 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.819004+0000 osd.7 (osd.7) 151 : cluster 0 2.7 scrub starts 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.826866+0000 osd.7 (osd.7) 152 : cluster 0 2.7 scrub ok 2024-04-06T03:28:00.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.838912+0000 osd.3 (osd.3) 173 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:00.600 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:00 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.838966+0000 osd.3 (osd.3) 174 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:00.612 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:00.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:00.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:00.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:00.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.114234+0000 osd.0 (osd.0) 138 : cluster 0 2.4 scrub starts 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.123503+0000 osd.0 (osd.0) 139 : cluster 0 2.4 scrub ok 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: audit 2024-04-06T03:27:59.294408+0000 mgr.y (mgr.24368) 263 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.778582+0000 osd.2 (osd.2) 125 : cluster 0 2.3 scrub starts 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.787746+0000 osd.2 (osd.2) 126 : cluster 0 2.3 scrub ok 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.819004+0000 osd.7 (osd.7) 151 : cluster 0 2.7 scrub starts 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.826866+0000 osd.7 (osd.7) 152 : cluster 0 2.7 scrub ok 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.838912+0000 osd.3 (osd.3) 173 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.838966+0000 osd.3 (osd.3) 174 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.114234+0000 osd.0 (osd.0) 138 : cluster 0 2.4 scrub starts 2024-04-06T03:28:00.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.123503+0000 osd.0 (osd.0) 139 : cluster 0 2.4 scrub ok 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: audit 2024-04-06T03:27:59.294408+0000 mgr.y (mgr.24368) 263 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.778582+0000 osd.2 (osd.2) 125 : cluster 0 2.3 scrub starts 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.787746+0000 osd.2 (osd.2) 126 : cluster 0 2.3 scrub ok 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.819004+0000 osd.7 (osd.7) 151 : cluster 0 2.7 scrub starts 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.826866+0000 osd.7 (osd.7) 152 : cluster 0 2.7 scrub ok 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.838912+0000 osd.3 (osd.3) 173 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:00.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:00 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.838966+0000 osd.3 (osd.3) 174 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:00.876 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:00.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:00.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:01.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:01.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:01.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:01.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:01.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:01.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:01.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:01.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:28:01.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:28:01.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:01.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:01.709 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:01 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.734604+0000 osd.4 (osd.4) 143 : cluster 0 6.e scrub starts 2024-04-06T03:28:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:01 smithi153 bash[23852]: cluster 2024-04-06T03:27:59.738872+0000 osd.4 (osd.4) 144 : cluster 0 6.e scrub ok 2024-04-06T03:28:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:01 smithi153 bash[23852]: cluster 2024-04-06T03:28:00.101202+0000 mgr.y (mgr.24368) 264 : cluster 0 pgmap v252: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:28:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:01 smithi153 bash[23852]: cluster 2024-04-06T03:28:00.554315+0000 mon.a (mon.0) 1090 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-04-06T03:28:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:01 smithi153 bash[23852]: cluster 2024-04-06T03:28:00.783776+0000 osd.2 (osd.2) 127 : cluster 0 2.10 scrub starts 2024-04-06T03:28:01.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:01 smithi153 bash[23852]: cluster 2024-04-06T03:28:00.785473+0000 osd.2 (osd.2) 128 : cluster 0 2.10 scrub ok 2024-04-06T03:28:01.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:01.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.734604+0000 osd.4 (osd.4) 143 : cluster 0 6.e scrub starts 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[21467]: cluster 2024-04-06T03:27:59.738872+0000 osd.4 (osd.4) 144 : cluster 0 6.e scrub ok 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[21467]: cluster 2024-04-06T03:28:00.101202+0000 mgr.y (mgr.24368) 264 : cluster 0 pgmap v252: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[21467]: cluster 2024-04-06T03:28:00.554315+0000 mon.a (mon.0) 1090 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[21467]: cluster 2024-04-06T03:28:00.783776+0000 osd.2 (osd.2) 127 : cluster 0 2.10 scrub starts 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[21467]: cluster 2024-04-06T03:28:00.785473+0000 osd.2 (osd.2) 128 : cluster 0 2.10 scrub ok 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.734604+0000 osd.4 (osd.4) 143 : cluster 0 6.e scrub starts 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[29221]: cluster 2024-04-06T03:27:59.738872+0000 osd.4 (osd.4) 144 : cluster 0 6.e scrub ok 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[29221]: cluster 2024-04-06T03:28:00.101202+0000 mgr.y (mgr.24368) 264 : cluster 0 pgmap v252: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 879 op/s 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[29221]: cluster 2024-04-06T03:28:00.554315+0000 mon.a (mon.0) 1090 : cluster 0 osdmap e162: 8 total, 8 up, 8 in 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[29221]: cluster 2024-04-06T03:28:00.783776+0000 osd.2 (osd.2) 127 : cluster 0 2.10 scrub starts 2024-04-06T03:28:01.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:01 smithi050 bash[29221]: cluster 2024-04-06T03:28:00.785473+0000 osd.2 (osd.2) 128 : cluster 0 2.10 scrub ok 2024-04-06T03:28:01.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:01.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:01.979 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:01.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:02.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:02.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:02.184 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:02.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:02.256 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:02.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:02.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:02.520 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:02.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:02.656 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:02 smithi050 bash[21467]: cluster 2024-04-06T03:28:01.771144+0000 osd.2 (osd.2) 129 : cluster 0 6.14 scrub starts 2024-04-06T03:28:02.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:02 smithi050 bash[21467]: cluster 2024-04-06T03:28:01.772860+0000 osd.2 (osd.2) 130 : cluster 0 6.14 scrub ok 2024-04-06T03:28:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:02 smithi050 bash[29221]: cluster 2024-04-06T03:28:01.771144+0000 osd.2 (osd.2) 129 : cluster 0 6.14 scrub starts 2024-04-06T03:28:02.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:02 smithi050 bash[29221]: cluster 2024-04-06T03:28:01.772860+0000 osd.2 (osd.2) 130 : cluster 0 6.14 scrub ok 2024-04-06T03:28:02.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:02.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:02.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:02 smithi153 bash[23852]: cluster 2024-04-06T03:28:01.771144+0000 osd.2 (osd.2) 129 : cluster 0 6.14 scrub starts 2024-04-06T03:28:02.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:02 smithi153 bash[23852]: cluster 2024-04-06T03:28:01.772860+0000 osd.2 (osd.2) 130 : cluster 0 6.14 scrub ok 2024-04-06T03:28:02.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:02.916 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:02.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:03.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:03.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:03.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:03.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:03.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:03.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:03.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:03.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:03.551 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:03.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:03.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:01.727084+0000 osd.4 (osd.4) 145 : cluster 0 3.1d scrub starts 2024-04-06T03:28:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:01.731043+0000 osd.4 (osd.4) 146 : cluster 0 3.1d scrub ok 2024-04-06T03:28:03.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.101580+0000 mgr.y (mgr.24368) 265 : cluster 0 pgmap v254: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 990 op/s 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.102119+0000 osd.1 (osd.1) 137 : cluster 0 2.9 scrub starts 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.107678+0000 osd.1 (osd.1) 138 : cluster 0 2.9 scrub ok 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.717439+0000 osd.7 (osd.7) 153 : cluster 0 6.8 scrub starts 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.718665+0000 osd.7 (osd.7) 154 : cluster 0 6.8 scrub ok 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:01.727084+0000 osd.4 (osd.4) 145 : cluster 0 3.1d scrub starts 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:01.731043+0000 osd.4 (osd.4) 146 : cluster 0 3.1d scrub ok 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.101580+0000 mgr.y (mgr.24368) 265 : cluster 0 pgmap v254: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 990 op/s 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.102119+0000 osd.1 (osd.1) 137 : cluster 0 2.9 scrub starts 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.107678+0000 osd.1 (osd.1) 138 : cluster 0 2.9 scrub ok 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.717439+0000 osd.7 (osd.7) 153 : cluster 0 6.8 scrub starts 2024-04-06T03:28:03.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:03 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.718665+0000 osd.7 (osd.7) 154 : cluster 0 6.8 scrub ok 2024-04-06T03:28:03.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:03.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:01.727084+0000 osd.4 (osd.4) 145 : cluster 0 3.1d scrub starts 2024-04-06T03:28:03.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:01.731043+0000 osd.4 (osd.4) 146 : cluster 0 3.1d scrub ok 2024-04-06T03:28:03.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.101580+0000 mgr.y (mgr.24368) 265 : cluster 0 pgmap v254: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 990 op/s 2024-04-06T03:28:03.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.102119+0000 osd.1 (osd.1) 137 : cluster 0 2.9 scrub starts 2024-04-06T03:28:03.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.107678+0000 osd.1 (osd.1) 138 : cluster 0 2.9 scrub ok 2024-04-06T03:28:03.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.717439+0000 osd.7 (osd.7) 153 : cluster 0 6.8 scrub starts 2024-04-06T03:28:03.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:03 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.718665+0000 osd.7 (osd.7) 154 : cluster 0 6.8 scrub ok 2024-04-06T03:28:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:03.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:03.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:03.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:03.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:04.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:04.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:04.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:04.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:04.222 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:04.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:04.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:04.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:04.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:04.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:04.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:04.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:04.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:04.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:04.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.722823+0000 osd.4 (osd.4) 147 : cluster 0 4.1 scrub starts 2024-04-06T03:28:04.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:02.724699+0000 osd.4 (osd.4) 148 : cluster 0 4.1 scrub ok 2024-04-06T03:28:04.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:03.094550+0000 osd.1 (osd.1) 139 : cluster 0 6.1 scrub starts 2024-04-06T03:28:04.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:03.097554+0000 osd.1 (osd.1) 140 : cluster 0 6.1 scrub ok 2024-04-06T03:28:04.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:03.280818+0000 osd.0 (osd.0) 140 : cluster 0 7.f scrub starts 2024-04-06T03:28:04.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:03.685038+0000 osd.7 (osd.7) 155 : cluster 0 2.8 scrub starts 2024-04-06T03:28:04.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: cluster 2024-04-06T03:28:03.691549+0000 osd.7 (osd.7) 156 : cluster 0 2.8 scrub ok 2024-04-06T03:28:04.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:04 smithi153 bash[23852]: audit 2024-04-06T03:28:03.737807+0000 mon.a (mon.0) 1091 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:04.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.722823+0000 osd.4 (osd.4) 147 : cluster 0 4.1 scrub starts 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:02.724699+0000 osd.4 (osd.4) 148 : cluster 0 4.1 scrub ok 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:03.094550+0000 osd.1 (osd.1) 139 : cluster 0 6.1 scrub starts 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:03.097554+0000 osd.1 (osd.1) 140 : cluster 0 6.1 scrub ok 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:03.280818+0000 osd.0 (osd.0) 140 : cluster 0 7.f scrub starts 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:03.685038+0000 osd.7 (osd.7) 155 : cluster 0 2.8 scrub starts 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: cluster 2024-04-06T03:28:03.691549+0000 osd.7 (osd.7) 156 : cluster 0 2.8 scrub ok 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[21467]: audit 2024-04-06T03:28:03.737807+0000 mon.a (mon.0) 1091 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.722823+0000 osd.4 (osd.4) 147 : cluster 0 4.1 scrub starts 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:02.724699+0000 osd.4 (osd.4) 148 : cluster 0 4.1 scrub ok 2024-04-06T03:28:04.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:03.094550+0000 osd.1 (osd.1) 139 : cluster 0 6.1 scrub starts 2024-04-06T03:28:04.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:03.097554+0000 osd.1 (osd.1) 140 : cluster 0 6.1 scrub ok 2024-04-06T03:28:04.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:03.280818+0000 osd.0 (osd.0) 140 : cluster 0 7.f scrub starts 2024-04-06T03:28:04.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:03.685038+0000 osd.7 (osd.7) 155 : cluster 0 2.8 scrub starts 2024-04-06T03:28:04.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: cluster 2024-04-06T03:28:03.691549+0000 osd.7 (osd.7) 156 : cluster 0 2.8 scrub ok 2024-04-06T03:28:04.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:04 smithi050 bash[29221]: audit 2024-04-06T03:28:03.737807+0000 mon.a (mon.0) 1091 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:04.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:04.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:04.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:04.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:04.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:04.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:05.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:05.129 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:05.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:05.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:05.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:05.393 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:05.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:05.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:05.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:05.646 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:05 smithi153 bash[23852]: cluster 2024-04-06T03:28:03.922908+0000 osd.0 (osd.0) 141 : cluster 0 7.f scrub ok 2024-04-06T03:28:05.647 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:05 smithi153 bash[23852]: cluster 2024-04-06T03:28:04.102280+0000 mgr.y (mgr.24368) 266 : cluster 0 pgmap v255: 148 pgs: 148 active+clean; 3.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s 2024-04-06T03:28:05.647 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:05 smithi153 bash[23852]: cluster 2024-04-06T03:28:04.229684+0000 osd.0 (osd.0) 142 : cluster 0 7.9 scrub starts 2024-04-06T03:28:05.647 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:05 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.091906+0000 osd.3 (osd.3) 175 : cluster 0 3.8 scrub starts 2024-04-06T03:28:05.647 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:05 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.096740+0000 osd.3 (osd.3) 176 : cluster 0 3.8 scrub ok 2024-04-06T03:28:05.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:05.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[21467]: cluster 2024-04-06T03:28:03.922908+0000 osd.0 (osd.0) 141 : cluster 0 7.f scrub ok 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[21467]: cluster 2024-04-06T03:28:04.102280+0000 mgr.y (mgr.24368) 266 : cluster 0 pgmap v255: 148 pgs: 148 active+clean; 3.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[21467]: cluster 2024-04-06T03:28:04.229684+0000 osd.0 (osd.0) 142 : cluster 0 7.9 scrub starts 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.091906+0000 osd.3 (osd.3) 175 : cluster 0 3.8 scrub starts 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.096740+0000 osd.3 (osd.3) 176 : cluster 0 3.8 scrub ok 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[29221]: cluster 2024-04-06T03:28:03.922908+0000 osd.0 (osd.0) 141 : cluster 0 7.f scrub ok 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[29221]: cluster 2024-04-06T03:28:04.102280+0000 mgr.y (mgr.24368) 266 : cluster 0 pgmap v255: 148 pgs: 148 active+clean; 3.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[29221]: cluster 2024-04-06T03:28:04.229684+0000 osd.0 (osd.0) 142 : cluster 0 7.9 scrub starts 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.091906+0000 osd.3 (osd.3) 175 : cluster 0 3.8 scrub starts 2024-04-06T03:28:05.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:05 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.096740+0000 osd.3 (osd.3) 176 : cluster 0 3.8 scrub ok 2024-04-06T03:28:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:05.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:05.922 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:05.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:06.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:06.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:06.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:06.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:06.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:06.258 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:06.260 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:06.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:06.314 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:06.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:06.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:06.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:06.601 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:06.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:06.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:06.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[21467]: cluster 2024-04-06T03:28:04.705029+0000 osd.4 (osd.4) 149 : cluster 0 5.1 scrub starts 2024-04-06T03:28:06.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[21467]: cluster 2024-04-06T03:28:04.706404+0000 osd.4 (osd.4) 150 : cluster 0 5.1 scrub ok 2024-04-06T03:28:06.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.550868+0000 osd.0 (osd.0) 143 : cluster 0 7.9 scrub ok 2024-04-06T03:28:06.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.737617+0000 osd.7 (osd.7) 157 : cluster 0 3.d scrub starts 2024-04-06T03:28:06.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.740752+0000 osd.7 (osd.7) 158 : cluster 0 3.d scrub ok 2024-04-06T03:28:06.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[29221]: cluster 2024-04-06T03:28:04.705029+0000 osd.4 (osd.4) 149 : cluster 0 5.1 scrub starts 2024-04-06T03:28:06.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[29221]: cluster 2024-04-06T03:28:04.706404+0000 osd.4 (osd.4) 150 : cluster 0 5.1 scrub ok 2024-04-06T03:28:06.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.550868+0000 osd.0 (osd.0) 143 : cluster 0 7.9 scrub ok 2024-04-06T03:28:06.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.737617+0000 osd.7 (osd.7) 157 : cluster 0 3.d scrub starts 2024-04-06T03:28:06.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:06 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.740752+0000 osd.7 (osd.7) 158 : cluster 0 3.d scrub ok 2024-04-06T03:28:06.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:06.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:06 smithi153 bash[23852]: cluster 2024-04-06T03:28:04.705029+0000 osd.4 (osd.4) 149 : cluster 0 5.1 scrub starts 2024-04-06T03:28:06.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:06 smithi153 bash[23852]: cluster 2024-04-06T03:28:04.706404+0000 osd.4 (osd.4) 150 : cluster 0 5.1 scrub ok 2024-04-06T03:28:06.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:06 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.550868+0000 osd.0 (osd.0) 143 : cluster 0 7.9 scrub ok 2024-04-06T03:28:06.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:06 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.737617+0000 osd.7 (osd.7) 157 : cluster 0 3.d scrub starts 2024-04-06T03:28:06.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:06 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.740752+0000 osd.7 (osd.7) 158 : cluster 0 3.d scrub ok 2024-04-06T03:28:06.849 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:06.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:06.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:06.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:07.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:07.113 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:07.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:07.242 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:07.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:07.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:07.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:07.502 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:07.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:07.635 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.606575+0000 osd.5 (osd.5) 123 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.606656+0000 osd.5 (osd.5) 124 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[21467]: cluster 2024-04-06T03:28:05.727479+0000 osd.4 (osd.4) 151 : cluster 0 7.1 scrub starts 2024-04-06T03:28:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.103366+0000 mgr.y (mgr.24368) 267 : cluster 0 pgmap v256: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:28:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.489636+0000 osd.4 (osd.4) 152 : cluster 0 7.1 scrub ok 2024-04-06T03:28:07.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.586363+0000 mon.a (mon.0) 1092 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-04-06T03:28:07.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.606575+0000 osd.5 (osd.5) 123 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:07.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.606656+0000 osd.5 (osd.5) 124 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:07.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[29221]: cluster 2024-04-06T03:28:05.727479+0000 osd.4 (osd.4) 151 : cluster 0 7.1 scrub starts 2024-04-06T03:28:07.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.103366+0000 mgr.y (mgr.24368) 267 : cluster 0 pgmap v256: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:28:07.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.489636+0000 osd.4 (osd.4) 152 : cluster 0 7.1 scrub ok 2024-04-06T03:28:07.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:07 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.586363+0000 mon.a (mon.0) 1092 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-04-06T03:28:07.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:07.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:07 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.606575+0000 osd.5 (osd.5) 123 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:07.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:07 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.606656+0000 osd.5 (osd.5) 124 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:07.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:07 smithi153 bash[23852]: cluster 2024-04-06T03:28:05.727479+0000 osd.4 (osd.4) 151 : cluster 0 7.1 scrub starts 2024-04-06T03:28:07.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:07 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.103366+0000 mgr.y (mgr.24368) 267 : cluster 0 pgmap v256: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:28:07.748 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:07 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.489636+0000 osd.4 (osd.4) 152 : cluster 0 7.1 scrub ok 2024-04-06T03:28:07.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:07 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.586363+0000 mon.a (mon.0) 1092 : cluster 0 osdmap e163: 8 total, 8 up, 8 in 2024-04-06T03:28:07.761 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:07.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:07.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:07.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:07.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:08.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:08.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:08.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:08.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:08.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:08.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:08.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:08.405 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:08.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:08.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:08.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:08.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:08.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.722101+0000 osd.4 (osd.4) 153 : cluster 0 6.1a scrub starts 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.722983+0000 osd.4 (osd.4) 154 : cluster 0 6.1a scrub ok 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.890273+0000 osd.2 (osd.2) 131 : cluster 0 2.e scrub starts 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:06.893772+0000 osd.2 (osd.2) 132 : cluster 0 2.e scrub ok 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:07.164905+0000 osd.1 (osd.1) 141 : cluster 0 2.6 scrub starts 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:07.168569+0000 osd.1 (osd.1) 142 : cluster 0 2.6 scrub ok 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:08.054162+0000 osd.3 (osd.3) 177 : cluster 0 6.c scrub starts 2024-04-06T03:28:08.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[21467]: cluster 2024-04-06T03:28:08.055699+0000 osd.3 (osd.3) 178 : cluster 0 6.c scrub ok 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.722101+0000 osd.4 (osd.4) 153 : cluster 0 6.1a scrub starts 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.722983+0000 osd.4 (osd.4) 154 : cluster 0 6.1a scrub ok 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.890273+0000 osd.2 (osd.2) 131 : cluster 0 2.e scrub starts 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:06.893772+0000 osd.2 (osd.2) 132 : cluster 0 2.e scrub ok 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:07.164905+0000 osd.1 (osd.1) 141 : cluster 0 2.6 scrub starts 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:07.168569+0000 osd.1 (osd.1) 142 : cluster 0 2.6 scrub ok 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:08.054162+0000 osd.3 (osd.3) 177 : cluster 0 6.c scrub starts 2024-04-06T03:28:08.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:08 smithi050 bash[29221]: cluster 2024-04-06T03:28:08.055699+0000 osd.3 (osd.3) 178 : cluster 0 6.c scrub ok 2024-04-06T03:28:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:08.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:08.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:08.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.722101+0000 osd.4 (osd.4) 153 : cluster 0 6.1a scrub starts 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.722983+0000 osd.4 (osd.4) 154 : cluster 0 6.1a scrub ok 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.890273+0000 osd.2 (osd.2) 131 : cluster 0 2.e scrub starts 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:06.893772+0000 osd.2 (osd.2) 132 : cluster 0 2.e scrub ok 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:07.164905+0000 osd.1 (osd.1) 141 : cluster 0 2.6 scrub starts 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:07.168569+0000 osd.1 (osd.1) 142 : cluster 0 2.6 scrub ok 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:08.054162+0000 osd.3 (osd.3) 177 : cluster 0 6.c scrub starts 2024-04-06T03:28:09.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:08 smithi153 bash[23852]: cluster 2024-04-06T03:28:08.055699+0000 osd.3 (osd.3) 178 : cluster 0 6.c scrub ok 2024-04-06T03:28:09.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:09.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:09.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:09.235 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:09.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:09.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:09.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:09.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:09.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:28:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:28:09.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:09.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:28:09.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:09.629 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:09.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:07.757626+0000 osd.4 (osd.4) 155 : cluster 0 2.f scrub starts 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:07.759564+0000 osd.4 (osd.4) 156 : cluster 0 2.f scrub ok 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:07.763966+0000 osd.2 (osd.2) 133 : cluster 0 2.17 scrub starts 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:07.768604+0000 osd.2 (osd.2) 134 : cluster 0 2.17 scrub ok 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:08.103875+0000 mgr.y (mgr.24368) 268 : cluster 0 pgmap v258: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:08.200511+0000 osd.1 (osd.1) 143 : cluster 0 7.b scrub starts 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:08.221563+0000 osd.0 (osd.0) 144 : cluster 0 6.9 scrub starts 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:08.224496+0000 osd.0 (osd.0) 145 : cluster 0 6.9 scrub ok 2024-04-06T03:28:09.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.072475+0000 osd.3 (osd.3) 179 : cluster 0 7.0 scrub starts 2024-04-06T03:28:09.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:09.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:07.757626+0000 osd.4 (osd.4) 155 : cluster 0 2.f scrub starts 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:07.759564+0000 osd.4 (osd.4) 156 : cluster 0 2.f scrub ok 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:07.763966+0000 osd.2 (osd.2) 133 : cluster 0 2.17 scrub starts 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:07.768604+0000 osd.2 (osd.2) 134 : cluster 0 2.17 scrub ok 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:08.103875+0000 mgr.y (mgr.24368) 268 : cluster 0 pgmap v258: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:08.200511+0000 osd.1 (osd.1) 143 : cluster 0 7.b scrub starts 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:08.221563+0000 osd.0 (osd.0) 144 : cluster 0 6.9 scrub starts 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:08.224496+0000 osd.0 (osd.0) 145 : cluster 0 6.9 scrub ok 2024-04-06T03:28:09.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:09 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.072475+0000 osd.3 (osd.3) 179 : cluster 0 7.0 scrub starts 2024-04-06T03:28:09.755 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:09.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:07.757626+0000 osd.4 (osd.4) 155 : cluster 0 2.f scrub starts 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:07.759564+0000 osd.4 (osd.4) 156 : cluster 0 2.f scrub ok 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:07.763966+0000 osd.2 (osd.2) 133 : cluster 0 2.17 scrub starts 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:07.768604+0000 osd.2 (osd.2) 134 : cluster 0 2.17 scrub ok 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:08.103875+0000 mgr.y (mgr.24368) 268 : cluster 0 pgmap v258: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 561 op/s 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:08.200511+0000 osd.1 (osd.1) 143 : cluster 0 7.b scrub starts 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:08.221563+0000 osd.0 (osd.0) 144 : cluster 0 6.9 scrub starts 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:08.224496+0000 osd.0 (osd.0) 145 : cluster 0 6.9 scrub ok 2024-04-06T03:28:09.869 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:09 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.072475+0000 osd.3 (osd.3) 179 : cluster 0 7.0 scrub starts 2024-04-06T03:28:09.886 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:09.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:09.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:09.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:10.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:10.145 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:10.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:10.266 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:10.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:10.391 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:10.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:10.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:10.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:10.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:10.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:10.797 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:10.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:10.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:10.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.211022+0000 osd.0 (osd.0) 146 : cluster 0 7.6 scrub starts 2024-04-06T03:28:10.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.246210+0000 osd.1 (osd.1) 144 : cluster 0 7.b scrub ok 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[21467]: audit 2024-04-06T03:28:09.304396+0000 mgr.y (mgr.24368) 269 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.764074+0000 osd.7 (osd.7) 159 : cluster 0 2.b scrub starts 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.765110+0000 osd.7 (osd.7) 160 : cluster 0 2.b scrub ok 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.211022+0000 osd.0 (osd.0) 146 : cluster 0 7.6 scrub starts 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.246210+0000 osd.1 (osd.1) 144 : cluster 0 7.b scrub ok 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[29221]: audit 2024-04-06T03:28:09.304396+0000 mgr.y (mgr.24368) 269 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.764074+0000 osd.7 (osd.7) 159 : cluster 0 2.b scrub starts 2024-04-06T03:28:10.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:10 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.765110+0000 osd.7 (osd.7) 160 : cluster 0 2.b scrub ok 2024-04-06T03:28:10.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:10.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:11.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:11.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:10 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.211022+0000 osd.0 (osd.0) 146 : cluster 0 7.6 scrub starts 2024-04-06T03:28:11.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:10 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.246210+0000 osd.1 (osd.1) 144 : cluster 0 7.b scrub ok 2024-04-06T03:28:11.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:10 smithi153 bash[23852]: audit 2024-04-06T03:28:09.304396+0000 mgr.y (mgr.24368) 269 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:11.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:10 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.764074+0000 osd.7 (osd.7) 159 : cluster 0 2.b scrub starts 2024-04-06T03:28:11.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:10 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.765110+0000 osd.7 (osd.7) 160 : cluster 0 2.b scrub ok 2024-04-06T03:28:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:11.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:11.195 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:11.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:11.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:11.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:11.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:11.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:11.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:11.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:11.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.727425+0000 osd.4 (osd.4) 157 : cluster 0 3.a scrub starts 2024-04-06T03:28:11.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:09.733824+0000 osd.4 (osd.4) 158 : cluster 0 3.a scrub ok 2024-04-06T03:28:11.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.104685+0000 mgr.y (mgr.24368) 270 : cluster 0 pgmap v259: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:28:11.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.454063+0000 osd.3 (osd.3) 180 : cluster 0 7.0 scrub ok 2024-04-06T03:28:11.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.639369+0000 osd.0 (osd.0) 147 : cluster 0 7.6 scrub ok 2024-04-06T03:28:11.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.748056+0000 osd.7 (osd.7) 161 : cluster 0 3.e scrub starts 2024-04-06T03:28:11.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:11 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.750577+0000 osd.7 (osd.7) 162 : cluster 0 3.e scrub ok 2024-04-06T03:28:11.715 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:11.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:11.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:11.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:11.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:11.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:12.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:12.113 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:12.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:12.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.727425+0000 osd.4 (osd.4) 157 : cluster 0 3.a scrub starts 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:09.733824+0000 osd.4 (osd.4) 158 : cluster 0 3.a scrub ok 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.104685+0000 mgr.y (mgr.24368) 270 : cluster 0 pgmap v259: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.454063+0000 osd.3 (osd.3) 180 : cluster 0 7.0 scrub ok 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.639369+0000 osd.0 (osd.0) 147 : cluster 0 7.6 scrub ok 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.748056+0000 osd.7 (osd.7) 161 : cluster 0 3.e scrub starts 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.750577+0000 osd.7 (osd.7) 162 : cluster 0 3.e scrub ok 2024-04-06T03:28:12.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.727425+0000 osd.4 (osd.4) 157 : cluster 0 3.a scrub starts 2024-04-06T03:28:12.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:09.733824+0000 osd.4 (osd.4) 158 : cluster 0 3.a scrub ok 2024-04-06T03:28:12.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.104685+0000 mgr.y (mgr.24368) 270 : cluster 0 pgmap v259: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:28:12.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.454063+0000 osd.3 (osd.3) 180 : cluster 0 7.0 scrub ok 2024-04-06T03:28:12.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.639369+0000 osd.0 (osd.0) 147 : cluster 0 7.6 scrub ok 2024-04-06T03:28:12.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.748056+0000 osd.7 (osd.7) 161 : cluster 0 3.e scrub starts 2024-04-06T03:28:12.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:11 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.750577+0000 osd.7 (osd.7) 162 : cluster 0 3.e scrub ok 2024-04-06T03:28:12.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:12.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:12.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:12.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:12.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:12 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.763184+0000 osd.4 (osd.4) 159 : cluster 0 3.3 scrub starts 2024-04-06T03:28:12.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:12 smithi050 bash[21467]: cluster 2024-04-06T03:28:10.765791+0000 osd.4 (osd.4) 160 : cluster 0 3.3 scrub ok 2024-04-06T03:28:12.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:12 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.763184+0000 osd.4 (osd.4) 159 : cluster 0 3.3 scrub starts 2024-04-06T03:28:12.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:12 smithi050 bash[29221]: cluster 2024-04-06T03:28:10.765791+0000 osd.4 (osd.4) 160 : cluster 0 3.3 scrub ok 2024-04-06T03:28:12.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:12.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:12.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:12.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:13.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:12 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.763184+0000 osd.4 (osd.4) 159 : cluster 0 3.3 scrub starts 2024-04-06T03:28:13.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:12 smithi153 bash[23852]: cluster 2024-04-06T03:28:10.765791+0000 osd.4 (osd.4) 160 : cluster 0 3.3 scrub ok 2024-04-06T03:28:13.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:13.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:13.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:13.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:13.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:13.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:13.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:13.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:13.466 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:13.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:13.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:13.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:13.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:13 smithi050 bash[21467]: cluster 2024-04-06T03:28:12.105298+0000 mgr.y (mgr.24368) 271 : cluster 0 pgmap v260: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:28:13.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:13 smithi050 bash[21467]: cluster 2024-04-06T03:28:12.152397+0000 osd.1 (osd.1) 145 : cluster 0 4.2 scrub starts 2024-04-06T03:28:13.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:13 smithi050 bash[21467]: cluster 2024-04-06T03:28:12.154953+0000 osd.1 (osd.1) 146 : cluster 0 4.2 scrub ok 2024-04-06T03:28:13.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:13 smithi050 bash[29221]: cluster 2024-04-06T03:28:12.105298+0000 mgr.y (mgr.24368) 271 : cluster 0 pgmap v260: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:28:13.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:13 smithi050 bash[29221]: cluster 2024-04-06T03:28:12.152397+0000 osd.1 (osd.1) 145 : cluster 0 4.2 scrub starts 2024-04-06T03:28:13.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:13 smithi050 bash[29221]: cluster 2024-04-06T03:28:12.154953+0000 osd.1 (osd.1) 146 : cluster 0 4.2 scrub ok 2024-04-06T03:28:13.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:13.847 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:13.854 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:13 smithi153 bash[23852]: cluster 2024-04-06T03:28:12.105298+0000 mgr.y (mgr.24368) 271 : cluster 0 pgmap v260: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 2.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:28:13.854 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:13 smithi153 bash[23852]: cluster 2024-04-06T03:28:12.152397+0000 osd.1 (osd.1) 145 : cluster 0 4.2 scrub starts 2024-04-06T03:28:13.854 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:13 smithi153 bash[23852]: cluster 2024-04-06T03:28:12.154953+0000 osd.1 (osd.1) 146 : cluster 0 4.2 scrub ok 2024-04-06T03:28:13.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:13.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:13.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:13.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:14.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:14.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:14.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:14.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:14.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:14.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:14.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:14.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:14.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:14.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:14.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:14.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[21467]: cluster 2024-04-06T03:28:13.333808+0000 osd.0 (osd.0) 148 : cluster 0 5.15 scrub starts 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[21467]: cluster 2024-04-06T03:28:13.337147+0000 osd.0 (osd.0) 149 : cluster 0 5.15 scrub ok 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[21467]: cluster 2024-04-06T03:28:13.694649+0000 osd.4 (osd.4) 161 : cluster 0 6.1e scrub starts 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[21467]: cluster 2024-04-06T03:28:13.696460+0000 osd.4 (osd.4) 162 : cluster 0 6.1e scrub ok 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[21467]: cluster 2024-04-06T03:28:13.773369+0000 osd.7 (osd.7) 163 : cluster 0 5.9 scrub starts 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[21467]: cluster 2024-04-06T03:28:13.775027+0000 osd.7 (osd.7) 164 : cluster 0 5.9 scrub ok 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[29221]: cluster 2024-04-06T03:28:13.333808+0000 osd.0 (osd.0) 148 : cluster 0 5.15 scrub starts 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[29221]: cluster 2024-04-06T03:28:13.337147+0000 osd.0 (osd.0) 149 : cluster 0 5.15 scrub ok 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[29221]: cluster 2024-04-06T03:28:13.694649+0000 osd.4 (osd.4) 161 : cluster 0 6.1e scrub starts 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[29221]: cluster 2024-04-06T03:28:13.696460+0000 osd.4 (osd.4) 162 : cluster 0 6.1e scrub ok 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[29221]: cluster 2024-04-06T03:28:13.773369+0000 osd.7 (osd.7) 163 : cluster 0 5.9 scrub starts 2024-04-06T03:28:14.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:14 smithi050 bash[29221]: cluster 2024-04-06T03:28:13.775027+0000 osd.7 (osd.7) 164 : cluster 0 5.9 scrub ok 2024-04-06T03:28:14.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:14.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:14.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:14 smithi153 bash[23852]: cluster 2024-04-06T03:28:13.333808+0000 osd.0 (osd.0) 148 : cluster 0 5.15 scrub starts 2024-04-06T03:28:14.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:14 smithi153 bash[23852]: cluster 2024-04-06T03:28:13.337147+0000 osd.0 (osd.0) 149 : cluster 0 5.15 scrub ok 2024-04-06T03:28:14.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:14 smithi153 bash[23852]: cluster 2024-04-06T03:28:13.694649+0000 osd.4 (osd.4) 161 : cluster 0 6.1e scrub starts 2024-04-06T03:28:14.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:14 smithi153 bash[23852]: cluster 2024-04-06T03:28:13.696460+0000 osd.4 (osd.4) 162 : cluster 0 6.1e scrub ok 2024-04-06T03:28:14.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:14 smithi153 bash[23852]: cluster 2024-04-06T03:28:13.773369+0000 osd.7 (osd.7) 163 : cluster 0 5.9 scrub starts 2024-04-06T03:28:14.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:14 smithi153 bash[23852]: cluster 2024-04-06T03:28:13.775027+0000 osd.7 (osd.7) 164 : cluster 0 5.9 scrub ok 2024-04-06T03:28:14.896 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:14.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:14.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.025 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:15.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:15.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:15.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.270 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:15.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:15.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:15.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:15.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:15.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:15.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:15.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:15 smithi050 bash[21467]: cluster 2024-04-06T03:28:14.105987+0000 mgr.y (mgr.24368) 272 : cluster 0 pgmap v261: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 2.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s 2024-04-06T03:28:15.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:15 smithi050 bash[21467]: cluster 2024-04-06T03:28:14.451000+0000 osd.3 (osd.3) 181 : cluster 0 5.13 scrub starts 2024-04-06T03:28:15.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:15 smithi050 bash[21467]: cluster 2024-04-06T03:28:14.454603+0000 osd.3 (osd.3) 182 : cluster 0 5.13 scrub ok 2024-04-06T03:28:15.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:15 smithi050 bash[29221]: cluster 2024-04-06T03:28:14.105987+0000 mgr.y (mgr.24368) 272 : cluster 0 pgmap v261: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 2.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s 2024-04-06T03:28:15.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:15 smithi050 bash[29221]: cluster 2024-04-06T03:28:14.451000+0000 osd.3 (osd.3) 181 : cluster 0 5.13 scrub starts 2024-04-06T03:28:15.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:15 smithi050 bash[29221]: cluster 2024-04-06T03:28:14.454603+0000 osd.3 (osd.3) 182 : cluster 0 5.13 scrub ok 2024-04-06T03:28:15.777 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:15.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:15 smithi153 bash[23852]: cluster 2024-04-06T03:28:14.105987+0000 mgr.y (mgr.24368) 272 : cluster 0 pgmap v261: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 2.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s 2024-04-06T03:28:15.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:15 smithi153 bash[23852]: cluster 2024-04-06T03:28:14.451000+0000 osd.3 (osd.3) 181 : cluster 0 5.13 scrub starts 2024-04-06T03:28:15.893 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:15 smithi153 bash[23852]: cluster 2024-04-06T03:28:14.454603+0000 osd.3 (osd.3) 182 : cluster 0 5.13 scrub ok 2024-04-06T03:28:15.906 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:15.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:16.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:16.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:16.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:16.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:16.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:16.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:16.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:16.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:16.557 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:16.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:16.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:16.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:16.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:16 smithi050 bash[21467]: cluster 2024-04-06T03:28:15.760938+0000 osd.7 (osd.7) 165 : cluster 0 3.11 scrub starts 2024-04-06T03:28:16.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:16 smithi050 bash[21467]: cluster 2024-04-06T03:28:15.764229+0000 osd.7 (osd.7) 166 : cluster 0 3.11 scrub ok 2024-04-06T03:28:16.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:16 smithi050 bash[29221]: cluster 2024-04-06T03:28:15.760938+0000 osd.7 (osd.7) 165 : cluster 0 3.11 scrub starts 2024-04-06T03:28:16.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:16 smithi050 bash[29221]: cluster 2024-04-06T03:28:15.764229+0000 osd.7 (osd.7) 166 : cluster 0 3.11 scrub ok 2024-04-06T03:28:16.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:16.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:16 smithi153 bash[23852]: cluster 2024-04-06T03:28:15.760938+0000 osd.7 (osd.7) 165 : cluster 0 3.11 scrub starts 2024-04-06T03:28:16.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:16 smithi153 bash[23852]: cluster 2024-04-06T03:28:15.764229+0000 osd.7 (osd.7) 166 : cluster 0 3.11 scrub ok 2024-04-06T03:28:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:16.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:16.951 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:16.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:17.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:17.080 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:17.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:17.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:17.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:17.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:17.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:17.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:17.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:17.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:17.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:17.600 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:17.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:17.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:17.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:17 smithi050 bash[21467]: cluster 2024-04-06T03:28:15.917800+0000 osd.2 (osd.2) 135 : cluster 0 6.f scrub starts 2024-04-06T03:28:17.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:17 smithi050 bash[21467]: cluster 2024-04-06T03:28:15.918996+0000 osd.2 (osd.2) 136 : cluster 0 6.f scrub ok 2024-04-06T03:28:17.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:17 smithi050 bash[21467]: cluster 2024-04-06T03:28:16.106604+0000 mgr.y (mgr.24368) 273 : cluster 0 pgmap v262: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 751 op/s 2024-04-06T03:28:17.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:17 smithi050 bash[29221]: cluster 2024-04-06T03:28:15.917800+0000 osd.2 (osd.2) 135 : cluster 0 6.f scrub starts 2024-04-06T03:28:17.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:17 smithi050 bash[29221]: cluster 2024-04-06T03:28:15.918996+0000 osd.2 (osd.2) 136 : cluster 0 6.f scrub ok 2024-04-06T03:28:17.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:17 smithi050 bash[29221]: cluster 2024-04-06T03:28:16.106604+0000 mgr.y (mgr.24368) 273 : cluster 0 pgmap v262: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 751 op/s 2024-04-06T03:28:17.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:17.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:17.871 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:17 smithi153 bash[23852]: cluster 2024-04-06T03:28:15.917800+0000 osd.2 (osd.2) 135 : cluster 0 6.f scrub starts 2024-04-06T03:28:17.871 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:17 smithi153 bash[23852]: cluster 2024-04-06T03:28:15.918996+0000 osd.2 (osd.2) 136 : cluster 0 6.f scrub ok 2024-04-06T03:28:17.871 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:17 smithi153 bash[23852]: cluster 2024-04-06T03:28:16.106604+0000 mgr.y (mgr.24368) 273 : cluster 0 pgmap v262: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 751 op/s 2024-04-06T03:28:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:17.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:17.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:18.001 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:18.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:18.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:18.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:18.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:18.403 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:18.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:18.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:18.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:18.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:18.652 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:18 smithi153 bash[23852]: cluster 2024-04-06T03:28:16.900539+0000 osd.2 (osd.2) 137 : cluster 0 2.16 scrub starts 2024-04-06T03:28:18.652 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:18 smithi153 bash[23852]: cluster 2024-04-06T03:28:16.903982+0000 osd.2 (osd.2) 138 : cluster 0 2.16 scrub ok 2024-04-06T03:28:18.665 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:18.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:18.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:18 smithi050 bash[29221]: cluster 2024-04-06T03:28:16.900539+0000 osd.2 (osd.2) 137 : cluster 0 2.16 scrub starts 2024-04-06T03:28:18.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:18 smithi050 bash[29221]: cluster 2024-04-06T03:28:16.903982+0000 osd.2 (osd.2) 138 : cluster 0 2.16 scrub ok 2024-04-06T03:28:18.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:18 smithi050 bash[21467]: cluster 2024-04-06T03:28:16.900539+0000 osd.2 (osd.2) 137 : cluster 0 2.16 scrub starts 2024-04-06T03:28:18.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:18 smithi050 bash[21467]: cluster 2024-04-06T03:28:16.903982+0000 osd.2 (osd.2) 138 : cluster 0 2.16 scrub ok 2024-04-06T03:28:18.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:18.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:18.920 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:18.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:19.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:19.052 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:19.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:19.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:19.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:19.322 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:19.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:19.437 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:28:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:28:19.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:19.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:19.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:19.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:28:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:28:19.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:19.712 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[29221]: cluster 2024-04-06T03:28:18.106956+0000 mgr.y (mgr.24368) 274 : cluster 0 pgmap v263: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 799 B/s rd, 0 B/s wr, 651 op/s 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[29221]: audit 2024-04-06T03:28:18.758142+0000 mon.a (mon.0) 1093 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[29221]: cluster 2024-04-06T03:28:18.785038+0000 osd.7 (osd.7) 167 : cluster 0 2.1d scrub starts 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[29221]: cluster 2024-04-06T03:28:18.790023+0000 osd.7 (osd.7) 168 : cluster 0 2.1d scrub ok 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[21467]: cluster 2024-04-06T03:28:18.106956+0000 mgr.y (mgr.24368) 274 : cluster 0 pgmap v263: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 799 B/s rd, 0 B/s wr, 651 op/s 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[21467]: audit 2024-04-06T03:28:18.758142+0000 mon.a (mon.0) 1093 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[21467]: cluster 2024-04-06T03:28:18.785038+0000 osd.7 (osd.7) 167 : cluster 0 2.1d scrub starts 2024-04-06T03:28:19.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:19 smithi050 bash[21467]: cluster 2024-04-06T03:28:18.790023+0000 osd.7 (osd.7) 168 : cluster 0 2.1d scrub ok 2024-04-06T03:28:19.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:19.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:19.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:19.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:20.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:19 smithi153 bash[23852]: cluster 2024-04-06T03:28:18.106956+0000 mgr.y (mgr.24368) 274 : cluster 0 pgmap v263: 148 pgs: 148 active+clean; 2.4 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 799 B/s rd, 0 B/s wr, 651 op/s 2024-04-06T03:28:20.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:19 smithi153 bash[23852]: audit 2024-04-06T03:28:18.758142+0000 mon.a (mon.0) 1093 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:20.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:19 smithi153 bash[23852]: cluster 2024-04-06T03:28:18.785038+0000 osd.7 (osd.7) 167 : cluster 0 2.1d scrub starts 2024-04-06T03:28:20.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:19 smithi153 bash[23852]: cluster 2024-04-06T03:28:18.790023+0000 osd.7 (osd.7) 168 : cluster 0 2.1d scrub ok 2024-04-06T03:28:20.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:20.113 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:20.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:20.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:20.377 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:20.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:20.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:20.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:20.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:20.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:20.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:20.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:20.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:20.896 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:18.915957+0000 osd.2 (osd.2) 139 : cluster 0 3.4 scrub starts 2024-04-06T03:28:20.896 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:18.921289+0000 osd.2 (osd.2) 140 : cluster 0 3.4 scrub ok 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: audit 2024-04-06T03:28:19.312286+0000 mgr.y (mgr.24368) 275 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.397902+0000 osd.1 (osd.1) 147 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.397972+0000 osd.1 (osd.1) 148 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.656820+0000 mon.a (mon.0) 1094 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.792139+0000 osd.0 (osd.0) 150 : cluster 0 7.d scrub starts 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.798078+0000 osd.7 (osd.7) 169 : cluster 0 3.16 scrub starts 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.801826+0000 osd.7 (osd.7) 170 : cluster 0 3.16 scrub ok 2024-04-06T03:28:20.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:20 smithi153 bash[23852]: cluster 2024-04-06T03:28:20.428095+0000 osd.3 (osd.3) 183 : cluster 0 7.3 scrub starts 2024-04-06T03:28:20.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:20.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:20.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:18.915957+0000 osd.2 (osd.2) 139 : cluster 0 3.4 scrub starts 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:18.921289+0000 osd.2 (osd.2) 140 : cluster 0 3.4 scrub ok 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: audit 2024-04-06T03:28:19.312286+0000 mgr.y (mgr.24368) 275 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.397902+0000 osd.1 (osd.1) 147 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.397972+0000 osd.1 (osd.1) 148 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.656820+0000 mon.a (mon.0) 1094 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.792139+0000 osd.0 (osd.0) 150 : cluster 0 7.d scrub starts 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.798078+0000 osd.7 (osd.7) 169 : cluster 0 3.16 scrub starts 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.801826+0000 osd.7 (osd.7) 170 : cluster 0 3.16 scrub ok 2024-04-06T03:28:20.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[29221]: cluster 2024-04-06T03:28:20.428095+0000 osd.3 (osd.3) 183 : cluster 0 7.3 scrub starts 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:18.915957+0000 osd.2 (osd.2) 139 : cluster 0 3.4 scrub starts 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:18.921289+0000 osd.2 (osd.2) 140 : cluster 0 3.4 scrub ok 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: audit 2024-04-06T03:28:19.312286+0000 mgr.y (mgr.24368) 275 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.397902+0000 osd.1 (osd.1) 147 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.397972+0000 osd.1 (osd.1) 148 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.656820+0000 mon.a (mon.0) 1094 : cluster 0 osdmap e164: 8 total, 8 up, 8 in 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.792139+0000 osd.0 (osd.0) 150 : cluster 0 7.d scrub starts 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.798078+0000 osd.7 (osd.7) 169 : cluster 0 3.16 scrub starts 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.801826+0000 osd.7 (osd.7) 170 : cluster 0 3.16 scrub ok 2024-04-06T03:28:20.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:20 smithi050 bash[21467]: cluster 2024-04-06T03:28:20.428095+0000 osd.3 (osd.3) 183 : cluster 0 7.3 scrub starts 2024-04-06T03:28:21.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:21.039 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:21.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:21.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:21.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:21.293 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:21.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:21.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:21.589 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:21.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:21.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:21.720 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.913970+0000 osd.2 (osd.2) 141 : cluster 0 5.6 scrub starts 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:19.915181+0000 osd.2 (osd.2) 142 : cluster 0 5.6 scrub ok 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:20.107618+0000 mgr.y (mgr.24368) 276 : cluster 0 pgmap v265: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 695 op/s 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:20.783980+0000 osd.0 (osd.0) 151 : cluster 0 3.1f scrub starts 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:20.790064+0000 osd.0 (osd.0) 152 : cluster 0 3.1f scrub ok 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:20.834773+0000 osd.7 (osd.7) 171 : cluster 0 3.f scrub starts 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:20.853919+0000 osd.7 (osd.7) 172 : cluster 0 3.f scrub ok 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.045571+0000 osd.3 (osd.3) 184 : cluster 0 7.3 scrub ok 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[21467]: audit 2024-04-06T03:28:21.423142+0000 mon.a (mon.0) 1095 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.913970+0000 osd.2 (osd.2) 141 : cluster 0 5.6 scrub starts 2024-04-06T03:28:21.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:19.915181+0000 osd.2 (osd.2) 142 : cluster 0 5.6 scrub ok 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:20.107618+0000 mgr.y (mgr.24368) 276 : cluster 0 pgmap v265: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 695 op/s 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:20.783980+0000 osd.0 (osd.0) 151 : cluster 0 3.1f scrub starts 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:20.790064+0000 osd.0 (osd.0) 152 : cluster 0 3.1f scrub ok 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:20.834773+0000 osd.7 (osd.7) 171 : cluster 0 3.f scrub starts 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:20.853919+0000 osd.7 (osd.7) 172 : cluster 0 3.f scrub ok 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.045571+0000 osd.3 (osd.3) 184 : cluster 0 7.3 scrub ok 2024-04-06T03:28:21.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:21 smithi050 bash[29221]: audit 2024-04-06T03:28:21.423142+0000 mon.a (mon.0) 1095 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:21.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:21.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:21.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:21.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:21.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.913970+0000 osd.2 (osd.2) 141 : cluster 0 5.6 scrub starts 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:19.915181+0000 osd.2 (osd.2) 142 : cluster 0 5.6 scrub ok 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:20.107618+0000 mgr.y (mgr.24368) 276 : cluster 0 pgmap v265: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 695 op/s 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:20.783980+0000 osd.0 (osd.0) 151 : cluster 0 3.1f scrub starts 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:20.790064+0000 osd.0 (osd.0) 152 : cluster 0 3.1f scrub ok 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:20.834773+0000 osd.7 (osd.7) 171 : cluster 0 3.f scrub starts 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:20.853919+0000 osd.7 (osd.7) 172 : cluster 0 3.f scrub ok 2024-04-06T03:28:22.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.045571+0000 osd.3 (osd.3) 184 : cluster 0 7.3 scrub ok 2024-04-06T03:28:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:21 smithi153 bash[23852]: audit 2024-04-06T03:28:21.423142+0000 mon.a (mon.0) 1095 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:28:22.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:22.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:22.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:22.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:22.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:22.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:22.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:22.514 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:22.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:22.637 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:22.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.132451+0000 osd.0 (osd.0) 153 : cluster 0 7.d scrub ok 2024-04-06T03:28:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.163052+0000 osd.1 (osd.1) 149 : cluster 0 2.d scrub starts 2024-04-06T03:28:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.171933+0000 osd.1 (osd.1) 150 : cluster 0 2.d scrub ok 2024-04-06T03:28:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.881354+0000 osd.0 (osd.0) 154 : cluster 0 2.13 scrub starts 2024-04-06T03:28:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.887735+0000 osd.0 (osd.0) 155 : cluster 0 2.13 scrub ok 2024-04-06T03:28:22.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: cluster 2024-04-06T03:28:22.108019+0000 mgr.y (mgr.24368) 277 : cluster 0 pgmap v266: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 696 op/s 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: audit 2024-04-06T03:28:22.200810+0000 mon.a (mon.0) 1096 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: audit 2024-04-06T03:28:22.202250+0000 mon.a (mon.0) 1097 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[21467]: audit 2024-04-06T03:28:22.209324+0000 mon.a (mon.0) 1098 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.132451+0000 osd.0 (osd.0) 153 : cluster 0 7.d scrub ok 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.163052+0000 osd.1 (osd.1) 149 : cluster 0 2.d scrub starts 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.171933+0000 osd.1 (osd.1) 150 : cluster 0 2.d scrub ok 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.881354+0000 osd.0 (osd.0) 154 : cluster 0 2.13 scrub starts 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.887735+0000 osd.0 (osd.0) 155 : cluster 0 2.13 scrub ok 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: cluster 2024-04-06T03:28:22.108019+0000 mgr.y (mgr.24368) 277 : cluster 0 pgmap v266: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 696 op/s 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: audit 2024-04-06T03:28:22.200810+0000 mon.a (mon.0) 1096 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:22.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: audit 2024-04-06T03:28:22.202250+0000 mon.a (mon.0) 1097 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:28:22.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:22 smithi050 bash[29221]: audit 2024-04-06T03:28:22.209324+0000 mon.a (mon.0) 1098 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:28:22.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:22.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:22.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:22.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:22.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:22.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:23.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.132451+0000 osd.0 (osd.0) 153 : cluster 0 7.d scrub ok 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.163052+0000 osd.1 (osd.1) 149 : cluster 0 2.d scrub starts 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.171933+0000 osd.1 (osd.1) 150 : cluster 0 2.d scrub ok 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.881354+0000 osd.0 (osd.0) 154 : cluster 0 2.13 scrub starts 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.887735+0000 osd.0 (osd.0) 155 : cluster 0 2.13 scrub ok 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: cluster 2024-04-06T03:28:22.108019+0000 mgr.y (mgr.24368) 277 : cluster 0 pgmap v266: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 696 op/s 2024-04-06T03:28:23.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: audit 2024-04-06T03:28:22.200810+0000 mon.a (mon.0) 1096 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:28:23.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: audit 2024-04-06T03:28:22.202250+0000 mon.a (mon.0) 1097 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:28:23.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:22 smithi153 bash[23852]: audit 2024-04-06T03:28:22.209324+0000 mon.a (mon.0) 1098 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:28:23.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:23.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:23.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:23.287 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:23.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:23.445 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:23.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:23.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:23.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:23.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:23.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:23.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:23.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:23 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.929409+0000 osd.2 (osd.2) 143 : cluster 0 5.8 scrub starts 2024-04-06T03:28:23.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:23 smithi050 bash[21467]: cluster 2024-04-06T03:28:21.946756+0000 osd.2 (osd.2) 144 : cluster 0 5.8 scrub ok 2024-04-06T03:28:23.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:23 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.929409+0000 osd.2 (osd.2) 143 : cluster 0 5.8 scrub starts 2024-04-06T03:28:23.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:23 smithi050 bash[29221]: cluster 2024-04-06T03:28:21.946756+0000 osd.2 (osd.2) 144 : cluster 0 5.8 scrub ok 2024-04-06T03:28:23.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:23.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:24.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:23 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.929409+0000 osd.2 (osd.2) 143 : cluster 0 5.8 scrub starts 2024-04-06T03:28:24.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:23 smithi153 bash[23852]: cluster 2024-04-06T03:28:21.946756+0000 osd.2 (osd.2) 144 : cluster 0 5.8 scrub ok 2024-04-06T03:28:24.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:24.092 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:24.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:24.225 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:28:24.225 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:28:24.226 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:24.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:24.361 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:24.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:24.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:24.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:24.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:24.679 INFO:teuthology.orchestra.run.smithi050.stderr:set 6.16 pg_upmap mapping to [3,0,6] 2024-04-06T03:28:24.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[29221]: cluster 2024-04-06T03:28:22.966568+0000 osd.2 (osd.2) 145 : cluster 0 5.b scrub starts 2024-04-06T03:28:24.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[29221]: cluster 2024-04-06T03:28:22.970564+0000 osd.2 (osd.2) 146 : cluster 0 5.b scrub ok 2024-04-06T03:28:24.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[29221]: cluster 2024-04-06T03:28:24.108659+0000 mgr.y (mgr.24368) 278 : cluster 0 pgmap v267: 148 pgs: 148 active+clean; 2.2 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 697 op/s 2024-04-06T03:28:24.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[29221]: audit 2024-04-06T03:28:24.354721+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.50:0/3948919792' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]: dispatch 2024-04-06T03:28:24.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[29221]: audit 2024-04-06T03:28:24.355236+0000 mon.a (mon.0) 1099 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]: dispatch 2024-04-06T03:28:24.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[21467]: cluster 2024-04-06T03:28:22.966568+0000 osd.2 (osd.2) 145 : cluster 0 5.b scrub starts 2024-04-06T03:28:24.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[21467]: cluster 2024-04-06T03:28:22.970564+0000 osd.2 (osd.2) 146 : cluster 0 5.b scrub ok 2024-04-06T03:28:24.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[21467]: cluster 2024-04-06T03:28:24.108659+0000 mgr.y (mgr.24368) 278 : cluster 0 pgmap v267: 148 pgs: 148 active+clean; 2.2 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 697 op/s 2024-04-06T03:28:24.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[21467]: audit 2024-04-06T03:28:24.354721+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.50:0/3948919792' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]: dispatch 2024-04-06T03:28:24.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:24 smithi050 bash[21467]: audit 2024-04-06T03:28:24.355236+0000 mon.a (mon.0) 1099 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]: dispatch 2024-04-06T03:28:24.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:24.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:24.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:24.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:24.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:25.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.035 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:24 smithi153 bash[23852]: cluster 2024-04-06T03:28:22.966568+0000 osd.2 (osd.2) 145 : cluster 0 5.b scrub starts 2024-04-06T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:24 smithi153 bash[23852]: cluster 2024-04-06T03:28:22.970564+0000 osd.2 (osd.2) 146 : cluster 0 5.b scrub ok 2024-04-06T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:24 smithi153 bash[23852]: cluster 2024-04-06T03:28:24.108659+0000 mgr.y (mgr.24368) 278 : cluster 0 pgmap v267: 148 pgs: 148 active+clean; 2.2 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 697 op/s 2024-04-06T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:24 smithi153 bash[23852]: audit 2024-04-06T03:28:24.354721+0000 mon.c (mon.2) 48 : audit 1 from='client.? 172.21.15.50:0/3948919792' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]: dispatch 2024-04-06T03:28:25.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:24 smithi153 bash[23852]: audit 2024-04-06T03:28:24.355236+0000 mon.a (mon.0) 1099 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]: dispatch 2024-04-06T03:28:25.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:25.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:25.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:25.453 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:25.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:25.587 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:25.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:25.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:25.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:25 smithi153 bash[23852]: audit 2024-04-06T03:28:24.667504+0000 mon.a (mon.0) 1100 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]': finished 2024-04-06T03:28:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:25 smithi153 bash[23852]: cluster 2024-04-06T03:28:24.672167+0000 mon.a (mon.0) 1101 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-04-06T03:28:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:25 smithi153 bash[23852]: cluster 2024-04-06T03:28:24.884672+0000 osd.0 (osd.0) 156 : cluster 0 7.4 scrub starts 2024-04-06T03:28:25.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:25.839 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:25.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:25 smithi050 bash[21467]: audit 2024-04-06T03:28:24.667504+0000 mon.a (mon.0) 1100 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]': finished 2024-04-06T03:28:25.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:25 smithi050 bash[21467]: cluster 2024-04-06T03:28:24.672167+0000 mon.a (mon.0) 1101 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-04-06T03:28:25.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:25 smithi050 bash[21467]: cluster 2024-04-06T03:28:24.884672+0000 osd.0 (osd.0) 156 : cluster 0 7.4 scrub starts 2024-04-06T03:28:25.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:25 smithi050 bash[29221]: audit 2024-04-06T03:28:24.667504+0000 mon.a (mon.0) 1100 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.16", "id": [3, 0, 6]}]': finished 2024-04-06T03:28:25.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:25 smithi050 bash[29221]: cluster 2024-04-06T03:28:24.672167+0000 mon.a (mon.0) 1101 : cluster 0 osdmap e165: 8 total, 8 up, 8 in 2024-04-06T03:28:25.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:25 smithi050 bash[29221]: cluster 2024-04-06T03:28:24.884672+0000 osd.0 (osd.0) 156 : cluster 0 7.4 scrub starts 2024-04-06T03:28:25.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:25.967 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:25.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:26.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:26.093 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:26.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:26.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:26.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:26.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:26.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:26.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:26.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:26.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:26.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:26.718 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:28:26.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:26.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:26.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[21467]: cluster 2024-04-06T03:28:24.768809+0000 osd.4 (osd.4) 163 : cluster 0 6.1d scrub starts 2024-04-06T03:28:26.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[21467]: cluster 2024-04-06T03:28:24.772374+0000 osd.4 (osd.4) 164 : cluster 0 6.1d scrub ok 2024-04-06T03:28:26.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[21467]: cluster 2024-04-06T03:28:25.671624+0000 mon.a (mon.0) 1102 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.109034+0000 mgr.y (mgr.24368) 279 : cluster 0 pgmap v270: 148 pgs: 1 peering, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 742 op/s 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[21467]: audit 2024-04-06T03:28:26.448642+0000 mon.a (mon.0) 1103 : audit 1 from='client.? 172.21.15.50:0/2706461357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[29221]: cluster 2024-04-06T03:28:24.768809+0000 osd.4 (osd.4) 163 : cluster 0 6.1d scrub starts 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[29221]: cluster 2024-04-06T03:28:24.772374+0000 osd.4 (osd.4) 164 : cluster 0 6.1d scrub ok 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[29221]: cluster 2024-04-06T03:28:25.671624+0000 mon.a (mon.0) 1102 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.109034+0000 mgr.y (mgr.24368) 279 : cluster 0 pgmap v270: 148 pgs: 1 peering, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 742 op/s 2024-04-06T03:28:26.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:26 smithi050 bash[29221]: audit 2024-04-06T03:28:26.448642+0000 mon.a (mon.0) 1103 : audit 1 from='client.? 172.21.15.50:0/2706461357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:28:26.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:26.871 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:26 smithi153 bash[23852]: cluster 2024-04-06T03:28:24.768809+0000 osd.4 (osd.4) 163 : cluster 0 6.1d scrub starts 2024-04-06T03:28:26.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:26 smithi153 bash[23852]: cluster 2024-04-06T03:28:24.772374+0000 osd.4 (osd.4) 164 : cluster 0 6.1d scrub ok 2024-04-06T03:28:26.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:26 smithi153 bash[23852]: cluster 2024-04-06T03:28:25.671624+0000 mon.a (mon.0) 1102 : cluster 0 osdmap e166: 8 total, 8 up, 8 in 2024-04-06T03:28:26.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:26 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.109034+0000 mgr.y (mgr.24368) 279 : cluster 0 pgmap v270: 148 pgs: 1 peering, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 742 op/s 2024-04-06T03:28:26.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:26 smithi153 bash[23852]: audit 2024-04-06T03:28:26.448642+0000 mon.a (mon.0) 1103 : audit 1 from='client.? 172.21.15.50:0/2706461357' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:28:26.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:26.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:26.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:27.012 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:27.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:27.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:27.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:27.285 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:27.308 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:28:27.334 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:28:27.354 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:28:27.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:27.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:27.481 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:28:27.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:27.514 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:28:27.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:27.561 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:28:27.601 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:28:27.637 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:28:27.646 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:28:27.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:27.677 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:27.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-06T03:28:27.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:27.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:25.758130+0000 osd.4 (osd.4) 165 : cluster 0 3.14 scrub starts 2024-04-06T03:28:27.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:25.763031+0000 osd.4 (osd.4) 166 : cluster 0 3.14 scrub ok 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.350063+0000 osd.0 (osd.0) 157 : cluster 0 7.4 scrub ok 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.675538+0000 mon.a (mon.0) 1104 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.675816+0000 mon.a (mon.0) 1105 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.675846+0000 mon.a (mon.0) 1106 : cluster 1 Cluster is now healthy 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: audit 2024-04-06T03:28:26.682975+0000 mon.a (mon.0) 1107 : audit 1 from='client.? 172.21.15.50:0/2706461357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.688307+0000 mon.a (mon.0) 1108 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[29221]: audit 2024-04-06T03:28:27.278598+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.50:0/782254970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:25.758130+0000 osd.4 (osd.4) 165 : cluster 0 3.14 scrub starts 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:25.763031+0000 osd.4 (osd.4) 166 : cluster 0 3.14 scrub ok 2024-04-06T03:28:27.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.350063+0000 osd.0 (osd.0) 157 : cluster 0 7.4 scrub ok 2024-04-06T03:28:27.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.675538+0000 mon.a (mon.0) 1104 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-06T03:28:27.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.675816+0000 mon.a (mon.0) 1105 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:28:27.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.675846+0000 mon.a (mon.0) 1106 : cluster 1 Cluster is now healthy 2024-04-06T03:28:27.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: audit 2024-04-06T03:28:26.682975+0000 mon.a (mon.0) 1107 : audit 1 from='client.? 172.21.15.50:0/2706461357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:28:27.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.688307+0000 mon.a (mon.0) 1108 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-04-06T03:28:27.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:27 smithi050 bash[21467]: audit 2024-04-06T03:28:27.278598+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.50:0/782254970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:28:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:27.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:27.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:27.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:25.758130+0000 osd.4 (osd.4) 165 : cluster 0 3.14 scrub starts 2024-04-06T03:28:27.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:25.763031+0000 osd.4 (osd.4) 166 : cluster 0 3.14 scrub ok 2024-04-06T03:28:27.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.350063+0000 osd.0 (osd.0) 157 : cluster 0 7.4 scrub ok 2024-04-06T03:28:27.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.675538+0000 mon.a (mon.0) 1104 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-06T03:28:27.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.675816+0000 mon.a (mon.0) 1105 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:28:27.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.675846+0000 mon.a (mon.0) 1106 : cluster 1 Cluster is now healthy 2024-04-06T03:28:27.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: audit 2024-04-06T03:28:26.682975+0000 mon.a (mon.0) 1107 : audit 1 from='client.? 172.21.15.50:0/2706461357' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:28:27.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.688307+0000 mon.a (mon.0) 1108 : cluster 0 osdmap e167: 8 total, 8 up, 8 in 2024-04-06T03:28:27.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:27 smithi153 bash[23852]: audit 2024-04-06T03:28:27.278598+0000 mon.c (mon.2) 49 : audit 0 from='client.? 172.21.15.50:0/782254970' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:28:27.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:27.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:28.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:28.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:28.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:28.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:28.349 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:28.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:28.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:28.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:28.618 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:28.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.796944+0000 osd.4 (osd.4) 167 : cluster 0 5.e scrub starts 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:26.798287+0000 osd.4 (osd.4) 168 : cluster 0 5.e scrub ok 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:27.095806+0000 osd.0 (osd.0) 158 : cluster 0 3.12 scrub starts 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:27.101674+0000 osd.0 (osd.0) 159 : cluster 0 3.12 scrub ok 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.109800+0000 mgr.y (mgr.24368) 280 : cluster 0 pgmap v272: 148 pgs: 1 peering, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 619 op/s 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.452334+0000 osd.3 (osd.3) 185 : cluster 0 2.19 scrub starts 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.461913+0000 osd.3 (osd.3) 186 : cluster 0 2.19 scrub ok 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.796944+0000 osd.4 (osd.4) 167 : cluster 0 5.e scrub starts 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:26.798287+0000 osd.4 (osd.4) 168 : cluster 0 5.e scrub ok 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:27.095806+0000 osd.0 (osd.0) 158 : cluster 0 3.12 scrub starts 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:27.101674+0000 osd.0 (osd.0) 159 : cluster 0 3.12 scrub ok 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.109800+0000 mgr.y (mgr.24368) 280 : cluster 0 pgmap v272: 148 pgs: 1 peering, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 619 op/s 2024-04-06T03:28:28.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.452334+0000 osd.3 (osd.3) 185 : cluster 0 2.19 scrub starts 2024-04-06T03:28:28.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:28 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.461913+0000 osd.3 (osd.3) 186 : cluster 0 2.19 scrub ok 2024-04-06T03:28:28.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:28.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.796944+0000 osd.4 (osd.4) 167 : cluster 0 5.e scrub starts 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:26.798287+0000 osd.4 (osd.4) 168 : cluster 0 5.e scrub ok 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:27.095806+0000 osd.0 (osd.0) 158 : cluster 0 3.12 scrub starts 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:27.101674+0000 osd.0 (osd.0) 159 : cluster 0 3.12 scrub ok 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.109800+0000 mgr.y (mgr.24368) 280 : cluster 0 pgmap v272: 148 pgs: 1 peering, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 619 op/s 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.452334+0000 osd.3 (osd.3) 185 : cluster 0 2.19 scrub starts 2024-04-06T03:28:28.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:28 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.461913+0000 osd.3 (osd.3) 186 : cluster 0 2.19 scrub ok 2024-04-06T03:28:28.885 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:28.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:28.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:29.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:29.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:29.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:29.272 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:29.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:29.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:29.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:29.523 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:28:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:28:29.527 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:29.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:29.639 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:28:29.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: cluster 2024-04-06T03:28:27.793714+0000 osd.4 (osd.4) 169 : cluster 0 2.18 scrub starts 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: cluster 2024-04-06T03:28:27.794662+0000 osd.4 (osd.4) 170 : cluster 0 2.18 scrub ok 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.292970+0000 osd.1 (osd.1) 151 : cluster 0 3.19 deep-scrub starts 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.306981+0000 osd.1 (osd.1) 152 : cluster 0 3.19 deep-scrub ok 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: audit 2024-04-06T03:28:29.320260+0000 mgr.y (mgr.24368) 281 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: cluster 2024-04-06T03:28:29.453084+0000 osd.3 (osd.3) 187 : cluster 0 6.1b scrub starts 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[21467]: cluster 2024-04-06T03:28:29.455587+0000 osd.3 (osd.3) 188 : cluster 0 6.1b scrub ok 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: cluster 2024-04-06T03:28:27.793714+0000 osd.4 (osd.4) 169 : cluster 0 2.18 scrub starts 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: cluster 2024-04-06T03:28:27.794662+0000 osd.4 (osd.4) 170 : cluster 0 2.18 scrub ok 2024-04-06T03:28:29.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.292970+0000 osd.1 (osd.1) 151 : cluster 0 3.19 deep-scrub starts 2024-04-06T03:28:29.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.306981+0000 osd.1 (osd.1) 152 : cluster 0 3.19 deep-scrub ok 2024-04-06T03:28:29.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: audit 2024-04-06T03:28:29.320260+0000 mgr.y (mgr.24368) 281 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:29.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: cluster 2024-04-06T03:28:29.453084+0000 osd.3 (osd.3) 187 : cluster 0 6.1b scrub starts 2024-04-06T03:28:29.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:29 smithi050 bash[29221]: cluster 2024-04-06T03:28:29.455587+0000 osd.3 (osd.3) 188 : cluster 0 6.1b scrub ok 2024-04-06T03:28:29.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:29.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: cluster 2024-04-06T03:28:27.793714+0000 osd.4 (osd.4) 169 : cluster 0 2.18 scrub starts 2024-04-06T03:28:29.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: cluster 2024-04-06T03:28:27.794662+0000 osd.4 (osd.4) 170 : cluster 0 2.18 scrub ok 2024-04-06T03:28:29.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.292970+0000 osd.1 (osd.1) 151 : cluster 0 3.19 deep-scrub starts 2024-04-06T03:28:29.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.306981+0000 osd.1 (osd.1) 152 : cluster 0 3.19 deep-scrub ok 2024-04-06T03:28:29.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: audit 2024-04-06T03:28:29.320260+0000 mgr.y (mgr.24368) 281 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:29.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: cluster 2024-04-06T03:28:29.453084+0000 osd.3 (osd.3) 187 : cluster 0 6.1b scrub starts 2024-04-06T03:28:29.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:29 smithi153 bash[23852]: cluster 2024-04-06T03:28:29.455587+0000 osd.3 (osd.3) 188 : cluster 0 6.1b scrub ok 2024-04-06T03:28:29.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:29.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:29.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:30.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:30.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:30.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:30.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:30.314 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:30.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:30.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:30.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:30.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:30.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:30.715 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:30.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:30.843 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:30.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:30.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.763026+0000 osd.4 (osd.4) 171 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:28.764127+0000 osd.4 (osd.4) 172 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:29.103576+0000 osd.0 (osd.0) 160 : cluster 0 7.7 scrub starts 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:29.881642+0000 osd.2 (osd.2) 147 : cluster 0 7.e scrub starts 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.066197+0000 osd.6 (osd.6) 81 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.066264+0000 osd.6 (osd.6) 82 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:30.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:30 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.110726+0000 mgr.y (mgr.24368) 282 : cluster 0 pgmap v273: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:28:30.972 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:30.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.763026+0000 osd.4 (osd.4) 171 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:28.764127+0000 osd.4 (osd.4) 172 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:29.103576+0000 osd.0 (osd.0) 160 : cluster 0 7.7 scrub starts 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:29.881642+0000 osd.2 (osd.2) 147 : cluster 0 7.e scrub starts 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.066197+0000 osd.6 (osd.6) 81 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.066264+0000 osd.6 (osd.6) 82 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.110726+0000 mgr.y (mgr.24368) 282 : cluster 0 pgmap v273: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:28:30.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.763026+0000 osd.4 (osd.4) 171 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:28:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:28.764127+0000 osd.4 (osd.4) 172 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:28:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:29.103576+0000 osd.0 (osd.0) 160 : cluster 0 7.7 scrub starts 2024-04-06T03:28:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:29.881642+0000 osd.2 (osd.2) 147 : cluster 0 7.e scrub starts 2024-04-06T03:28:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.066197+0000 osd.6 (osd.6) 81 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.066264+0000 osd.6 (osd.6) 82 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:30 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.110726+0000 mgr.y (mgr.24368) 282 : cluster 0 pgmap v273: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:28:31.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:31.108 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:31.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:31.233 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:31.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:31.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:31.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:31.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:31.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:31.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:31.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:31.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:31.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:28:31.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:31.883 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:31.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:31.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:29.754887+0000 osd.4 (osd.4) 173 : cluster 0 5.c scrub starts 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:29.756370+0000 osd.4 (osd.4) 174 : cluster 0 5.c scrub ok 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.193326+0000 osd.1 (osd.1) 153 : cluster 0 2.15 deep-scrub starts 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.204983+0000 osd.1 (osd.1) 154 : cluster 0 2.15 deep-scrub ok 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.396717+0000 osd.0 (osd.0) 161 : cluster 0 7.7 scrub ok 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.557537+0000 osd.2 (osd.2) 148 : cluster 0 7.e scrub ok 2024-04-06T03:28:31.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.700971+0000 mon.a (mon.0) 1109 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.701009+0000 mon.a (mon.0) 1110 : cluster 1 Cluster is now healthy 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.717371+0000 mon.a (mon.0) 1111 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.025757+0000 osd.7 (osd.7) 173 : cluster 0 7.5 scrub starts 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.032144+0000 osd.0 (osd.0) 162 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.042194+0000 osd.0 (osd.0) 163 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.457217+0000 osd.3 (osd.3) 189 : cluster 0 6.7 scrub starts 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.463368+0000 osd.3 (osd.3) 190 : cluster 0 6.7 scrub ok 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:29.754887+0000 osd.4 (osd.4) 173 : cluster 0 5.c scrub starts 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:29.756370+0000 osd.4 (osd.4) 174 : cluster 0 5.c scrub ok 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.193326+0000 osd.1 (osd.1) 153 : cluster 0 2.15 deep-scrub starts 2024-04-06T03:28:31.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.204983+0000 osd.1 (osd.1) 154 : cluster 0 2.15 deep-scrub ok 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.396717+0000 osd.0 (osd.0) 161 : cluster 0 7.7 scrub ok 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.557537+0000 osd.2 (osd.2) 148 : cluster 0 7.e scrub ok 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.700971+0000 mon.a (mon.0) 1109 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.701009+0000 mon.a (mon.0) 1110 : cluster 1 Cluster is now healthy 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.717371+0000 mon.a (mon.0) 1111 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.025757+0000 osd.7 (osd.7) 173 : cluster 0 7.5 scrub starts 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.032144+0000 osd.0 (osd.0) 162 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.042194+0000 osd.0 (osd.0) 163 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.457217+0000 osd.3 (osd.3) 189 : cluster 0 6.7 scrub starts 2024-04-06T03:28:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:31 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.463368+0000 osd.3 (osd.3) 190 : cluster 0 6.7 scrub ok 2024-04-06T03:28:31.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:32.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:32.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:29.754887+0000 osd.4 (osd.4) 173 : cluster 0 5.c scrub starts 2024-04-06T03:28:32.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:29.756370+0000 osd.4 (osd.4) 174 : cluster 0 5.c scrub ok 2024-04-06T03:28:32.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.193326+0000 osd.1 (osd.1) 153 : cluster 0 2.15 deep-scrub starts 2024-04-06T03:28:32.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.204983+0000 osd.1 (osd.1) 154 : cluster 0 2.15 deep-scrub ok 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.396717+0000 osd.0 (osd.0) 161 : cluster 0 7.7 scrub ok 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.557537+0000 osd.2 (osd.2) 148 : cluster 0 7.e scrub ok 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.700971+0000 mon.a (mon.0) 1109 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.701009+0000 mon.a (mon.0) 1110 : cluster 1 Cluster is now healthy 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.717371+0000 mon.a (mon.0) 1111 : cluster 0 osdmap e168: 8 total, 8 up, 8 in 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.025757+0000 osd.7 (osd.7) 173 : cluster 0 7.5 scrub starts 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.032144+0000 osd.0 (osd.0) 162 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.042194+0000 osd.0 (osd.0) 163 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.457217+0000 osd.3 (osd.3) 189 : cluster 0 6.7 scrub starts 2024-04-06T03:28:32.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:31 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.463368+0000 osd.3 (osd.3) 190 : cluster 0 6.7 scrub ok 2024-04-06T03:28:32.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:32.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:32.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:32.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:32.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:32.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:32.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:32.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:32.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:28:32.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:32.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:32.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:32.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:32.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:32.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.756783+0000 osd.4 (osd.4) 175 : cluster 0 3.1a scrub starts 2024-04-06T03:28:32.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:30.760478+0000 osd.4 (osd.4) 176 : cluster 0 3.1a scrub ok 2024-04-06T03:28:32.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.181631+0000 osd.1 (osd.1) 155 : cluster 0 2.2 scrub starts 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.183661+0000 osd.1 (osd.1) 156 : cluster 0 2.2 scrub ok 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.444969+0000 osd.7 (osd.7) 174 : cluster 0 7.5 scrub ok 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.704495+0000 osd.4 (osd.4) 177 : cluster 0 6.17 scrub starts 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.705775+0000 osd.4 (osd.4) 178 : cluster 0 6.17 scrub ok 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.889648+0000 osd.2 (osd.2) 149 : cluster 0 5.a deep-scrub starts 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:31.906513+0000 osd.2 (osd.2) 150 : cluster 0 5.a deep-scrub ok 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.076452+0000 osd.0 (osd.0) 164 : cluster 0 6.a scrub starts 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.080915+0000 osd.0 (osd.0) 165 : cluster 0 6.a scrub ok 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.111144+0000 mgr.y (mgr.24368) 283 : cluster 0 pgmap v275: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 515 op/s 2024-04-06T03:28:32.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.756783+0000 osd.4 (osd.4) 175 : cluster 0 3.1a scrub starts 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:30.760478+0000 osd.4 (osd.4) 176 : cluster 0 3.1a scrub ok 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.181631+0000 osd.1 (osd.1) 155 : cluster 0 2.2 scrub starts 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.183661+0000 osd.1 (osd.1) 156 : cluster 0 2.2 scrub ok 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.444969+0000 osd.7 (osd.7) 174 : cluster 0 7.5 scrub ok 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.704495+0000 osd.4 (osd.4) 177 : cluster 0 6.17 scrub starts 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.705775+0000 osd.4 (osd.4) 178 : cluster 0 6.17 scrub ok 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.889648+0000 osd.2 (osd.2) 149 : cluster 0 5.a deep-scrub starts 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:31.906513+0000 osd.2 (osd.2) 150 : cluster 0 5.a deep-scrub ok 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.076452+0000 osd.0 (osd.0) 164 : cluster 0 6.a scrub starts 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.080915+0000 osd.0 (osd.0) 165 : cluster 0 6.a scrub ok 2024-04-06T03:28:32.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:32 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.111144+0000 mgr.y (mgr.24368) 283 : cluster 0 pgmap v275: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 515 op/s 2024-04-06T03:28:32.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:32.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:33.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.756783+0000 osd.4 (osd.4) 175 : cluster 0 3.1a scrub starts 2024-04-06T03:28:33.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:30.760478+0000 osd.4 (osd.4) 176 : cluster 0 3.1a scrub ok 2024-04-06T03:28:33.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.181631+0000 osd.1 (osd.1) 155 : cluster 0 2.2 scrub starts 2024-04-06T03:28:33.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.183661+0000 osd.1 (osd.1) 156 : cluster 0 2.2 scrub ok 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.444969+0000 osd.7 (osd.7) 174 : cluster 0 7.5 scrub ok 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.704495+0000 osd.4 (osd.4) 177 : cluster 0 6.17 scrub starts 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.705775+0000 osd.4 (osd.4) 178 : cluster 0 6.17 scrub ok 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.889648+0000 osd.2 (osd.2) 149 : cluster 0 5.a deep-scrub starts 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:31.906513+0000 osd.2 (osd.2) 150 : cluster 0 5.a deep-scrub ok 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.076452+0000 osd.0 (osd.0) 164 : cluster 0 6.a scrub starts 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.080915+0000 osd.0 (osd.0) 165 : cluster 0 6.a scrub ok 2024-04-06T03:28:33.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:32 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.111144+0000 mgr.y (mgr.24368) 283 : cluster 0 pgmap v275: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.8 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 515 op/s 2024-04-06T03:28:33.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:33.083 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:33.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:33.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:33.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:33.351 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:33.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:33.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:33.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:33.666 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:33.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:33.775 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:33.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:33.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:33.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.130955+0000 osd.1 (osd.1) 157 : cluster 0 6.19 scrub starts 2024-04-06T03:28:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.135809+0000 osd.1 (osd.1) 158 : cluster 0 6.19 scrub ok 2024-04-06T03:28:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.492828+0000 osd.3 (osd.3) 191 : cluster 0 7.c deep-scrub starts 2024-04-06T03:28:33.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[21467]: cluster 2024-04-06T03:28:32.873305+0000 osd.2 (osd.2) 151 : cluster 0 7.8 scrub starts 2024-04-06T03:28:33.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.130955+0000 osd.1 (osd.1) 157 : cluster 0 6.19 scrub starts 2024-04-06T03:28:33.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.135809+0000 osd.1 (osd.1) 158 : cluster 0 6.19 scrub ok 2024-04-06T03:28:33.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.492828+0000 osd.3 (osd.3) 191 : cluster 0 7.c deep-scrub starts 2024-04-06T03:28:33.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:33 smithi050 bash[29221]: cluster 2024-04-06T03:28:32.873305+0000 osd.2 (osd.2) 151 : cluster 0 7.8 scrub starts 2024-04-06T03:28:34.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:33 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.130955+0000 osd.1 (osd.1) 157 : cluster 0 6.19 scrub starts 2024-04-06T03:28:34.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:33 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.135809+0000 osd.1 (osd.1) 158 : cluster 0 6.19 scrub ok 2024-04-06T03:28:34.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:33 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.492828+0000 osd.3 (osd.3) 191 : cluster 0 7.c deep-scrub starts 2024-04-06T03:28:34.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:33 smithi153 bash[23852]: cluster 2024-04-06T03:28:32.873305+0000 osd.2 (osd.2) 151 : cluster 0 7.8 scrub starts 2024-04-06T03:28:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:34.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:34.170 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:34.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:34.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:34.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.436 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:34.542 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:34.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:34.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.698 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:28:34.698 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:28:34.699 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-06T03:28:34.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-06T03:28:34.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:34.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:33.341144+0000 osd.2 (osd.2) 152 : cluster 0 7.8 scrub ok 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: audit 2024-04-06T03:28:33.775011+0000 mon.a (mon.0) 1112 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: audit 2024-04-06T03:28:33.776861+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:33.881614+0000 osd.2 (osd.2) 153 : cluster 0 2.1 scrub starts 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:33.883399+0000 osd.2 (osd.2) 154 : cluster 0 2.1 scrub ok 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.049742+0000 osd.7 (osd.7) 175 : cluster 0 2.5 deep-scrub starts 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.057994+0000 osd.7 (osd.7) 176 : cluster 0 2.5 deep-scrub ok 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.111834+0000 mgr.y (mgr.24368) 284 : cluster 0 pgmap v276: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 514 op/s 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.683203+0000 osd.4 (osd.4) 179 : cluster 0 2.11 scrub starts 2024-04-06T03:28:34.823 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:34 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.687265+0000 osd.4 (osd.4) 180 : cluster 0 2.11 scrub ok 2024-04-06T03:28:34.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:34.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:33.341144+0000 osd.2 (osd.2) 152 : cluster 0 7.8 scrub ok 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: audit 2024-04-06T03:28:33.775011+0000 mon.a (mon.0) 1112 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: audit 2024-04-06T03:28:33.776861+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:33.881614+0000 osd.2 (osd.2) 153 : cluster 0 2.1 scrub starts 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:33.883399+0000 osd.2 (osd.2) 154 : cluster 0 2.1 scrub ok 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.049742+0000 osd.7 (osd.7) 175 : cluster 0 2.5 deep-scrub starts 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.057994+0000 osd.7 (osd.7) 176 : cluster 0 2.5 deep-scrub ok 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.111834+0000 mgr.y (mgr.24368) 284 : cluster 0 pgmap v276: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 514 op/s 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.683203+0000 osd.4 (osd.4) 179 : cluster 0 2.11 scrub starts 2024-04-06T03:28:34.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.687265+0000 osd.4 (osd.4) 180 : cluster 0 2.11 scrub ok 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:33.341144+0000 osd.2 (osd.2) 152 : cluster 0 7.8 scrub ok 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: audit 2024-04-06T03:28:33.775011+0000 mon.a (mon.0) 1112 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: audit 2024-04-06T03:28:33.776861+0000 mon.a (mon.0) 1113 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:33.881614+0000 osd.2 (osd.2) 153 : cluster 0 2.1 scrub starts 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:33.883399+0000 osd.2 (osd.2) 154 : cluster 0 2.1 scrub ok 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.049742+0000 osd.7 (osd.7) 175 : cluster 0 2.5 deep-scrub starts 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.057994+0000 osd.7 (osd.7) 176 : cluster 0 2.5 deep-scrub ok 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.111834+0000 mgr.y (mgr.24368) 284 : cluster 0 pgmap v276: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 514 op/s 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.683203+0000 osd.4 (osd.4) 179 : cluster 0 2.11 scrub starts 2024-04-06T03:28:34.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:34 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.687265+0000 osd.4 (osd.4) 180 : cluster 0 2.11 scrub ok 2024-04-06T03:28:34.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:34.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:35.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:35.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:35.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:35.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:35.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:35.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:35.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:35.492 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:35.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:35.622 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:35.692 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:35.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:35.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:35.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:35.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:35.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:35.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.118322+0000 osd.0 (osd.0) 166 : cluster 0 5.1b scrub starts 2024-04-06T03:28:35.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[21467]: cluster 2024-04-06T03:28:34.122060+0000 osd.0 (osd.0) 167 : cluster 0 5.1b scrub ok 2024-04-06T03:28:35.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[21467]: cluster 2024-04-06T03:28:35.027360+0000 osd.7 (osd.7) 177 : cluster 0 6.5 scrub starts 2024-04-06T03:28:35.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[21467]: cluster 2024-04-06T03:28:35.028494+0000 osd.7 (osd.7) 178 : cluster 0 6.5 scrub ok 2024-04-06T03:28:35.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.118322+0000 osd.0 (osd.0) 166 : cluster 0 5.1b scrub starts 2024-04-06T03:28:35.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[29221]: cluster 2024-04-06T03:28:34.122060+0000 osd.0 (osd.0) 167 : cluster 0 5.1b scrub ok 2024-04-06T03:28:35.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[29221]: cluster 2024-04-06T03:28:35.027360+0000 osd.7 (osd.7) 177 : cluster 0 6.5 scrub starts 2024-04-06T03:28:35.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:35 smithi050 bash[29221]: cluster 2024-04-06T03:28:35.028494+0000 osd.7 (osd.7) 178 : cluster 0 6.5 scrub ok 2024-04-06T03:28:36.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:36.033 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:36.034 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:35 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.118322+0000 osd.0 (osd.0) 166 : cluster 0 5.1b scrub starts 2024-04-06T03:28:36.035 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:35 smithi153 bash[23852]: cluster 2024-04-06T03:28:34.122060+0000 osd.0 (osd.0) 167 : cluster 0 5.1b scrub ok 2024-04-06T03:28:36.035 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:35 smithi153 bash[23852]: cluster 2024-04-06T03:28:35.027360+0000 osd.7 (osd.7) 177 : cluster 0 6.5 scrub starts 2024-04-06T03:28:36.035 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:35 smithi153 bash[23852]: cluster 2024-04-06T03:28:35.028494+0000 osd.7 (osd.7) 178 : cluster 0 6.5 scrub ok 2024-04-06T03:28:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:36.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:36.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:36.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:36.317 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:36.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:36.444 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:36.548 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:36.568 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:36.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:36.694 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:36.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:36.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:36.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:36 smithi153 bash[23852]: cluster 2024-04-06T03:28:35.578213+0000 osd.3 (osd.3) 192 : cluster 0 7.a scrub starts 2024-04-06T03:28:36.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:36 smithi153 bash[23852]: cluster 2024-04-06T03:28:36.112802+0000 mgr.y (mgr.24368) 285 : cluster 0 pgmap v277: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 808 op/s 2024-04-06T03:28:36.826 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:36 smithi153 bash[23852]: cluster 2024-04-06T03:28:36.362727+0000 osd.3 (osd.3) 193 : cluster 0 7.a scrub ok 2024-04-06T03:28:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:36.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:36.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:37.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:37.075 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:36 smithi050 bash[21467]: cluster 2024-04-06T03:28:35.578213+0000 osd.3 (osd.3) 192 : cluster 0 7.a scrub starts 2024-04-06T03:28:37.075 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:36 smithi050 bash[21467]: cluster 2024-04-06T03:28:36.112802+0000 mgr.y (mgr.24368) 285 : cluster 0 pgmap v277: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 808 op/s 2024-04-06T03:28:37.075 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:36 smithi050 bash[21467]: cluster 2024-04-06T03:28:36.362727+0000 osd.3 (osd.3) 193 : cluster 0 7.a scrub ok 2024-04-06T03:28:37.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:36 smithi050 bash[29221]: cluster 2024-04-06T03:28:35.578213+0000 osd.3 (osd.3) 192 : cluster 0 7.a scrub starts 2024-04-06T03:28:37.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:36 smithi050 bash[29221]: cluster 2024-04-06T03:28:36.112802+0000 mgr.y (mgr.24368) 285 : cluster 0 pgmap v277: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 808 op/s 2024-04-06T03:28:37.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:36 smithi050 bash[29221]: cluster 2024-04-06T03:28:36.362727+0000 osd.3 (osd.3) 193 : cluster 0 7.a scrub ok 2024-04-06T03:28:37.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:37.195 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:37.218 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:37.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:37.342 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:37.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:37.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:37.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:37.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:37.720 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:37.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:37.848 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:37.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:37.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:37.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:38.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:37 smithi153 bash[23852]: cluster 2024-04-06T03:28:36.946002+0000 osd.7 (osd.7) 179 : cluster 0 6.15 deep-scrub starts 2024-04-06T03:28:38.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:37 smithi153 bash[23852]: cluster 2024-04-06T03:28:36.947223+0000 osd.7 (osd.7) 180 : cluster 0 6.15 deep-scrub ok 2024-04-06T03:28:38.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:38.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:38.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:38.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:37 smithi050 bash[21467]: cluster 2024-04-06T03:28:36.946002+0000 osd.7 (osd.7) 179 : cluster 0 6.15 deep-scrub starts 2024-04-06T03:28:38.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:37 smithi050 bash[21467]: cluster 2024-04-06T03:28:36.947223+0000 osd.7 (osd.7) 180 : cluster 0 6.15 deep-scrub ok 2024-04-06T03:28:38.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:37 smithi050 bash[29221]: cluster 2024-04-06T03:28:36.946002+0000 osd.7 (osd.7) 179 : cluster 0 6.15 deep-scrub starts 2024-04-06T03:28:38.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:37 smithi050 bash[29221]: cluster 2024-04-06T03:28:36.947223+0000 osd.7 (osd.7) 180 : cluster 0 6.15 deep-scrub ok 2024-04-06T03:28:38.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:38.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:38.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:38.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:38.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:38.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:38.747 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:28:38.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:38.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:38.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:38.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:38 smithi153 bash[23852]: cluster 2024-04-06T03:28:37.626494+0000 osd.3 (osd.3) 194 : cluster 0 3.c scrub starts 2024-04-06T03:28:38.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:38 smithi153 bash[23852]: cluster 2024-04-06T03:28:37.635104+0000 osd.3 (osd.3) 195 : cluster 0 3.c scrub ok 2024-04-06T03:28:38.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:38 smithi153 bash[23852]: cluster 2024-04-06T03:28:37.844072+0000 osd.4 (osd.4) 181 : cluster 0 5.1c scrub starts 2024-04-06T03:28:38.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:38 smithi153 bash[23852]: cluster 2024-04-06T03:28:37.848848+0000 osd.4 (osd.4) 182 : cluster 0 5.1c scrub ok 2024-04-06T03:28:38.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:38 smithi153 bash[23852]: cluster 2024-04-06T03:28:38.113271+0000 mgr.y (mgr.24368) 286 : cluster 0 pgmap v278: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-04-06T03:28:38.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:38.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:39.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:39.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:39.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.165 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:39.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[29221]: cluster 2024-04-06T03:28:37.626494+0000 osd.3 (osd.3) 194 : cluster 0 3.c scrub starts 2024-04-06T03:28:39.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[29221]: cluster 2024-04-06T03:28:37.635104+0000 osd.3 (osd.3) 195 : cluster 0 3.c scrub ok 2024-04-06T03:28:39.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[29221]: cluster 2024-04-06T03:28:37.844072+0000 osd.4 (osd.4) 181 : cluster 0 5.1c scrub starts 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[29221]: cluster 2024-04-06T03:28:37.848848+0000 osd.4 (osd.4) 182 : cluster 0 5.1c scrub ok 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[29221]: cluster 2024-04-06T03:28:38.113271+0000 mgr.y (mgr.24368) 286 : cluster 0 pgmap v278: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[21467]: cluster 2024-04-06T03:28:37.626494+0000 osd.3 (osd.3) 194 : cluster 0 3.c scrub starts 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[21467]: cluster 2024-04-06T03:28:37.635104+0000 osd.3 (osd.3) 195 : cluster 0 3.c scrub ok 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[21467]: cluster 2024-04-06T03:28:37.844072+0000 osd.4 (osd.4) 181 : cluster 0 5.1c scrub starts 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[21467]: cluster 2024-04-06T03:28:37.848848+0000 osd.4 (osd.4) 182 : cluster 0 5.1c scrub ok 2024-04-06T03:28:39.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:38 smithi050 bash[21467]: cluster 2024-04-06T03:28:38.113271+0000 mgr.y (mgr.24368) 286 : cluster 0 pgmap v278: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.5 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-04-06T03:28:39.294 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:39.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:39.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:39.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:28:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:28:39.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:39.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:39.733 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:28:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:28:39.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:39.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:39.926 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.933 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:39 smithi153 bash[23852]: cluster 2024-04-06T03:28:38.691699+0000 osd.3 (osd.3) 196 : cluster 0 7.2 scrub starts 2024-04-06T03:28:39.933 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:39 smithi153 bash[23852]: cluster 2024-04-06T03:28:38.992705+0000 osd.7 (osd.7) 181 : cluster 0 5.1a scrub starts 2024-04-06T03:28:39.933 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:39 smithi153 bash[23852]: cluster 2024-04-06T03:28:38.995889+0000 osd.7 (osd.7) 182 : cluster 0 5.1a scrub ok 2024-04-06T03:28:39.934 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:39 smithi153 bash[23852]: audit 2024-04-06T03:28:39.328021+0000 mgr.y (mgr.24368) 287 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:39.946 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:39.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:40.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:40.076 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:40.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:40.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:40.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[21467]: cluster 2024-04-06T03:28:38.691699+0000 osd.3 (osd.3) 196 : cluster 0 7.2 scrub starts 2024-04-06T03:28:40.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[21467]: cluster 2024-04-06T03:28:38.992705+0000 osd.7 (osd.7) 181 : cluster 0 5.1a scrub starts 2024-04-06T03:28:40.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[21467]: cluster 2024-04-06T03:28:38.995889+0000 osd.7 (osd.7) 182 : cluster 0 5.1a scrub ok 2024-04-06T03:28:40.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[21467]: audit 2024-04-06T03:28:39.328021+0000 mgr.y (mgr.24368) 287 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:40.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[29221]: cluster 2024-04-06T03:28:38.691699+0000 osd.3 (osd.3) 196 : cluster 0 7.2 scrub starts 2024-04-06T03:28:40.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[29221]: cluster 2024-04-06T03:28:38.992705+0000 osd.7 (osd.7) 181 : cluster 0 5.1a scrub starts 2024-04-06T03:28:40.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[29221]: cluster 2024-04-06T03:28:38.995889+0000 osd.7 (osd.7) 182 : cluster 0 5.1a scrub ok 2024-04-06T03:28:40.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:39 smithi050 bash[29221]: audit 2024-04-06T03:28:39.328021+0000 mgr.y (mgr.24368) 287 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:40.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:40.327 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:40.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:40.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:40.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:40.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:40.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:40.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:40.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:40.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:40.941 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:40.966 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:40.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:41.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:41.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:41.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:39.303195+0000 osd.1 (osd.1) 159 : cluster 0 6.4 scrub starts 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:39.309953+0000 osd.1 (osd.1) 160 : cluster 0 6.4 scrub ok 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:39.619477+0000 osd.3 (osd.3) 197 : cluster 0 7.2 scrub ok 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:39.957681+0000 osd.7 (osd.7) 183 : cluster 0 3.13 deep-scrub starts 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:39.973799+0000 osd.7 (osd.7) 184 : cluster 0 3.13 deep-scrub ok 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:40.114053+0000 mgr.y (mgr.24368) 288 : cluster 0 pgmap v279: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 798 op/s 2024-04-06T03:28:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:40 smithi153 bash[23852]: cluster 2024-04-06T03:28:40.285403+0000 osd.3 (osd.3) 198 : cluster 0 7.c deep-scrub ok 2024-04-06T03:28:41.228 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:41.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:41.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:39.303195+0000 osd.1 (osd.1) 159 : cluster 0 6.4 scrub starts 2024-04-06T03:28:41.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:39.309953+0000 osd.1 (osd.1) 160 : cluster 0 6.4 scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:39.619477+0000 osd.3 (osd.3) 197 : cluster 0 7.2 scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:39.957681+0000 osd.7 (osd.7) 183 : cluster 0 3.13 deep-scrub starts 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:39.973799+0000 osd.7 (osd.7) 184 : cluster 0 3.13 deep-scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:40.114053+0000 mgr.y (mgr.24368) 288 : cluster 0 pgmap v279: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 798 op/s 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:40.285403+0000 osd.3 (osd.3) 198 : cluster 0 7.c deep-scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:39.303195+0000 osd.1 (osd.1) 159 : cluster 0 6.4 scrub starts 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:39.309953+0000 osd.1 (osd.1) 160 : cluster 0 6.4 scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:39.619477+0000 osd.3 (osd.3) 197 : cluster 0 7.2 scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:39.957681+0000 osd.7 (osd.7) 183 : cluster 0 3.13 deep-scrub starts 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:39.973799+0000 osd.7 (osd.7) 184 : cluster 0 3.13 deep-scrub ok 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:40.114053+0000 mgr.y (mgr.24368) 288 : cluster 0 pgmap v279: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 798 op/s 2024-04-06T03:28:41.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:40.285403+0000 osd.3 (osd.3) 198 : cluster 0 7.c deep-scrub ok 2024-04-06T03:28:41.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:41.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:41.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:41.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:41.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:41.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:41.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:41.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:41.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:40.169518+0000 osd.0 (osd.0) 168 : cluster 0 3.10 scrub starts 2024-04-06T03:28:41.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:40.174412+0000 osd.0 (osd.0) 169 : cluster 0 3.10 scrub ok 2024-04-06T03:28:41.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:40.716566+0000 osd.3 (osd.3) 199 : cluster 0 3.5 deep-scrub starts 2024-04-06T03:28:41.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[21467]: cluster 2024-04-06T03:28:40.744509+0000 osd.3 (osd.3) 200 : cluster 0 3.5 deep-scrub ok 2024-04-06T03:28:41.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:40.169518+0000 osd.0 (osd.0) 168 : cluster 0 3.10 scrub starts 2024-04-06T03:28:41.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:40.174412+0000 osd.0 (osd.0) 169 : cluster 0 3.10 scrub ok 2024-04-06T03:28:41.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:40.716566+0000 osd.3 (osd.3) 199 : cluster 0 3.5 deep-scrub starts 2024-04-06T03:28:41.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:41 smithi050 bash[29221]: cluster 2024-04-06T03:28:40.744509+0000 osd.3 (osd.3) 200 : cluster 0 3.5 deep-scrub ok 2024-04-06T03:28:41.877 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:41.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:41.979 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:42.003 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:42.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:42.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:42.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:42.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:42.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:41 smithi153 bash[23852]: cluster 2024-04-06T03:28:40.169518+0000 osd.0 (osd.0) 168 : cluster 0 3.10 scrub starts 2024-04-06T03:28:42.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:41 smithi153 bash[23852]: cluster 2024-04-06T03:28:40.174412+0000 osd.0 (osd.0) 169 : cluster 0 3.10 scrub ok 2024-04-06T03:28:42.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:41 smithi153 bash[23852]: cluster 2024-04-06T03:28:40.716566+0000 osd.3 (osd.3) 199 : cluster 0 3.5 deep-scrub starts 2024-04-06T03:28:42.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:41 smithi153 bash[23852]: cluster 2024-04-06T03:28:40.744509+0000 osd.3 (osd.3) 200 : cluster 0 3.5 deep-scrub ok 2024-04-06T03:28:42.261 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:42.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:42.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:42.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:42.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:42.516 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:42.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:42.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:42.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:42.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:42.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:42.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:42.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:43.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:41.186670+0000 osd.1 (osd.1) 161 : cluster 0 5.12 scrub starts 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:41.198501+0000 osd.1 (osd.1) 162 : cluster 0 5.12 scrub ok 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:41.911243+0000 osd.4 (osd.4) 183 : cluster 0 5.f deep-scrub starts 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:41.912376+0000 osd.4 (osd.4) 184 : cluster 0 5.f deep-scrub ok 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:42.002384+0000 osd.7 (osd.7) 185 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:42.005335+0000 osd.7 (osd.7) 186 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:28:43.021 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[21467]: cluster 2024-04-06T03:28:42.114518+0000 mgr.y (mgr.24368) 289 : cluster 0 pgmap v280: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 700 op/s 2024-04-06T03:28:43.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:41.186670+0000 osd.1 (osd.1) 161 : cluster 0 5.12 scrub starts 2024-04-06T03:28:43.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:41.198501+0000 osd.1 (osd.1) 162 : cluster 0 5.12 scrub ok 2024-04-06T03:28:43.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:41.911243+0000 osd.4 (osd.4) 183 : cluster 0 5.f deep-scrub starts 2024-04-06T03:28:43.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:41.912376+0000 osd.4 (osd.4) 184 : cluster 0 5.f deep-scrub ok 2024-04-06T03:28:43.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:42.002384+0000 osd.7 (osd.7) 185 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:28:43.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:42.005335+0000 osd.7 (osd.7) 186 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:28:43.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:42 smithi050 bash[29221]: cluster 2024-04-06T03:28:42.114518+0000 mgr.y (mgr.24368) 289 : cluster 0 pgmap v280: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 700 op/s 2024-04-06T03:28:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:41.186670+0000 osd.1 (osd.1) 161 : cluster 0 5.12 scrub starts 2024-04-06T03:28:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:41.198501+0000 osd.1 (osd.1) 162 : cluster 0 5.12 scrub ok 2024-04-06T03:28:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:41.911243+0000 osd.4 (osd.4) 183 : cluster 0 5.f deep-scrub starts 2024-04-06T03:28:43.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:41.912376+0000 osd.4 (osd.4) 184 : cluster 0 5.f deep-scrub ok 2024-04-06T03:28:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:42.002384+0000 osd.7 (osd.7) 185 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:28:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:42.005335+0000 osd.7 (osd.7) 186 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:28:43.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:42 smithi153 bash[23852]: cluster 2024-04-06T03:28:42.114518+0000 mgr.y (mgr.24368) 289 : cluster 0 pgmap v280: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 897 B/s rd, 0 B/s wr, 700 op/s 2024-04-06T03:28:43.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:43.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:43.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:43.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:43.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:43.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:43.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:43.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:43.601 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:43.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:43.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:43.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:43.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:43.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:43.995 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:43.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:44.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:43 smithi153 bash[23852]: cluster 2024-04-06T03:28:43.073155+0000 osd.0 (osd.0) 170 : cluster 0 6.10 scrub starts 2024-04-06T03:28:44.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:43 smithi153 bash[23852]: cluster 2024-04-06T03:28:43.074305+0000 osd.0 (osd.0) 171 : cluster 0 6.10 scrub ok 2024-04-06T03:28:44.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:44.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:44.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:44.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:43 smithi050 bash[21467]: cluster 2024-04-06T03:28:43.073155+0000 osd.0 (osd.0) 170 : cluster 0 6.10 scrub starts 2024-04-06T03:28:44.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:43 smithi050 bash[21467]: cluster 2024-04-06T03:28:43.074305+0000 osd.0 (osd.0) 171 : cluster 0 6.10 scrub ok 2024-04-06T03:28:44.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:43 smithi050 bash[29221]: cluster 2024-04-06T03:28:43.073155+0000 osd.0 (osd.0) 170 : cluster 0 6.10 scrub starts 2024-04-06T03:28:44.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:43 smithi050 bash[29221]: cluster 2024-04-06T03:28:43.074305+0000 osd.0 (osd.0) 171 : cluster 0 6.10 scrub ok 2024-04-06T03:28:44.251 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:44.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:44.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:44.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:44.491 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:44.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:44.620 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:44.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:44.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:44.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:44.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:44.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:44.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:45.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:44 smithi153 bash[23852]: cluster 2024-04-06T03:28:44.115135+0000 mgr.y (mgr.24368) 290 : cluster 0 pgmap v281: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 706 op/s 2024-04-06T03:28:45.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:45.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:45.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:45.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:44 smithi050 bash[21467]: cluster 2024-04-06T03:28:44.115135+0000 mgr.y (mgr.24368) 290 : cluster 0 pgmap v281: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 706 op/s 2024-04-06T03:28:45.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:44 smithi050 bash[29221]: cluster 2024-04-06T03:28:44.115135+0000 mgr.y (mgr.24368) 290 : cluster 0 pgmap v281: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.3 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 706 op/s 2024-04-06T03:28:45.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:45.283 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:45.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:45.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:45.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:45.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:45.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:45.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:45.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:45.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:45.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:45.942 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:45.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:46.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[29221]: cluster 2024-04-06T03:28:44.940660+0000 osd.4 (osd.4) 185 : cluster 0 5.11 scrub starts 2024-04-06T03:28:46.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[29221]: cluster 2024-04-06T03:28:44.944093+0000 osd.4 (osd.4) 186 : cluster 0 5.11 scrub ok 2024-04-06T03:28:46.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[29221]: cluster 2024-04-06T03:28:44.971600+0000 osd.2 (osd.2) 155 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:46.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[29221]: cluster 2024-04-06T03:28:44.971672+0000 osd.2 (osd.2) 156 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:46.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[21467]: cluster 2024-04-06T03:28:44.940660+0000 osd.4 (osd.4) 185 : cluster 0 5.11 scrub starts 2024-04-06T03:28:46.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[21467]: cluster 2024-04-06T03:28:44.944093+0000 osd.4 (osd.4) 186 : cluster 0 5.11 scrub ok 2024-04-06T03:28:46.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[21467]: cluster 2024-04-06T03:28:44.971600+0000 osd.2 (osd.2) 155 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:46.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:45 smithi050 bash[21467]: cluster 2024-04-06T03:28:44.971672+0000 osd.2 (osd.2) 156 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:46.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:46.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:46.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:46.189 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:45 smithi153 bash[23852]: cluster 2024-04-06T03:28:44.940660+0000 osd.4 (osd.4) 185 : cluster 0 5.11 scrub starts 2024-04-06T03:28:46.189 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:45 smithi153 bash[23852]: cluster 2024-04-06T03:28:44.944093+0000 osd.4 (osd.4) 186 : cluster 0 5.11 scrub ok 2024-04-06T03:28:46.189 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:45 smithi153 bash[23852]: cluster 2024-04-06T03:28:44.971600+0000 osd.2 (osd.2) 155 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:46.189 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:45 smithi153 bash[23852]: cluster 2024-04-06T03:28:44.971672+0000 osd.2 (osd.2) 156 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:46.204 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:46.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:46.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:46.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:46.475 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:46.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:46.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:46.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:46.730 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:46.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:46.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:46.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:46.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:46.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:47.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:46 smithi153 bash[23852]: cluster 2024-04-06T03:28:45.848659+0000 osd.3 (osd.3) 201 : cluster 0 2.1b scrub starts 2024-04-06T03:28:47.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:46 smithi153 bash[23852]: cluster 2024-04-06T03:28:45.853010+0000 osd.3 (osd.3) 202 : cluster 0 2.1b scrub ok 2024-04-06T03:28:47.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:46 smithi153 bash[23852]: cluster 2024-04-06T03:28:45.870503+0000 mon.a (mon.0) 1114 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-04-06T03:28:47.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:46 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.083594+0000 osd.0 (osd.0) 172 : cluster 0 5.3 scrub starts 2024-04-06T03:28:47.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:46 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.085669+0000 osd.0 (osd.0) 173 : cluster 0 5.3 scrub ok 2024-04-06T03:28:47.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:46 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.115558+0000 mgr.y (mgr.24368) 291 : cluster 0 pgmap v283: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 770 op/s 2024-04-06T03:28:47.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:47.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:47.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[21467]: cluster 2024-04-06T03:28:45.848659+0000 osd.3 (osd.3) 201 : cluster 0 2.1b scrub starts 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[21467]: cluster 2024-04-06T03:28:45.853010+0000 osd.3 (osd.3) 202 : cluster 0 2.1b scrub ok 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[21467]: cluster 2024-04-06T03:28:45.870503+0000 mon.a (mon.0) 1114 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.083594+0000 osd.0 (osd.0) 172 : cluster 0 5.3 scrub starts 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.085669+0000 osd.0 (osd.0) 173 : cluster 0 5.3 scrub ok 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.115558+0000 mgr.y (mgr.24368) 291 : cluster 0 pgmap v283: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 770 op/s 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[29221]: cluster 2024-04-06T03:28:45.848659+0000 osd.3 (osd.3) 201 : cluster 0 2.1b scrub starts 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[29221]: cluster 2024-04-06T03:28:45.853010+0000 osd.3 (osd.3) 202 : cluster 0 2.1b scrub ok 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[29221]: cluster 2024-04-06T03:28:45.870503+0000 mon.a (mon.0) 1114 : cluster 0 osdmap e169: 8 total, 8 up, 8 in 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.083594+0000 osd.0 (osd.0) 172 : cluster 0 5.3 scrub starts 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.085669+0000 osd.0 (osd.0) 173 : cluster 0 5.3 scrub ok 2024-04-06T03:28:47.213 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:46 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.115558+0000 mgr.y (mgr.24368) 291 : cluster 0 pgmap v283: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 770 op/s 2024-04-06T03:28:47.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:47.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:47.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:47.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:47.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:47.509 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:47.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:47.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:47.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:47.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:47.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:47.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:47.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:48.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:48.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.164005+0000 osd.1 (osd.1) 163 : cluster 0 5.7 scrub starts 2024-04-06T03:28:48.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.166908+0000 osd.1 (osd.1) 164 : cluster 0 5.7 scrub ok 2024-04-06T03:28:48.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.298505+0000 osd.7 (osd.7) 187 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:48.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.298603+0000 osd.7 (osd.7) 188 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:48.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.887007+0000 mon.a (mon.0) 1115 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-04-06T03:28:48.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.914792+0000 osd.4 (osd.4) 187 : cluster 0 2.1a scrub starts 2024-04-06T03:28:48.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:46.915843+0000 osd.4 (osd.4) 188 : cluster 0 2.1a scrub ok 2024-04-06T03:28:48.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:47.030801+0000 osd.7 (osd.7) 189 : cluster 0 5.19 scrub starts 2024-04-06T03:28:48.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:47 smithi153 bash[23852]: cluster 2024-04-06T03:28:47.032286+0000 osd.7 (osd.7) 190 : cluster 0 5.19 scrub ok 2024-04-06T03:28:48.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.164005+0000 osd.1 (osd.1) 163 : cluster 0 5.7 scrub starts 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.166908+0000 osd.1 (osd.1) 164 : cluster 0 5.7 scrub ok 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.298505+0000 osd.7 (osd.7) 187 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.298603+0000 osd.7 (osd.7) 188 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.887007+0000 mon.a (mon.0) 1115 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.914792+0000 osd.4 (osd.4) 187 : cluster 0 2.1a scrub starts 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:46.915843+0000 osd.4 (osd.4) 188 : cluster 0 2.1a scrub ok 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:47.030801+0000 osd.7 (osd.7) 189 : cluster 0 5.19 scrub starts 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[21467]: cluster 2024-04-06T03:28:47.032286+0000 osd.7 (osd.7) 190 : cluster 0 5.19 scrub ok 2024-04-06T03:28:48.145 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.164005+0000 osd.1 (osd.1) 163 : cluster 0 5.7 scrub starts 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.166908+0000 osd.1 (osd.1) 164 : cluster 0 5.7 scrub ok 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.298505+0000 osd.7 (osd.7) 187 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.298603+0000 osd.7 (osd.7) 188 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.887007+0000 mon.a (mon.0) 1115 : cluster 0 osdmap e170: 8 total, 8 up, 8 in 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.914792+0000 osd.4 (osd.4) 187 : cluster 0 2.1a scrub starts 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:46.915843+0000 osd.4 (osd.4) 188 : cluster 0 2.1a scrub ok 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:47.030801+0000 osd.7 (osd.7) 189 : cluster 0 5.19 scrub starts 2024-04-06T03:28:48.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:47 smithi050 bash[29221]: cluster 2024-04-06T03:28:47.032286+0000 osd.7 (osd.7) 190 : cluster 0 5.19 scrub ok 2024-04-06T03:28:48.156 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:48.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:48.286 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:48.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:48.416 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:48.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:48.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.680 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:48.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:48.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:48.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:47.871533+0000 osd.3 (osd.3) 203 : cluster 0 5.0 scrub starts 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:47.877509+0000 osd.3 (osd.3) 204 : cluster 0 5.0 scrub ok 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:47.943518+0000 osd.4 (osd.4) 189 : cluster 0 3.1c scrub starts 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:47.948363+0000 osd.4 (osd.4) 190 : cluster 0 3.1c scrub ok 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:48.085598+0000 osd.0 (osd.0) 174 : cluster 0 6.0 scrub starts 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:48.093012+0000 osd.0 (osd.0) 175 : cluster 0 6.0 scrub ok 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: cluster 2024-04-06T03:28:48.115909+0000 mgr.y (mgr.24368) 292 : cluster 0 pgmap v285: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 502 op/s 2024-04-06T03:28:48.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[21467]: audit 2024-04-06T03:28:48.794999+0000 mon.a (mon.0) 1116 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:47.871533+0000 osd.3 (osd.3) 203 : cluster 0 5.0 scrub starts 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:47.877509+0000 osd.3 (osd.3) 204 : cluster 0 5.0 scrub ok 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:47.943518+0000 osd.4 (osd.4) 189 : cluster 0 3.1c scrub starts 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:47.948363+0000 osd.4 (osd.4) 190 : cluster 0 3.1c scrub ok 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:48.085598+0000 osd.0 (osd.0) 174 : cluster 0 6.0 scrub starts 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:48.093012+0000 osd.0 (osd.0) 175 : cluster 0 6.0 scrub ok 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: cluster 2024-04-06T03:28:48.115909+0000 mgr.y (mgr.24368) 292 : cluster 0 pgmap v285: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 502 op/s 2024-04-06T03:28:48.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:48 smithi050 bash[29221]: audit 2024-04-06T03:28:48.794999+0000 mon.a (mon.0) 1116 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:48.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:48.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:49.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:47.871533+0000 osd.3 (osd.3) 203 : cluster 0 5.0 scrub starts 2024-04-06T03:28:49.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:47.877509+0000 osd.3 (osd.3) 204 : cluster 0 5.0 scrub ok 2024-04-06T03:28:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:47.943518+0000 osd.4 (osd.4) 189 : cluster 0 3.1c scrub starts 2024-04-06T03:28:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:47.948363+0000 osd.4 (osd.4) 190 : cluster 0 3.1c scrub ok 2024-04-06T03:28:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:48.085598+0000 osd.0 (osd.0) 174 : cluster 0 6.0 scrub starts 2024-04-06T03:28:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:48.093012+0000 osd.0 (osd.0) 175 : cluster 0 6.0 scrub ok 2024-04-06T03:28:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: cluster 2024-04-06T03:28:48.115909+0000 mgr.y (mgr.24368) 292 : cluster 0 pgmap v285: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 502 op/s 2024-04-06T03:28:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:48 smithi153 bash[23852]: audit 2024-04-06T03:28:48.794999+0000 mon.a (mon.0) 1116 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:28:49.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:49.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:49.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:49.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:49.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:49.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:49.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:49.458 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:28:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:28:49.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:49.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:49.601 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:49.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:49.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:49.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:28:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:28:49.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:49.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:49.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:49.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:49.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[21467]: cluster 2024-04-06T03:28:48.950202+0000 osd.4 (osd.4) 191 : cluster 0 5.1f scrub starts 2024-04-06T03:28:49.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[21467]: cluster 2024-04-06T03:28:48.953721+0000 osd.4 (osd.4) 192 : cluster 0 5.1f scrub ok 2024-04-06T03:28:49.983 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[21467]: audit 2024-04-06T03:28:49.338802+0000 mgr.y (mgr.24368) 293 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:49.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[29221]: cluster 2024-04-06T03:28:48.950202+0000 osd.4 (osd.4) 191 : cluster 0 5.1f scrub starts 2024-04-06T03:28:49.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[29221]: cluster 2024-04-06T03:28:48.953721+0000 osd.4 (osd.4) 192 : cluster 0 5.1f scrub ok 2024-04-06T03:28:49.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:49 smithi050 bash[29221]: audit 2024-04-06T03:28:49.338802+0000 mgr.y (mgr.24368) 293 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:50.000 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:50.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:50.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:50.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:50.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:50.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:49 smithi153 bash[23852]: cluster 2024-04-06T03:28:48.950202+0000 osd.4 (osd.4) 191 : cluster 0 5.1f scrub starts 2024-04-06T03:28:50.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:49 smithi153 bash[23852]: cluster 2024-04-06T03:28:48.953721+0000 osd.4 (osd.4) 192 : cluster 0 5.1f scrub ok 2024-04-06T03:28:50.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:49 smithi153 bash[23852]: audit 2024-04-06T03:28:49.338802+0000 mgr.y (mgr.24368) 293 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:28:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:50.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:50.394 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:50.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:50.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:50.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:50.664 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:50.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:50.803 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:50.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:50.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:50.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:51.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:51.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:50 smithi050 bash[21467]: cluster 2024-04-06T03:28:49.999030+0000 osd.7 (osd.7) 191 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:28:51.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:50 smithi050 bash[21467]: cluster 2024-04-06T03:28:50.000209+0000 osd.7 (osd.7) 192 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:28:51.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:50 smithi050 bash[21467]: cluster 2024-04-06T03:28:50.117397+0000 mgr.y (mgr.24368) 294 : cluster 0 pgmap v286: 148 pgs: 148 active+clean; 829 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:28:51.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:50 smithi050 bash[29221]: cluster 2024-04-06T03:28:49.999030+0000 osd.7 (osd.7) 191 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:28:51.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:50 smithi050 bash[29221]: cluster 2024-04-06T03:28:50.000209+0000 osd.7 (osd.7) 192 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:28:51.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:50 smithi050 bash[29221]: cluster 2024-04-06T03:28:50.117397+0000 mgr.y (mgr.24368) 294 : cluster 0 pgmap v286: 148 pgs: 148 active+clean; 829 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:28:51.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:51.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:51.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:50 smithi153 bash[23852]: cluster 2024-04-06T03:28:49.999030+0000 osd.7 (osd.7) 191 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:28:51.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:50 smithi153 bash[23852]: cluster 2024-04-06T03:28:50.000209+0000 osd.7 (osd.7) 192 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:28:51.184 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:50 smithi153 bash[23852]: cluster 2024-04-06T03:28:50.117397+0000 mgr.y (mgr.24368) 294 : cluster 0 pgmap v286: 148 pgs: 148 active+clean; 829 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:28:51.199 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:51.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:51.330 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:51.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:51.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:51.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:51.587 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:51.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:51.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:51.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:51.850 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:51.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:51.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:51.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:52.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:51 smithi153 bash[23852]: cluster 2024-04-06T03:28:50.178822+0000 osd.1 (osd.1) 165 : cluster 0 5.1d deep-scrub starts 2024-04-06T03:28:52.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:51 smithi153 bash[23852]: cluster 2024-04-06T03:28:50.181447+0000 osd.1 (osd.1) 166 : cluster 0 5.1d deep-scrub ok 2024-04-06T03:28:52.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:52.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:52.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:51 smithi050 bash[21467]: cluster 2024-04-06T03:28:50.178822+0000 osd.1 (osd.1) 165 : cluster 0 5.1d deep-scrub starts 2024-04-06T03:28:52.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:51 smithi050 bash[21467]: cluster 2024-04-06T03:28:50.181447+0000 osd.1 (osd.1) 166 : cluster 0 5.1d deep-scrub ok 2024-04-06T03:28:52.201 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:51 smithi050 bash[29221]: cluster 2024-04-06T03:28:50.178822+0000 osd.1 (osd.1) 165 : cluster 0 5.1d deep-scrub starts 2024-04-06T03:28:52.201 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:51 smithi050 bash[29221]: cluster 2024-04-06T03:28:50.181447+0000 osd.1 (osd.1) 166 : cluster 0 5.1d deep-scrub ok 2024-04-06T03:28:52.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:52.241 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:52.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:52.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:52.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:52.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:52.623 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:52.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:52.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:52.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:52.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:52.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:52.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:53.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:53.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:53.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:53.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:52 smithi050 bash[21467]: cluster 2024-04-06T03:28:51.856157+0000 osd.3 (osd.3) 205 : cluster 0 6.13 scrub starts 2024-04-06T03:28:53.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:52 smithi050 bash[21467]: cluster 2024-04-06T03:28:51.861801+0000 osd.3 (osd.3) 206 : cluster 0 6.13 scrub ok 2024-04-06T03:28:53.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:52 smithi050 bash[21467]: cluster 2024-04-06T03:28:52.117947+0000 mgr.y (mgr.24368) 295 : cluster 0 pgmap v287: 148 pgs: 148 active+clean; 829 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 955 op/s 2024-04-06T03:28:53.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:52 smithi050 bash[29221]: cluster 2024-04-06T03:28:51.856157+0000 osd.3 (osd.3) 205 : cluster 0 6.13 scrub starts 2024-04-06T03:28:53.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:52 smithi050 bash[29221]: cluster 2024-04-06T03:28:51.861801+0000 osd.3 (osd.3) 206 : cluster 0 6.13 scrub ok 2024-04-06T03:28:53.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:52 smithi050 bash[29221]: cluster 2024-04-06T03:28:52.117947+0000 mgr.y (mgr.24368) 295 : cluster 0 pgmap v287: 148 pgs: 148 active+clean; 829 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 955 op/s 2024-04-06T03:28:53.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:53.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:52 smithi153 bash[23852]: cluster 2024-04-06T03:28:51.856157+0000 osd.3 (osd.3) 205 : cluster 0 6.13 scrub starts 2024-04-06T03:28:53.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:52 smithi153 bash[23852]: cluster 2024-04-06T03:28:51.861801+0000 osd.3 (osd.3) 206 : cluster 0 6.13 scrub ok 2024-04-06T03:28:53.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:52 smithi153 bash[23852]: cluster 2024-04-06T03:28:52.117947+0000 mgr.y (mgr.24368) 295 : cluster 0 pgmap v287: 148 pgs: 148 active+clean; 829 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 955 op/s 2024-04-06T03:28:53.159 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:53.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:53.390 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:28:53.394 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:53.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:53.434 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:28:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:28:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:28:53.434 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:28:53.435 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:28:53.444 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:28:53.516 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:28:53.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:53.570 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:28:53.612 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:28:53.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:53.658 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:28:53.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:53.699 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:28:53.745 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:28:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:28:53.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:53.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:53.838 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:28:53.841 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 171, 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-06T03:10:26.969128+0000", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-06T03:28:52.895247+0000", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-06T03:20:09.072730+0000", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-06T03:19:42.545394+0000", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5799936, 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-06T03:28:53.842 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 18, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 7, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-06T03:28:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:15:48.096900+0000", 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-06T03:28:53.844 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.845 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.846 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.847 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 6, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 6, 2024-04-06T03:28:53.848 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.5, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:50.011930+0000", 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.849 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:28:53.850 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "79", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:28:53.851 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.852 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:28:53.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.3200000524520874, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.3200000524520874, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.854 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:51.687715+0000", 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.855 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "81", 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.856 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.857 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.858 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:28:53.859 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.301921+0000", 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-06T03:28:53.860 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-06T03:28:53.861 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "88", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-06T03:28:53.862 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 88, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.863 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.864 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.1699999570846558, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.810316+0000", 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.866 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:28:53.867 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "83", 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.868 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.869 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.870 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.871 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:56.321502+0000", 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.872 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "85", 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.874 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.875 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:28:53.876 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.690000057220459, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.690000057220459, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.25, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.25, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 7, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "unique_pool_0", 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:25:19.407695+0000", 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:28:53.878 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 16, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 16, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 16, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 16, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 16, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "130", 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:28:53.879 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:28:53.880 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:28:53.881 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 2.2599999904632568, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 2.2599999904632568, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:28:53.882 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "e607f527-4592-4acd-a47f-76575615ae1b", 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.883 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.884 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/3531754646", 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/3531754646", 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/3531754646", 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/3531754646", 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10", 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.888 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0.5613250732421875, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.890 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.891 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-06T03:28:53.892 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/4128410069", 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/4128410069", 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/4128410069", 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/4128410069", 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b", 2024-04-06T03:28:53.893 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 19, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.894 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-06T03:28:53.895 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/1307247904", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/1307247904", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/1307247904", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/1307247904", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.898 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240", 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 165, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.899 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/1397888508", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/1397888508", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/1397888508", 2024-04-06T03:28:53.903 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/1397888508", 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48", 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.904 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6800", 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6801", 2024-04-06T03:28:53.905 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6802", 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6803", 2024-04-06T03:28:53.906 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6806", 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6807", 2024-04-06T03:28:53.907 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6804", 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6805", 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:28:53.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6801/3377683473", 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6803/3377683473", 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6807/3377683473", 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6805/3377683473", 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241", 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.909 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 46, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 127, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.910 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6808", 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6809", 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6810", 2024-04-06T03:28:53.911 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6811", 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6814", 2024-04-06T03:28:53.912 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6815", 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6812", 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6813", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6809/357179117", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6811/357179117", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6815/357179117", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6813/357179117", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "98027d53-4324-44d1-89ce-8da85be58efb", 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.915 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6816", 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6817", 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6818", 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6819", 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6822", 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6823", 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.918 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6820", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6821", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6817/1552538208", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6819/1552538208", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6823/1552538208", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6821/1552538208", 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd", 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 138, 2024-04-06T03:28:53.920 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6824", 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6825", 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6826", 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6827", 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.922 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6830", 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6831", 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.923 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6828", 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6829", 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6825/2479193924", 2024-04-06T03:28:53.924 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6827/2479193924", 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6831/2479193924", 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6829/2479193924", 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.925 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:52.398069+0000", 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:19.397975+0000", 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:44.971675+0000", 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.927 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:27:59.838970+0000", 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:27:44.208049+0000", 2024-04-06T03:28:53.928 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:05.606659+0000", 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.929 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:30.066267+0000", 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:28:53.930 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:46.298606+0000", 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [ 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "6.16", 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: 3, 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: 0, 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: 6 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.931 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.1f", 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/283275195": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2738019433": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:28:53.932 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1268273764": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/567103210": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2891059051": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2170219983": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/4082841962": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/570623774": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1379780741": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2880368026": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2044358308": "2024-04-07T03:21:47.915758+0000" 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-06T03:28:53.933 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-06T03:28:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-06T03:28:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-06T03:28:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-06T03:28:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-06T03:28:53.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:28:53.935 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:28:53.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:28:53.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:53.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:54.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: cluster 2024-04-06T03:28:52.397990+0000 osd.0 (osd.0) 176 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:54.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: cluster 2024-04-06T03:28:52.398066+0000 osd.0 (osd.0) 177 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: cluster 2024-04-06T03:28:52.882911+0000 osd.3 (osd.3) 207 : cluster 0 6.11 scrub starts 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: cluster 2024-04-06T03:28:52.889173+0000 osd.3 (osd.3) 208 : cluster 0 6.11 scrub ok 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: audit 2024-04-06T03:28:52.895162+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.50:0/1399515812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: audit 2024-04-06T03:28:52.895775+0000 mon.a (mon.0) 1117 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: cluster 2024-04-06T03:28:52.922715+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: audit 2024-04-06T03:28:53.334035+0000 mon.c (mon.2) 51 : audit 0 from='client.? 172.21.15.50:0/1455669906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:28:54.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:53 smithi153 bash[23852]: audit 2024-04-06T03:28:53.840773+0000 mon.a (mon.0) 1119 : audit 0 from='client.? 172.21.15.50:0/582562183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:28:54.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:54.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:54.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:54.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: cluster 2024-04-06T03:28:52.397990+0000 osd.0 (osd.0) 176 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: cluster 2024-04-06T03:28:52.398066+0000 osd.0 (osd.0) 177 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: cluster 2024-04-06T03:28:52.882911+0000 osd.3 (osd.3) 207 : cluster 0 6.11 scrub starts 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: cluster 2024-04-06T03:28:52.889173+0000 osd.3 (osd.3) 208 : cluster 0 6.11 scrub ok 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: audit 2024-04-06T03:28:52.895162+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.50:0/1399515812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: audit 2024-04-06T03:28:52.895775+0000 mon.a (mon.0) 1117 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: cluster 2024-04-06T03:28:52.922715+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: audit 2024-04-06T03:28:53.334035+0000 mon.c (mon.2) 51 : audit 0 from='client.? 172.21.15.50:0/1455669906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:28:54.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[21467]: audit 2024-04-06T03:28:53.840773+0000 mon.a (mon.0) 1119 : audit 0 from='client.? 172.21.15.50:0/582562183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: cluster 2024-04-06T03:28:52.397990+0000 osd.0 (osd.0) 176 : cluster 0 purged_snaps scrub starts 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: cluster 2024-04-06T03:28:52.398066+0000 osd.0 (osd.0) 177 : cluster 0 purged_snaps scrub ok 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: cluster 2024-04-06T03:28:52.882911+0000 osd.3 (osd.3) 207 : cluster 0 6.11 scrub starts 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: cluster 2024-04-06T03:28:52.889173+0000 osd.3 (osd.3) 208 : cluster 0 6.11 scrub ok 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: audit 2024-04-06T03:28:52.895162+0000 mon.c (mon.2) 50 : audit 1 from='client.? 172.21.15.50:0/1399515812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: audit 2024-04-06T03:28:52.895775+0000 mon.a (mon.0) 1117 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: cluster 2024-04-06T03:28:52.922715+0000 mon.a (mon.0) 1118 : cluster 0 osdmap e171: 8 total, 8 up, 8 in 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: audit 2024-04-06T03:28:53.334035+0000 mon.c (mon.2) 51 : audit 0 from='client.? 172.21.15.50:0/1455669906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:28:54.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:53 smithi050 bash[29221]: audit 2024-04-06T03:28:53.840773+0000 mon.a (mon.0) 1119 : audit 0 from='client.? 172.21.15.50:0/582562183' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:28:54.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:54.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:54.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:54.464 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:54.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:54.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:54.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:54.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:54.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:54.967 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:54.986 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:54.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:55.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:55.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:55.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:54 smithi050 bash[21467]: cluster 2024-04-06T03:28:53.911780+0000 mon.a (mon.0) 1120 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:54 smithi050 bash[21467]: audit 2024-04-06T03:28:53.913777+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:54 smithi050 bash[21467]: cluster 2024-04-06T03:28:53.920039+0000 mon.a (mon.0) 1122 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:54 smithi050 bash[21467]: cluster 2024-04-06T03:28:54.118615+0000 mgr.y (mgr.24368) 296 : cluster 0 pgmap v290: 148 pgs: 148 active+clean; 801 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 0 B/s wr, 632 op/s 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:55 smithi050 bash[29221]: cluster 2024-04-06T03:28:53.911780+0000 mon.a (mon.0) 1120 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:55 smithi050 bash[29221]: audit 2024-04-06T03:28:53.913777+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:55 smithi050 bash[29221]: cluster 2024-04-06T03:28:53.920039+0000 mon.a (mon.0) 1122 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-04-06T03:28:55.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:55 smithi050 bash[29221]: cluster 2024-04-06T03:28:54.118615+0000 mgr.y (mgr.24368) 296 : cluster 0 pgmap v290: 148 pgs: 148 active+clean; 801 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 0 B/s wr, 632 op/s 2024-04-06T03:28:55.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:55.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:54 smithi153 bash[23852]: cluster 2024-04-06T03:28:53.911780+0000 mon.a (mon.0) 1120 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:28:55.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:54 smithi153 bash[23852]: audit 2024-04-06T03:28:53.913777+0000 mon.a (mon.0) 1121 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:28:55.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:54 smithi153 bash[23852]: cluster 2024-04-06T03:28:53.920039+0000 mon.a (mon.0) 1122 : cluster 0 osdmap e172: 8 total, 8 up, 8 in 2024-04-06T03:28:55.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:54 smithi153 bash[23852]: cluster 2024-04-06T03:28:54.118615+0000 mgr.y (mgr.24368) 296 : cluster 0 pgmap v290: 148 pgs: 148 active+clean; 801 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 707 B/s rd, 0 B/s wr, 632 op/s 2024-04-06T03:28:55.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:55.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:28:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:28:55.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:55.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:55.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:55.619 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:55.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:55.744 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:55.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:55.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:55.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:55.994 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:55.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:56.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:56.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:56.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:56.245 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:56.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-06T03:28:56.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:56.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:56.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:56.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:56.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:56.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:56.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:56.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:56.757 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:56.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:56.887 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:56.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:56.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:57.013 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:57.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:57.144 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:57.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:57 smithi050 bash[21467]: cluster 2024-04-06T03:28:56.119325+0000 mgr.y (mgr.24368) 297 : cluster 0 pgmap v291: 148 pgs: 148 active+clean; 570 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:28:57.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:57.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:57.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:57 smithi153 bash[23852]: cluster 2024-04-06T03:28:56.119325+0000 mgr.y (mgr.24368) 297 : cluster 0 pgmap v291: 148 pgs: 148 active+clean; 570 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:28:57.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:57.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:57.483 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:57 smithi050 bash[29221]: cluster 2024-04-06T03:28:56.119325+0000 mgr.y (mgr.24368) 297 : cluster 0 pgmap v291: 148 pgs: 148 active+clean; 570 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:28:57.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:57.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:28:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:28:57.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:57.700 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:57.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:57.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:57.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:57.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:58.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:58.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:58.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:58.209 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:58.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:58.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:58.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:58.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:58.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:58.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:58.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:28:58.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:58.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:58.904 INFO:tasks.thrashosds.thrasher:j is {'epoch': 171, 'fsid': '0313e520-f3c3-11ee-b649-cb9ed24678a4', 'created': '2024-04-06T03:10:26.969128+0000', 'modified': '2024-04-06T03:28:52.895247+0000', 'last_up_change': '2024-04-06T03:20:09.072730+0000', 'last_in_change': '2024-04-06T03:19:42.545394+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:15:48.096900+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-06T03:20:50.011930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-06T03:20:51.687715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-06T03:20:53.301921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-06T03:20:53.810316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-06T03:20:56.321502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-04-06T03:25:19.407695+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.259999990463257, 'score_stable': 2.259999990463257, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'e607f527-4592-4acd-a47f-76575615ae1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 3531754646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 3531754646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3531754646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 3531754646}]}, 'public_addr': '172.21.15.50:6803/3531754646', 'cluster_addr': '172.21.15.50:6805/3531754646', 'heartbeat_back_addr': '172.21.15.50:6809/3531754646', 'heartbeat_front_addr': '172.21.15.50:6807/3531754646', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2ae8f784-d36a-4633-8630-2aed04e4ff10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5613250732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 4128410069}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 4128410069}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 4128410069}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 4128410069}]}, 'public_addr': '172.21.15.50:6811/4128410069', 'cluster_addr': '172.21.15.50:6813/4128410069', 'heartbeat_back_addr': '172.21.15.50:6817/4128410069', 'heartbeat_front_addr': '172.21.15.50:6815/4128410069', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '43f31d09-5c16-4aea-8822-e0d877cc931b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1307247904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1307247904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 1307247904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1307247904}]}, 'public_addr': '172.21.15.50:6819/1307247904', 'cluster_addr': '172.21.15.50:6821/1307247904', 'heartbeat_back_addr': '172.21.15.50:6825/1307247904', 'heartbeat_front_addr': '172.21.15.50:6823/1307247904', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4c913325-442e-4ee5-94f1-8b5bca0c3240', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 165, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 1397888508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 1397888508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 1397888508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 1397888508}]}, 'public_addr': '172.21.15.50:6827/1397888508', 'cluster_addr': '172.21.15.50:6829/1397888508', 'heartbeat_back_addr': '172.21.15.50:6833/1397888508', 'heartbeat_front_addr': '172.21.15.50:6831/1397888508', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50c02bfc-c2f2-4aae-b1d5-c38a86726b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6800', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6801', 'nonce': 3377683473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6802', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6803', 'nonce': 3377683473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6806', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6807', 'nonce': 3377683473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6804', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6805', 'nonce': 3377683473}]}, 'public_addr': '172.21.15.153:6801/3377683473', 'cluster_addr': '172.21.15.153:6803/3377683473', 'heartbeat_back_addr': '172.21.15.153:6807/3377683473', 'heartbeat_front_addr': '172.21.15.153:6805/3377683473', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'c3e8194b-2c20-4ffb-859d-00c33902b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6808', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6809', 'nonce': 357179117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6810', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6811', 'nonce': 357179117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6814', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6815', 'nonce': 357179117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6812', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6813', 'nonce': 357179117}]}, 'public_addr': '172.21.15.153:6809/357179117', 'cluster_addr': '172.21.15.153:6811/357179117', 'heartbeat_back_addr': '172.21.15.153:6815/357179117', 'heartbeat_front_addr': '172.21.15.153:6813/357179117', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98027d53-4324-44d1-89ce-8da85be58efb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6816', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6817', 'nonce': 1552538208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6818', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6819', 'nonce': 1552538208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6822', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6823', 'nonce': 1552538208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6820', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6821', 'nonce': 1552538208}]}, 'public_addr': '172.21.15.153:6817/1552538208', 'cluster_addr': '172.21.15.153:6819/1552538208', 'heartbeat_back_addr': '172.21.15.153:6823/1552538208', 'heartbeat_front_addr': '172.21.15.153:6821/1552538208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6b47d95a-1228-4fb1-a315-5371d9b42bbd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6824', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6825', 'nonce': 2479193924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6826', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6827', 'nonce': 2479193924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6830', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6831', 'nonce': 2479193924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6828', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6829', 'nonce': 2479193924}]}, 'public_addr': '172.21.15.153:6825/2479193924', 'cluster_addr': '172.21.15.153:6827/2479193924', 'heartbeat_back_addr': '172.21.15.153:6831/2479193924', 'heartbeat_front_addr': '172.21.15.153:6829/2479193924', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:52.398069+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:19.397975+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:44.971675+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:27:59.838970+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:27:44.208049+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:05.606659+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:30.066267+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:46.298606+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.16', 'osds': [3, 0, 6]}], 'pg_upmap_items': [{'pgid': '2.1f', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/283275195': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2738019433': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1268273764': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/567103210': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/2891059051': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2170219983': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/4082841962': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6800/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/570623774': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6801/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1379780741': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2880368026': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2044358308': '2024-04-07T03:21:47.915758+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-06T03:28:58.904 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.16 2024-04-06T03:28:58.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd rm-pg-upmap 6.16 2024-04-06T03:28:58.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:58.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:28:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:58.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:28:59.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:59.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:59.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:59.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:28:59 smithi050 bash[21467]: cluster 2024-04-06T03:28:58.119758+0000 mgr.y (mgr.24368) 298 : cluster 0 pgmap v292: 148 pgs: 148 active+clean; 570 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 519 op/s 2024-04-06T03:28:59.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:28:59 smithi050 bash[29221]: cluster 2024-04-06T03:28:58.119758+0000 mgr.y (mgr.24368) 298 : cluster 0 pgmap v292: 148 pgs: 148 active+clean; 570 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 519 op/s 2024-04-06T03:28:59.245 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:28:59.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:28:59.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:59.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:59.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:59.508 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:28:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:28:59.508 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:28:59 smithi153 bash[23852]: cluster 2024-04-06T03:28:58.119758+0000 mgr.y (mgr.24368) 298 : cluster 0 pgmap v292: 148 pgs: 148 active+clean; 570 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 519 op/s 2024-04-06T03:28:59.520 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:28:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:28:59.524 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:28:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:28:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:28:59.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:59.653 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:28:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:28:59.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:59.776 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:28:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:28:59.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:59.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:28:59.909 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:28:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:28:59.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:00.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:00.038 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:00.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:00.160 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:00.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:00.284 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:00.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:00 smithi153 bash[23852]: audit 2024-04-06T03:28:59.347758+0000 mgr.y (mgr.24368) 299 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:00.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:00.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:00.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:00 smithi050 bash[21467]: audit 2024-04-06T03:28:59.347758+0000 mgr.y (mgr.24368) 299 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:00.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:00 smithi050 bash[29221]: audit 2024-04-06T03:28:59.347758+0000 mgr.y (mgr.24368) 299 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:00.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:00.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:00.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:00.690 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:00.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:00.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:00.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:00.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:29:00.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:00.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:01.034 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:01.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:01.129 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:01.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:01.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:01.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:01.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:01.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:01 smithi153 bash[23852]: cluster 2024-04-06T03:29:00.120667+0000 mgr.y (mgr.24368) 300 : cluster 0 pgmap v293: 148 pgs: 148 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:29:01.403 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:01.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:01 smithi050 bash[21467]: cluster 2024-04-06T03:29:00.120667+0000 mgr.y (mgr.24368) 300 : cluster 0 pgmap v293: 148 pgs: 148 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:29:01.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:01 smithi050 bash[29221]: cluster 2024-04-06T03:29:00.120667+0000 mgr.y (mgr.24368) 300 : cluster 0 pgmap v293: 148 pgs: 148 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:29:01.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:01.525 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:01.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:01.647 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:01.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:01.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:01.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:01.894 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:01.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:01.997 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:02.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:02.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:02.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:02.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:02.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:02.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:02.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:02.549 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:02.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:02.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:02.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:02.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:02.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:02.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:02.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:03.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:03.071 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:03.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:03.196 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:03.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:03.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:03.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:03 smithi050 bash[21467]: cluster 2024-04-06T03:29:02.121110+0000 mgr.y (mgr.24368) 301 : cluster 0 pgmap v294: 148 pgs: 148 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 885 op/s 2024-04-06T03:29:03.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:03 smithi050 bash[29221]: cluster 2024-04-06T03:29:02.121110+0000 mgr.y (mgr.24368) 301 : cluster 0 pgmap v294: 148 pgs: 148 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 885 op/s 2024-04-06T03:29:03.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:03.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:03 smithi153 bash[23852]: cluster 2024-04-06T03:29:02.121110+0000 mgr.y (mgr.24368) 301 : cluster 0 pgmap v294: 148 pgs: 148 active+clean; 320 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 885 op/s 2024-04-06T03:29:03.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:03.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:03.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:03.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:03.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:03.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:03.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:03.981 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:03.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:04.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:04.090 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:04.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:04.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:04.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:04.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:04.372 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:04.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:04.506 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:04.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:04.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:04.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:04.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:04.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:04.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:04.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:04.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:05.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:04 smithi153 bash[23852]: audit 2024-04-06T03:29:03.820451+0000 mon.a (mon.0) 1123 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:05.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:04 smithi153 bash[23852]: audit 2024-04-06T03:29:03.821890+0000 mon.a (mon.0) 1124 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:05.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:04 smithi153 bash[23852]: cluster 2024-04-06T03:29:04.123608+0000 mgr.y (mgr.24368) 302 : cluster 0 pgmap v295: 148 pgs: 148 active+clean; 289 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1003 B/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:29:05.020 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:05.089 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:04 smithi050 bash[21467]: audit 2024-04-06T03:29:03.820451+0000 mon.a (mon.0) 1123 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:05.089 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:04 smithi050 bash[21467]: audit 2024-04-06T03:29:03.821890+0000 mon.a (mon.0) 1124 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:05.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:04 smithi050 bash[21467]: cluster 2024-04-06T03:29:04.123608+0000 mgr.y (mgr.24368) 302 : cluster 0 pgmap v295: 148 pgs: 148 active+clean; 289 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1003 B/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:29:05.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:04 smithi050 bash[29221]: audit 2024-04-06T03:29:03.820451+0000 mon.a (mon.0) 1123 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:05.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:04 smithi050 bash[29221]: audit 2024-04-06T03:29:03.821890+0000 mon.a (mon.0) 1124 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:05.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:04 smithi050 bash[29221]: cluster 2024-04-06T03:29:04.123608+0000 mgr.y (mgr.24368) 302 : cluster 0 pgmap v295: 148 pgs: 148 active+clean; 289 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1003 B/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:29:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:05.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:05.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:05.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:05.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:05.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:05.437 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:05.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:05.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:05.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:05.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:05.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:05.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:05.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:05.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:05.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:05.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:06.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:06.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:06.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:06.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:06.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:06.369 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:06.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:06.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Removed 96747 objects 2024-04-06T03:29:06.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Clean up completed and total clean up time :119.21 2024-04-06T03:29:06.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:06.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:06.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:06.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:06.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:06.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:06.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:06.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:06.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:07.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:07.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:07.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:07.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:07.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:07.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:07.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:07.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:07 smithi153 bash[23852]: cluster 2024-04-06T03:29:06.124356+0000 mgr.y (mgr.24368) 303 : cluster 0 pgmap v296: 148 pgs: 148 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 961 op/s 2024-04-06T03:29:07.516 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:07.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:07 smithi050 bash[21467]: cluster 2024-04-06T03:29:06.124356+0000 mgr.y (mgr.24368) 303 : cluster 0 pgmap v296: 148 pgs: 148 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 961 op/s 2024-04-06T03:29:07.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:07 smithi050 bash[29221]: cluster 2024-04-06T03:29:06.124356+0000 mgr.y (mgr.24368) 303 : cluster 0 pgmap v296: 148 pgs: 148 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 961 op/s 2024-04-06T03:29:07.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:07.647 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:07.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:07.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:07.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:07.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:07.908 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:07.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:08.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:08.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:08.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:08.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:08.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:08.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:08.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:08.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:08.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:08.562 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:08.664 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:08.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:08.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:08.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:08.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:08.942 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:08.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:09.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:09.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:09.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:09.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:09.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:09 smithi153 bash[23852]: cluster 2024-04-06T03:29:08.124863+0000 mgr.y (mgr.24368) 304 : cluster 0 pgmap v297: 148 pgs: 148 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 653 op/s 2024-04-06T03:29:09.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:09.340 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:09.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:09.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:09 smithi050 bash[21467]: cluster 2024-04-06T03:29:08.124863+0000 mgr.y (mgr.24368) 304 : cluster 0 pgmap v297: 148 pgs: 148 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 653 op/s 2024-04-06T03:29:09.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:09 smithi050 bash[29221]: cluster 2024-04-06T03:29:08.124863+0000 mgr.y (mgr.24368) 304 : cluster 0 pgmap v297: 148 pgs: 148 active+clean; 80 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 653 op/s 2024-04-06T03:29:09.455 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:29:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:29:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:29:09.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:09.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:29:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:29:09.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:09.602 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:09.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:09.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:09.777 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-06T03:29:09.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-06T03:29:09.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:09.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:09.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:09.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:09.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:09.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:10.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:10.116 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:10.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:10.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:10.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:10.357 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:10 smithi153 bash[23852]: audit 2024-04-06T03:29:09.355928+0000 mgr.y (mgr.24368) 305 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:10.373 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:10.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:10.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:10 smithi050 bash[29221]: audit 2024-04-06T03:29:09.355928+0000 mgr.y (mgr.24368) 305 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:10.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:10 smithi050 bash[21467]: audit 2024-04-06T03:29:09.355928+0000 mgr.y (mgr.24368) 305 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:10.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:10.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:10.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:10.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:10.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:10.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:10.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:10.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:10.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:10.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:10.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:11.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:11.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:11.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:11.275 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:11.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:11.389 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:11 smithi050 bash[21467]: cluster 2024-04-06T03:29:10.125629+0000 mgr.y (mgr.24368) 306 : cluster 0 pgmap v298: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s 2024-04-06T03:29:11.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:11 smithi050 bash[29221]: cluster 2024-04-06T03:29:10.125629+0000 mgr.y (mgr.24368) 306 : cluster 0 pgmap v298: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s 2024-04-06T03:29:11.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:11.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:11.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:11.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:11.544 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:11 smithi153 bash[23852]: cluster 2024-04-06T03:29:10.125629+0000 mgr.y (mgr.24368) 306 : cluster 0 pgmap v298: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s 2024-04-06T03:29:11.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:11.667 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:11.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:11.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:11.927 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:11.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:12.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.056 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:12.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:12.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:12.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:12.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:12.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:12.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.581 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:12.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:12.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:12.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:12.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.972 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:12.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:13.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:13.103 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:13.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:13.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:13.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:13.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:13.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:13.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:13 smithi153 bash[23852]: cluster 2024-04-06T03:29:11.784755+0000 osd.4 (osd.4) 193 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:13.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:13 smithi153 bash[23852]: cluster 2024-04-06T03:29:11.784869+0000 osd.4 (osd.4) 194 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:13.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:13 smithi153 bash[23852]: cluster 2024-04-06T03:29:12.126295+0000 mgr.y (mgr.24368) 307 : cluster 0 pgmap v299: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 427 op/s 2024-04-06T03:29:13.479 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:13 smithi153 bash[23852]: cluster 2024-04-06T03:29:12.206693+0000 mon.a (mon.0) 1125 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-04-06T03:29:13.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:13.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:13.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:13.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[21467]: cluster 2024-04-06T03:29:11.784755+0000 osd.4 (osd.4) 193 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:13.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[21467]: cluster 2024-04-06T03:29:11.784869+0000 osd.4 (osd.4) 194 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:13.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[21467]: cluster 2024-04-06T03:29:12.126295+0000 mgr.y (mgr.24368) 307 : cluster 0 pgmap v299: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 427 op/s 2024-04-06T03:29:13.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[21467]: cluster 2024-04-06T03:29:12.206693+0000 mon.a (mon.0) 1125 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-04-06T03:29:13.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[29221]: cluster 2024-04-06T03:29:11.784755+0000 osd.4 (osd.4) 193 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:13.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[29221]: cluster 2024-04-06T03:29:11.784869+0000 osd.4 (osd.4) 194 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:13.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[29221]: cluster 2024-04-06T03:29:12.126295+0000 mgr.y (mgr.24368) 307 : cluster 0 pgmap v299: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 427 op/s 2024-04-06T03:29:13.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:13 smithi050 bash[29221]: cluster 2024-04-06T03:29:12.206693+0000 mon.a (mon.0) 1125 : cluster 0 osdmap e173: 8 total, 8 up, 8 in 2024-04-06T03:29:13.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:13.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:13.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:13.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:13.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:14.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:14.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:14.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:14.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:14.264 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:14 smithi153 bash[23852]: cluster 2024-04-06T03:29:12.792766+0000 osd.5 (osd.5) 125 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:14.264 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:14 smithi153 bash[23852]: cluster 2024-04-06T03:29:12.792860+0000 osd.5 (osd.5) 126 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:14.264 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:14 smithi153 bash[23852]: cluster 2024-04-06T03:29:13.210320+0000 mon.a (mon.0) 1126 : cluster 0 osdmap e174: 8 total, 8 up, 8 in 2024-04-06T03:29:14.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:14.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:14.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:14.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:14 smithi050 bash[21467]: cluster 2024-04-06T03:29:12.792766+0000 osd.5 (osd.5) 125 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:14.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:14 smithi050 bash[21467]: cluster 2024-04-06T03:29:12.792860+0000 osd.5 (osd.5) 126 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:14.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:14 smithi050 bash[21467]: cluster 2024-04-06T03:29:13.210320+0000 mon.a (mon.0) 1126 : cluster 0 osdmap e174: 8 total, 8 up, 8 in 2024-04-06T03:29:14.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:14 smithi050 bash[29221]: cluster 2024-04-06T03:29:12.792766+0000 osd.5 (osd.5) 125 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:14.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:14 smithi050 bash[29221]: cluster 2024-04-06T03:29:12.792860+0000 osd.5 (osd.5) 126 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:14.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:14 smithi050 bash[29221]: cluster 2024-04-06T03:29:13.210320+0000 mon.a (mon.0) 1126 : cluster 0 osdmap e174: 8 total, 8 up, 8 in 2024-04-06T03:29:14.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:14.541 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:14.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:14.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:14.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:14.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:14.809 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:29:14.809 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:29:14.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:14.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:14.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:15.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:15.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:15.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:15.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:15.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:15.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:15.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:15.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:15.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:15.573 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:15.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:15.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:15.703 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:15.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:15 smithi050 bash[29221]: cluster 2024-04-06T03:29:14.127320+0000 mgr.y (mgr.24368) 308 : cluster 0 pgmap v302: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 160 op/s 2024-04-06T03:29:15.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:15 smithi050 bash[21467]: cluster 2024-04-06T03:29:14.127320+0000 mgr.y (mgr.24368) 308 : cluster 0 pgmap v302: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 160 op/s 2024-04-06T03:29:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:15 smithi153 bash[23852]: cluster 2024-04-06T03:29:14.127320+0000 mgr.y (mgr.24368) 308 : cluster 0 pgmap v302: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 160 op/s 2024-04-06T03:29:15.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:15.832 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:15.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:15.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:15.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:16.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:16.097 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:16.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:16.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:16.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:16.360 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:16.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:16.490 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:16.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:16.621 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:16.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:16.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:16.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:16.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:16.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:16.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:17.008 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:17.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:17.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:17.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:17.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:17.399 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:17.487 INFO:teuthology.orchestra.run.smithi050.stderr:clear 6.16 pg_upmap mapping 2024-04-06T03:29:17.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:17.529 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:17.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:17.643 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:17 smithi153 bash[23852]: cluster 2024-04-06T03:29:16.128261+0000 mgr.y (mgr.24368) 309 : cluster 0 pgmap v303: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 160 op/s 2024-04-06T03:29:17.643 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:17 smithi153 bash[23852]: audit 2024-04-06T03:29:16.839006+0000 mon.a (mon.0) 1127 : audit 1 from='client.? 172.21.15.50:0/1130142860' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.16"}]: dispatch 2024-04-06T03:29:17.660 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:17.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-06T03:29:17.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:17.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:17 smithi050 bash[29221]: cluster 2024-04-06T03:29:16.128261+0000 mgr.y (mgr.24368) 309 : cluster 0 pgmap v303: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 160 op/s 2024-04-06T03:29:17.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:17 smithi050 bash[29221]: audit 2024-04-06T03:29:16.839006+0000 mon.a (mon.0) 1127 : audit 1 from='client.? 172.21.15.50:0/1130142860' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.16"}]: dispatch 2024-04-06T03:29:17.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:17 smithi050 bash[21467]: cluster 2024-04-06T03:29:16.128261+0000 mgr.y (mgr.24368) 309 : cluster 0 pgmap v303: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 160 op/s 2024-04-06T03:29:17.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:17 smithi050 bash[21467]: audit 2024-04-06T03:29:16.839006+0000 mon.a (mon.0) 1127 : audit 1 from='client.? 172.21.15.50:0/1130142860' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.16"}]: dispatch 2024-04-06T03:29:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:17.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:17.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:17.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:17.973 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:29:17.993 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:29:18.013 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:29:18.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:18.034 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:29:18.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:18.058 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:29:18.080 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:29:18.103 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:29:18.127 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:29:18.129 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:29:18.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:18.185 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:18.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:18.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:18.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:18.438 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:18.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:18.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[21467]: audit 2024-04-06T03:29:17.484524+0000 mon.a (mon.0) 1128 : audit 1 from='client.? 172.21.15.50:0/1130142860' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.16"}]': finished 2024-04-06T03:29:18.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[21467]: cluster 2024-04-06T03:29:17.496892+0000 mon.a (mon.0) 1129 : cluster 0 osdmap e175: 8 total, 8 up, 8 in 2024-04-06T03:29:18.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[21467]: audit 2024-04-06T03:29:17.960063+0000 mon.a (mon.0) 1130 : audit 0 from='client.? 172.21.15.50:0/145817273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:18.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[21467]: cluster 2024-04-06T03:29:18.493319+0000 mon.a (mon.0) 1131 : cluster 0 osdmap e176: 8 total, 8 up, 8 in 2024-04-06T03:29:18.551 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[29221]: audit 2024-04-06T03:29:17.484524+0000 mon.a (mon.0) 1128 : audit 1 from='client.? 172.21.15.50:0/1130142860' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.16"}]': finished 2024-04-06T03:29:18.551 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[29221]: cluster 2024-04-06T03:29:17.496892+0000 mon.a (mon.0) 1129 : cluster 0 osdmap e175: 8 total, 8 up, 8 in 2024-04-06T03:29:18.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[29221]: audit 2024-04-06T03:29:17.960063+0000 mon.a (mon.0) 1130 : audit 0 from='client.? 172.21.15.50:0/145817273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:18.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:18 smithi050 bash[29221]: cluster 2024-04-06T03:29:18.493319+0000 mon.a (mon.0) 1131 : cluster 0 osdmap e176: 8 total, 8 up, 8 in 2024-04-06T03:29:18.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:18.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:18.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:18.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:18 smithi153 bash[23852]: audit 2024-04-06T03:29:17.484524+0000 mon.a (mon.0) 1128 : audit 1 from='client.? 172.21.15.50:0/1130142860' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.16"}]': finished 2024-04-06T03:29:18.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:18 smithi153 bash[23852]: cluster 2024-04-06T03:29:17.496892+0000 mon.a (mon.0) 1129 : cluster 0 osdmap e175: 8 total, 8 up, 8 in 2024-04-06T03:29:18.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:18 smithi153 bash[23852]: audit 2024-04-06T03:29:17.960063+0000 mon.a (mon.0) 1130 : audit 0 from='client.? 172.21.15.50:0/145817273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:18.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:18 smithi153 bash[23852]: cluster 2024-04-06T03:29:18.493319+0000 mon.a (mon.0) 1131 : cluster 0 osdmap e176: 8 total, 8 up, 8 in 2024-04-06T03:29:18.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:18.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:18.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:18.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:18.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:19.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:19.080 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:19.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:19.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:19.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:19.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:19.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:19.456 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:29:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:29:19.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:19.517 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:29:19.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:19.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:19.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[21467]: cluster 2024-04-06T03:29:18.128704+0000 mgr.y (mgr.24368) 310 : cluster 0 pgmap v305: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[21467]: audit 2024-04-06T03:29:18.624428+0000 mon.c (mon.2) 52 : audit 1 from='client.? 172.21.15.50:0/1825873553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[21467]: audit 2024-04-06T03:29:18.624754+0000 mon.a (mon.0) 1132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[21467]: audit 2024-04-06T03:29:18.842368+0000 mon.a (mon.0) 1133 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:19.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[29221]: cluster 2024-04-06T03:29:18.128704+0000 mgr.y (mgr.24368) 310 : cluster 0 pgmap v305: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[29221]: audit 2024-04-06T03:29:18.624428+0000 mon.c (mon.2) 52 : audit 1 from='client.? 172.21.15.50:0/1825873553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[29221]: audit 2024-04-06T03:29:18.624754+0000 mon.a (mon.0) 1132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:19 smithi050 bash[29221]: audit 2024-04-06T03:29:18.842368+0000 mon.a (mon.0) 1133 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:19.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:19.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:29:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:29:19.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:19 smithi153 bash[23852]: cluster 2024-04-06T03:29:18.128704+0000 mgr.y (mgr.24368) 310 : cluster 0 pgmap v305: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:19.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:19 smithi153 bash[23852]: audit 2024-04-06T03:29:18.624428+0000 mon.c (mon.2) 52 : audit 1 from='client.? 172.21.15.50:0/1825873553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:19.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:19 smithi153 bash[23852]: audit 2024-04-06T03:29:18.624754+0000 mon.a (mon.0) 1132 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:19.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:19 smithi153 bash[23852]: audit 2024-04-06T03:29:18.842368+0000 mon.a (mon.0) 1133 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:19.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:19.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:19.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:19.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:20.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:20.121 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:20.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:20.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:20.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:20.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:20.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:20.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:20.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:20.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:20 smithi153 bash[23852]: audit 2024-04-06T03:29:19.363870+0000 mgr.y (mgr.24368) 311 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:20.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:20 smithi153 bash[23852]: cluster 2024-04-06T03:29:19.508438+0000 mon.a (mon.0) 1134 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:29:20.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:20 smithi153 bash[23852]: audit 2024-04-06T03:29:19.516565+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:29:20.620 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:20 smithi153 bash[23852]: cluster 2024-04-06T03:29:19.520362+0000 mon.a (mon.0) 1136 : cluster 0 osdmap e177: 8 total, 8 up, 8 in 2024-04-06T03:29:20.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:20.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:20.765 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:20.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[21467]: audit 2024-04-06T03:29:19.363870+0000 mgr.y (mgr.24368) 311 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[21467]: cluster 2024-04-06T03:29:19.508438+0000 mon.a (mon.0) 1134 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:29:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[21467]: audit 2024-04-06T03:29:19.516565+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:29:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[21467]: cluster 2024-04-06T03:29:19.520362+0000 mon.a (mon.0) 1136 : cluster 0 osdmap e177: 8 total, 8 up, 8 in 2024-04-06T03:29:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[29221]: audit 2024-04-06T03:29:19.363870+0000 mgr.y (mgr.24368) 311 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[29221]: cluster 2024-04-06T03:29:19.508438+0000 mon.a (mon.0) 1134 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:29:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[29221]: audit 2024-04-06T03:29:19.516565+0000 mon.a (mon.0) 1135 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:29:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:20 smithi050 bash[29221]: cluster 2024-04-06T03:29:19.520362+0000 mon.a (mon.0) 1136 : cluster 0 osdmap e177: 8 total, 8 up, 8 in 2024-04-06T03:29:20.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:20.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:20.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:21.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:21.074 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:21.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:21.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:21.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.284 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:21.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:21.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:21.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:21.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:21.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[21467]: cluster 2024-04-06T03:29:20.129763+0000 mgr.y (mgr.24368) 312 : cluster 0 pgmap v308: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:29:21.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[21467]: cluster 2024-04-06T03:29:20.375646+0000 osd.1 (osd.1) 167 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:21.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[21467]: cluster 2024-04-06T03:29:20.375767+0000 osd.1 (osd.1) 168 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:21.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[21467]: cluster 2024-04-06T03:29:20.537753+0000 mon.a (mon.0) 1137 : cluster 0 osdmap e178: 8 total, 8 up, 8 in 2024-04-06T03:29:21.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[29221]: cluster 2024-04-06T03:29:20.129763+0000 mgr.y (mgr.24368) 312 : cluster 0 pgmap v308: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:29:21.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[29221]: cluster 2024-04-06T03:29:20.375646+0000 osd.1 (osd.1) 167 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:21.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[29221]: cluster 2024-04-06T03:29:20.375767+0000 osd.1 (osd.1) 168 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:21.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:21 smithi050 bash[29221]: cluster 2024-04-06T03:29:20.537753+0000 mon.a (mon.0) 1137 : cluster 0 osdmap e178: 8 total, 8 up, 8 in 2024-04-06T03:29:21.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:21 smithi153 bash[23852]: cluster 2024-04-06T03:29:20.129763+0000 mgr.y (mgr.24368) 312 : cluster 0 pgmap v308: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:29:21.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:21 smithi153 bash[23852]: cluster 2024-04-06T03:29:20.375646+0000 osd.1 (osd.1) 167 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:21.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:21 smithi153 bash[23852]: cluster 2024-04-06T03:29:20.375767+0000 osd.1 (osd.1) 168 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:21.788 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:21 smithi153 bash[23852]: cluster 2024-04-06T03:29:20.537753+0000 mon.a (mon.0) 1137 : cluster 0 osdmap e178: 8 total, 8 up, 8 in 2024-04-06T03:29:21.804 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:21.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:21.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:22.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:22.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:29:22.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:22.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:22.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:22.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:22.322 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:22.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:22.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:22.518 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:29:22.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:22.560 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:22 smithi153 bash[23852]: audit 2024-04-06T03:29:22.321896+0000 mon.a (mon.0) 1138 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:22.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:22.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:29:22.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:22.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:22.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:22.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:22 smithi050 bash[21467]: audit 2024-04-06T03:29:22.321896+0000 mon.a (mon.0) 1138 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:22.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:22 smithi050 bash[29221]: audit 2024-04-06T03:29:22.321896+0000 mon.a (mon.0) 1138 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:29:22.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:22.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:22.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:22.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:22.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:23.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:23.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:23.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:23.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:23.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:23.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:23.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:23.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:23.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:23.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:23.704 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[21467]: cluster 2024-04-06T03:29:22.130216+0000 mgr.y (mgr.24368) 313 : cluster 0 pgmap v310: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:23.704 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[21467]: audit 2024-04-06T03:29:23.002997+0000 mon.a (mon.0) 1139 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:23.704 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[21467]: audit 2024-04-06T03:29:23.003893+0000 mon.a (mon.0) 1140 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:23.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[21467]: audit 2024-04-06T03:29:23.009215+0000 mon.a (mon.0) 1141 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:23.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[29221]: cluster 2024-04-06T03:29:22.130216+0000 mgr.y (mgr.24368) 313 : cluster 0 pgmap v310: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:23.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[29221]: audit 2024-04-06T03:29:23.002997+0000 mon.a (mon.0) 1139 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:23.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[29221]: audit 2024-04-06T03:29:23.003893+0000 mon.a (mon.0) 1140 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:23.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:23 smithi050 bash[29221]: audit 2024-04-06T03:29:23.009215+0000 mon.a (mon.0) 1141 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:23.720 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:23.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:23 smithi153 bash[23852]: cluster 2024-04-06T03:29:22.130216+0000 mgr.y (mgr.24368) 313 : cluster 0 pgmap v310: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:23.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:23 smithi153 bash[23852]: audit 2024-04-06T03:29:23.002997+0000 mon.a (mon.0) 1139 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:29:23.771 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:23 smithi153 bash[23852]: audit 2024-04-06T03:29:23.003893+0000 mon.a (mon.0) 1140 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:29:23.771 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:23 smithi153 bash[23852]: audit 2024-04-06T03:29:23.009215+0000 mon.a (mon.0) 1141 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:23.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:23.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:23.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:23.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:24.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:24.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:24.219 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:24.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:29:24.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:29:24.326 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-06T03:29:24.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-06T03:29:24.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:24.376 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:24.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:24.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:24.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:24.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:24.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:24.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:24.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:24.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:24.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:24.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:25.001 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:25.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:25.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:25.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:25.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:25.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:25.396 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:25.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:25.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:25.552 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_0' removed 2024-04-06T03:29:25.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:25.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:25.677 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:29:25.677 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:29:25.677 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:29:25.677 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:29:25.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:25 smithi153 bash[23852]: cluster 2024-04-06T03:29:24.131064+0000 mgr.y (mgr.24368) 314 : cluster 0 pgmap v311: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:25.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:25 smithi153 bash[23852]: audit 2024-04-06T03:29:24.831384+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.50:0/3132680827' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:29:25.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:25 smithi153 bash[23852]: audit 2024-04-06T03:29:24.831726+0000 mon.a (mon.0) 1142 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:29:25.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:25 smithi153 bash[23852]: cluster 2024-04-06T03:29:25.463449+0000 osd.3 (osd.3) 209 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:25.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:25 smithi153 bash[23852]: cluster 2024-04-06T03:29:25.463575+0000 osd.3 (osd.3) 210 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:25.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[21467]: cluster 2024-04-06T03:29:24.131064+0000 mgr.y (mgr.24368) 314 : cluster 0 pgmap v311: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:25.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[21467]: audit 2024-04-06T03:29:24.831384+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.50:0/3132680827' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:29:25.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[21467]: audit 2024-04-06T03:29:24.831726+0000 mon.a (mon.0) 1142 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:29:25.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[21467]: cluster 2024-04-06T03:29:25.463449+0000 osd.3 (osd.3) 209 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:25.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[21467]: cluster 2024-04-06T03:29:25.463575+0000 osd.3 (osd.3) 210 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:25.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[29221]: cluster 2024-04-06T03:29:24.131064+0000 mgr.y (mgr.24368) 314 : cluster 0 pgmap v311: 148 pgs: 148 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:25.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[29221]: audit 2024-04-06T03:29:24.831384+0000 mon.c (mon.2) 53 : audit 1 from='client.? 172.21.15.50:0/3132680827' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:29:25.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[29221]: audit 2024-04-06T03:29:24.831726+0000 mon.a (mon.0) 1142 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:29:25.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[29221]: cluster 2024-04-06T03:29:25.463449+0000 osd.3 (osd.3) 209 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:25.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:25 smithi050 bash[29221]: cluster 2024-04-06T03:29:25.463575+0000 osd.3 (osd.3) 210 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:25.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:25.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:25.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:25.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:26.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:26.039 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:26.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:26.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:26.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:26.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:26.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-06T03:29:26.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:26.447 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:26.454 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-06T03:29:26.455 INFO:tasks.radosbench:Beginning radosbench... 2024-04-06T03:29:26.455 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-04-06T03:29:26.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool create unique_pool_1 16 2024-04-06T03:29:26.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:26.573 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:26.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:26.703 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:26.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:26 smithi050 bash[29221]: audit 2024-04-06T03:29:25.545087+0000 mon.a (mon.0) 1143 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:29:26.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:26 smithi050 bash[29221]: cluster 2024-04-06T03:29:25.549016+0000 mon.a (mon.0) 1144 : cluster 0 osdmap e179: 8 total, 8 up, 8 in 2024-04-06T03:29:26.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:26 smithi050 bash[21467]: audit 2024-04-06T03:29:25.545087+0000 mon.a (mon.0) 1143 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:29:26.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:26 smithi050 bash[21467]: cluster 2024-04-06T03:29:25.549016+0000 mon.a (mon.0) 1144 : cluster 0 osdmap e179: 8 total, 8 up, 8 in 2024-04-06T03:29:26.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:26.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:26 smithi153 bash[23852]: audit 2024-04-06T03:29:25.545087+0000 mon.a (mon.0) 1143 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:29:26.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:26 smithi153 bash[23852]: cluster 2024-04-06T03:29:25.549016+0000 mon.a (mon.0) 1144 : cluster 0 osdmap e179: 8 total, 8 up, 8 in 2024-04-06T03:29:26.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:26.930 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:26.930 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:26.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:26.970 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:26.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:27.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:27.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:27.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:27.236 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:27.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:27.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:27.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:27.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:27.610 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:27 smithi153 bash[23852]: cluster 2024-04-06T03:29:26.131736+0000 mgr.y (mgr.24368) 315 : cluster 0 pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:29:27.626 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:27.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:27.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:27 smithi050 bash[29221]: cluster 2024-04-06T03:29:26.131736+0000 mgr.y (mgr.24368) 315 : cluster 0 pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:29:27.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:27 smithi050 bash[21467]: cluster 2024-04-06T03:29:26.131736+0000 mgr.y (mgr.24368) 315 : cluster 0 pgmap v313: 132 pgs: 132 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:29:27.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:27.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:27.887 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:27.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:27.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:28.017 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:28.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:28.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:28.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:28.275 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:28.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:28.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:28.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:28.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:28.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:28.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:28.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:28.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:28.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:28.915 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:28.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:29.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:29.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:29.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:29.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:29.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:29.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:29.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:29.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:29.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:29.544 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:29:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:29:29.560 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:29.568 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:29:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:29:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:29:29.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:29.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:29 smithi050 bash[21467]: cluster 2024-04-06T03:29:28.132306+0000 mgr.y (mgr.24368) 316 : cluster 0 pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:29:29.676 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:29 smithi050 bash[29221]: cluster 2024-04-06T03:29:28.132306+0000 mgr.y (mgr.24368) 316 : cluster 0 pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:29:29.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:29.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:29 smithi153 bash[23852]: cluster 2024-04-06T03:29:28.132306+0000 mgr.y (mgr.24368) 316 : cluster 0 pgmap v314: 132 pgs: 132 active+clean; 582 KiB data, 822 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:29:29.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:29.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:29.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:29.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:30.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:30.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:30.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:30.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:30.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:30.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:30.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:30.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:30.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:30.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:30.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:30 smithi153 bash[23852]: audit 2024-04-06T03:29:29.375226+0000 mgr.y (mgr.24368) 317 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:30.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:30.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:30.954 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:30 smithi050 bash[21467]: audit 2024-04-06T03:29:29.375226+0000 mgr.y (mgr.24368) 317 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:30.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:30 smithi050 bash[29221]: audit 2024-04-06T03:29:29.375226+0000 mgr.y (mgr.24368) 317 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:30.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:30.997 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:30.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:30.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:31.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:31.151 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:29:31.166 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:31.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:31.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:31.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:31.385 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:31.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:31.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:31.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:31.644 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:31.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:31.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:31 smithi153 bash[23852]: cluster 2024-04-06T03:29:30.133139+0000 mgr.y (mgr.24368) 318 : cluster 0 pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-06T03:29:31.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:31 smithi153 bash[23852]: cluster 2024-04-06T03:29:30.799778+0000 osd.6 (osd.6) 83 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:31.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:31 smithi153 bash[23852]: cluster 2024-04-06T03:29:30.799876+0000 osd.6 (osd.6) 84 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:31.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:31.906 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:31.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:31 smithi050 bash[21467]: cluster 2024-04-06T03:29:30.133139+0000 mgr.y (mgr.24368) 318 : cluster 0 pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-06T03:29:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:31 smithi050 bash[21467]: cluster 2024-04-06T03:29:30.799778+0000 osd.6 (osd.6) 83 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:31.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:31 smithi050 bash[21467]: cluster 2024-04-06T03:29:30.799876+0000 osd.6 (osd.6) 84 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:31.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:31 smithi050 bash[29221]: cluster 2024-04-06T03:29:30.133139+0000 mgr.y (mgr.24368) 318 : cluster 0 pgmap v315: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-06T03:29:31.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:31 smithi050 bash[29221]: cluster 2024-04-06T03:29:30.799778+0000 osd.6 (osd.6) 83 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:31.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:31 smithi050 bash[29221]: cluster 2024-04-06T03:29:30.799876+0000 osd.6 (osd.6) 84 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:32.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:32.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:32.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:32.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:32.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:32.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:32.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:32.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:32.425 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:32.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:32.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:32.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:32.677 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:32.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:32 smithi050 bash[21467]: cluster 2024-04-06T03:29:31.591945+0000 mon.a (mon.0) 1145 : cluster 0 osdmap e180: 8 total, 8 up, 8 in 2024-04-06T03:29:32.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:32 smithi050 bash[29221]: cluster 2024-04-06T03:29:31.591945+0000 mon.a (mon.0) 1145 : cluster 0 osdmap e180: 8 total, 8 up, 8 in 2024-04-06T03:29:32.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:32.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:32 smithi153 bash[23852]: cluster 2024-04-06T03:29:31.591945+0000 mon.a (mon.0) 1145 : cluster 0 osdmap e180: 8 total, 8 up, 8 in 2024-04-06T03:29:32.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:32.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:32.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:32.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:33.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:33.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:33.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:33.191 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:33.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:33.320 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:33.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:33.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:33.577 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:33.578 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:29:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:33.601 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:29:33.621 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:29:33.642 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:29:33.666 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:29:33.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:33.691 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:29:33.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:33.714 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:29:33.737 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:29:33.738 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:29:33.739 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:29:33.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:33.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:33.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:33 smithi050 bash[21467]: cluster 2024-04-06T03:29:32.133806+0000 mgr.y (mgr.24368) 319 : cluster 0 pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:33.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:33 smithi050 bash[21467]: audit 2024-04-06T03:29:33.563949+0000 mon.c (mon.2) 54 : audit 0 from='client.? 172.21.15.50:0/174285694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:33.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:33 smithi050 bash[29221]: cluster 2024-04-06T03:29:32.133806+0000 mgr.y (mgr.24368) 319 : cluster 0 pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:33.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:33 smithi050 bash[29221]: audit 2024-04-06T03:29:33.563949+0000 mon.c (mon.2) 54 : audit 0 from='client.? 172.21.15.50:0/174285694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:33.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:33.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:33 smithi153 bash[23852]: cluster 2024-04-06T03:29:32.133806+0000 mgr.y (mgr.24368) 319 : cluster 0 pgmap v317: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:33.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:33 smithi153 bash[23852]: audit 2024-04-06T03:29:33.563949+0000 mon.c (mon.2) 54 : audit 0 from='client.? 172.21.15.50:0/174285694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:33.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:33.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:34.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:34.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:34.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:34.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:34.288 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:29:34.288 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:29:34.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:34.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:34.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:34.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:34.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:34.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:34.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:34.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:34 smithi050 bash[21467]: audit 2024-04-06T03:29:33.856102+0000 mon.a (mon.0) 1146 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:34.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:34 smithi050 bash[29221]: audit 2024-04-06T03:29:33.856102+0000 mon.a (mon.0) 1146 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:34.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:34.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:34 smithi153 bash[23852]: audit 2024-04-06T03:29:33.856102+0000 mon.a (mon.0) 1146 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:34.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:34.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:34.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:34.992 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:34.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:35.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-06T03:29:35.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:35.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:35.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:35.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:35.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:35.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:35.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:35.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:35.619 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:29:35.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:35.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:35.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:35 smithi050 bash[21467]: cluster 2024-04-06T03:29:34.134852+0000 mgr.y (mgr.24368) 320 : cluster 0 pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-04-06T03:29:35.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:35 smithi050 bash[21467]: audit 2024-04-06T03:29:35.162364+0000 mon.c (mon.2) 55 : audit 1 from='client.? 172.21.15.50:0/314140038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:29:35.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:35 smithi050 bash[21467]: audit 2024-04-06T03:29:35.162732+0000 mon.a (mon.0) 1147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:29:35.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:35 smithi050 bash[29221]: cluster 2024-04-06T03:29:34.134852+0000 mgr.y (mgr.24368) 320 : cluster 0 pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-04-06T03:29:35.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:35 smithi050 bash[29221]: audit 2024-04-06T03:29:35.162364+0000 mon.c (mon.2) 55 : audit 1 from='client.? 172.21.15.50:0/314140038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:29:35.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:35 smithi050 bash[29221]: audit 2024-04-06T03:29:35.162732+0000 mon.a (mon.0) 1147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:29:35.759 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:35.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:35.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:35 smithi153 bash[23852]: cluster 2024-04-06T03:29:34.134852+0000 mgr.y (mgr.24368) 320 : cluster 0 pgmap v318: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-04-06T03:29:35.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:35 smithi153 bash[23852]: audit 2024-04-06T03:29:35.162364+0000 mon.c (mon.2) 55 : audit 1 from='client.? 172.21.15.50:0/314140038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:29:35.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:35 smithi153 bash[23852]: audit 2024-04-06T03:29:35.162732+0000 mon.a (mon.0) 1147 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:29:35.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:35.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:35.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:36.017 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:36.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:36.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:36.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:36.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:36.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:36.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:36.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:36.539 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:36.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:36.654 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:36 smithi153 bash[23852]: cluster 2024-04-06T03:29:35.609794+0000 mon.a (mon.0) 1148 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:29:36.654 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:36 smithi153 bash[23852]: audit 2024-04-06T03:29:35.612230+0000 mon.a (mon.0) 1149 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:29:36.654 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:36 smithi153 bash[23852]: cluster 2024-04-06T03:29:35.619045+0000 mon.a (mon.0) 1150 : cluster 0 osdmap e181: 8 total, 8 up, 8 in 2024-04-06T03:29:36.654 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:36 smithi153 bash[23852]: cluster 2024-04-06T03:29:35.901603+0000 osd.3 (osd.3) 211 : cluster 0 6.1b scrub starts 2024-04-06T03:29:36.654 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:36 smithi153 bash[23852]: cluster 2024-04-06T03:29:35.947256+0000 osd.3 (osd.3) 212 : cluster 0 6.1b scrub ok 2024-04-06T03:29:36.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:36.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:36.798 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:36.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:36.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[21467]: cluster 2024-04-06T03:29:35.609794+0000 mon.a (mon.0) 1148 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[21467]: audit 2024-04-06T03:29:35.612230+0000 mon.a (mon.0) 1149 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[21467]: cluster 2024-04-06T03:29:35.619045+0000 mon.a (mon.0) 1150 : cluster 0 osdmap e181: 8 total, 8 up, 8 in 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[21467]: cluster 2024-04-06T03:29:35.901603+0000 osd.3 (osd.3) 211 : cluster 0 6.1b scrub starts 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[21467]: cluster 2024-04-06T03:29:35.947256+0000 osd.3 (osd.3) 212 : cluster 0 6.1b scrub ok 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[29221]: cluster 2024-04-06T03:29:35.609794+0000 mon.a (mon.0) 1148 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[29221]: audit 2024-04-06T03:29:35.612230+0000 mon.a (mon.0) 1149 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[29221]: cluster 2024-04-06T03:29:35.619045+0000 mon.a (mon.0) 1150 : cluster 0 osdmap e181: 8 total, 8 up, 8 in 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[29221]: cluster 2024-04-06T03:29:35.901603+0000 osd.3 (osd.3) 211 : cluster 0 6.1b scrub starts 2024-04-06T03:29:36.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:36 smithi050 bash[29221]: cluster 2024-04-06T03:29:35.947256+0000 osd.3 (osd.3) 212 : cluster 0 6.1b scrub ok 2024-04-06T03:29:36.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:36.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:37.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:37.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:37.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:37.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:37.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:37.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:37.321 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:37.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:37.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:37.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:37.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:37.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:37.714 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:37.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:37.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:37 smithi050 bash[29221]: cluster 2024-04-06T03:29:35.714341+0000 osd.4 (osd.4) 195 : cluster 0 2.18 scrub starts 2024-04-06T03:29:37.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:37 smithi050 bash[29221]: cluster 2024-04-06T03:29:35.717669+0000 osd.4 (osd.4) 196 : cluster 0 2.18 scrub ok 2024-04-06T03:29:37.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:37 smithi050 bash[29221]: cluster 2024-04-06T03:29:36.135645+0000 mgr.y (mgr.24368) 321 : cluster 0 pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:29:37.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:37 smithi050 bash[21467]: cluster 2024-04-06T03:29:35.714341+0000 osd.4 (osd.4) 195 : cluster 0 2.18 scrub starts 2024-04-06T03:29:37.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:37 smithi050 bash[21467]: cluster 2024-04-06T03:29:35.717669+0000 osd.4 (osd.4) 196 : cluster 0 2.18 scrub ok 2024-04-06T03:29:37.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:37 smithi050 bash[21467]: cluster 2024-04-06T03:29:36.135645+0000 mgr.y (mgr.24368) 321 : cluster 0 pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:29:37.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:37.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:37.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:37.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:38.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:37 smithi153 bash[23852]: cluster 2024-04-06T03:29:35.714341+0000 osd.4 (osd.4) 195 : cluster 0 2.18 scrub starts 2024-04-06T03:29:38.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:37 smithi153 bash[23852]: cluster 2024-04-06T03:29:35.717669+0000 osd.4 (osd.4) 196 : cluster 0 2.18 scrub ok 2024-04-06T03:29:38.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:37 smithi153 bash[23852]: cluster 2024-04-06T03:29:36.135645+0000 mgr.y (mgr.24368) 321 : cluster 0 pgmap v320: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:29:38.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:38.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:38.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:38.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:38.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:38.370 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:38.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:38.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:38.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:38.634 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:38.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:38.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:38.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:38.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:38 smithi153 bash[23852]: cluster 2024-04-06T03:29:36.717406+0000 osd.4 (osd.4) 197 : cluster 0 6.1a scrub starts 2024-04-06T03:29:38.879 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:38 smithi153 bash[23852]: cluster 2024-04-06T03:29:36.732204+0000 osd.4 (osd.4) 198 : cluster 0 6.1a scrub ok 2024-04-06T03:29:38.896 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:38.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:38.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:38 smithi050 bash[21467]: cluster 2024-04-06T03:29:36.717406+0000 osd.4 (osd.4) 197 : cluster 0 6.1a scrub starts 2024-04-06T03:29:38.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:38 smithi050 bash[21467]: cluster 2024-04-06T03:29:36.732204+0000 osd.4 (osd.4) 198 : cluster 0 6.1a scrub ok 2024-04-06T03:29:38.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:38 smithi050 bash[29221]: cluster 2024-04-06T03:29:36.717406+0000 osd.4 (osd.4) 197 : cluster 0 6.1a scrub starts 2024-04-06T03:29:38.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:38 smithi050 bash[29221]: cluster 2024-04-06T03:29:36.732204+0000 osd.4 (osd.4) 198 : cluster 0 6.1a scrub ok 2024-04-06T03:29:38.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:39.025 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:39.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:39.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:39.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 181, 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-06T03:10:26.969128+0000", 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-06T03:29:35.609434+0000", 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-06T03:20:09.072730+0000", 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-06T03:19:42.545394+0000", 2024-04-06T03:29:39.259 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5804032, 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "nodeep-scrub", 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 18, 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-06T03:29:39.260 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 7, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:15:48.096900+0000", 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:29:39.261 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:29:39.262 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:29:39.263 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:29:39.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-06T03:29:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 6, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 6, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.5, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:29:39.266 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:50.011930+0000", 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:29:39.267 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.268 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "79", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:29:39.269 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:29:39.270 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.3200000524520874, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.3200000524520874, 2024-04-06T03:29:39.271 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:51.687715+0000", 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:29:39.272 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:29:39.273 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "81", 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:29:39.274 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:29:39.275 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:29:39.276 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.301921+0000", 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:29:39.278 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "88", 2024-04-06T03:29:39.279 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 88, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:29:39.280 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:29:39.281 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-06T03:29:39.282 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.1699999570846558, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.810316+0000", 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:29:39.283 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:29:39.284 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "83", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:29:39.285 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:29:39.286 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:29:39.287 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.288 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:56.321502+0000", 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:29:39.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "85", 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:29:39.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:29:39.291 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:29:39.292 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:29:39.293 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "e607f527-4592-4acd-a47f-76575615ae1b", 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.294 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 175, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.295 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-06T03:29:39.296 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.297 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/3531754646", 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/3531754646", 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/3531754646", 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/3531754646", 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10", 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.299 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0.5613250732421875, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-06T03:29:39.300 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.302 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/4128410069", 2024-04-06T03:29:39.303 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/4128410069", 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/4128410069", 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/4128410069", 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b", 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.304 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 19, 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-06T03:29:39.305 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.306 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/1307247904", 2024-04-06T03:29:39.308 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/1307247904", 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/1307247904", 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/1307247904", 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240", 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.309 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 165, 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.310 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.311 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/1397888508", 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/1397888508", 2024-04-06T03:29:39.313 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/1397888508", 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/1397888508", 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48", 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:29:39.314 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 153, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6800", 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6801", 2024-04-06T03:29:39.315 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6802", 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6803", 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.316 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6806", 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6807", 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6804", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6805", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6801/3377683473", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6803/3377683473", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6807/3377683473", 2024-04-06T03:29:39.318 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6805/3377683473", 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241", 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 46, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 127, 2024-04-06T03:29:39.319 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6808", 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6809", 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.320 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6810", 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6811", 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.321 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6814", 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6815", 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.322 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6812", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6813", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6809/357179117", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6811/357179117", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6815/357179117", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6813/357179117", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.323 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "98027d53-4324-44d1-89ce-8da85be58efb", 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6816", 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6817", 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.325 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6818", 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6819", 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6822", 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6823", 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6820", 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6821", 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6817/1552538208", 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6819/1552538208", 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6823/1552538208", 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6821/1552538208", 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:29:39.328 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd", 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 138, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6824", 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6825", 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6826", 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6827", 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6830", 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6831", 2024-04-06T03:29:39.331 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6828", 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6829", 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6825/2479193924", 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6827/2479193924", 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6831/2479193924", 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6829/2479193924", 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.333 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:52.398069+0000", 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.334 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:29:20.375772+0000", 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:44.971675+0000", 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:29:39.335 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:29:25.463584+0000", 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.336 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:29:11.784875+0000", 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:29:12.792864+0000", 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.337 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:29:30.799882+0000", 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:29:39.338 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:28:46.298606+0000", 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.1f", 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/283275195": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2738019433": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1268273764": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/567103210": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2891059051": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2170219983": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:29:39.340 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/4082841962": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/570623774": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1379780741": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2880368026": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2044358308": "2024-04-07T03:21:47.915758+0000" 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-06T03:29:39.341 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-06T03:29:39.342 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-06T03:29:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-06T03:29:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-06T03:29:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-06T03:29:39.343 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:29:39.343 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:29:39.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:39.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:39.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:39.483 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:29:39.484 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:29:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:29:39.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:39.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:39.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:39.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:39.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:39.829 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:29:39.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:39.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:39.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:39.977 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_1' created 2024-04-06T03:29:39.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:40.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:40.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:40.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:40.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.136345+0000 mgr.y (mgr.24368) 322 : cluster 0 pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:29:40.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.277662+0000 osd.7 (osd.7) 193 : cluster 0 6.3 scrub starts 2024-04-06T03:29:40.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.304804+0000 osd.7 (osd.7) 194 : cluster 0 6.3 scrub ok 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.648590+0000 osd.4 (osd.4) 199 : cluster 0 5.1c scrub starts 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.649971+0000 osd.4 (osd.4) 200 : cluster 0 5.1c scrub ok 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[29221]: audit 2024-04-06T03:29:39.262920+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.50:0/1201996152' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.136345+0000 mgr.y (mgr.24368) 322 : cluster 0 pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.277662+0000 osd.7 (osd.7) 193 : cluster 0 6.3 scrub starts 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.304804+0000 osd.7 (osd.7) 194 : cluster 0 6.3 scrub ok 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.648590+0000 osd.4 (osd.4) 199 : cluster 0 5.1c scrub starts 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.649971+0000 osd.4 (osd.4) 200 : cluster 0 5.1c scrub ok 2024-04-06T03:29:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:39 smithi050 bash[21467]: audit 2024-04-06T03:29:39.262920+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.50:0/1201996152' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:29:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:40.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:40.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:40.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.136345+0000 mgr.y (mgr.24368) 322 : cluster 0 pgmap v321: 132 pgs: 132 active+clean; 582 KiB data, 840 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:29:40.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.277662+0000 osd.7 (osd.7) 193 : cluster 0 6.3 scrub starts 2024-04-06T03:29:40.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.304804+0000 osd.7 (osd.7) 194 : cluster 0 6.3 scrub ok 2024-04-06T03:29:40.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.648590+0000 osd.4 (osd.4) 199 : cluster 0 5.1c scrub starts 2024-04-06T03:29:40.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.649971+0000 osd.4 (osd.4) 200 : cluster 0 5.1c scrub ok 2024-04-06T03:29:40.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:39 smithi153 bash[23852]: audit 2024-04-06T03:29:39.262920+0000 mon.c (mon.2) 56 : audit 0 from='client.? 172.21.15.50:0/1201996152' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:29:40.346 INFO:tasks.thrashosds.thrasher:j is {'epoch': 181, 'fsid': '0313e520-f3c3-11ee-b649-cb9ed24678a4', 'created': '2024-04-06T03:10:26.969128+0000', 'modified': '2024-04-06T03:29:35.609434+0000', 'last_up_change': '2024-04-06T03:20:09.072730+0000', 'last_in_change': '2024-04-06T03:19:42.545394+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:15:48.096900+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-06T03:20:50.011930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-06T03:20:51.687715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-06T03:20:53.301921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-06T03:20:53.810316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-06T03:20:56.321502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'e607f527-4592-4acd-a47f-76575615ae1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 175, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 3531754646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 3531754646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3531754646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 3531754646}]}, 'public_addr': '172.21.15.50:6803/3531754646', 'cluster_addr': '172.21.15.50:6805/3531754646', 'heartbeat_back_addr': '172.21.15.50:6809/3531754646', 'heartbeat_front_addr': '172.21.15.50:6807/3531754646', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2ae8f784-d36a-4633-8630-2aed04e4ff10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5613250732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 4128410069}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 4128410069}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 4128410069}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 4128410069}]}, 'public_addr': '172.21.15.50:6811/4128410069', 'cluster_addr': '172.21.15.50:6813/4128410069', 'heartbeat_back_addr': '172.21.15.50:6817/4128410069', 'heartbeat_front_addr': '172.21.15.50:6815/4128410069', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '43f31d09-5c16-4aea-8822-e0d877cc931b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1307247904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1307247904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 1307247904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1307247904}]}, 'public_addr': '172.21.15.50:6819/1307247904', 'cluster_addr': '172.21.15.50:6821/1307247904', 'heartbeat_back_addr': '172.21.15.50:6825/1307247904', 'heartbeat_front_addr': '172.21.15.50:6823/1307247904', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4c913325-442e-4ee5-94f1-8b5bca0c3240', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 165, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 1397888508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 1397888508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 1397888508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 1397888508}]}, 'public_addr': '172.21.15.50:6827/1397888508', 'cluster_addr': '172.21.15.50:6829/1397888508', 'heartbeat_back_addr': '172.21.15.50:6833/1397888508', 'heartbeat_front_addr': '172.21.15.50:6831/1397888508', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50c02bfc-c2f2-4aae-b1d5-c38a86726b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 153, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6800', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6801', 'nonce': 3377683473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6802', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6803', 'nonce': 3377683473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6806', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6807', 'nonce': 3377683473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6804', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6805', 'nonce': 3377683473}]}, 'public_addr': '172.21.15.153:6801/3377683473', 'cluster_addr': '172.21.15.153:6803/3377683473', 'heartbeat_back_addr': '172.21.15.153:6807/3377683473', 'heartbeat_front_addr': '172.21.15.153:6805/3377683473', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'c3e8194b-2c20-4ffb-859d-00c33902b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6808', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6809', 'nonce': 357179117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6810', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6811', 'nonce': 357179117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6814', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6815', 'nonce': 357179117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6812', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6813', 'nonce': 357179117}]}, 'public_addr': '172.21.15.153:6809/357179117', 'cluster_addr': '172.21.15.153:6811/357179117', 'heartbeat_back_addr': '172.21.15.153:6815/357179117', 'heartbeat_front_addr': '172.21.15.153:6813/357179117', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98027d53-4324-44d1-89ce-8da85be58efb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6816', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6817', 'nonce': 1552538208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6818', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6819', 'nonce': 1552538208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6822', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6823', 'nonce': 1552538208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6820', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6821', 'nonce': 1552538208}]}, 'public_addr': '172.21.15.153:6817/1552538208', 'cluster_addr': '172.21.15.153:6819/1552538208', 'heartbeat_back_addr': '172.21.15.153:6823/1552538208', 'heartbeat_front_addr': '172.21.15.153:6821/1552538208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6b47d95a-1228-4fb1-a315-5371d9b42bbd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6824', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6825', 'nonce': 2479193924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6826', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6827', 'nonce': 2479193924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6830', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6831', 'nonce': 2479193924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6828', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6829', 'nonce': 2479193924}]}, 'public_addr': '172.21.15.153:6825/2479193924', 'cluster_addr': '172.21.15.153:6827/2479193924', 'heartbeat_back_addr': '172.21.15.153:6831/2479193924', 'heartbeat_front_addr': '172.21.15.153:6829/2479193924', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:52.398069+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:29:20.375772+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:44.971675+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:29:25.463584+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:29:11.784875+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:29:12.792864+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:29:30.799882+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:28:46.298606+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1f', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/283275195': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2738019433': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1268273764': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/567103210': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/2891059051': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2170219983': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/4082841962': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6800/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/570623774': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6801/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1379780741': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2880368026': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2044358308': '2024-04-07T03:21:47.915758+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-06T03:29:40.346 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.1f 2024-04-06T03:29:40.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd rm-pg-upmap-items 2.1f 2024-04-06T03:29:40.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:40.385 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:40.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:40.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:29:40.516 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:40.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:40.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:40.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:40.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:40.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:40.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:40.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:41.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.906059+0000 osd.3 (osd.3) 213 : cluster 0 2.a scrub starts 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:38.907222+0000 osd.3 (osd.3) 214 : cluster 0 2.a scrub ok 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: audit 2024-04-06T03:29:39.383727+0000 mgr.y (mgr.24368) 323 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.414543+0000 osd.0 (osd.0) 178 : cluster 0 2.1f scrub starts 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.415746+0000 osd.0 (osd.0) 179 : cluster 0 2.1f scrub ok 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.440952+0000 osd.1 (osd.1) 169 : cluster 0 2.6 scrub starts 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.442766+0000 osd.1 (osd.1) 170 : cluster 0 2.6 scrub ok 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: audit 2024-04-06T03:29:39.480861+0000 mon.c (mon.2) 57 : audit 1 from='client.? 172.21.15.50:0/1116574603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: audit 2024-04-06T03:29:39.481442+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.937883+0000 osd.2 (osd.2) 157 : cluster 0 3.0 scrub starts 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.942544+0000 osd.2 (osd.2) 158 : cluster 0 3.0 scrub ok 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: audit 2024-04-06T03:29:39.970259+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:39.973673+0000 mon.a (mon.0) 1153 : cluster 0 osdmap e182: 8 total, 8 up, 8 in 2024-04-06T03:29:41.017 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:40 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.136852+0000 mgr.y (mgr.24368) 324 : cluster 0 pgmap v323: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:29:41.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:41.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-04-06T03:29:41.089 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.906059+0000 osd.3 (osd.3) 213 : cluster 0 2.a scrub starts 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:38.907222+0000 osd.3 (osd.3) 214 : cluster 0 2.a scrub ok 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: audit 2024-04-06T03:29:39.383727+0000 mgr.y (mgr.24368) 323 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.414543+0000 osd.0 (osd.0) 178 : cluster 0 2.1f scrub starts 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.415746+0000 osd.0 (osd.0) 179 : cluster 0 2.1f scrub ok 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.440952+0000 osd.1 (osd.1) 169 : cluster 0 2.6 scrub starts 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.442766+0000 osd.1 (osd.1) 170 : cluster 0 2.6 scrub ok 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: audit 2024-04-06T03:29:39.480861+0000 mon.c (mon.2) 57 : audit 1 from='client.? 172.21.15.50:0/1116574603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: audit 2024-04-06T03:29:39.481442+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.937883+0000 osd.2 (osd.2) 157 : cluster 0 3.0 scrub starts 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.942544+0000 osd.2 (osd.2) 158 : cluster 0 3.0 scrub ok 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: audit 2024-04-06T03:29:39.970259+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:39.973673+0000 mon.a (mon.0) 1153 : cluster 0 osdmap e182: 8 total, 8 up, 8 in 2024-04-06T03:29:41.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.136852+0000 mgr.y (mgr.24368) 324 : cluster 0 pgmap v323: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.906059+0000 osd.3 (osd.3) 213 : cluster 0 2.a scrub starts 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:38.907222+0000 osd.3 (osd.3) 214 : cluster 0 2.a scrub ok 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: audit 2024-04-06T03:29:39.383727+0000 mgr.y (mgr.24368) 323 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.414543+0000 osd.0 (osd.0) 178 : cluster 0 2.1f scrub starts 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.415746+0000 osd.0 (osd.0) 179 : cluster 0 2.1f scrub ok 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.440952+0000 osd.1 (osd.1) 169 : cluster 0 2.6 scrub starts 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.442766+0000 osd.1 (osd.1) 170 : cluster 0 2.6 scrub ok 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: audit 2024-04-06T03:29:39.480861+0000 mon.c (mon.2) 57 : audit 1 from='client.? 172.21.15.50:0/1116574603' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-06T03:29:41.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: audit 2024-04-06T03:29:39.481442+0000 mon.a (mon.0) 1151 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-04-06T03:29:41.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.937883+0000 osd.2 (osd.2) 157 : cluster 0 3.0 scrub starts 2024-04-06T03:29:41.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.942544+0000 osd.2 (osd.2) 158 : cluster 0 3.0 scrub ok 2024-04-06T03:29:41.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: audit 2024-04-06T03:29:39.970259+0000 mon.a (mon.0) 1152 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-04-06T03:29:41.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:39.973673+0000 mon.a (mon.0) 1153 : cluster 0 osdmap e182: 8 total, 8 up, 8 in 2024-04-06T03:29:41.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:40 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.136852+0000 mgr.y (mgr.24368) 324 : cluster 0 pgmap v323: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:29:41.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:41.155 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:41.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:41.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:41.382 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:41.408 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:41.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:41.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:41.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:41.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:41.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:41.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:41.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:41.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:41.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:41.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:42.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:42.052 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:42.053 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:41 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.428237+0000 osd.1 (osd.1) 171 : cluster 0 4.2 scrub starts 2024-04-06T03:29:42.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:41 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.430224+0000 osd.1 (osd.1) 172 : cluster 0 4.2 scrub ok 2024-04-06T03:29:42.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:41 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.953964+0000 osd.2 (osd.2) 159 : cluster 0 2.10 scrub starts 2024-04-06T03:29:42.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:41 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.956513+0000 osd.2 (osd.2) 160 : cluster 0 2.10 scrub ok 2024-04-06T03:29:42.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:41 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.970397+0000 mon.a (mon.0) 1154 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:29:42.054 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:41 smithi153 bash[23852]: cluster 2024-04-06T03:29:40.989340+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e183: 8 total, 8 up, 8 in 2024-04-06T03:29:42.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:42.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:42.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.428237+0000 osd.1 (osd.1) 171 : cluster 0 4.2 scrub starts 2024-04-06T03:29:42.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.430224+0000 osd.1 (osd.1) 172 : cluster 0 4.2 scrub ok 2024-04-06T03:29:42.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.953964+0000 osd.2 (osd.2) 159 : cluster 0 2.10 scrub starts 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.956513+0000 osd.2 (osd.2) 160 : cluster 0 2.10 scrub ok 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.970397+0000 mon.a (mon.0) 1154 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[21467]: cluster 2024-04-06T03:29:40.989340+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e183: 8 total, 8 up, 8 in 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.428237+0000 osd.1 (osd.1) 171 : cluster 0 4.2 scrub starts 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.430224+0000 osd.1 (osd.1) 172 : cluster 0 4.2 scrub ok 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.953964+0000 osd.2 (osd.2) 159 : cluster 0 2.10 scrub starts 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.956513+0000 osd.2 (osd.2) 160 : cluster 0 2.10 scrub ok 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.970397+0000 mon.a (mon.0) 1154 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:29:42.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:41 smithi050 bash[29221]: cluster 2024-04-06T03:29:40.989340+0000 mon.a (mon.0) 1155 : cluster 0 osdmap e183: 8 total, 8 up, 8 in 2024-04-06T03:29:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:42.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:42.317 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:42.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:42.446 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:42.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:42.577 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:42.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:42.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:42.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:42.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:42.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:42.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:42.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:42.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:43.008 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:29:43.008 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:29:43.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:43.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:42 smithi153 bash[23852]: cluster 2024-04-06T03:29:41.274494+0000 osd.7 (osd.7) 195 : cluster 0 2.1c scrub starts 2024-04-06T03:29:43.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:42 smithi153 bash[23852]: cluster 2024-04-06T03:29:41.276096+0000 osd.7 (osd.7) 196 : cluster 0 2.1c scrub ok 2024-04-06T03:29:43.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:42 smithi153 bash[23852]: cluster 2024-04-06T03:29:41.992374+0000 mon.a (mon.0) 1156 : cluster 0 osdmap e184: 8 total, 8 up, 8 in 2024-04-06T03:29:43.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:42 smithi153 bash[23852]: cluster 2024-04-06T03:29:42.137545+0000 mgr.y (mgr.24368) 325 : cluster 0 pgmap v326: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:43.085 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:43.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:43.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:43.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:43.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:43.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:43.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:43.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[29221]: cluster 2024-04-06T03:29:41.274494+0000 osd.7 (osd.7) 195 : cluster 0 2.1c scrub starts 2024-04-06T03:29:43.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[29221]: cluster 2024-04-06T03:29:41.276096+0000 osd.7 (osd.7) 196 : cluster 0 2.1c scrub ok 2024-04-06T03:29:43.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[29221]: cluster 2024-04-06T03:29:41.992374+0000 mon.a (mon.0) 1156 : cluster 0 osdmap e184: 8 total, 8 up, 8 in 2024-04-06T03:29:43.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[29221]: cluster 2024-04-06T03:29:42.137545+0000 mgr.y (mgr.24368) 325 : cluster 0 pgmap v326: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:43.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[21467]: cluster 2024-04-06T03:29:41.274494+0000 osd.7 (osd.7) 195 : cluster 0 2.1c scrub starts 2024-04-06T03:29:43.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[21467]: cluster 2024-04-06T03:29:41.276096+0000 osd.7 (osd.7) 196 : cluster 0 2.1c scrub ok 2024-04-06T03:29:43.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[21467]: cluster 2024-04-06T03:29:41.992374+0000 mon.a (mon.0) 1156 : cluster 0 osdmap e184: 8 total, 8 up, 8 in 2024-04-06T03:29:43.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:42 smithi050 bash[21467]: cluster 2024-04-06T03:29:42.137545+0000 mgr.y (mgr.24368) 325 : cluster 0 pgmap v326: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:43.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:43.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:43.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:43.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:43.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-06T03:29:43.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:43.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:43.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:43.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:43.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:44.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:44.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:44.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:44.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:44.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:44 smithi050 bash[29221]: cluster 2024-04-06T03:29:42.256433+0000 osd.7 (osd.7) 197 : cluster 0 3.15 scrub starts 2024-04-06T03:29:44.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:44 smithi050 bash[29221]: cluster 2024-04-06T03:29:42.260401+0000 osd.7 (osd.7) 198 : cluster 0 3.15 scrub ok 2024-04-06T03:29:44.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:44 smithi050 bash[29221]: cluster 2024-04-06T03:29:42.924945+0000 osd.2 (osd.2) 161 : cluster 0 2.c scrub starts 2024-04-06T03:29:44.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:44 smithi050 bash[29221]: cluster 2024-04-06T03:29:42.926377+0000 osd.2 (osd.2) 162 : cluster 0 2.c scrub ok 2024-04-06T03:29:44.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:43 smithi050 bash[21467]: cluster 2024-04-06T03:29:42.256433+0000 osd.7 (osd.7) 197 : cluster 0 3.15 scrub starts 2024-04-06T03:29:44.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:43 smithi050 bash[21467]: cluster 2024-04-06T03:29:42.260401+0000 osd.7 (osd.7) 198 : cluster 0 3.15 scrub ok 2024-04-06T03:29:44.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:43 smithi050 bash[21467]: cluster 2024-04-06T03:29:42.924945+0000 osd.2 (osd.2) 161 : cluster 0 2.c scrub starts 2024-04-06T03:29:44.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:43 smithi050 bash[21467]: cluster 2024-04-06T03:29:42.926377+0000 osd.2 (osd.2) 162 : cluster 0 2.c scrub ok 2024-04-06T03:29:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:44.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:43 smithi153 bash[23852]: cluster 2024-04-06T03:29:42.256433+0000 osd.7 (osd.7) 197 : cluster 0 3.15 scrub starts 2024-04-06T03:29:44.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:43 smithi153 bash[23852]: cluster 2024-04-06T03:29:42.260401+0000 osd.7 (osd.7) 198 : cluster 0 3.15 scrub ok 2024-04-06T03:29:44.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:43 smithi153 bash[23852]: cluster 2024-04-06T03:29:42.924945+0000 osd.2 (osd.2) 161 : cluster 0 2.c scrub starts 2024-04-06T03:29:44.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:43 smithi153 bash[23852]: cluster 2024-04-06T03:29:42.926377+0000 osd.2 (osd.2) 162 : cluster 0 2.c scrub ok 2024-04-06T03:29:44.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:44.385 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:44.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:44.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:44.608 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:44.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:44.645 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:44.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:44.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:44.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:44.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:44.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:45.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:45.033 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:45.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:45.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:45.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:45.274 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:45 smithi153 bash[23852]: cluster 2024-04-06T03:29:44.138754+0000 mgr.y (mgr.24368) 326 : cluster 0 pgmap v327: 148 pgs: 5 creating+peering, 7 unknown, 136 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:45.292 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:45.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:45.421 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:45.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:45 smithi050 bash[21467]: cluster 2024-04-06T03:29:44.138754+0000 mgr.y (mgr.24368) 326 : cluster 0 pgmap v327: 148 pgs: 5 creating+peering, 7 unknown, 136 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:45.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:45 smithi050 bash[29221]: cluster 2024-04-06T03:29:44.138754+0000 mgr.y (mgr.24368) 326 : cluster 0 pgmap v327: 148 pgs: 5 creating+peering, 7 unknown, 136 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:29:45.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:45.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:45.627 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:45.627 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:45.627 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:29:45.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:45.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:45.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:45.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:45.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:45.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:46.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:46.087 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[21467]: cluster 2024-04-06T03:29:44.264922+0000 osd.7 (osd.7) 199 : cluster 0 5.4 scrub starts 2024-04-06T03:29:46.087 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[21467]: cluster 2024-04-06T03:29:44.267111+0000 osd.7 (osd.7) 200 : cluster 0 5.4 scrub ok 2024-04-06T03:29:46.087 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[21467]: cluster 2024-04-06T03:29:44.500383+0000 osd.1 (osd.1) 173 : cluster 0 6.1 scrub starts 2024-04-06T03:29:46.088 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[21467]: cluster 2024-04-06T03:29:44.517712+0000 osd.1 (osd.1) 174 : cluster 0 6.1 scrub ok 2024-04-06T03:29:46.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[29221]: cluster 2024-04-06T03:29:44.264922+0000 osd.7 (osd.7) 199 : cluster 0 5.4 scrub starts 2024-04-06T03:29:46.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[29221]: cluster 2024-04-06T03:29:44.267111+0000 osd.7 (osd.7) 200 : cluster 0 5.4 scrub ok 2024-04-06T03:29:46.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[29221]: cluster 2024-04-06T03:29:44.500383+0000 osd.1 (osd.1) 173 : cluster 0 6.1 scrub starts 2024-04-06T03:29:46.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:46 smithi050 bash[29221]: cluster 2024-04-06T03:29:44.517712+0000 osd.1 (osd.1) 174 : cluster 0 6.1 scrub ok 2024-04-06T03:29:46.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:46.193 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:46.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:46 smithi153 bash[23852]: cluster 2024-04-06T03:29:44.264922+0000 osd.7 (osd.7) 199 : cluster 0 5.4 scrub starts 2024-04-06T03:29:46.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:46 smithi153 bash[23852]: cluster 2024-04-06T03:29:44.267111+0000 osd.7 (osd.7) 200 : cluster 0 5.4 scrub ok 2024-04-06T03:29:46.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:46 smithi153 bash[23852]: cluster 2024-04-06T03:29:44.500383+0000 osd.1 (osd.1) 173 : cluster 0 6.1 scrub starts 2024-04-06T03:29:46.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:46 smithi153 bash[23852]: cluster 2024-04-06T03:29:44.517712+0000 osd.1 (osd.1) 174 : cluster 0 6.1 scrub ok 2024-04-06T03:29:46.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:46.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:46.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:46.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:46.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:46.587 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:46.690 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:46.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:46.848 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:46.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:46.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:46.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:46.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:47.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:47.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:47.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:45.258567+0000 osd.7 (osd.7) 201 : cluster 0 2.7 scrub starts 2024-04-06T03:29:47.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:45.261070+0000 osd.7 (osd.7) 202 : cluster 0 2.7 scrub ok 2024-04-06T03:29:47.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:45.974782+0000 osd.2 (osd.2) 163 : cluster 0 6.14 scrub starts 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:45.976519+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub ok 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:46.139694+0000 mgr.y (mgr.24368) 327 : cluster 0 pgmap v328: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:46.588547+0000 osd.4 (osd.4) 201 : cluster 0 6.2 scrub starts 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[29221]: cluster 2024-04-06T03:29:46.589862+0000 osd.4 (osd.4) 202 : cluster 0 6.2 scrub ok 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:45.258567+0000 osd.7 (osd.7) 201 : cluster 0 2.7 scrub starts 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:45.261070+0000 osd.7 (osd.7) 202 : cluster 0 2.7 scrub ok 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:45.974782+0000 osd.2 (osd.2) 163 : cluster 0 6.14 scrub starts 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:45.976519+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub ok 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:46.139694+0000 mgr.y (mgr.24368) 327 : cluster 0 pgmap v328: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:46.588547+0000 osd.4 (osd.4) 201 : cluster 0 6.2 scrub starts 2024-04-06T03:29:47.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:47 smithi050 bash[21467]: cluster 2024-04-06T03:29:46.589862+0000 osd.4 (osd.4) 202 : cluster 0 6.2 scrub ok 2024-04-06T03:29:47.241 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:45.258567+0000 osd.7 (osd.7) 201 : cluster 0 2.7 scrub starts 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:45.261070+0000 osd.7 (osd.7) 202 : cluster 0 2.7 scrub ok 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:45.974782+0000 osd.2 (osd.2) 163 : cluster 0 6.14 scrub starts 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:45.976519+0000 osd.2 (osd.2) 164 : cluster 0 6.14 scrub ok 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:46.139694+0000 mgr.y (mgr.24368) 327 : cluster 0 pgmap v328: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:46.588547+0000 osd.4 (osd.4) 201 : cluster 0 6.2 scrub starts 2024-04-06T03:29:47.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:47 smithi153 bash[23852]: cluster 2024-04-06T03:29:46.589862+0000 osd.4 (osd.4) 202 : cluster 0 6.2 scrub ok 2024-04-06T03:29:47.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:47.371 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:47.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:47.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:47.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:47.600 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:47.628 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:47.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:47.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:47.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:47.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:47.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:47.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:48.015 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:48.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:48.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:48.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:48.270 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[29221]: cluster 2024-04-06T03:29:46.944573+0000 osd.2 (osd.2) 165 : cluster 0 2.3 scrub starts 2024-04-06T03:29:48.270 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[29221]: cluster 2024-04-06T03:29:46.947706+0000 osd.2 (osd.2) 166 : cluster 0 2.3 scrub ok 2024-04-06T03:29:48.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.365671+0000 osd.0 (osd.0) 180 : cluster 0 6.18 scrub starts 2024-04-06T03:29:48.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.382871+0000 osd.0 (osd.0) 181 : cluster 0 6.18 scrub ok 2024-04-06T03:29:48.271 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[21467]: cluster 2024-04-06T03:29:46.944573+0000 osd.2 (osd.2) 165 : cluster 0 2.3 scrub starts 2024-04-06T03:29:48.271 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[21467]: cluster 2024-04-06T03:29:46.947706+0000 osd.2 (osd.2) 166 : cluster 0 2.3 scrub ok 2024-04-06T03:29:48.271 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.365671+0000 osd.0 (osd.0) 180 : cluster 0 6.18 scrub starts 2024-04-06T03:29:48.271 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:48 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.382871+0000 osd.0 (osd.0) 181 : cluster 0 6.18 scrub ok 2024-04-06T03:29:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:48.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:48 smithi153 bash[23852]: cluster 2024-04-06T03:29:46.944573+0000 osd.2 (osd.2) 165 : cluster 0 2.3 scrub starts 2024-04-06T03:29:48.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:48 smithi153 bash[23852]: cluster 2024-04-06T03:29:46.947706+0000 osd.2 (osd.2) 166 : cluster 0 2.3 scrub ok 2024-04-06T03:29:48.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:48 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.365671+0000 osd.0 (osd.0) 180 : cluster 0 6.18 scrub starts 2024-04-06T03:29:48.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:48 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.382871+0000 osd.0 (osd.0) 181 : cluster 0 6.18 scrub ok 2024-04-06T03:29:48.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:48.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:48.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:48.566 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:29:48.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:48.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:48.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:48.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:48.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:48.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:49.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.439880+0000 osd.1 (osd.1) 175 : cluster 0 2.9 scrub starts 2024-04-06T03:29:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.442001+0000 osd.1 (osd.1) 176 : cluster 0 2.9 scrub ok 2024-04-06T03:29:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.779088+0000 osd.3 (osd.3) 215 : cluster 0 3.7 scrub starts 2024-04-06T03:29:49.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.782706+0000 osd.3 (osd.3) 216 : cluster 0 3.7 scrub ok 2024-04-06T03:29:49.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.931437+0000 osd.2 (osd.2) 167 : cluster 0 2.16 scrub starts 2024-04-06T03:29:49.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:47.933667+0000 osd.2 (osd.2) 168 : cluster 0 2.16 scrub ok 2024-04-06T03:29:49.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: cluster 2024-04-06T03:29:48.140139+0000 mgr.y (mgr.24368) 328 : cluster 0 pgmap v329: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:29:49.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: audit 2024-04-06T03:29:48.875302+0000 mon.a (mon.0) 1157 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:49.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[23852]: audit 2024-04-06T03:29:48.877499+0000 mon.a (mon.0) 1158 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:49.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:49.181 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:49.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:49.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.439880+0000 osd.1 (osd.1) 175 : cluster 0 2.9 scrub starts 2024-04-06T03:29:49.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.442001+0000 osd.1 (osd.1) 176 : cluster 0 2.9 scrub ok 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.779088+0000 osd.3 (osd.3) 215 : cluster 0 3.7 scrub starts 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.782706+0000 osd.3 (osd.3) 216 : cluster 0 3.7 scrub ok 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.931437+0000 osd.2 (osd.2) 167 : cluster 0 2.16 scrub starts 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:47.933667+0000 osd.2 (osd.2) 168 : cluster 0 2.16 scrub ok 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: cluster 2024-04-06T03:29:48.140139+0000 mgr.y (mgr.24368) 328 : cluster 0 pgmap v329: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: audit 2024-04-06T03:29:48.875302+0000 mon.a (mon.0) 1157 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[29221]: audit 2024-04-06T03:29:48.877499+0000 mon.a (mon.0) 1158 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.439880+0000 osd.1 (osd.1) 175 : cluster 0 2.9 scrub starts 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.442001+0000 osd.1 (osd.1) 176 : cluster 0 2.9 scrub ok 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.779088+0000 osd.3 (osd.3) 215 : cluster 0 3.7 scrub starts 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.782706+0000 osd.3 (osd.3) 216 : cluster 0 3.7 scrub ok 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.931437+0000 osd.2 (osd.2) 167 : cluster 0 2.16 scrub starts 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:47.933667+0000 osd.2 (osd.2) 168 : cluster 0 2.16 scrub ok 2024-04-06T03:29:49.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: cluster 2024-04-06T03:29:48.140139+0000 mgr.y (mgr.24368) 328 : cluster 0 pgmap v329: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:29:49.298 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: audit 2024-04-06T03:29:48.875302+0000 mon.a (mon.0) 1157 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:29:49.298 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21467]: audit 2024-04-06T03:29:48.877499+0000 mon.a (mon.0) 1158 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:29:49.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:49.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:49.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:49.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:49.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:29:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:29:49.556 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:29:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:29:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:29:49.572 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:49.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:49.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:49.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:49.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:49.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:49.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:49.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:50.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:50 smithi050 bash[21467]: cluster 2024-04-06T03:29:48.893626+0000 osd.2 (osd.2) 169 : cluster 0 3.4 scrub starts 2024-04-06T03:29:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:50 smithi050 bash[21467]: cluster 2024-04-06T03:29:48.898415+0000 osd.2 (osd.2) 170 : cluster 0 3.4 scrub ok 2024-04-06T03:29:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:50 smithi050 bash[21467]: audit 2024-04-06T03:29:49.391537+0000 mgr.y (mgr.24368) 329 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:50.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:50 smithi050 bash[29221]: cluster 2024-04-06T03:29:48.893626+0000 osd.2 (osd.2) 169 : cluster 0 3.4 scrub starts 2024-04-06T03:29:50.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:50 smithi050 bash[29221]: cluster 2024-04-06T03:29:48.898415+0000 osd.2 (osd.2) 170 : cluster 0 3.4 scrub ok 2024-04-06T03:29:50.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:50 smithi050 bash[29221]: audit 2024-04-06T03:29:49.391537+0000 mgr.y (mgr.24368) 329 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:50.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:50.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:50 smithi153 bash[23852]: cluster 2024-04-06T03:29:48.893626+0000 osd.2 (osd.2) 169 : cluster 0 3.4 scrub starts 2024-04-06T03:29:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:50 smithi153 bash[23852]: cluster 2024-04-06T03:29:48.898415+0000 osd.2 (osd.2) 170 : cluster 0 3.4 scrub ok 2024-04-06T03:29:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:50 smithi153 bash[23852]: audit 2024-04-06T03:29:49.391537+0000 mgr.y (mgr.24368) 329 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:29:50.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:50.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:50.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:50.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:50.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:50.595 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:50.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:50.666 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:29:50.687 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:29:50.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:50.707 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:29:50.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:50.728 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:29:50.752 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:29:50.773 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:29:50.794 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:29:50.818 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:29:50.819 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:29:50.819 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:29:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:29:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:29:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:29:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:29:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:29:50.820 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:29:50.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:50.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:50.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:50.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:50.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:51.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[29221]: cluster 2024-04-06T03:29:49.710046+0000 osd.3 (osd.3) 217 : cluster 0 3.18 scrub starts 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[29221]: cluster 2024-04-06T03:29:49.712004+0000 osd.3 (osd.3) 218 : cluster 0 3.18 scrub ok 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.140918+0000 mgr.y (mgr.24368) 330 : cluster 0 pgmap v330: 148 pgs: 148 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[29221]: audit 2024-04-06T03:29:50.650624+0000 mon.a (mon.0) 1159 : audit 0 from='client.? 172.21.15.50:0/3509348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[21467]: cluster 2024-04-06T03:29:49.710046+0000 osd.3 (osd.3) 217 : cluster 0 3.18 scrub starts 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[21467]: cluster 2024-04-06T03:29:49.712004+0000 osd.3 (osd.3) 218 : cluster 0 3.18 scrub ok 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.140918+0000 mgr.y (mgr.24368) 330 : cluster 0 pgmap v330: 148 pgs: 148 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:51.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:51 smithi050 bash[21467]: audit 2024-04-06T03:29:50.650624+0000 mon.a (mon.0) 1159 : audit 0 from='client.? 172.21.15.50:0/3509348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:51.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:51.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:51.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:51 smithi153 bash[23852]: cluster 2024-04-06T03:29:49.710046+0000 osd.3 (osd.3) 217 : cluster 0 3.18 scrub starts 2024-04-06T03:29:51.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:51 smithi153 bash[23852]: cluster 2024-04-06T03:29:49.712004+0000 osd.3 (osd.3) 218 : cluster 0 3.18 scrub ok 2024-04-06T03:29:51.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:51 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.140918+0000 mgr.y (mgr.24368) 330 : cluster 0 pgmap v330: 148 pgs: 148 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:29:51.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:51 smithi153 bash[23852]: audit 2024-04-06T03:29:50.650624+0000 mon.a (mon.0) 1159 : audit 0 from='client.? 172.21.15.50:0/3509348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:29:51.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:51.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:51.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:51.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:51.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:51.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:51.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:51.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:51.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:51.757 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:51.779 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:29:51.779 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:29:51.779 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:29:51.779 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:29:51.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:51.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:51.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:51.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:52.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:52.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.586677+0000 osd.2 (osd.2) 171 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.586751+0000 osd.2 (osd.2) 172 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.672492+0000 osd.3 (osd.3) 219 : cluster 0 3.1e scrub starts 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.675114+0000 osd.3 (osd.3) 220 : cluster 0 3.1e scrub ok 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.891369+0000 osd.2 (osd.2) 173 : cluster 0 5.d scrub starts 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:50.893490+0000 osd.2 (osd.2) 174 : cluster 0 5.d scrub ok 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.045436+0000 mon.a (mon.0) 1160 : cluster 0 osdmap e185: 8 total, 8 up, 8 in 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.597937+0000 osd.4 (osd.4) 203 : cluster 0 6.e scrub starts 2024-04-06T03:29:52.139 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.612568+0000 osd.4 (osd.4) 204 : cluster 0 6.e scrub ok 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.586677+0000 osd.2 (osd.2) 171 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.586751+0000 osd.2 (osd.2) 172 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.672492+0000 osd.3 (osd.3) 219 : cluster 0 3.1e scrub starts 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.675114+0000 osd.3 (osd.3) 220 : cluster 0 3.1e scrub ok 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.891369+0000 osd.2 (osd.2) 173 : cluster 0 5.d scrub starts 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:50.893490+0000 osd.2 (osd.2) 174 : cluster 0 5.d scrub ok 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.045436+0000 mon.a (mon.0) 1160 : cluster 0 osdmap e185: 8 total, 8 up, 8 in 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.597937+0000 osd.4 (osd.4) 203 : cluster 0 6.e scrub starts 2024-04-06T03:29:52.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:52 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.612568+0000 osd.4 (osd.4) 204 : cluster 0 6.e scrub ok 2024-04-06T03:29:52.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:52.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:52.270 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.586677+0000 osd.2 (osd.2) 171 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:52.270 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.586751+0000 osd.2 (osd.2) 172 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:52.270 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.672492+0000 osd.3 (osd.3) 219 : cluster 0 3.1e scrub starts 2024-04-06T03:29:52.270 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.675114+0000 osd.3 (osd.3) 220 : cluster 0 3.1e scrub ok 2024-04-06T03:29:52.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.891369+0000 osd.2 (osd.2) 173 : cluster 0 5.d scrub starts 2024-04-06T03:29:52.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:50.893490+0000 osd.2 (osd.2) 174 : cluster 0 5.d scrub ok 2024-04-06T03:29:52.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.045436+0000 mon.a (mon.0) 1160 : cluster 0 osdmap e185: 8 total, 8 up, 8 in 2024-04-06T03:29:52.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.597937+0000 osd.4 (osd.4) 203 : cluster 0 6.e scrub starts 2024-04-06T03:29:52.271 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:52 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.612568+0000 osd.4 (osd.4) 204 : cluster 0 6.e scrub ok 2024-04-06T03:29:52.285 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:52.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:52.414 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:52.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:52.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:52.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:52.677 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:52.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-06T03:29:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:52.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:52.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:52.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:52.932 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:52.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:53.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:53.044 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.206301+0000 osd.7 (osd.7) 203 : cluster 0 1.0 scrub starts 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.211556+0000 osd.7 (osd.7) 204 : cluster 0 1.0 scrub ok 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.856939+0000 osd.2 (osd.2) 175 : cluster 0 2.17 scrub starts 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:51.858939+0000 osd.2 (osd.2) 176 : cluster 0 2.17 scrub ok 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.141553+0000 mgr.y (mgr.24368) 331 : cluster 0 pgmap v332: 148 pgs: 148 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.623903+0000 osd.4 (osd.4) 205 : cluster 0 3.1d scrub starts 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.628063+0000 osd.4 (osd.4) 206 : cluster 0 3.1d scrub ok 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.206301+0000 osd.7 (osd.7) 203 : cluster 0 1.0 scrub starts 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.211556+0000 osd.7 (osd.7) 204 : cluster 0 1.0 scrub ok 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.856939+0000 osd.2 (osd.2) 175 : cluster 0 2.17 scrub starts 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:51.858939+0000 osd.2 (osd.2) 176 : cluster 0 2.17 scrub ok 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.141553+0000 mgr.y (mgr.24368) 331 : cluster 0 pgmap v332: 148 pgs: 148 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.623903+0000 osd.4 (osd.4) 205 : cluster 0 3.1d scrub starts 2024-04-06T03:29:53.045 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:53 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.628063+0000 osd.4 (osd.4) 206 : cluster 0 3.1d scrub ok 2024-04-06T03:29:53.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:53.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.206301+0000 osd.7 (osd.7) 203 : cluster 0 1.0 scrub starts 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.211556+0000 osd.7 (osd.7) 204 : cluster 0 1.0 scrub ok 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.856939+0000 osd.2 (osd.2) 175 : cluster 0 2.17 scrub starts 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:51.858939+0000 osd.2 (osd.2) 176 : cluster 0 2.17 scrub ok 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.141553+0000 mgr.y (mgr.24368) 331 : cluster 0 pgmap v332: 148 pgs: 148 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.623903+0000 osd.4 (osd.4) 205 : cluster 0 3.1d scrub starts 2024-04-06T03:29:53.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:53 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.628063+0000 osd.4 (osd.4) 206 : cluster 0 3.1d scrub ok 2024-04-06T03:29:53.191 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:53.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:53.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:53.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:53.454 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:53.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:53.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:53.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:53.712 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:53.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:53.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:53.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:53.972 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:53.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:53.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:54.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:54.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:54.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:54.218 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:54 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.548965+0000 osd.1 (osd.1) 177 : cluster 0 2.2 scrub starts 2024-04-06T03:29:54.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:54 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.551318+0000 osd.1 (osd.1) 178 : cluster 0 2.2 scrub ok 2024-04-06T03:29:54.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:54 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.854521+0000 osd.2 (osd.2) 177 : cluster 0 2.e scrub starts 2024-04-06T03:29:54.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:54 smithi153 bash[23852]: cluster 2024-04-06T03:29:52.855991+0000 osd.2 (osd.2) 178 : cluster 0 2.e scrub ok 2024-04-06T03:29:54.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.548965+0000 osd.1 (osd.1) 177 : cluster 0 2.2 scrub starts 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.551318+0000 osd.1 (osd.1) 178 : cluster 0 2.2 scrub ok 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.854521+0000 osd.2 (osd.2) 177 : cluster 0 2.e scrub starts 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[29221]: cluster 2024-04-06T03:29:52.855991+0000 osd.2 (osd.2) 178 : cluster 0 2.e scrub ok 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.548965+0000 osd.1 (osd.1) 177 : cluster 0 2.2 scrub starts 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.551318+0000 osd.1 (osd.1) 178 : cluster 0 2.2 scrub ok 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.854521+0000 osd.2 (osd.2) 177 : cluster 0 2.e scrub starts 2024-04-06T03:29:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:54 smithi050 bash[21467]: cluster 2024-04-06T03:29:52.855991+0000 osd.2 (osd.2) 178 : cluster 0 2.e scrub ok 2024-04-06T03:29:54.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:54.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:54.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:54.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:54.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:54.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:54.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:54.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:54.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:54.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:54.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:54.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:54.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:29:55.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:55.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:55.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.275 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:55.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:55.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:55 smithi153 bash[23852]: cluster 2024-04-06T03:29:53.632609+0000 osd.3 (osd.3) 221 : cluster 0 6.1f scrub starts 2024-04-06T03:29:55.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:55 smithi153 bash[23852]: cluster 2024-04-06T03:29:53.697368+0000 osd.3 (osd.3) 222 : cluster 0 6.1f scrub ok 2024-04-06T03:29:55.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:55 smithi153 bash[23852]: cluster 2024-04-06T03:29:53.828013+0000 osd.0 (osd.0) 182 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:55.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:55 smithi153 bash[23852]: cluster 2024-04-06T03:29:53.828126+0000 osd.0 (osd.0) 183 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:55.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:55 smithi153 bash[23852]: cluster 2024-04-06T03:29:54.142748+0000 mgr.y (mgr.24368) 332 : cluster 0 pgmap v333: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:29:55.390 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:55 smithi153 bash[23852]: cluster 2024-04-06T03:29:54.495534+0000 mon.a (mon.0) 1161 : cluster 0 osdmap e186: 8 total, 8 up, 8 in 2024-04-06T03:29:55.405 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[21467]: cluster 2024-04-06T03:29:53.632609+0000 osd.3 (osd.3) 221 : cluster 0 6.1f scrub starts 2024-04-06T03:29:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[21467]: cluster 2024-04-06T03:29:53.697368+0000 osd.3 (osd.3) 222 : cluster 0 6.1f scrub ok 2024-04-06T03:29:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[21467]: cluster 2024-04-06T03:29:53.828013+0000 osd.0 (osd.0) 182 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[21467]: cluster 2024-04-06T03:29:53.828126+0000 osd.0 (osd.0) 183 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[21467]: cluster 2024-04-06T03:29:54.142748+0000 mgr.y (mgr.24368) 332 : cluster 0 pgmap v333: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[21467]: cluster 2024-04-06T03:29:54.495534+0000 mon.a (mon.0) 1161 : cluster 0 osdmap e186: 8 total, 8 up, 8 in 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[29221]: cluster 2024-04-06T03:29:53.632609+0000 osd.3 (osd.3) 221 : cluster 0 6.1f scrub starts 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[29221]: cluster 2024-04-06T03:29:53.697368+0000 osd.3 (osd.3) 222 : cluster 0 6.1f scrub ok 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[29221]: cluster 2024-04-06T03:29:53.828013+0000 osd.0 (osd.0) 182 : cluster 0 purged_snaps scrub starts 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[29221]: cluster 2024-04-06T03:29:53.828126+0000 osd.0 (osd.0) 183 : cluster 0 purged_snaps scrub ok 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[29221]: cluster 2024-04-06T03:29:54.142748+0000 mgr.y (mgr.24368) 332 : cluster 0 pgmap v333: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:29:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:55 smithi050 bash[29221]: cluster 2024-04-06T03:29:54.495534+0000 mon.a (mon.0) 1161 : cluster 0 osdmap e186: 8 total, 8 up, 8 in 2024-04-06T03:29:55.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:55.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:29:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:29:55.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:55.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:55.781 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:55.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:55.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:55.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:55.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:56.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:56.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:56.056 INFO:teuthology.orchestra.run.smithi050.stderr:clear 2.1f pg_upmap_items mapping 2024-04-06T03:29:56.101 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[29221]: cluster 2024-04-06T03:29:54.282859+0000 osd.7 (osd.7) 205 : cluster 0 6.1c scrub starts 2024-04-06T03:29:56.101 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[29221]: cluster 2024-04-06T03:29:54.284833+0000 osd.7 (osd.7) 206 : cluster 0 6.1c scrub ok 2024-04-06T03:29:56.101 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[29221]: cluster 2024-04-06T03:29:55.556250+0000 osd.4 (osd.4) 207 : cluster 0 3.a scrub starts 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[29221]: cluster 2024-04-06T03:29:55.561327+0000 osd.4 (osd.4) 208 : cluster 0 3.a scrub ok 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[29221]: audit 2024-04-06T03:29:55.858460+0000 mon.a (mon.0) 1162 : audit 1 from='client.? 172.21.15.50:0/4160071767' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1f"}]: dispatch 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[21467]: cluster 2024-04-06T03:29:54.282859+0000 osd.7 (osd.7) 205 : cluster 0 6.1c scrub starts 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[21467]: cluster 2024-04-06T03:29:54.284833+0000 osd.7 (osd.7) 206 : cluster 0 6.1c scrub ok 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[21467]: cluster 2024-04-06T03:29:55.556250+0000 osd.4 (osd.4) 207 : cluster 0 3.a scrub starts 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[21467]: cluster 2024-04-06T03:29:55.561327+0000 osd.4 (osd.4) 208 : cluster 0 3.a scrub ok 2024-04-06T03:29:56.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:56 smithi050 bash[21467]: audit 2024-04-06T03:29:55.858460+0000 mon.a (mon.0) 1162 : audit 1 from='client.? 172.21.15.50:0/4160071767' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1f"}]: dispatch 2024-04-06T03:29:56.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:56.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:56.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:56.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:56.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:56.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:56.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:56.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:56.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:56 smithi153 bash[23852]: cluster 2024-04-06T03:29:54.282859+0000 osd.7 (osd.7) 205 : cluster 0 6.1c scrub starts 2024-04-06T03:29:56.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:56 smithi153 bash[23852]: cluster 2024-04-06T03:29:54.284833+0000 osd.7 (osd.7) 206 : cluster 0 6.1c scrub ok 2024-04-06T03:29:56.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:56 smithi153 bash[23852]: cluster 2024-04-06T03:29:55.556250+0000 osd.4 (osd.4) 207 : cluster 0 3.a scrub starts 2024-04-06T03:29:56.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:56 smithi153 bash[23852]: cluster 2024-04-06T03:29:55.561327+0000 osd.4 (osd.4) 208 : cluster 0 3.a scrub ok 2024-04-06T03:29:56.513 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:56 smithi153 bash[23852]: audit 2024-04-06T03:29:55.858460+0000 mon.a (mon.0) 1162 : audit 1 from='client.? 172.21.15.50:0/4160071767' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1f"}]: dispatch 2024-04-06T03:29:56.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:56.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:56.661 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:56.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:56.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:56.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:56.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:57.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:57.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-04-06T03:29:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:29:57.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:57.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:29:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:29:57.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:57.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: cluster 2024-04-06T03:29:55.303254+0000 osd.7 (osd.7) 207 : cluster 0 6.8 scrub starts 2024-04-06T03:29:57.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: cluster 2024-04-06T03:29:55.308186+0000 osd.7 (osd.7) 208 : cluster 0 6.8 scrub ok 2024-04-06T03:29:57.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: audit 2024-04-06T03:29:56.057503+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.50:0/4160071767' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1f"}]': finished 2024-04-06T03:29:57.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: cluster 2024-04-06T03:29:56.061011+0000 mon.a (mon.0) 1164 : cluster 0 osdmap e187: 8 total, 8 up, 8 in 2024-04-06T03:29:57.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: cluster 2024-04-06T03:29:56.143305+0000 mgr.y (mgr.24368) 333 : cluster 0 pgmap v336: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:57.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: audit 2024-04-06T03:29:56.294997+0000 mon.a (mon.0) 1165 : audit 1 from='client.? 172.21.15.50:0/1064374190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:57.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: cluster 2024-04-06T03:29:56.326869+0000 osd.0 (osd.0) 184 : cluster 0 3.1b scrub starts 2024-04-06T03:29:57.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: cluster 2024-04-06T03:29:56.329255+0000 osd.0 (osd.0) 185 : cluster 0 3.1b scrub ok 2024-04-06T03:29:57.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:57 smithi153 bash[23852]: audit 2024-04-06T03:29:56.350547+0000 mon.a (mon.0) 1166 : audit 1 from='client.? 172.21.15.50:0/937645417' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:29:57.305 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:57.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: cluster 2024-04-06T03:29:55.303254+0000 osd.7 (osd.7) 207 : cluster 0 6.8 scrub starts 2024-04-06T03:29:57.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: cluster 2024-04-06T03:29:55.308186+0000 osd.7 (osd.7) 208 : cluster 0 6.8 scrub ok 2024-04-06T03:29:57.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: audit 2024-04-06T03:29:56.057503+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.50:0/4160071767' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1f"}]': finished 2024-04-06T03:29:57.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: cluster 2024-04-06T03:29:56.061011+0000 mon.a (mon.0) 1164 : cluster 0 osdmap e187: 8 total, 8 up, 8 in 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: cluster 2024-04-06T03:29:56.143305+0000 mgr.y (mgr.24368) 333 : cluster 0 pgmap v336: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: audit 2024-04-06T03:29:56.294997+0000 mon.a (mon.0) 1165 : audit 1 from='client.? 172.21.15.50:0/1064374190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: cluster 2024-04-06T03:29:56.326869+0000 osd.0 (osd.0) 184 : cluster 0 3.1b scrub starts 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: cluster 2024-04-06T03:29:56.329255+0000 osd.0 (osd.0) 185 : cluster 0 3.1b scrub ok 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[29221]: audit 2024-04-06T03:29:56.350547+0000 mon.a (mon.0) 1166 : audit 1 from='client.? 172.21.15.50:0/937645417' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: cluster 2024-04-06T03:29:55.303254+0000 osd.7 (osd.7) 207 : cluster 0 6.8 scrub starts 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: cluster 2024-04-06T03:29:55.308186+0000 osd.7 (osd.7) 208 : cluster 0 6.8 scrub ok 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: audit 2024-04-06T03:29:56.057503+0000 mon.a (mon.0) 1163 : audit 1 from='client.? 172.21.15.50:0/4160071767' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.1f"}]': finished 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: cluster 2024-04-06T03:29:56.061011+0000 mon.a (mon.0) 1164 : cluster 0 osdmap e187: 8 total, 8 up, 8 in 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: cluster 2024-04-06T03:29:56.143305+0000 mgr.y (mgr.24368) 333 : cluster 0 pgmap v336: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: audit 2024-04-06T03:29:56.294997+0000 mon.a (mon.0) 1165 : audit 1 from='client.? 172.21.15.50:0/1064374190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: cluster 2024-04-06T03:29:56.326869+0000 osd.0 (osd.0) 184 : cluster 0 3.1b scrub starts 2024-04-06T03:29:57.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: cluster 2024-04-06T03:29:56.329255+0000 osd.0 (osd.0) 185 : cluster 0 3.1b scrub ok 2024-04-06T03:29:57.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:57 smithi050 bash[21467]: audit 2024-04-06T03:29:56.350547+0000 mon.a (mon.0) 1166 : audit 1 from='client.? 172.21.15.50:0/937645417' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:29:57.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:57.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:57.504 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:29:57.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:57.569 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:57.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:57.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:57.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:57.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:29:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:29:57.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:57.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:57.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:58.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:58.090 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:58.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:58.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: cluster 2024-04-06T03:29:56.266509+0000 osd.7 (osd.7) 209 : cluster 0 3.e scrub starts 2024-04-06T03:29:58.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: cluster 2024-04-06T03:29:56.270723+0000 osd.7 (osd.7) 210 : cluster 0 3.e scrub ok 2024-04-06T03:29:58.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: cluster 2024-04-06T03:29:57.066322+0000 mon.a (mon.0) 1167 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:29:58.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: audit 2024-04-06T03:29:57.076246+0000 mon.a (mon.0) 1168 : audit 1 from='client.? 172.21.15.50:0/1064374190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:29:58.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: audit 2024-04-06T03:29:57.076312+0000 mon.a (mon.0) 1169 : audit 1 from='client.? 172.21.15.50:0/937645417' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:29:58.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: cluster 2024-04-06T03:29:57.081447+0000 mon.a (mon.0) 1170 : cluster 0 osdmap e188: 8 total, 8 up, 8 in 2024-04-06T03:29:58.205 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: cluster 2024-04-06T03:29:57.363448+0000 osd.0 (osd.0) 186 : cluster 0 6.16 scrub starts 2024-04-06T03:29:58.205 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:58 smithi153 bash[23852]: cluster 2024-04-06T03:29:57.382369+0000 osd.0 (osd.0) 187 : cluster 0 6.16 scrub ok 2024-04-06T03:29:58.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:58.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:58.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: cluster 2024-04-06T03:29:56.266509+0000 osd.7 (osd.7) 209 : cluster 0 3.e scrub starts 2024-04-06T03:29:58.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: cluster 2024-04-06T03:29:56.270723+0000 osd.7 (osd.7) 210 : cluster 0 3.e scrub ok 2024-04-06T03:29:58.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: cluster 2024-04-06T03:29:57.066322+0000 mon.a (mon.0) 1167 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:29:58.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: audit 2024-04-06T03:29:57.076246+0000 mon.a (mon.0) 1168 : audit 1 from='client.? 172.21.15.50:0/1064374190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:29:58.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: audit 2024-04-06T03:29:57.076312+0000 mon.a (mon.0) 1169 : audit 1 from='client.? 172.21.15.50:0/937645417' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:29:58.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: cluster 2024-04-06T03:29:57.081447+0000 mon.a (mon.0) 1170 : cluster 0 osdmap e188: 8 total, 8 up, 8 in 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: cluster 2024-04-06T03:29:57.363448+0000 osd.0 (osd.0) 186 : cluster 0 6.16 scrub starts 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[29221]: cluster 2024-04-06T03:29:57.382369+0000 osd.0 (osd.0) 187 : cluster 0 6.16 scrub ok 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: cluster 2024-04-06T03:29:56.266509+0000 osd.7 (osd.7) 209 : cluster 0 3.e scrub starts 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: cluster 2024-04-06T03:29:56.270723+0000 osd.7 (osd.7) 210 : cluster 0 3.e scrub ok 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: cluster 2024-04-06T03:29:57.066322+0000 mon.a (mon.0) 1167 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: audit 2024-04-06T03:29:57.076246+0000 mon.a (mon.0) 1168 : audit 1 from='client.? 172.21.15.50:0/1064374190' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: audit 2024-04-06T03:29:57.076312+0000 mon.a (mon.0) 1169 : audit 1 from='client.? 172.21.15.50:0/937645417' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: cluster 2024-04-06T03:29:57.081447+0000 mon.a (mon.0) 1170 : cluster 0 osdmap e188: 8 total, 8 up, 8 in 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: cluster 2024-04-06T03:29:57.363448+0000 osd.0 (osd.0) 186 : cluster 0 6.16 scrub starts 2024-04-06T03:29:58.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:58 smithi050 bash[21467]: cluster 2024-04-06T03:29:57.382369+0000 osd.0 (osd.0) 187 : cluster 0 6.16 scrub ok 2024-04-06T03:29:58.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:58.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:58.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:58.488 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:58.591 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:58.611 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:29:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:29:58.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:58.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:58.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:58.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:58.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:59.001 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:29:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:29:59.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:59.116 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[29221]: cluster 2024-04-06T03:29:57.300837+0000 osd.7 (osd.7) 211 : cluster 0 2.0 scrub starts 2024-04-06T03:29:59.116 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[29221]: cluster 2024-04-06T03:29:57.303081+0000 osd.7 (osd.7) 212 : cluster 0 2.0 scrub ok 2024-04-06T03:29:59.116 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[29221]: cluster 2024-04-06T03:29:58.143993+0000 mgr.y (mgr.24368) 334 : cluster 0 pgmap v338: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:59.116 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[29221]: cluster 2024-04-06T03:29:58.525440+0000 osd.4 (osd.4) 209 : cluster 0 5.1 scrub starts 2024-04-06T03:29:59.116 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[29221]: cluster 2024-04-06T03:29:58.526804+0000 osd.4 (osd.4) 210 : cluster 0 5.1 scrub ok 2024-04-06T03:29:59.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[21467]: cluster 2024-04-06T03:29:57.300837+0000 osd.7 (osd.7) 211 : cluster 0 2.0 scrub starts 2024-04-06T03:29:59.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[21467]: cluster 2024-04-06T03:29:57.303081+0000 osd.7 (osd.7) 212 : cluster 0 2.0 scrub ok 2024-04-06T03:29:59.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[21467]: cluster 2024-04-06T03:29:58.143993+0000 mgr.y (mgr.24368) 334 : cluster 0 pgmap v338: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:59.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[21467]: cluster 2024-04-06T03:29:58.525440+0000 osd.4 (osd.4) 209 : cluster 0 5.1 scrub starts 2024-04-06T03:29:59.117 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[21467]: cluster 2024-04-06T03:29:58.526804+0000 osd.4 (osd.4) 210 : cluster 0 5.1 scrub ok 2024-04-06T03:29:59.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:59.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:59.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:59 smithi153 bash[23852]: cluster 2024-04-06T03:29:57.300837+0000 osd.7 (osd.7) 211 : cluster 0 2.0 scrub starts 2024-04-06T03:29:59.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:59 smithi153 bash[23852]: cluster 2024-04-06T03:29:57.303081+0000 osd.7 (osd.7) 212 : cluster 0 2.0 scrub ok 2024-04-06T03:29:59.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:59 smithi153 bash[23852]: cluster 2024-04-06T03:29:58.143993+0000 mgr.y (mgr.24368) 334 : cluster 0 pgmap v338: 148 pgs: 148 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:29:59.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:59 smithi153 bash[23852]: cluster 2024-04-06T03:29:58.525440+0000 osd.4 (osd.4) 209 : cluster 0 5.1 scrub starts 2024-04-06T03:29:59.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:29:59 smithi153 bash[23852]: cluster 2024-04-06T03:29:58.526804+0000 osd.4 (osd.4) 210 : cluster 0 5.1 scrub ok 2024-04-06T03:29:59.262 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:59.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:59.377 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:29:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:29:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:29:59.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:59.397 DEBUG:teuthology.orchestra.run.smithi050:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-04-06T03:29:59.439 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-06T03:29:59.439 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-06T03:29:59.486 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:hints = 1 2024-04-06T03:29:59.486 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-06T03:29:59.486 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object prefix: benchmark_data_smithi050_86143 2024-04-06T03:29:59.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:29:59.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 0 0 0 0 0 0 - 0 2024-04-06T03:29:59.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:59.525 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:29:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:29:59.546 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:29:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:29:59.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:59.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:29:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:29:59.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:59.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:29:59.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:59.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:29:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:29:59.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:00.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.052 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:00.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.182 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:00.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:29:58.675977+0000 osd.3 (osd.3) 223 : cluster 0 5.14 scrub starts 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:29:58.678818+0000 osd.3 (osd.3) 224 : cluster 0 5.14 scrub ok 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:29:59.069485+0000 mon.a (mon.0) 1171 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:29:59.069543+0000 mon.a (mon.0) 1172 : cluster 1 Cluster is now healthy 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: audit 2024-04-06T03:29:59.402660+0000 mgr.y (mgr.24368) 335 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:29:59.514589+0000 osd.4 (osd.4) 211 : cluster 0 3.9 scrub starts 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:29:59.531468+0000 osd.4 (osd.4) 212 : cluster 0 3.9 scrub ok 2024-04-06T03:30:00.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:00 smithi153 bash[23852]: cluster 2024-04-06T03:30:00.000072+0000 mon.a (mon.0) 1173 : cluster 1 overall HEALTH_OK 2024-04-06T03:30:00.313 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:00.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:29:58.675977+0000 osd.3 (osd.3) 223 : cluster 0 5.14 scrub starts 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:29:58.678818+0000 osd.3 (osd.3) 224 : cluster 0 5.14 scrub ok 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:29:59.069485+0000 mon.a (mon.0) 1171 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:29:59.069543+0000 mon.a (mon.0) 1172 : cluster 1 Cluster is now healthy 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: audit 2024-04-06T03:29:59.402660+0000 mgr.y (mgr.24368) 335 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:29:59.514589+0000 osd.4 (osd.4) 211 : cluster 0 3.9 scrub starts 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:29:59.531468+0000 osd.4 (osd.4) 212 : cluster 0 3.9 scrub ok 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[21467]: cluster 2024-04-06T03:30:00.000072+0000 mon.a (mon.0) 1173 : cluster 1 overall HEALTH_OK 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:29:58.675977+0000 osd.3 (osd.3) 223 : cluster 0 5.14 scrub starts 2024-04-06T03:30:00.395 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:29:58.678818+0000 osd.3 (osd.3) 224 : cluster 0 5.14 scrub ok 2024-04-06T03:30:00.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:29:59.069485+0000 mon.a (mon.0) 1171 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:30:00.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:29:59.069543+0000 mon.a (mon.0) 1172 : cluster 1 Cluster is now healthy 2024-04-06T03:30:00.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: audit 2024-04-06T03:29:59.402660+0000 mgr.y (mgr.24368) 335 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:00.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:29:59.514589+0000 osd.4 (osd.4) 211 : cluster 0 3.9 scrub starts 2024-04-06T03:30:00.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:29:59.531468+0000 osd.4 (osd.4) 212 : cluster 0 3.9 scrub ok 2024-04-06T03:30:00.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:00 smithi050 bash[29221]: cluster 2024-04-06T03:30:00.000072+0000 mon.a (mon.0) 1173 : cluster 1 overall HEALTH_OK 2024-04-06T03:30:00.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:00.441 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:00.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 1 16 2731 2715 169.679 169.688 0.00596341 0.00586403 2024-04-06T03:30:00.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:00.580 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:30:00.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:30:00.580 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:30:00.580 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:30:00.581 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:30:00.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:00.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:00.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:00.925 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:00.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:00.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:01.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:01.093 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:01.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:01.219 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:01.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:01 smithi050 bash[21467]: cluster 2024-04-06T03:29:59.717063+0000 osd.3 (osd.3) 225 : cluster 0 2.14 scrub starts 2024-04-06T03:30:01.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:01 smithi050 bash[21467]: cluster 2024-04-06T03:29:59.720676+0000 osd.3 (osd.3) 226 : cluster 0 2.14 scrub ok 2024-04-06T03:30:01.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:01 smithi050 bash[21467]: cluster 2024-04-06T03:30:00.144813+0000 mgr.y (mgr.24368) 336 : cluster 0 pgmap v339: 148 pgs: 148 active+clean; 43 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.1 MiB/s wr, 114 op/s 2024-04-06T03:30:01.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:01 smithi050 bash[29221]: cluster 2024-04-06T03:29:59.717063+0000 osd.3 (osd.3) 225 : cluster 0 2.14 scrub starts 2024-04-06T03:30:01.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:01 smithi050 bash[29221]: cluster 2024-04-06T03:29:59.720676+0000 osd.3 (osd.3) 226 : cluster 0 2.14 scrub ok 2024-04-06T03:30:01.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:01 smithi050 bash[29221]: cluster 2024-04-06T03:30:00.144813+0000 mgr.y (mgr.24368) 336 : cluster 0 pgmap v339: 148 pgs: 148 active+clean; 43 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.1 MiB/s wr, 114 op/s 2024-04-06T03:30:01.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:01.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:01.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:01.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:01 smithi153 bash[23852]: cluster 2024-04-06T03:29:59.717063+0000 osd.3 (osd.3) 225 : cluster 0 2.14 scrub starts 2024-04-06T03:30:01.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:01 smithi153 bash[23852]: cluster 2024-04-06T03:29:59.720676+0000 osd.3 (osd.3) 226 : cluster 0 2.14 scrub ok 2024-04-06T03:30:01.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:01 smithi153 bash[23852]: cluster 2024-04-06T03:30:00.144813+0000 mgr.y (mgr.24368) 336 : cluster 0 pgmap v339: 148 pgs: 148 active+clean; 43 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.1 MiB/s wr, 114 op/s 2024-04-06T03:30:01.477 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:01.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 2 16 5221 5205 162.646 155.625 0.00348772 0.00613241 2024-04-06T03:30:01.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:01.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:01.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:01.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:01.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-06T03:30:01.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:01.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:01.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:02.044 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:30:02.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 0 1 2024-04-06T03:30:02.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:02.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:02.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:02.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:02 smithi153 bash[23852]: cluster 2024-04-06T03:30:00.646387+0000 osd.1 (osd.1) 179 : cluster 0 2.15 scrub starts 2024-04-06T03:30:02.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:02 smithi153 bash[23852]: cluster 2024-04-06T03:30:00.651674+0000 osd.1 (osd.1) 180 : cluster 0 2.15 scrub ok 2024-04-06T03:30:02.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:02 smithi153 bash[23852]: cluster 2024-04-06T03:30:00.871302+0000 osd.2 (osd.2) 179 : cluster 0 6.f scrub starts 2024-04-06T03:30:02.232 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:02 smithi153 bash[23852]: cluster 2024-04-06T03:30:00.933705+0000 osd.2 (osd.2) 180 : cluster 0 6.f scrub ok 2024-04-06T03:30:02.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:02 smithi153 bash[23852]: cluster 2024-04-06T03:30:01.525688+0000 osd.4 (osd.4) 213 : cluster 0 2.f deep-scrub starts 2024-04-06T03:30:02.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:02 smithi153 bash[23852]: cluster 2024-04-06T03:30:01.533596+0000 osd.4 (osd.4) 214 : cluster 0 2.f deep-scrub ok 2024-04-06T03:30:02.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:02.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[21467]: cluster 2024-04-06T03:30:00.646387+0000 osd.1 (osd.1) 179 : cluster 0 2.15 scrub starts 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[21467]: cluster 2024-04-06T03:30:00.651674+0000 osd.1 (osd.1) 180 : cluster 0 2.15 scrub ok 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[21467]: cluster 2024-04-06T03:30:00.871302+0000 osd.2 (osd.2) 179 : cluster 0 6.f scrub starts 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[21467]: cluster 2024-04-06T03:30:00.933705+0000 osd.2 (osd.2) 180 : cluster 0 6.f scrub ok 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[21467]: cluster 2024-04-06T03:30:01.525688+0000 osd.4 (osd.4) 213 : cluster 0 2.f deep-scrub starts 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[21467]: cluster 2024-04-06T03:30:01.533596+0000 osd.4 (osd.4) 214 : cluster 0 2.f deep-scrub ok 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[29221]: cluster 2024-04-06T03:30:00.646387+0000 osd.1 (osd.1) 179 : cluster 0 2.15 scrub starts 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[29221]: cluster 2024-04-06T03:30:00.651674+0000 osd.1 (osd.1) 180 : cluster 0 2.15 scrub ok 2024-04-06T03:30:02.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[29221]: cluster 2024-04-06T03:30:00.871302+0000 osd.2 (osd.2) 179 : cluster 0 6.f scrub starts 2024-04-06T03:30:02.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[29221]: cluster 2024-04-06T03:30:00.933705+0000 osd.2 (osd.2) 180 : cluster 0 6.f scrub ok 2024-04-06T03:30:02.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[29221]: cluster 2024-04-06T03:30:01.525688+0000 osd.4 (osd.4) 213 : cluster 0 2.f deep-scrub starts 2024-04-06T03:30:02.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:02 smithi050 bash[29221]: cluster 2024-04-06T03:30:01.533596+0000 osd.4 (osd.4) 214 : cluster 0 2.f deep-scrub ok 2024-04-06T03:30:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:02.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 3 16 6359 6343 132.136 71.125 0.00423757 0.0075493 2024-04-06T03:30:02.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:02.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:02.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:02.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:02.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:02.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:02.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:02.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:02.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:02.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:03.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:03.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:03.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:03.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:03.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:03.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:03.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 4 16 7900 7884 123.178 96.3125 0.0100078 0.00810062 2024-04-06T03:30:03.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:03.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:01.253013+0000 osd.7 (osd.7) 213 : cluster 0 2.8 scrub starts 2024-04-06T03:30:03.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:01.254826+0000 osd.7 (osd.7) 214 : cluster 0 2.8 scrub ok 2024-04-06T03:30:03.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:01.435762+0000 osd.0 (osd.0) 188 : cluster 0 3.17 scrub starts 2024-04-06T03:30:03.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:01.439405+0000 osd.0 (osd.0) 189 : cluster 0 3.17 scrub ok 2024-04-06T03:30:03.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:02.145174+0000 mgr.y (mgr.24368) 337 : cluster 0 pgmap v340: 148 pgs: 148 active+clean; 43 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.6 MiB/s wr, 90 op/s 2024-04-06T03:30:03.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:02.490943+0000 osd.4 (osd.4) 215 : cluster 0 6.1e scrub starts 2024-04-06T03:30:03.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:03 smithi153 bash[23852]: cluster 2024-04-06T03:30:02.501352+0000 osd.4 (osd.4) 216 : cluster 0 6.1e scrub ok 2024-04-06T03:30:03.541 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:03.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:03.671 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:03.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:01.253013+0000 osd.7 (osd.7) 213 : cluster 0 2.8 scrub starts 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:01.254826+0000 osd.7 (osd.7) 214 : cluster 0 2.8 scrub ok 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:01.435762+0000 osd.0 (osd.0) 188 : cluster 0 3.17 scrub starts 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:01.439405+0000 osd.0 (osd.0) 189 : cluster 0 3.17 scrub ok 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:02.145174+0000 mgr.y (mgr.24368) 337 : cluster 0 pgmap v340: 148 pgs: 148 active+clean; 43 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.6 MiB/s wr, 90 op/s 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:02.490943+0000 osd.4 (osd.4) 215 : cluster 0 6.1e scrub starts 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[21467]: cluster 2024-04-06T03:30:02.501352+0000 osd.4 (osd.4) 216 : cluster 0 6.1e scrub ok 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:01.253013+0000 osd.7 (osd.7) 213 : cluster 0 2.8 scrub starts 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:01.254826+0000 osd.7 (osd.7) 214 : cluster 0 2.8 scrub ok 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:01.435762+0000 osd.0 (osd.0) 188 : cluster 0 3.17 scrub starts 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:01.439405+0000 osd.0 (osd.0) 189 : cluster 0 3.17 scrub ok 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:02.145174+0000 mgr.y (mgr.24368) 337 : cluster 0 pgmap v340: 148 pgs: 148 active+clean; 43 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.6 MiB/s wr, 90 op/s 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:02.490943+0000 osd.4 (osd.4) 215 : cluster 0 6.1e scrub starts 2024-04-06T03:30:03.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:03 smithi050 bash[29221]: cluster 2024-04-06T03:30:02.501352+0000 osd.4 (osd.4) 216 : cluster 0 6.1e scrub ok 2024-04-06T03:30:03.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:03.801 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:03.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:03.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:03.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:03.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:04.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:04.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[21467]: cluster 2024-04-06T03:30:02.596670+0000 osd.1 (osd.1) 181 : cluster 0 2.d scrub starts 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[21467]: cluster 2024-04-06T03:30:02.600806+0000 osd.1 (osd.1) 182 : cluster 0 2.d scrub ok 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[21467]: audit 2024-04-06T03:30:03.913206+0000 mon.a (mon.0) 1174 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[21467]: audit 2024-04-06T03:30:03.914802+0000 mon.a (mon.0) 1175 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[29221]: cluster 2024-04-06T03:30:02.596670+0000 osd.1 (osd.1) 181 : cluster 0 2.d scrub starts 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[29221]: cluster 2024-04-06T03:30:02.600806+0000 osd.1 (osd.1) 182 : cluster 0 2.d scrub ok 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[29221]: audit 2024-04-06T03:30:03.913206+0000 mon.a (mon.0) 1174 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:04.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:04 smithi050 bash[29221]: audit 2024-04-06T03:30:03.914802+0000 mon.a (mon.0) 1175 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:04.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:04.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:04.345 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:04.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.471 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:04.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 5 15 10300 10285 128.552 150.062 0.00244333 0.00776931 2024-04-06T03:30:04.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:04 smithi153 bash[23852]: cluster 2024-04-06T03:30:02.596670+0000 osd.1 (osd.1) 181 : cluster 0 2.d scrub starts 2024-04-06T03:30:04.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:04 smithi153 bash[23852]: cluster 2024-04-06T03:30:02.600806+0000 osd.1 (osd.1) 182 : cluster 0 2.d scrub ok 2024-04-06T03:30:04.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:04 smithi153 bash[23852]: audit 2024-04-06T03:30:03.913206+0000 mon.a (mon.0) 1174 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:04.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:04 smithi153 bash[23852]: audit 2024-04-06T03:30:03.914802+0000 mon.a (mon.0) 1175 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:04.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:04.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:04.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:04.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:04.862 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:04.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:04.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:04.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:05.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:05.120 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:05.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:03.432769+0000 osd.0 (osd.0) 190 : cluster 0 5.1e scrub starts 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:03.437491+0000 osd.0 (osd.0) 191 : cluster 0 5.1e scrub ok 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:03.577815+0000 osd.1 (osd.1) 183 : cluster 0 6.4 scrub starts 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:03.580379+0000 osd.1 (osd.1) 184 : cluster 0 6.4 scrub ok 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:04.145723+0000 mgr.y (mgr.24368) 338 : cluster 0 pgmap v341: 148 pgs: 148 active+clean; 195 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 633 B/s rd, 24 MiB/s wr, 385 op/s 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:04.185670+0000 osd.2 (osd.2) 181 : cluster 0 5.6 scrub starts 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[21467]: cluster 2024-04-06T03:30:04.186788+0000 osd.2 (osd.2) 182 : cluster 0 5.6 scrub ok 2024-04-06T03:30:05.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:03.432769+0000 osd.0 (osd.0) 190 : cluster 0 5.1e scrub starts 2024-04-06T03:30:05.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:03.437491+0000 osd.0 (osd.0) 191 : cluster 0 5.1e scrub ok 2024-04-06T03:30:05.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:03.577815+0000 osd.1 (osd.1) 183 : cluster 0 6.4 scrub starts 2024-04-06T03:30:05.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:03.580379+0000 osd.1 (osd.1) 184 : cluster 0 6.4 scrub ok 2024-04-06T03:30:05.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:04.145723+0000 mgr.y (mgr.24368) 338 : cluster 0 pgmap v341: 148 pgs: 148 active+clean; 195 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 633 B/s rd, 24 MiB/s wr, 385 op/s 2024-04-06T03:30:05.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:04.185670+0000 osd.2 (osd.2) 181 : cluster 0 5.6 scrub starts 2024-04-06T03:30:05.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:05 smithi050 bash[29221]: cluster 2024-04-06T03:30:04.186788+0000 osd.2 (osd.2) 182 : cluster 0 5.6 scrub ok 2024-04-06T03:30:05.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:05.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:05.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:05.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 6 15 12619 12604 131.281 144.938 0.00290848 0.00760515 2024-04-06T03:30:05.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:05.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:03.432769+0000 osd.0 (osd.0) 190 : cluster 0 5.1e scrub starts 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:03.437491+0000 osd.0 (osd.0) 191 : cluster 0 5.1e scrub ok 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:03.577815+0000 osd.1 (osd.1) 183 : cluster 0 6.4 scrub starts 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:03.580379+0000 osd.1 (osd.1) 184 : cluster 0 6.4 scrub ok 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:04.145723+0000 mgr.y (mgr.24368) 338 : cluster 0 pgmap v341: 148 pgs: 148 active+clean; 195 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 633 B/s rd, 24 MiB/s wr, 385 op/s 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:04.185670+0000 osd.2 (osd.2) 181 : cluster 0 5.6 scrub starts 2024-04-06T03:30:05.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:05 smithi153 bash[23852]: cluster 2024-04-06T03:30:04.186788+0000 osd.2 (osd.2) 182 : cluster 0 5.6 scrub ok 2024-04-06T03:30:05.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:05.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:05.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:05.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:05.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:05.899 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:05.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:06.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:06.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:06.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:06.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:06.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:06.266 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:06 smithi153 bash[23852]: cluster 2024-04-06T03:30:04.596101+0000 osd.1 (osd.1) 185 : cluster 0 6.19 scrub starts 2024-04-06T03:30:06.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:06 smithi153 bash[23852]: cluster 2024-04-06T03:30:04.645038+0000 osd.1 (osd.1) 186 : cluster 0 6.19 scrub ok 2024-04-06T03:30:06.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:06 smithi153 bash[23852]: cluster 2024-04-06T03:30:05.466516+0000 osd.4 (osd.4) 217 : cluster 0 5.e deep-scrub starts 2024-04-06T03:30:06.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:06 smithi153 bash[23852]: cluster 2024-04-06T03:30:05.475367+0000 osd.4 (osd.4) 218 : cluster 0 5.e deep-scrub ok 2024-04-06T03:30:06.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:06.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[21467]: cluster 2024-04-06T03:30:04.596101+0000 osd.1 (osd.1) 185 : cluster 0 6.19 scrub starts 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[21467]: cluster 2024-04-06T03:30:04.645038+0000 osd.1 (osd.1) 186 : cluster 0 6.19 scrub ok 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[21467]: cluster 2024-04-06T03:30:05.466516+0000 osd.4 (osd.4) 217 : cluster 0 5.e deep-scrub starts 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[21467]: cluster 2024-04-06T03:30:05.475367+0000 osd.4 (osd.4) 218 : cluster 0 5.e deep-scrub ok 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[29221]: cluster 2024-04-06T03:30:04.596101+0000 osd.1 (osd.1) 185 : cluster 0 6.19 scrub starts 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[29221]: cluster 2024-04-06T03:30:04.645038+0000 osd.1 (osd.1) 186 : cluster 0 6.19 scrub ok 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[29221]: cluster 2024-04-06T03:30:05.466516+0000 osd.4 (osd.4) 217 : cluster 0 5.e deep-scrub starts 2024-04-06T03:30:06.394 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:06 smithi050 bash[29221]: cluster 2024-04-06T03:30:05.475367+0000 osd.4 (osd.4) 218 : cluster 0 5.e deep-scrub ok 2024-04-06T03:30:06.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:06.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:06.487 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 7 16 14923 14907 133.088 143.938 0.00971071 0.00750578 2024-04-06T03:30:06.520 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:06.544 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:06.605 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:30:06.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:06.648 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:06.649 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:06.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:06.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:06.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:06.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:06.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:06.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:07.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:07.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:07.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:07.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:07.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:07.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:07.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:07.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:07 smithi050 bash[21467]: cluster 2024-04-06T03:30:05.204011+0000 osd.7 (osd.7) 215 : cluster 0 3.16 scrub starts 2024-04-06T03:30:07.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:07 smithi050 bash[21467]: cluster 2024-04-06T03:30:05.214624+0000 osd.7 (osd.7) 216 : cluster 0 3.16 scrub ok 2024-04-06T03:30:07.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:07 smithi050 bash[21467]: cluster 2024-04-06T03:30:06.146573+0000 mgr.y (mgr.24368) 339 : cluster 0 pgmap v342: 148 pgs: 148 active+clean; 654 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-06T03:30:07.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:07 smithi050 bash[29221]: cluster 2024-04-06T03:30:05.204011+0000 osd.7 (osd.7) 215 : cluster 0 3.16 scrub starts 2024-04-06T03:30:07.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:07 smithi050 bash[29221]: cluster 2024-04-06T03:30:05.214624+0000 osd.7 (osd.7) 216 : cluster 0 3.16 scrub ok 2024-04-06T03:30:07.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:07 smithi050 bash[29221]: cluster 2024-04-06T03:30:06.146573+0000 mgr.y (mgr.24368) 339 : cluster 0 pgmap v342: 148 pgs: 148 active+clean; 654 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-06T03:30:07.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:07.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 8 15 16008 15993 124.935 67.875 0.00855487 0.00794444 2024-04-06T03:30:07.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:07.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:07 smithi153 bash[23852]: cluster 2024-04-06T03:30:05.204011+0000 osd.7 (osd.7) 215 : cluster 0 3.16 scrub starts 2024-04-06T03:30:07.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:07 smithi153 bash[23852]: cluster 2024-04-06T03:30:05.214624+0000 osd.7 (osd.7) 216 : cluster 0 3.16 scrub ok 2024-04-06T03:30:07.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:07 smithi153 bash[23852]: cluster 2024-04-06T03:30:06.146573+0000 mgr.y (mgr.24368) 339 : cluster 0 pgmap v342: 148 pgs: 148 active+clean; 654 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-06T03:30:07.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:07.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:07.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:07.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:07.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:07.951 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:07.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:08.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:08.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:08.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:08.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:08.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:08.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:08.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:08 smithi050 bash[21467]: cluster 2024-04-06T03:30:06.428719+0000 osd.0 (osd.0) 192 : cluster 0 3.1 scrub starts 2024-04-06T03:30:08.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:08 smithi050 bash[21467]: cluster 2024-04-06T03:30:06.434435+0000 osd.0 (osd.0) 193 : cluster 0 3.1 scrub ok 2024-04-06T03:30:08.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:08 smithi050 bash[29221]: cluster 2024-04-06T03:30:06.428719+0000 osd.0 (osd.0) 192 : cluster 0 3.1 scrub starts 2024-04-06T03:30:08.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:08 smithi050 bash[29221]: cluster 2024-04-06T03:30:06.434435+0000 osd.0 (osd.0) 193 : cluster 0 3.1 scrub ok 2024-04-06T03:30:08.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:08.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:08.454 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:08 smithi153 bash[23852]: cluster 2024-04-06T03:30:06.428719+0000 osd.0 (osd.0) 192 : cluster 0 3.1 scrub starts 2024-04-06T03:30:08.454 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:08 smithi153 bash[23852]: cluster 2024-04-06T03:30:06.434435+0000 osd.0 (osd.0) 193 : cluster 0 3.1 scrub ok 2024-04-06T03:30:08.472 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:08.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 9 16 17613 17597 122.191 100.25 0.00149661 0.00815817 2024-04-06T03:30:08.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:08.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:08.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:08.730 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:08.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:08.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:08.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:08.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:09.002 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:09.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:09.127 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:09.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:09.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[21467]: cluster 2024-04-06T03:30:07.199255+0000 osd.7 (osd.7) 217 : cluster 0 3.d deep-scrub starts 2024-04-06T03:30:09.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[21467]: cluster 2024-04-06T03:30:07.204760+0000 osd.7 (osd.7) 218 : cluster 0 3.d deep-scrub ok 2024-04-06T03:30:09.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[21467]: cluster 2024-04-06T03:30:07.483748+0000 osd.2 (osd.2) 183 : cluster 0 5.b scrub starts 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[21467]: cluster 2024-04-06T03:30:07.486488+0000 osd.2 (osd.2) 184 : cluster 0 5.b scrub ok 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[21467]: cluster 2024-04-06T03:30:08.147022+0000 mgr.y (mgr.24368) 340 : cluster 0 pgmap v343: 148 pgs: 148 active+clean; 654 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 925 B/s rd, 59 MiB/s wr, 945 op/s 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[29221]: cluster 2024-04-06T03:30:07.199255+0000 osd.7 (osd.7) 217 : cluster 0 3.d deep-scrub starts 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[29221]: cluster 2024-04-06T03:30:07.204760+0000 osd.7 (osd.7) 218 : cluster 0 3.d deep-scrub ok 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[29221]: cluster 2024-04-06T03:30:07.483748+0000 osd.2 (osd.2) 183 : cluster 0 5.b scrub starts 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[29221]: cluster 2024-04-06T03:30:07.486488+0000 osd.2 (osd.2) 184 : cluster 0 5.b scrub ok 2024-04-06T03:30:09.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[29221]: cluster 2024-04-06T03:30:08.147022+0000 mgr.y (mgr.24368) 340 : cluster 0 pgmap v343: 148 pgs: 148 active+clean; 654 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 925 B/s rd, 59 MiB/s wr, 945 op/s 2024-04-06T03:30:09.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:09.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:09.377 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:09 smithi153 bash[23852]: cluster 2024-04-06T03:30:07.199255+0000 osd.7 (osd.7) 217 : cluster 0 3.d deep-scrub starts 2024-04-06T03:30:09.377 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:09 smithi153 bash[23852]: cluster 2024-04-06T03:30:07.204760+0000 osd.7 (osd.7) 218 : cluster 0 3.d deep-scrub ok 2024-04-06T03:30:09.377 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:09 smithi153 bash[23852]: cluster 2024-04-06T03:30:07.483748+0000 osd.2 (osd.2) 183 : cluster 0 5.b scrub starts 2024-04-06T03:30:09.377 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:09 smithi153 bash[23852]: cluster 2024-04-06T03:30:07.486488+0000 osd.2 (osd.2) 184 : cluster 0 5.b scrub ok 2024-04-06T03:30:09.377 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:09 smithi153 bash[23852]: cluster 2024-04-06T03:30:08.147022+0000 mgr.y (mgr.24368) 340 : cluster 0 pgmap v343: 148 pgs: 148 active+clean; 654 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 925 B/s rd, 59 MiB/s wr, 945 op/s 2024-04-06T03:30:09.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:09.484 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:30:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:30:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:30:09.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 10 16 18602 18586 116.153 61.8125 0.00479458 0.00860124 2024-04-06T03:30:09.498 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:09.524 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:09.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:09.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:09.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:09.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:30:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:30:09.796 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:09.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:09.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:09.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:10.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:10.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:10.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:10.197 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:10.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:10 smithi153 bash[23852]: cluster 2024-04-06T03:30:08.251490+0000 osd.2 (osd.2) 185 : cluster 0 2.1 deep-scrub starts 2024-04-06T03:30:10.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:10 smithi153 bash[23852]: cluster 2024-04-06T03:30:08.253643+0000 osd.2 (osd.2) 186 : cluster 0 2.1 deep-scrub ok 2024-04-06T03:30:10.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:10 smithi153 bash[23852]: cluster 2024-04-06T03:30:08.812998+0000 osd.3 (osd.3) 227 : cluster 0 4.0 scrub starts 2024-04-06T03:30:10.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:10 smithi153 bash[23852]: cluster 2024-04-06T03:30:08.822348+0000 osd.3 (osd.3) 228 : cluster 0 4.0 scrub ok 2024-04-06T03:30:10.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:10 smithi153 bash[23852]: audit 2024-04-06T03:30:09.411742+0000 mgr.y (mgr.24368) 341 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:10.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:10.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:10.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:10.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:10.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:10.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 11 16 20533 20517 116.563 120.688 0.00189737 0.00851808 2024-04-06T03:30:10.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:10.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[21467]: cluster 2024-04-06T03:30:08.251490+0000 osd.2 (osd.2) 185 : cluster 0 2.1 deep-scrub starts 2024-04-06T03:30:10.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[21467]: cluster 2024-04-06T03:30:08.253643+0000 osd.2 (osd.2) 186 : cluster 0 2.1 deep-scrub ok 2024-04-06T03:30:10.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[21467]: cluster 2024-04-06T03:30:08.812998+0000 osd.3 (osd.3) 227 : cluster 0 4.0 scrub starts 2024-04-06T03:30:10.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[21467]: cluster 2024-04-06T03:30:08.822348+0000 osd.3 (osd.3) 228 : cluster 0 4.0 scrub ok 2024-04-06T03:30:10.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[21467]: audit 2024-04-06T03:30:09.411742+0000 mgr.y (mgr.24368) 341 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:10.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[29221]: cluster 2024-04-06T03:30:08.251490+0000 osd.2 (osd.2) 185 : cluster 0 2.1 deep-scrub starts 2024-04-06T03:30:10.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[29221]: cluster 2024-04-06T03:30:08.253643+0000 osd.2 (osd.2) 186 : cluster 0 2.1 deep-scrub ok 2024-04-06T03:30:10.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[29221]: cluster 2024-04-06T03:30:08.812998+0000 osd.3 (osd.3) 227 : cluster 0 4.0 scrub starts 2024-04-06T03:30:10.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[29221]: cluster 2024-04-06T03:30:08.822348+0000 osd.3 (osd.3) 228 : cluster 0 4.0 scrub ok 2024-04-06T03:30:10.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:10 smithi050 bash[29221]: audit 2024-04-06T03:30:09.411742+0000 mgr.y (mgr.24368) 341 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:10.575 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:10.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:10.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:10.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:10.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:10.935 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:10.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:10.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:11.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:11.091 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:11.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:11.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:11.340 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:09.274229+0000 osd.7 (osd.7) 219 : cluster 0 5.9 scrub starts 2024-04-06T03:30:11.340 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:09.277329+0000 osd.7 (osd.7) 220 : cluster 0 5.9 scrub ok 2024-04-06T03:30:11.340 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:09.507312+0000 osd.0 (osd.0) 194 : cluster 0 3.6 scrub starts 2024-04-06T03:30:11.340 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:09.516295+0000 osd.0 (osd.0) 195 : cluster 0 3.6 scrub ok 2024-04-06T03:30:11.341 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:09.551764+0000 osd.1 (osd.1) 187 : cluster 0 3.19 scrub starts 2024-04-06T03:30:11.341 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:09.558410+0000 osd.1 (osd.1) 188 : cluster 0 3.19 scrub ok 2024-04-06T03:30:11.341 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.147929+0000 mgr.y (mgr.24368) 342 : cluster 0 pgmap v344: 148 pgs: 148 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-04-06T03:30:11.341 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.452565+0000 osd.4 (osd.4) 219 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:30:11.341 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:11 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.453417+0000 osd.4 (osd.4) 220 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:30:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:11.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:11.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:11.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:11.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:09.274229+0000 osd.7 (osd.7) 219 : cluster 0 5.9 scrub starts 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:09.277329+0000 osd.7 (osd.7) 220 : cluster 0 5.9 scrub ok 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:09.507312+0000 osd.0 (osd.0) 194 : cluster 0 3.6 scrub starts 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:09.516295+0000 osd.0 (osd.0) 195 : cluster 0 3.6 scrub ok 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:09.551764+0000 osd.1 (osd.1) 187 : cluster 0 3.19 scrub starts 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:09.558410+0000 osd.1 (osd.1) 188 : cluster 0 3.19 scrub ok 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.147929+0000 mgr.y (mgr.24368) 342 : cluster 0 pgmap v344: 148 pgs: 148 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.452565+0000 osd.4 (osd.4) 219 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.453417+0000 osd.4 (osd.4) 220 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:09.274229+0000 osd.7 (osd.7) 219 : cluster 0 5.9 scrub starts 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:09.277329+0000 osd.7 (osd.7) 220 : cluster 0 5.9 scrub ok 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:09.507312+0000 osd.0 (osd.0) 194 : cluster 0 3.6 scrub starts 2024-04-06T03:30:11.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:09.516295+0000 osd.0 (osd.0) 195 : cluster 0 3.6 scrub ok 2024-04-06T03:30:11.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:09.551764+0000 osd.1 (osd.1) 187 : cluster 0 3.19 scrub starts 2024-04-06T03:30:11.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:09.558410+0000 osd.1 (osd.1) 188 : cluster 0 3.19 scrub ok 2024-04-06T03:30:11.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.147929+0000 mgr.y (mgr.24368) 342 : cluster 0 pgmap v344: 148 pgs: 148 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-04-06T03:30:11.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.452565+0000 osd.4 (osd.4) 219 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:30:11.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:11 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.453417+0000 osd.4 (osd.4) 220 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:30:11.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 12 16 22444 22428 116.802 119.438 0.00641465 0.00855392 2024-04-06T03:30:11.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:11.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:11.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:11.741 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:11.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:11.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:11.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:11.916 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:30:11.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:11.987 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:30:12.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:12.039 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:30:12.087 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:30:12.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:12.129 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:30:12.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:12.188 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:30:12.229 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:30:12.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.523622+0000 osd.0 (osd.0) 196 : cluster 0 6.d scrub starts 2024-04-06T03:30:12.234 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.579848+0000 osd.1 (osd.1) 189 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.595150+0000 osd.1 (osd.1) 190 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: cluster 2024-04-06T03:30:10.609765+0000 osd.0 (osd.0) 197 : cluster 0 6.d scrub ok 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: cluster 2024-04-06T03:30:11.430785+0000 osd.4 (osd.4) 221 : cluster 0 6.1d scrub starts 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: cluster 2024-04-06T03:30:11.459394+0000 osd.4 (osd.4) 222 : cluster 0 6.1d scrub ok 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[21467]: audit 2024-04-06T03:30:11.876862+0000 mon.c (mon.2) 58 : audit 0 from='client.? 172.21.15.50:0/599910498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.523622+0000 osd.0 (osd.0) 196 : cluster 0 6.d scrub starts 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.579848+0000 osd.1 (osd.1) 189 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.595150+0000 osd.1 (osd.1) 190 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: cluster 2024-04-06T03:30:10.609765+0000 osd.0 (osd.0) 197 : cluster 0 6.d scrub ok 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: cluster 2024-04-06T03:30:11.430785+0000 osd.4 (osd.4) 221 : cluster 0 6.1d scrub starts 2024-04-06T03:30:12.235 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: cluster 2024-04-06T03:30:11.459394+0000 osd.4 (osd.4) 222 : cluster 0 6.1d scrub ok 2024-04-06T03:30:12.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:12 smithi050 bash[29221]: audit 2024-04-06T03:30:11.876862+0000 mon.c (mon.2) 58 : audit 0 from='client.? 172.21.15.50:0/599910498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:30:12.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:12.268 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:12.274 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:30:12.277 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:30:12.278 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:30:12.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:12.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.523622+0000 osd.0 (osd.0) 196 : cluster 0 6.d scrub starts 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.579848+0000 osd.1 (osd.1) 189 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.595150+0000 osd.1 (osd.1) 190 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: cluster 2024-04-06T03:30:10.609765+0000 osd.0 (osd.0) 197 : cluster 0 6.d scrub ok 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: cluster 2024-04-06T03:30:11.430785+0000 osd.4 (osd.4) 221 : cluster 0 6.1d scrub starts 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: cluster 2024-04-06T03:30:11.459394+0000 osd.4 (osd.4) 222 : cluster 0 6.1d scrub ok 2024-04-06T03:30:12.401 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:12 smithi153 bash[23852]: audit 2024-04-06T03:30:11.876862+0000 mon.c (mon.2) 58 : audit 0 from='client.? 172.21.15.50:0/599910498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:30:12.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 13 16 24191 24175 116.215 109.188 0.00246028 0.00859821 2024-04-06T03:30:12.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:12.529 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:12.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:12.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:12.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:12.800 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:30:12.800 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:30:12.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:12.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:12.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:13.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:13.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:13.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:11.511047+0000 osd.0 (osd.0) 198 : cluster 0 5.2 scrub starts 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:11.512920+0000 osd.0 (osd.0) 199 : cluster 0 5.2 scrub ok 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:11.563939+0000 osd.1 (osd.1) 191 : cluster 0 5.1d scrub starts 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:11.571415+0000 osd.1 (osd.1) 192 : cluster 0 5.1d scrub ok 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:12.148398+0000 mgr.y (mgr.24368) 343 : cluster 0 pgmap v345: 148 pgs: 148 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:12.434135+0000 osd.4 (osd.4) 223 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[21467]: cluster 2024-04-06T03:30:12.438402+0000 osd.4 (osd.4) 224 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:11.511047+0000 osd.0 (osd.0) 198 : cluster 0 5.2 scrub starts 2024-04-06T03:30:13.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:11.512920+0000 osd.0 (osd.0) 199 : cluster 0 5.2 scrub ok 2024-04-06T03:30:13.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:11.563939+0000 osd.1 (osd.1) 191 : cluster 0 5.1d scrub starts 2024-04-06T03:30:13.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:11.571415+0000 osd.1 (osd.1) 192 : cluster 0 5.1d scrub ok 2024-04-06T03:30:13.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:12.148398+0000 mgr.y (mgr.24368) 343 : cluster 0 pgmap v345: 148 pgs: 148 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-06T03:30:13.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:12.434135+0000 osd.4 (osd.4) 223 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:30:13.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:13 smithi050 bash[29221]: cluster 2024-04-06T03:30:12.438402+0000 osd.4 (osd.4) 224 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:30:13.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:13.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:13.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:11.511047+0000 osd.0 (osd.0) 198 : cluster 0 5.2 scrub starts 2024-04-06T03:30:13.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:11.512920+0000 osd.0 (osd.0) 199 : cluster 0 5.2 scrub ok 2024-04-06T03:30:13.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:11.563939+0000 osd.1 (osd.1) 191 : cluster 0 5.1d scrub starts 2024-04-06T03:30:13.293 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:11.571415+0000 osd.1 (osd.1) 192 : cluster 0 5.1d scrub ok 2024-04-06T03:30:13.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:12.148398+0000 mgr.y (mgr.24368) 343 : cluster 0 pgmap v345: 148 pgs: 148 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-04-06T03:30:13.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:12.434135+0000 osd.4 (osd.4) 223 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:30:13.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:13 smithi153 bash[23852]: cluster 2024-04-06T03:30:12.438402+0000 osd.4 (osd.4) 224 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:30:13.307 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:13.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:13.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:13.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 14 15 26116 26101 116.512 120.375 0.0150885 0.00857403 2024-04-06T03:30:13.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:13.563 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:13.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:13.691 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:13.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:13.819 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:13.921 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:13.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:13.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:13.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-06T03:30:14.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:14.103 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:14.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:14.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:14.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:14.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:14.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:14.488 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 15 16 28016 28000 116.655 118.688 0.0120085 0.00856453 2024-04-06T03:30:14.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:14.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:14.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:14.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:14 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.110238+0000 osd.2 (osd.2) 187 : cluster 0 5.8 scrub starts 2024-04-06T03:30:14.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:14 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.112019+0000 osd.2 (osd.2) 188 : cluster 0 5.8 scrub ok 2024-04-06T03:30:14.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:14 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.110238+0000 osd.2 (osd.2) 187 : cluster 0 5.8 scrub starts 2024-04-06T03:30:14.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:14 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.112019+0000 osd.2 (osd.2) 188 : cluster 0 5.8 scrub ok 2024-04-06T03:30:14.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:14.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:14.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:14 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.110238+0000 osd.2 (osd.2) 187 : cluster 0 5.8 scrub starts 2024-04-06T03:30:14.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:14 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.112019+0000 osd.2 (osd.2) 188 : cluster 0 5.8 scrub ok 2024-04-06T03:30:14.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:14.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:14.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:14.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:15.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:15.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:15.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:15.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:15.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:15.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:15.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:15.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 16 16 29844 29828 116.504 114.25 0.0132017 0.00857697 2024-04-06T03:30:15.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:15.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.149073+0000 mgr.y (mgr.24368) 344 : cluster 0 pgmap v346: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.260436+0000 osd.7 (osd.7) 221 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.264587+0000 osd.7 (osd.7) 222 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.563229+0000 osd.1 (osd.1) 193 : cluster 0 5.12 scrub starts 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.569702+0000 osd.1 (osd.1) 194 : cluster 0 5.12 scrub ok 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.576090+0000 osd.0 (osd.0) 200 : cluster 0 2.4 deep-scrub starts 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[21467]: cluster 2024-04-06T03:30:14.578514+0000 osd.0 (osd.0) 201 : cluster 0 2.4 deep-scrub ok 2024-04-06T03:30:15.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.149073+0000 mgr.y (mgr.24368) 344 : cluster 0 pgmap v346: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:30:15.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.260436+0000 osd.7 (osd.7) 221 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:30:15.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.264587+0000 osd.7 (osd.7) 222 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:30:15.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.563229+0000 osd.1 (osd.1) 193 : cluster 0 5.12 scrub starts 2024-04-06T03:30:15.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.569702+0000 osd.1 (osd.1) 194 : cluster 0 5.12 scrub ok 2024-04-06T03:30:15.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.576090+0000 osd.0 (osd.0) 200 : cluster 0 2.4 deep-scrub starts 2024-04-06T03:30:15.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:15 smithi050 bash[29221]: cluster 2024-04-06T03:30:14.578514+0000 osd.0 (osd.0) 201 : cluster 0 2.4 deep-scrub ok 2024-04-06T03:30:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:15.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:15.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:15.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.149073+0000 mgr.y (mgr.24368) 344 : cluster 0 pgmap v346: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.260436+0000 osd.7 (osd.7) 221 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.264587+0000 osd.7 (osd.7) 222 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.563229+0000 osd.1 (osd.1) 193 : cluster 0 5.12 scrub starts 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.569702+0000 osd.1 (osd.1) 194 : cluster 0 5.12 scrub ok 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.576090+0000 osd.0 (osd.0) 200 : cluster 0 2.4 deep-scrub starts 2024-04-06T03:30:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:15 smithi153 bash[23852]: cluster 2024-04-06T03:30:14.578514+0000 osd.0 (osd.0) 201 : cluster 0 2.4 deep-scrub ok 2024-04-06T03:30:15.815 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:15.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:15.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:15.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:16.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:16.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:16.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:16.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:16.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:16.339 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:16.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:16.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:16.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 17 16 31475 31459 115.647 101.938 0.0140943 0.0086404 2024-04-06T03:30:16.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:16.594 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:16.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:16.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:16.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:16 smithi050 bash[29221]: cluster 2024-04-06T03:30:15.307277+0000 osd.7 (osd.7) 223 : cluster 0 2.b scrub starts 2024-04-06T03:30:16.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:16 smithi050 bash[29221]: cluster 2024-04-06T03:30:15.309113+0000 osd.7 (osd.7) 224 : cluster 0 2.b scrub ok 2024-04-06T03:30:16.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:16 smithi050 bash[21467]: cluster 2024-04-06T03:30:15.307277+0000 osd.7 (osd.7) 223 : cluster 0 2.b scrub starts 2024-04-06T03:30:16.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:16 smithi050 bash[21467]: cluster 2024-04-06T03:30:15.309113+0000 osd.7 (osd.7) 224 : cluster 0 2.b scrub ok 2024-04-06T03:30:16.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:16 smithi153 bash[23852]: cluster 2024-04-06T03:30:15.307277+0000 osd.7 (osd.7) 223 : cluster 0 2.b scrub starts 2024-04-06T03:30:16.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:16 smithi153 bash[23852]: cluster 2024-04-06T03:30:15.309113+0000 osd.7 (osd.7) 224 : cluster 0 2.b scrub ok 2024-04-06T03:30:16.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:16.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:16.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:16.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:16.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:16.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:17.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:17.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:17.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:17.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:17.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:17.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:17.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:17.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 18 16 32661 32645 113.34 74.125 0.00407587 0.00881465 2024-04-06T03:30:17.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:17.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:17.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:17.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[21467]: cluster 2024-04-06T03:30:16.149853+0000 mgr.y (mgr.24368) 345 : cluster 0 pgmap v347: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[21467]: cluster 2024-04-06T03:30:16.314739+0000 osd.7 (osd.7) 225 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[21467]: cluster 2024-04-06T03:30:16.314811+0000 osd.7 (osd.7) 226 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[21467]: cluster 2024-04-06T03:30:16.361543+0000 osd.7 (osd.7) 227 : cluster 0 2.1d scrub starts 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[21467]: cluster 2024-04-06T03:30:16.365087+0000 osd.7 (osd.7) 228 : cluster 0 2.1d scrub ok 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[21467]: cluster 2024-04-06T03:30:16.542949+0000 mon.a (mon.0) 1176 : cluster 0 osdmap e189: 8 total, 8 up, 8 in 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[29221]: cluster 2024-04-06T03:30:16.149853+0000 mgr.y (mgr.24368) 345 : cluster 0 pgmap v347: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[29221]: cluster 2024-04-06T03:30:16.314739+0000 osd.7 (osd.7) 225 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:17.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[29221]: cluster 2024-04-06T03:30:16.314811+0000 osd.7 (osd.7) 226 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:17.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[29221]: cluster 2024-04-06T03:30:16.361543+0000 osd.7 (osd.7) 227 : cluster 0 2.1d scrub starts 2024-04-06T03:30:17.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[29221]: cluster 2024-04-06T03:30:16.365087+0000 osd.7 (osd.7) 228 : cluster 0 2.1d scrub ok 2024-04-06T03:30:17.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:17 smithi050 bash[29221]: cluster 2024-04-06T03:30:16.542949+0000 mon.a (mon.0) 1176 : cluster 0 osdmap e189: 8 total, 8 up, 8 in 2024-04-06T03:30:17.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:17.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:17 smithi153 bash[23852]: cluster 2024-04-06T03:30:16.149853+0000 mgr.y (mgr.24368) 345 : cluster 0 pgmap v347: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 130 MiB/s wr, 2.09k op/s 2024-04-06T03:30:17.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:17 smithi153 bash[23852]: cluster 2024-04-06T03:30:16.314739+0000 osd.7 (osd.7) 225 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:17.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:17 smithi153 bash[23852]: cluster 2024-04-06T03:30:16.314811+0000 osd.7 (osd.7) 226 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:17.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:17 smithi153 bash[23852]: cluster 2024-04-06T03:30:16.361543+0000 osd.7 (osd.7) 227 : cluster 0 2.1d scrub starts 2024-04-06T03:30:17.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:17 smithi153 bash[23852]: cluster 2024-04-06T03:30:16.365087+0000 osd.7 (osd.7) 228 : cluster 0 2.1d scrub ok 2024-04-06T03:30:17.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:17 smithi153 bash[23852]: cluster 2024-04-06T03:30:16.542949+0000 mon.a (mon.0) 1176 : cluster 0 osdmap e189: 8 total, 8 up, 8 in 2024-04-06T03:30:17.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:17.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:17.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:17.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:18.023 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:18.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:18.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:18.279 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:18.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:18.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:18.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 19 16 33851 33835 111.289 74.375 0.00159507 0.00885121 2024-04-06T03:30:18.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:18.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:18.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:18.676 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:18.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:18.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:18 smithi153 bash[23852]: cluster 2024-04-06T03:30:17.491284+0000 osd.4 (osd.4) 225 : cluster 0 3.3 deep-scrub starts 2024-04-06T03:30:18.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:18 smithi153 bash[23852]: cluster 2024-04-06T03:30:17.493996+0000 osd.4 (osd.4) 226 : cluster 0 3.3 deep-scrub ok 2024-04-06T03:30:18.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:18.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:18 smithi050 bash[21467]: cluster 2024-04-06T03:30:17.491284+0000 osd.4 (osd.4) 225 : cluster 0 3.3 deep-scrub starts 2024-04-06T03:30:18.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:18 smithi050 bash[21467]: cluster 2024-04-06T03:30:17.493996+0000 osd.4 (osd.4) 226 : cluster 0 3.3 deep-scrub ok 2024-04-06T03:30:18.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:18 smithi050 bash[29221]: cluster 2024-04-06T03:30:17.491284+0000 osd.4 (osd.4) 225 : cluster 0 3.3 deep-scrub starts 2024-04-06T03:30:18.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:18 smithi050 bash[29221]: cluster 2024-04-06T03:30:17.493996+0000 osd.4 (osd.4) 226 : cluster 0 3.3 deep-scrub ok 2024-04-06T03:30:18.835 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:30:18.912 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:18.931 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:18.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:19.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:19.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:19.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:19.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:19.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:19.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:19.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:19.444 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:30:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:30:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:30:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:19.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:30:19.496604+0000 min lat: 0.00122838 max lat: 0.562935 avg lat: 0.00917382 2024-04-06T03:30:19.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:30:19.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 20 16 34871 34855 108.912 63.75 0.00731514 0.00917382 2024-04-06T03:30:19.538 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:30:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:30:19.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:19.596 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:19.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:30:19.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:19.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:19 smithi050 bash[21467]: cluster 2024-04-06T03:30:18.150334+0000 mgr.y (mgr.24368) 346 : cluster 0 pgmap v349: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-04-06T03:30:19.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:19 smithi050 bash[21467]: audit 2024-04-06T03:30:18.936231+0000 mon.a (mon.0) 1177 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:19.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:19 smithi050 bash[29221]: cluster 2024-04-06T03:30:18.150334+0000 mgr.y (mgr.24368) 346 : cluster 0 pgmap v349: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-04-06T03:30:19.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:19 smithi050 bash[29221]: audit 2024-04-06T03:30:18.936231+0000 mon.a (mon.0) 1177 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:19.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:19 smithi153 bash[23852]: cluster 2024-04-06T03:30:18.150334+0000 mgr.y (mgr.24368) 346 : cluster 0 pgmap v349: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-04-06T03:30:19.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:19 smithi153 bash[23852]: audit 2024-04-06T03:30:18.936231+0000 mon.a (mon.0) 1177 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:19.725 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:19.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:19.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:19.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:19.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:19.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:20.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:20.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:20.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:20.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:20.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:20.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:20.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 21 16 36068 36052 107.287 74.8125 0.00230527 0.00921846 2024-04-06T03:30:20.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:20.527 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:20.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:20.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:20.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:20.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: cluster 2024-04-06T03:30:18.758185+0000 osd.3 (osd.3) 229 : cluster 0 2.1e scrub starts 2024-04-06T03:30:20.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: cluster 2024-04-06T03:30:18.764172+0000 osd.3 (osd.3) 230 : cluster 0 2.1e scrub ok 2024-04-06T03:30:20.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: audit 2024-04-06T03:30:19.419160+0000 mgr.y (mgr.24368) 347 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:20.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: cluster 2024-04-06T03:30:19.536208+0000 osd.4 (osd.4) 227 : cluster 0 6.17 scrub starts 2024-04-06T03:30:20.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: cluster 2024-04-06T03:30:19.570337+0000 osd.4 (osd.4) 228 : cluster 0 6.17 scrub ok 2024-04-06T03:30:20.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.171458+0000 osd.2 (osd.2) 189 : cluster 0 5.a scrub starts 2024-04-06T03:30:20.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:20 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.174411+0000 osd.2 (osd.2) 190 : cluster 0 5.a scrub ok 2024-04-06T03:30:20.791 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:20.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: cluster 2024-04-06T03:30:18.758185+0000 osd.3 (osd.3) 229 : cluster 0 2.1e scrub starts 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: cluster 2024-04-06T03:30:18.764172+0000 osd.3 (osd.3) 230 : cluster 0 2.1e scrub ok 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: audit 2024-04-06T03:30:19.419160+0000 mgr.y (mgr.24368) 347 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: cluster 2024-04-06T03:30:19.536208+0000 osd.4 (osd.4) 227 : cluster 0 6.17 scrub starts 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: cluster 2024-04-06T03:30:19.570337+0000 osd.4 (osd.4) 228 : cluster 0 6.17 scrub ok 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.171458+0000 osd.2 (osd.2) 189 : cluster 0 5.a scrub starts 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.174411+0000 osd.2 (osd.2) 190 : cluster 0 5.a scrub ok 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: cluster 2024-04-06T03:30:18.758185+0000 osd.3 (osd.3) 229 : cluster 0 2.1e scrub starts 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: cluster 2024-04-06T03:30:18.764172+0000 osd.3 (osd.3) 230 : cluster 0 2.1e scrub ok 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: audit 2024-04-06T03:30:19.419160+0000 mgr.y (mgr.24368) 347 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:20.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: cluster 2024-04-06T03:30:19.536208+0000 osd.4 (osd.4) 227 : cluster 0 6.17 scrub starts 2024-04-06T03:30:20.906 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: cluster 2024-04-06T03:30:19.570337+0000 osd.4 (osd.4) 228 : cluster 0 6.17 scrub ok 2024-04-06T03:30:20.906 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.171458+0000 osd.2 (osd.2) 189 : cluster 0 5.a scrub starts 2024-04-06T03:30:20.906 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:20 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.174411+0000 osd.2 (osd.2) 190 : cluster 0 5.a scrub ok 2024-04-06T03:30:20.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:20.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:21.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:21.055 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:21.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:21.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:21.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:21.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:21.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:21.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 22 16 37501 37485 106.481 89.5625 0.00987401 0.00938547 2024-04-06T03:30:21.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.151122+0000 mgr.y (mgr.24368) 348 : cluster 0 pgmap v350: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-06T03:30:21.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: audit 2024-04-06T03:30:20.610550+0000 mon.a (mon.0) 1178 : audit 1 from='client.? 172.21.15.50:0/1407838210' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:30:21.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: audit 2024-04-06T03:30:20.613376+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.50:0/3436472333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:30:21.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: audit 2024-04-06T03:30:20.613773+0000 mon.a (mon.0) 1179 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:30:21.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.639905+0000 osd.7 (osd.7) 229 : cluster 0 6.15 scrub starts 2024-04-06T03:30:21.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.686708+0000 osd.7 (osd.7) 230 : cluster 0 6.15 scrub ok 2024-04-06T03:30:21.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.755670+0000 osd.3 (osd.3) 231 : cluster 0 2.12 scrub starts 2024-04-06T03:30:21.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[21467]: cluster 2024-04-06T03:30:20.759665+0000 osd.3 (osd.3) 232 : cluster 0 2.12 scrub ok 2024-04-06T03:30:21.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:21.567 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:30:21.577 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-06T03:30:21.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:21.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:21.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.151122+0000 mgr.y (mgr.24368) 348 : cluster 0 pgmap v350: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-06T03:30:21.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: audit 2024-04-06T03:30:20.610550+0000 mon.a (mon.0) 1178 : audit 1 from='client.? 172.21.15.50:0/1407838210' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:30:21.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: audit 2024-04-06T03:30:20.613376+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.50:0/3436472333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:30:21.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: audit 2024-04-06T03:30:20.613773+0000 mon.a (mon.0) 1179 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:30:21.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.639905+0000 osd.7 (osd.7) 229 : cluster 0 6.15 scrub starts 2024-04-06T03:30:21.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.686708+0000 osd.7 (osd.7) 230 : cluster 0 6.15 scrub ok 2024-04-06T03:30:21.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.755670+0000 osd.3 (osd.3) 231 : cluster 0 2.12 scrub starts 2024-04-06T03:30:21.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:21 smithi153 bash[23852]: cluster 2024-04-06T03:30:20.759665+0000 osd.3 (osd.3) 232 : cluster 0 2.12 scrub ok 2024-04-06T03:30:21.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:21.854 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:21.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.151122+0000 mgr.y (mgr.24368) 348 : cluster 0 pgmap v350: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: audit 2024-04-06T03:30:20.610550+0000 mon.a (mon.0) 1178 : audit 1 from='client.? 172.21.15.50:0/1407838210' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: audit 2024-04-06T03:30:20.613376+0000 mon.c (mon.2) 59 : audit 1 from='client.? 172.21.15.50:0/3436472333' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: audit 2024-04-06T03:30:20.613773+0000 mon.a (mon.0) 1179 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.639905+0000 osd.7 (osd.7) 229 : cluster 0 6.15 scrub starts 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.686708+0000 osd.7 (osd.7) 230 : cluster 0 6.15 scrub ok 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.755670+0000 osd.3 (osd.3) 231 : cluster 0 2.12 scrub starts 2024-04-06T03:30:21.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:21 smithi050 bash[29221]: cluster 2024-04-06T03:30:20.759665+0000 osd.3 (osd.3) 232 : cluster 0 2.12 scrub ok 2024-04-06T03:30:21.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:21.985 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:21.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:22.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:22.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:22.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:22.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:22.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:22.371 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:22.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:22.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 23 16 39155 39139 106.346 103.375 0.00465659 0.00939553 2024-04-06T03:30:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:22.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:22.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:22 smithi153 bash[23852]: cluster 2024-04-06T03:30:21.494182+0000 osd.4 (osd.4) 229 : cluster 0 5.11 scrub starts 2024-04-06T03:30:22.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:22 smithi153 bash[23852]: cluster 2024-04-06T03:30:21.495329+0000 osd.4 (osd.4) 230 : cluster 0 5.11 scrub ok 2024-04-06T03:30:22.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:22 smithi153 bash[23852]: cluster 2024-04-06T03:30:21.553555+0000 mon.a (mon.0) 1180 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:30:22.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:22 smithi153 bash[23852]: audit 2024-04-06T03:30:21.563941+0000 mon.a (mon.0) 1181 : audit 1 from='client.? 172.21.15.50:0/1407838210' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:30:22.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:22 smithi153 bash[23852]: audit 2024-04-06T03:30:21.564369+0000 mon.a (mon.0) 1182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:30:22.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:22 smithi153 bash[23852]: cluster 2024-04-06T03:30:21.577188+0000 mon.a (mon.0) 1183 : cluster 0 osdmap e190: 8 total, 8 up, 8 in 2024-04-06T03:30:22.635 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:22.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:22.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[21467]: cluster 2024-04-06T03:30:21.494182+0000 osd.4 (osd.4) 229 : cluster 0 5.11 scrub starts 2024-04-06T03:30:22.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[21467]: cluster 2024-04-06T03:30:21.495329+0000 osd.4 (osd.4) 230 : cluster 0 5.11 scrub ok 2024-04-06T03:30:22.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[21467]: cluster 2024-04-06T03:30:21.553555+0000 mon.a (mon.0) 1180 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:30:22.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[21467]: audit 2024-04-06T03:30:21.563941+0000 mon.a (mon.0) 1181 : audit 1 from='client.? 172.21.15.50:0/1407838210' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:30:22.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[21467]: audit 2024-04-06T03:30:21.564369+0000 mon.a (mon.0) 1182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:30:22.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[21467]: cluster 2024-04-06T03:30:21.577188+0000 mon.a (mon.0) 1183 : cluster 0 osdmap e190: 8 total, 8 up, 8 in 2024-04-06T03:30:22.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[29221]: cluster 2024-04-06T03:30:21.494182+0000 osd.4 (osd.4) 229 : cluster 0 5.11 scrub starts 2024-04-06T03:30:22.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[29221]: cluster 2024-04-06T03:30:21.495329+0000 osd.4 (osd.4) 230 : cluster 0 5.11 scrub ok 2024-04-06T03:30:22.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[29221]: cluster 2024-04-06T03:30:21.553555+0000 mon.a (mon.0) 1180 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:30:22.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[29221]: audit 2024-04-06T03:30:21.563941+0000 mon.a (mon.0) 1181 : audit 1 from='client.? 172.21.15.50:0/1407838210' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:30:22.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[29221]: audit 2024-04-06T03:30:21.564369+0000 mon.a (mon.0) 1182 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:30:22.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:22 smithi050 bash[29221]: cluster 2024-04-06T03:30:21.577188+0000 mon.a (mon.0) 1183 : cluster 0 osdmap e190: 8 total, 8 up, 8 in 2024-04-06T03:30:22.766 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:22.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:22.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:22.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:23.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:23.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:23.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:23.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:23.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:23.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:23.409 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:23.490 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 24 16 40448 40432 105.281 80.8125 0.00252636 0.00943452 2024-04-06T03:30:23.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:23.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:23.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:23.681 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:23.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:23.813 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:23.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:23.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: cluster 2024-04-06T03:30:21.682876+0000 osd.5 (osd.5) 127 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:23.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: cluster 2024-04-06T03:30:21.682946+0000 osd.5 (osd.5) 128 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:23.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: cluster 2024-04-06T03:30:22.151562+0000 mgr.y (mgr.24368) 349 : cluster 0 pgmap v352: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-04-06T03:30:23.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: cluster 2024-04-06T03:30:22.580099+0000 mon.a (mon.0) 1184 : cluster 0 osdmap e191: 8 total, 8 up, 8 in 2024-04-06T03:30:23.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: cluster 2024-04-06T03:30:22.898943+0000 osd.1 (osd.1) 195 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:23.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: cluster 2024-04-06T03:30:22.899012+0000 osd.1 (osd.1) 196 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:23.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:23 smithi153 bash[23852]: audit 2024-04-06T03:30:23.068556+0000 mon.a (mon.0) 1185 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:23.947 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:23.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:23.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: cluster 2024-04-06T03:30:21.682876+0000 osd.5 (osd.5) 127 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:23.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: cluster 2024-04-06T03:30:21.682946+0000 osd.5 (osd.5) 128 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: cluster 2024-04-06T03:30:22.151562+0000 mgr.y (mgr.24368) 349 : cluster 0 pgmap v352: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: cluster 2024-04-06T03:30:22.580099+0000 mon.a (mon.0) 1184 : cluster 0 osdmap e191: 8 total, 8 up, 8 in 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: cluster 2024-04-06T03:30:22.898943+0000 osd.1 (osd.1) 195 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: cluster 2024-04-06T03:30:22.899012+0000 osd.1 (osd.1) 196 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[21467]: audit 2024-04-06T03:30:23.068556+0000 mon.a (mon.0) 1185 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: cluster 2024-04-06T03:30:21.682876+0000 osd.5 (osd.5) 127 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: cluster 2024-04-06T03:30:21.682946+0000 osd.5 (osd.5) 128 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: cluster 2024-04-06T03:30:22.151562+0000 mgr.y (mgr.24368) 349 : cluster 0 pgmap v352: 148 pgs: 148 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: cluster 2024-04-06T03:30:22.580099+0000 mon.a (mon.0) 1184 : cluster 0 osdmap e191: 8 total, 8 up, 8 in 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: cluster 2024-04-06T03:30:22.898943+0000 osd.1 (osd.1) 195 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: cluster 2024-04-06T03:30:22.899012+0000 osd.1 (osd.1) 196 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:23.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:23 smithi050 bash[29221]: audit 2024-04-06T03:30:23.068556+0000 mon.a (mon.0) 1185 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:30:24.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:24.071 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:24.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:24.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:24.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:24.429 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:30:24.430 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:30:24.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:24.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:24.489 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 25 16 41260 41244 103.1 50.75 0.00530434 0.00969347 2024-04-06T03:30:24.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:24.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:24.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:24.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[21467]: cluster 2024-04-06T03:30:23.579981+0000 mon.a (mon.0) 1186 : cluster 0 osdmap e192: 8 total, 8 up, 8 in 2024-04-06T03:30:24.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[21467]: audit 2024-04-06T03:30:23.855629+0000 mon.a (mon.0) 1187 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:24.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[21467]: audit 2024-04-06T03:30:23.857881+0000 mon.a (mon.0) 1188 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:24.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[21467]: audit 2024-04-06T03:30:23.865857+0000 mon.a (mon.0) 1189 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:24.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[29221]: cluster 2024-04-06T03:30:23.579981+0000 mon.a (mon.0) 1186 : cluster 0 osdmap e192: 8 total, 8 up, 8 in 2024-04-06T03:30:24.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[29221]: audit 2024-04-06T03:30:23.855629+0000 mon.a (mon.0) 1187 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:24.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[29221]: audit 2024-04-06T03:30:23.857881+0000 mon.a (mon.0) 1188 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:24.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:24 smithi050 bash[29221]: audit 2024-04-06T03:30:23.865857+0000 mon.a (mon.0) 1189 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:24.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:24.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:24.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:24.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:24.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:24 smithi153 bash[23852]: cluster 2024-04-06T03:30:23.579981+0000 mon.a (mon.0) 1186 : cluster 0 osdmap e192: 8 total, 8 up, 8 in 2024-04-06T03:30:24.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:24 smithi153 bash[23852]: audit 2024-04-06T03:30:23.855629+0000 mon.a (mon.0) 1187 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:30:24.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:24 smithi153 bash[23852]: audit 2024-04-06T03:30:23.857881+0000 mon.a (mon.0) 1188 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:30:24.981 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:24 smithi153 bash[23852]: audit 2024-04-06T03:30:23.865857+0000 mon.a (mon.0) 1189 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:24.994 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:24.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:25.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:25.124 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:25.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 1 1 2024-04-06T03:30:25.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:25.251 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:25.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-06T03:30:25.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:25.382 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:25.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:25.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:25.490 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 26 16 42775 42759 102.776 94.6875 0.00722936 0.00972387 2024-04-06T03:30:25.518 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:25.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:25.647 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:25.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:25.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:25 smithi153 bash[23852]: cluster 2024-04-06T03:30:24.152565+0000 mgr.y (mgr.24368) 350 : cluster 0 pgmap v355: 148 pgs: 148 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-04-06T03:30:25.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:25.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:25.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:25 smithi050 bash[21467]: cluster 2024-04-06T03:30:24.152565+0000 mgr.y (mgr.24368) 350 : cluster 0 pgmap v355: 148 pgs: 148 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-04-06T03:30:25.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:25 smithi050 bash[29221]: cluster 2024-04-06T03:30:24.152565+0000 mgr.y (mgr.24368) 350 : cluster 0 pgmap v355: 148 pgs: 148 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-04-06T03:30:25.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:25.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:26.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:26.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:26.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:26.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:26.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:26.414 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:26.490 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 27 16 44326 44310 102.559 96.9375 0.0059427 0.00974505 2024-04-06T03:30:26.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:26.551 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:26.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:26.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:26.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:26.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:26.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:26.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:26.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:26.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:27.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:27.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:27.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:27.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:30:27.208 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:27.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:27.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:27.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:27.493 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 28 16 45811 45795 102.201 92.8125 0.00491907 0.00977948 2024-04-06T03:30:27.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:27.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:27.692 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:27.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:27.732 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:27 smithi153 bash[23852]: cluster 2024-04-06T03:30:26.153336+0000 mgr.y (mgr.24368) 351 : cluster 0 pgmap v356: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:30:27.748 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:27.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:27.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:27 smithi050 bash[21467]: cluster 2024-04-06T03:30:26.153336+0000 mgr.y (mgr.24368) 351 : cluster 0 pgmap v356: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:30:27.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:27 smithi050 bash[29221]: cluster 2024-04-06T03:30:26.153336+0000 mgr.y (mgr.24368) 351 : cluster 0 pgmap v356: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:30:27.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:27.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:27.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:28.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:28.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:28.143 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:28.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:28.275 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:28.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:28.406 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:28.493 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 29 16 47274 47258 101.83 91.4375 0.00369896 0.00981394 2024-04-06T03:30:28.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:28.530 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:28.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:28.658 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:28.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:28.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:28.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:28.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:28.920 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:28.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:29.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:29.048 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:29.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:29.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:29.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:29.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:29.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:29.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:29.493 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 30 16 48683 48667 101.371 88.0625 0.0104637 0.00985833 2024-04-06T03:30:29.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:29.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:29.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:30:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:30:29.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:29 smithi153 bash[23852]: cluster 2024-04-06T03:30:28.153782+0000 mgr.y (mgr.24368) 352 : cluster 0 pgmap v357: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 777 B/s rd, 62 MiB/s wr, 992 op/s 2024-04-06T03:30:29.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:29 smithi153 bash[23852]: cluster 2024-04-06T03:30:28.252848+0000 osd.3 (osd.3) 233 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:29.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:29 smithi153 bash[23852]: cluster 2024-04-06T03:30:28.252907+0000 osd.3 (osd.3) 234 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:29.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:29 smithi153 bash[23852]: cluster 2024-04-06T03:30:28.608203+0000 mon.a (mon.0) 1190 : cluster 0 osdmap e193: 8 total, 8 up, 8 in 2024-04-06T03:30:29.707 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:29.725 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[21467]: cluster 2024-04-06T03:30:28.153782+0000 mgr.y (mgr.24368) 352 : cluster 0 pgmap v357: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 777 B/s rd, 62 MiB/s wr, 992 op/s 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[21467]: cluster 2024-04-06T03:30:28.252848+0000 osd.3 (osd.3) 233 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[21467]: cluster 2024-04-06T03:30:28.252907+0000 osd.3 (osd.3) 234 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[21467]: cluster 2024-04-06T03:30:28.608203+0000 mon.a (mon.0) 1190 : cluster 0 osdmap e193: 8 total, 8 up, 8 in 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:30:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[29221]: cluster 2024-04-06T03:30:28.153782+0000 mgr.y (mgr.24368) 352 : cluster 0 pgmap v357: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 777 B/s rd, 62 MiB/s wr, 992 op/s 2024-04-06T03:30:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[29221]: cluster 2024-04-06T03:30:28.252848+0000 osd.3 (osd.3) 233 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[29221]: cluster 2024-04-06T03:30:28.252907+0000 osd.3 (osd.3) 234 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:29 smithi050 bash[29221]: cluster 2024-04-06T03:30:28.608203+0000 mon.a (mon.0) 1190 : cluster 0 osdmap e193: 8 total, 8 up, 8 in 2024-04-06T03:30:29.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:29.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:29.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:29.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:29.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:30.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:30.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:30.171 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:30:30.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:30.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:30.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:30.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:30.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:30.487 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:30.493 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 31 16 49966 49950 100.687 80.1875 0.0184727 0.00992654 2024-04-06T03:30:30.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:30.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:30.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:30.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:30 smithi153 bash[23852]: audit 2024-04-06T03:30:29.428111+0000 mgr.y (mgr.24368) 353 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:30.850 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:30.874 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:30.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:30.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:30.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:30 smithi050 bash[21467]: audit 2024-04-06T03:30:29.428111+0000 mgr.y (mgr.24368) 353 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:30.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:30 smithi050 bash[29221]: audit 2024-04-06T03:30:29.428111+0000 mgr.y (mgr.24368) 353 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:31.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:31.135 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:31.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:31.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:31.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:31.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:31.493 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 32 16 51127 51111 99.8083 72.5625 0.00228432 0.00998671 2024-04-06T03:30:31.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:31.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:31.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:31.657 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:31.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:31 smithi050 bash[21467]: cluster 2024-04-06T03:30:30.154648+0000 mgr.y (mgr.24368) 354 : cluster 0 pgmap v359: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-04-06T03:30:31.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:31 smithi050 bash[29221]: cluster 2024-04-06T03:30:30.154648+0000 mgr.y (mgr.24368) 354 : cluster 0 pgmap v359: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-04-06T03:30:31.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:31.763 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:31.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:31.867 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:31 smithi153 bash[23852]: cluster 2024-04-06T03:30:30.154648+0000 mgr.y (mgr.24368) 354 : cluster 0 pgmap v359: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-04-06T03:30:31.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:31.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:31.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:32.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:32.056 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:32.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:32.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:32.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:32.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:32.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:32.493 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 33 16 51858 51842 98.1683 45.6875 0.00269268 0.0101357 2024-04-06T03:30:32.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:32.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:32.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:32.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:32.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:32.857 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:32.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:32.997 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:32.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:33.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:33.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:33.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:33.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:33.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:33.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:33.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 34 16 53163 53147 97.6688 81.5625 0.00997534 0.0102319 2024-04-06T03:30:33.510 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:33.613 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:33.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:33 smithi153 bash[23852]: cluster 2024-04-06T03:30:32.155111+0000 mgr.y (mgr.24368) 355 : cluster 0 pgmap v360: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:30:33.635 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:33.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:33.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:33.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:33.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:33 smithi050 bash[21467]: cluster 2024-04-06T03:30:32.155111+0000 mgr.y (mgr.24368) 355 : cluster 0 pgmap v360: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:30:33.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:33 smithi050 bash[29221]: cluster 2024-04-06T03:30:32.155111+0000 mgr.y (mgr.24368) 355 : cluster 0 pgmap v360: 148 pgs: 148 active+clean; 3.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:30:33.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:33.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:33.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:34.026 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:34.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:34.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:34.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:34.288 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:34.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:34.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 35 16 54275 54259 96.8639 69.5 0.0155554 0.0103166 2024-04-06T03:30:34.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:34.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:34.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:34.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:34.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:34.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:34.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:34.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:34.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:35.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:35.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:35 smithi050 bash[21467]: audit 2024-04-06T03:30:33.968291+0000 mon.a (mon.0) 1191 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:35.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:35 smithi050 bash[21467]: audit 2024-04-06T03:30:33.969505+0000 mon.a (mon.0) 1192 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:35.075 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:35 smithi050 bash[21467]: cluster 2024-04-06T03:30:34.155615+0000 mgr.y (mgr.24368) 356 : cluster 0 pgmap v361: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-06T03:30:35.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:35 smithi050 bash[29221]: audit 2024-04-06T03:30:33.968291+0000 mon.a (mon.0) 1191 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:35.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:35 smithi050 bash[29221]: audit 2024-04-06T03:30:33.969505+0000 mon.a (mon.0) 1192 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:35.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:35 smithi050 bash[29221]: cluster 2024-04-06T03:30:34.155615+0000 mgr.y (mgr.24368) 356 : cluster 0 pgmap v361: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-06T03:30:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:35.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:35.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:34 smithi153 bash[23852]: audit 2024-04-06T03:30:33.968291+0000 mon.a (mon.0) 1191 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:30:35.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:34 smithi153 bash[23852]: audit 2024-04-06T03:30:33.969505+0000 mon.a (mon.0) 1192 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:35.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:34 smithi153 bash[23852]: cluster 2024-04-06T03:30:34.155615+0000 mgr.y (mgr.24368) 356 : cluster 0 pgmap v361: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-04-06T03:30:35.225 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:35.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:35.429 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:30:35.429 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:30:35.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:35.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:35.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 36 16 55450 55434 96.2131 73.4375 0.0066317 0.0103881 2024-04-06T03:30:35.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:35.623 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:35.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.760 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:35.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:35.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:35.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:35.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:35.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:36.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:36.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:36.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:36.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:36.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:36.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:36.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:36.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-06T03:30:36.399 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:36.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 37 16 56786 56770 95.8695 83.5 0.0150743 0.0104255 2024-04-06T03:30:36.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:36.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:36.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:36.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:36.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:36.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:36.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:36.927 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:36.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:37.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:37.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:37.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:37.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:37.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:37.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:37.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:37.441 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:37.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 38 16 57553 57537 94.6082 47.9375 0.00240976 0.0105287 2024-04-06T03:30:37.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:37 smithi050 bash[21467]: cluster 2024-04-06T03:30:36.156522+0000 mgr.y (mgr.24368) 357 : cluster 0 pgmap v362: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:37 smithi050 bash[29221]: cluster 2024-04-06T03:30:36.156522+0000 mgr.y (mgr.24368) 357 : cluster 0 pgmap v362: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:30:37.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:37.545 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:37 smithi153 bash[23852]: cluster 2024-04-06T03:30:36.156522+0000 mgr.y (mgr.24368) 357 : cluster 0 pgmap v362: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:30:37.591 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:37.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:37.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:37.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:37.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:37.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:37.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:37.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:38.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:38.123 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:38.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:38.252 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:38.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:38.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:38.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:38.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 39 15 58834 58819 94.2367 80.125 0.0080534 0.010606 2024-04-06T03:30:38.516 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:38.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:38.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:38.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:38.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:38.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:38.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:38.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:38.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:39.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:39.038 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:39.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:39.164 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:39.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:39.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:39.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:39.422 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:39 smithi153 bash[23852]: cluster 2024-04-06T03:30:38.156949+0000 mgr.y (mgr.24368) 358 : cluster 0 pgmap v363: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:30:39.439 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:39.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:39 smithi050 bash[21467]: cluster 2024-04-06T03:30:38.156949+0000 mgr.y (mgr.24368) 358 : cluster 0 pgmap v363: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:30:39.485 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:30:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:30:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:30:39.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:39 smithi050 bash[29221]: cluster 2024-04-06T03:30:38.156949+0000 mgr.y (mgr.24368) 358 : cluster 0 pgmap v363: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-04-06T03:30:39.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:30:39.504941+0000 min lat: 0.00122838 max lat: 0.562935 avg lat: 0.0106961 2024-04-06T03:30:39.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:30:39.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 40 16 59832 59816 93.4386 62.3125 0.00923063 0.0106961 2024-04-06T03:30:39.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:39.545 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:30:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:30:39.577 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:39.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:39.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:39.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:39.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:39.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:39.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:39.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:40.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:40.082 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:40.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:40.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:40.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:40.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:40.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:40.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:40 smithi153 bash[23852]: audit 2024-04-06T03:30:39.436718+0000 mgr.y (mgr.24368) 359 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:40.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:40.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:40 smithi050 bash[21467]: audit 2024-04-06T03:30:39.436718+0000 mgr.y (mgr.24368) 359 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:40.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:40 smithi050 bash[29221]: audit 2024-04-06T03:30:39.436718+0000 mgr.y (mgr.24368) 359 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:40.497 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 41 16 60893 60877 92.7769 66.3125 0.0200866 0.0107733 2024-04-06T03:30:40.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:40.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:40.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:40.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:40.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:40.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:40.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:40.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:40.997 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:41.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:41.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:41.206 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:30:41.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:41.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:41.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:41.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:41 smithi050 bash[21467]: cluster 2024-04-06T03:30:40.157731+0000 mgr.y (mgr.24368) 360 : cluster 0 pgmap v364: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-04-06T03:30:41.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:41 smithi050 bash[21467]: cluster 2024-04-06T03:30:40.593462+0000 osd.4 (osd.4) 231 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:41.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:41 smithi050 bash[21467]: cluster 2024-04-06T03:30:40.593521+0000 osd.4 (osd.4) 232 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:41.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:41 smithi050 bash[29221]: cluster 2024-04-06T03:30:40.157731+0000 mgr.y (mgr.24368) 360 : cluster 0 pgmap v364: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-04-06T03:30:41.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:41 smithi050 bash[29221]: cluster 2024-04-06T03:30:40.593462+0000 osd.4 (osd.4) 231 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:41.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:41 smithi050 bash[29221]: cluster 2024-04-06T03:30:40.593521+0000 osd.4 (osd.4) 232 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:41.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:41.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 42 16 61812 61796 91.9355 57.4375 0.0176277 0.0108715 2024-04-06T03:30:41.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:41 smithi153 bash[23852]: cluster 2024-04-06T03:30:40.157731+0000 mgr.y (mgr.24368) 360 : cluster 0 pgmap v364: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 99 MiB/s wr, 1.59k op/s 2024-04-06T03:30:41.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:41 smithi153 bash[23852]: cluster 2024-04-06T03:30:40.593462+0000 osd.4 (osd.4) 231 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:41.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:41 smithi153 bash[23852]: cluster 2024-04-06T03:30:40.593521+0000 osd.4 (osd.4) 232 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:41.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:41.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:41.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:41.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:41.780 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:41.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:41.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:41.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:42.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:42.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:42.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:42.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:42.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:42.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:42 smithi050 bash[21467]: cluster 2024-04-06T03:30:41.227589+0000 mon.a (mon.0) 1193 : cluster 0 osdmap e194: 8 total, 8 up, 8 in 2024-04-06T03:30:42.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:42 smithi050 bash[29221]: cluster 2024-04-06T03:30:41.227589+0000 mon.a (mon.0) 1193 : cluster 0 osdmap e194: 8 total, 8 up, 8 in 2024-04-06T03:30:42.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:42.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:42.425 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:42.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:42.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 43 16 62630 62614 90.9865 51.125 0.0373414 0.0109837 2024-04-06T03:30:42.527 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:42.552 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:42.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:42.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:42.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:42.773 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:30:42.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:42 smithi153 bash[23852]: cluster 2024-04-06T03:30:41.227589+0000 mon.a (mon.0) 1193 : cluster 0 osdmap e194: 8 total, 8 up, 8 in 2024-04-06T03:30:42.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:42.813 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:30:42.844 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:42.860 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:30:42.913 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:30:42.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:42.958 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:30:42.974 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:42.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:43.003 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:30:43.043 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:30:43.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:43.086 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:30:43.097 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:30:43.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:43.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:43.244 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:43.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:43.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:43 smithi050 bash[21467]: cluster 2024-04-06T03:30:42.158057+0000 mgr.y (mgr.24368) 361 : cluster 0 pgmap v366: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:30:43.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:43 smithi050 bash[21467]: audit 2024-04-06T03:30:42.734263+0000 mon.b (mon.1) 36 : audit 0 from='client.? 172.21.15.50:0/3169219340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:30:43.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:43 smithi050 bash[29221]: cluster 2024-04-06T03:30:42.158057+0000 mgr.y (mgr.24368) 361 : cluster 0 pgmap v366: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:30:43.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:43 smithi050 bash[29221]: audit 2024-04-06T03:30:42.734263+0000 mon.b (mon.1) 36 : audit 0 from='client.? 172.21.15.50:0/3169219340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:30:43.394 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:43.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 44 16 63674 63658 90.4015 65.25 0.00689244 0.011051 2024-04-06T03:30:43.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:43 smithi153 bash[23852]: cluster 2024-04-06T03:30:42.158057+0000 mgr.y (mgr.24368) 361 : cluster 0 pgmap v366: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-04-06T03:30:43.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:43 smithi153 bash[23852]: audit 2024-04-06T03:30:42.734263+0000 mon.b (mon.1) 36 : audit 0 from='client.? 172.21.15.50:0/3169219340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:30:43.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:43.521 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:43.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:43.649 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:43.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:43.793 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:43.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:43.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:43.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:44.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:44.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:44.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:44.187 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:44.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:44.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:44.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:44.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:44.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 45 16 64851 64835 90.0272 73.5625 0.0119555 0.0111024 2024-04-06T03:30:44.516 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-04-06T03:30:44.534 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:30:44.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:44.582 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:44.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:44.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:44.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:44 smithi153 bash[23852]: audit 2024-04-06T03:30:43.807267+0000 mon.a (mon.0) 1194 : audit 1 from='client.? 172.21.15.50:0/1511231448' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:30:44.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:44 smithi153 bash[23852]: audit 2024-04-06T03:30:43.834132+0000 mon.c (mon.2) 60 : audit 1 from='client.? 172.21.15.50:0/1698249238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:30:44.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:44 smithi153 bash[23852]: audit 2024-04-06T03:30:43.834577+0000 mon.a (mon.0) 1195 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:30:44.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:44 smithi050 bash[21467]: audit 2024-04-06T03:30:43.807267+0000 mon.a (mon.0) 1194 : audit 1 from='client.? 172.21.15.50:0/1511231448' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:30:44.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:44 smithi050 bash[21467]: audit 2024-04-06T03:30:43.834132+0000 mon.c (mon.2) 60 : audit 1 from='client.? 172.21.15.50:0/1698249238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:30:44.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:44 smithi050 bash[21467]: audit 2024-04-06T03:30:43.834577+0000 mon.a (mon.0) 1195 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:30:44.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:44 smithi050 bash[29221]: audit 2024-04-06T03:30:43.807267+0000 mon.a (mon.0) 1194 : audit 1 from='client.? 172.21.15.50:0/1511231448' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:30:44.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:44 smithi050 bash[29221]: audit 2024-04-06T03:30:43.834132+0000 mon.c (mon.2) 60 : audit 1 from='client.? 172.21.15.50:0/1698249238' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:30:44.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:44 smithi050 bash[29221]: audit 2024-04-06T03:30:43.834577+0000 mon.a (mon.0) 1195 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:30:44.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:44.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:44.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:44.984 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:44.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:45.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:45.106 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:45.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:45.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:45.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:45.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:45.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:45.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:45.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 46 16 66081 66065 89.7412 76.875 0.0107884 0.0111378 2024-04-06T03:30:45.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:45.624 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:45.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:45 smithi153 bash[23852]: cluster 2024-04-06T03:30:44.160029+0000 mgr.y (mgr.24368) 362 : cluster 0 pgmap v367: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:30:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:45 smithi153 bash[23852]: cluster 2024-04-06T03:30:44.501890+0000 mon.a (mon.0) 1196 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:30:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:45 smithi153 bash[23852]: audit 2024-04-06T03:30:44.516519+0000 mon.a (mon.0) 1197 : audit 1 from='client.? 172.21.15.50:0/1511231448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:30:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:45 smithi153 bash[23852]: audit 2024-04-06T03:30:44.516651+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:30:45.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:45 smithi153 bash[23852]: cluster 2024-04-06T03:30:44.523781+0000 mon.a (mon.0) 1199 : cluster 0 osdmap e195: 8 total, 8 up, 8 in 2024-04-06T03:30:45.757 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:45.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:45.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[21467]: cluster 2024-04-06T03:30:44.160029+0000 mgr.y (mgr.24368) 362 : cluster 0 pgmap v367: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:30:45.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[21467]: cluster 2024-04-06T03:30:44.501890+0000 mon.a (mon.0) 1196 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:30:45.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[21467]: audit 2024-04-06T03:30:44.516519+0000 mon.a (mon.0) 1197 : audit 1 from='client.? 172.21.15.50:0/1511231448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:30:45.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[21467]: audit 2024-04-06T03:30:44.516651+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:30:45.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[21467]: cluster 2024-04-06T03:30:44.523781+0000 mon.a (mon.0) 1199 : cluster 0 osdmap e195: 8 total, 8 up, 8 in 2024-04-06T03:30:45.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[29221]: cluster 2024-04-06T03:30:44.160029+0000 mgr.y (mgr.24368) 362 : cluster 0 pgmap v367: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:30:45.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[29221]: cluster 2024-04-06T03:30:44.501890+0000 mon.a (mon.0) 1196 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:30:45.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[29221]: audit 2024-04-06T03:30:44.516519+0000 mon.a (mon.0) 1197 : audit 1 from='client.? 172.21.15.50:0/1511231448' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:30:45.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[29221]: audit 2024-04-06T03:30:44.516651+0000 mon.a (mon.0) 1198 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:30:45.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:45 smithi050 bash[29221]: cluster 2024-04-06T03:30:44.523781+0000 mon.a (mon.0) 1199 : cluster 0 osdmap e195: 8 total, 8 up, 8 in 2024-04-06T03:30:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:45.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:45.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:46.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:46.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:46.142 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:17585 7.7 7:e0372a13:::benchmark_data_smithi050_68188_object17584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:30:46.187 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:45.688227+0000", 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 300.498879128, 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.27866005999999999, 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 17585 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688227+0000", 2024-04-06T03:30:46.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688227+0000", 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688229+0000", 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1939999999999999e-06 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688265+0000", 2024-04-06T03:30:46.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6396000000000002e-05 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688274+0000", 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3219999999999999e-06 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688281+0000", 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7989999999999996e-06 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688328+0000", 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6937999999999997e-05 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688453+0000", 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012471100000000001 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.688560+0000", 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000107422 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.690503+0000", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001943391 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692497+0000", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001993945 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.966620+0000", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.27412304799999998 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.966776+0000", 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015500999999999999 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.966887+0000", 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111884 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:17592 7.7 7:e8487811:::benchmark_data_smithi050_68188_object17591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:45.691572+0000", 2024-04-06T03:30:46.193 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 300.49553425699997, 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30351632099999998, 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 17592 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691572+0000", 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691572+0000", 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691574+0000", 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3340000000000001e-06 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691635+0000", 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1068000000000006e-05 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.195 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691646+0000", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1538e-05 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691653+0000", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7789999999999998e-06 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691703+0000", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0278999999999998e-05 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691789+0000", 2024-04-06T03:30:46.196 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5521999999999999e-05 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.691902+0000", 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011333299999999999 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.693122+0000", 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0012195470000000001 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.967021+0000", 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.27389904300000001 2024-04-06T03:30:46.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.994793+0000", 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.027772721 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.994973+0000", 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017973300000000001 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.995089+0000", 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000115424 2024-04-06T03:30:46.198 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:17593 7.7 7:e0a366a2:::benchmark_data_smithi050_68188_object17592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:45.692106+0000", 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 300.49499998700003, 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.30348360499999999, 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 17593 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.199 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692106+0000", 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692106+0000", 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692107+0000", 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4399999999999998e-07 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692173+0000", 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5640000000000002e-05 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692185+0000", 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1868999999999999e-05 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692193+0000", 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7959999999999996e-06 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692534+0000", 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00034122299999999998 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692622+0000", 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7829000000000002e-05 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.692708+0000", 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6219999999999998e-05 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.693859+0000", 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0011506330000000001 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.967147+0000", 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.27328806 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.995213+0000", 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.028065903999999999 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.995466+0000", 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025313199999999999 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:45.995590+0000", 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012435500000000001 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:18847 7.d 7:bc3b19db:::benchmark_data_smithi050_68188_object18846:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-04-06T03:30:46.204 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:46.742342+0000", 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 299.44476421000002, 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.23187981899999999, 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 18847 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742342+0000", 2024-04-06T03:30:46.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742342+0000", 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742347+0000", 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6990000000000002e-06 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742409+0000", 2024-04-06T03:30:46.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1620999999999998e-05 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742422+0000", 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3132999999999999e-05 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742429+0000", 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9970000000000001e-06 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742493+0000", 2024-04-06T03:30:46.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4529999999999994e-05 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742631+0000", 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013795100000000001 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,1", 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.742740+0000", 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010849399999999999 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.743523+0000", 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00078345200000000002 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.743825+0000", 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00030151599999999999 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.973889+0000", 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.23006448900000001 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.974087+0000", 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019749000000000001 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:46.974222+0000", 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013544699999999999 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:25133 7.7 7:ea91ff98:::benchmark_data_smithi050_68188_object25132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:30:46.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:50.739388+0000", 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 295.447718678, 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20686870199999999, 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 25133 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739388+0000", 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739388+0000", 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739389+0000", 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1299999999999999e-07 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739499+0000", 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000109969 2024-04-06T03:30:46.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739506+0000", 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6090000000000003e-06 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739511+0000", 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6600000000000003e-06 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739550+0000", 2024-04-06T03:30:46.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9583999999999999e-05 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739627+0000", 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6878999999999994e-05 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739713+0000", 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5340000000000006e-05 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.740870+0000", 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001157541 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743441+0000", 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025704069999999998 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.945975+0000", 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.202533979 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.215 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.946137+0000", 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000162356 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.946256+0000", 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000119565 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:25134 7.7 7:ee4ca489:::benchmark_data_smithi050_68188_object25133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:50.739785+0000", 2024-04-06T03:30:46.216 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 295.44732167000001, 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.207047116, 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 25134 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739785+0000", 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.217 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739785+0000", 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739786+0000", 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2500000000000005e-07 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739858+0000", 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2245999999999999e-05 2024-04-06T03:30:46.218 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739865+0000", 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.481e-06 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.739870+0000", 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2899999999999996e-06 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.740132+0000", 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000262618 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.219 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.740219+0000", 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6538000000000004e-05 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.740313+0000", 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4728999999999994e-05 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.741020+0000", 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00070703300000000003 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.220 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.880230+0000", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.13920975199999999 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.946568+0000", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.066338012000000002 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.946681+0000", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011291800000000001 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.221 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.946832+0000", 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000150774 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:25145 7.7 7:ec4f8fd1:::benchmark_data_smithi050_68188_object25144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:50.743881+0000", 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 295.44322510699999, 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20340055400000001, 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.222 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 25145 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743881+0000", 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743881+0000", 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.223 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743892+0000", 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0190999999999999e-05 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743923+0000", 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1111000000000001e-05 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743935+0000", 2024-04-06T03:30:46.224 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2721000000000001e-05 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.743943+0000", 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4909999999999999e-06 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.744004+0000", 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0995000000000001e-05 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.744142+0000", 2024-04-06T03:30:46.225 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013834700000000001 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.744293+0000", 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000150836 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.745228+0000", 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00093535299999999999 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.880466+0000", 2024-04-06T03:30:46.226 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.13523722699999999 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.946997+0000", 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.066531007000000003 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.947159+0000", 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016198699999999999 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:50.947282+0000", 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012328799999999999 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.227 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:31538 7.7 7:e7f80031:::benchmark_data_smithi050_68188_object31537:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:55.825498+0000", 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 290.36160837, 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48900570399999999, 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 31538 2024-04-06T03:30:46.228 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825498+0000", 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825498+0000", 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825499+0000", 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3200000000000001e-06 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825539+0000", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9496999999999998e-05 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825551+0000", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1951e-05 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825559+0000", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1310000000000006e-06 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.230 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825681+0000", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012208100000000001 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825808+0000", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012645500000000001 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.825952+0000", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014447700000000001 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.827036+0000", 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0010841690000000001 2024-04-06T03:30:46.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.830517+0000", 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0034812810000000001 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.314145+0000", 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48362707300000002 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.314354+0000", 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020949599999999999 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.232 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.314504+0000", 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014977299999999999 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:31559 7.7 7:e8239444:::benchmark_data_smithi050_68188_object31558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:55.835503+0000", 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 290.35160365299998, 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.49507427599999998, 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.233 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 31559 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835503+0000", 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835503+0000", 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.234 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835504+0000", 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0580000000000001e-06 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835559+0000", 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4870999999999997e-05 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835570+0000", 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1095e-05 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.235 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835578+0000", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2719999999999997e-06 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835646+0000", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8336000000000004e-05 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.835823+0000", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017646200000000001 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.836009+0000", 2024-04-06T03:30:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018614599999999999 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.837504+0000", 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001494977 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.844228+0000", 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0067240499999999996 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.330465+0000", 2024-04-06T03:30:46.237 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.486236644 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.330523+0000", 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8612000000000001e-05 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.330577+0000", 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3752999999999998e-05 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:31573 7.7 7:e3b13f77:::benchmark_data_smithi050_68188_object31572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:25:55.846466+0000", 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 290.34064034800002, 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.490276463, 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 31573 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.239 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846466+0000", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846466+0000", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846467+0000", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3459999999999999e-06 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846526+0000", 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8202999999999998e-05 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846538+0000", 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1855000000000001e-05 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846543+0000", 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5400000000000003e-06 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846598+0000", 2024-04-06T03:30:46.241 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.448e-05 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846747+0000", 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014994800000000001 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.846909+0000", 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016110500000000001 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.848217+0000", 2024-04-06T03:30:46.242 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013085939999999999 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:55.850937+0000", 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027203100000000001 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.336630+0000", 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48569214900000002 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.336698+0000", 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8794999999999994e-05 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.243 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:25:56.336743+0000", 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4138000000000003e-05 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:36814 7.d 7:b1c53922:::benchmark_data_smithi050_68188_object36813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:26:00.169445+0000", 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 286.01766134600001, 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.13950212300000001, 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.244 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 36814 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169445+0000", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169445+0000", 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.245 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169446+0000", 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7599999999999996e-07 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169504+0000", 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8476000000000001e-05 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169514+0000", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7200000000000001e-06 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169520+0000", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6810000000000003e-06 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169562+0000", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2478000000000002e-05 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169648+0000", 2024-04-06T03:30:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5844e-05 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,1", 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.169737+0000", 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8593999999999999e-05 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.170673+0000", 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00093575199999999998 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.171494+0000", 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00082111100000000002 2024-04-06T03:30:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.308403+0000", 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.13690949599999999 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.308554+0000", 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015120699999999999 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:00.308947+0000", 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00039288799999999998 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:41516 7.7 7:e3748307:::benchmark_data_smithi050_68188_object41515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:26:03.572117+0000", 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 282.61498953, 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.16085392100000001, 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 41516 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572117+0000", 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572117+0000", 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572118+0000", 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9900000000000009e-07 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572173+0000", 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5158000000000003e-05 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572183+0000", 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0151e-05 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572188+0000", 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2530000000000004e-06 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572249+0000", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0485000000000001e-05 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572402+0000", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015291799999999999 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572566+0000", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000163721 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.575423+0000", 2024-04-06T03:30:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0028574889999999999 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.575989+0000", 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00056634899999999995 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.732837+0000", 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15684732900000001 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.732920+0000", 2024-04-06T03:30:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3066999999999996e-05 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.732971+0000", 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1001999999999997e-05 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:41517 7.d 7:b93110cc:::benchmark_data_smithi050_68188_object41516:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:26:03.572272+0000", 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 282.61483418099999, 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.16073689899999999, 2024-04-06T03:30:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 41517 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572272+0000", 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572272+0000", 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572273+0000", 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8700000000000005e-07 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572361+0000", 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7567999999999999e-05 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572369+0000", 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.789e-06 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572376+0000", 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.114e-06 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572434+0000", 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7224e-05 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.258 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572577+0000", 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014294700000000001 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,1", 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.572778+0000", 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020128299999999999 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.575421+0000", 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0026428340000000002 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.259 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.575856+0000", 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00043480299999999999 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.732836+0000", 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15698088199999999 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.732916+0000", 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9504000000000003e-05 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.260 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:03.733009+0000", 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3263999999999993e-05 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:48018 7.7 7:e9ac5bf7:::benchmark_data_smithi050_68188_object48017:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:26:08.706536+0000", 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 277.48057066400003, 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.33283994300000003, 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.261 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 48018 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706536+0000", 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.262 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706536+0000", 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706537+0000", 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6360000000000001e-06 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706582+0000", 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4860000000000001e-05 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706597+0000", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4440000000000001e-05 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706605+0000", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2600000000000005e-06 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.706671+0000", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6421999999999995e-05 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.710213+0000", 2024-04-06T03:30:46.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0035411290000000001 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.710378+0000", 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016500099999999999 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.713460+0000", 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0030822179999999999 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.034611+0000", 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.32115074700000001 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.039257+0000", 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0046466199999999997 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.039322+0000", 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4518999999999997e-05 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.039376+0000", 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4091000000000001e-05 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:48021 7.7 7:e866837a:::benchmark_data_smithi050_68188_object48020:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:26:08.712083+0000", 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 277.47502377900003, 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.33895791600000003, 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.267 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 48021 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712083+0000", 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712083+0000", 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712085+0000", 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1150000000000001e-06 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712126+0000", 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0939000000000002e-05 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712136+0000", 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0767e-05 2024-04-06T03:30:46.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712145+0000", 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5590000000000006e-06 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712195+0000", 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0395000000000001e-05 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712283+0000", 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7404999999999994e-05 2024-04-06T03:30:46.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.712388+0000", 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000104942 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.713575+0000", 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001186753 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.037340+0000", 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.323765091 2024-04-06T03:30:46.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.050538+0000", 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.01319857 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.050874+0000", 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033594000000000001 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.051041+0000", 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016644000000000001 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.272 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14745.0:48028 7.d 7:b25acc62:::benchmark_data_smithi050_68188_object48027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:26:08.715247+0000", 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 277.47185956599998, 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.324264208, 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14745", 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:30:46.273 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 48028 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715247+0000", 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715247+0000", 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715248+0000", 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.02e-07 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715315+0000", 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6981000000000001e-05 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715326+0000", 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1316e-05 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715332+0000", 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2430000000000002e-06 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715377+0000", 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4462999999999998e-05 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715462+0000", 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5589999999999999e-05 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,1", 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.715554+0000", 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1367999999999997e-05 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.716847+0000", 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001292978 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:08.717528+0000", 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00068087599999999996 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.039190+0000", 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.321661947 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.039397+0000", 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020776599999999999 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:26:09.039511+0000", 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011377799999999999 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.278 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17650 8.5 e188/182 8:a5b71103:::benchmark_data_smithi050_86143_object17649:head v 188'1100, mlcod=188'1098)", 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.618177+0000", 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 37.568929378, 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15274811899999999, 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618177+0000", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618177+0000", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618178+0000", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2800000000000005e-07 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618205+0000", 2024-04-06T03:30:46.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6792999999999999e-05 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618215+0000", 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0599e-05 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618221+0000", 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0090000000000004e-06 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618268+0000", 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6851999999999998e-05 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618289+0000", 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0798e-05 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619210+0000", 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000920838 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.282 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770925+0000", 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.151715302 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17641 8.9 e188/182 8:9cd1a4c4:::benchmark_data_smithi050_86143_object17640:head v 188'1118, mlcod=188'1115)", 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 37.567155915999997, 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15268031300000001, 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.283 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3499999999999995e-07 2024-04-06T03:30:46.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.620026+0000", 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5055999999999994e-05 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770855+0000", 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.150828727 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770876+0000", 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1024999999999999e-05 2024-04-06T03:30:46.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770972+0000", 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6123000000000001e-05 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771008+0000", 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5932000000000001e-05 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.772580+0000", 2024-04-06T03:30:46.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001571467 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.772631+0000", 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1248000000000001e-05 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17654 8.5 e188/182 8:ae505e77:::benchmark_data_smithi050_86143_object17653:head v 188'1101, mlcod=188'1098)", 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.622691+0000", 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 37.564415001, 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15084618, 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.287 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622691+0000", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622691+0000", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622692+0000", 2024-04-06T03:30:46.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0049999999999999e-06 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622721+0000", 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8861999999999999e-05 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770859+0000", 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.14813772 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770875+0000", 2024-04-06T03:30:46.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6096999999999999e-05 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770978+0000", 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000103112 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771008+0000", 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9629999999999999e-05 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773472+0000", 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024642190000000001 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773538+0000", 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5535000000000002e-05 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17669 8.3 e188/182 8:c5e49b00:::benchmark_data_smithi050_86143_object17668:head v 188'1081, mlcod=188'1079)", 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.634617+0000", 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 37.552488965999999, 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.138758459, 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:46.291 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634617+0000", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634617+0000", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634619+0000", 2024-04-06T03:30:46.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.883e-06 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634687+0000", 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7918999999999998e-05 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770893+0000", 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.136205936 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770905+0000", 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.223e-05 2024-04-06T03:30:46.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771008+0000", 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000102527 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771030+0000", 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2368e-05 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773273+0000", 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0022429630000000002 2024-04-06T03:30:46.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773376+0000", 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000102633 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:46.295 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:30:46.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:46.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:46.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:46.456 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:46.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 47 16 67055 67039 89.127 60.875 0.014615 0.0112147 2024-04-06T03:30:46.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:46.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:46.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:30:46.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:46.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:46.831 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:46.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:46.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 2 1 2024-04-06T03:30:46.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:46.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:46.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:47.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:47.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:47.216 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:47.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-06T03:30:47.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:47.354 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:47.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:47.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:47.498 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 48 16 67887 67871 88.3535 52 0.00900019 0.0113132 2024-04-06T03:30:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:47.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:47.628 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:47.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:47 smithi050 bash[21467]: cluster 2024-04-06T03:30:46.160680+0000 mgr.y (mgr.24368) 363 : cluster 0 pgmap v369: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-06T03:30:47.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:47 smithi050 bash[29221]: cluster 2024-04-06T03:30:46.160680+0000 mgr.y (mgr.24368) 363 : cluster 0 pgmap v369: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-06T03:30:47.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:47.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:47.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:47 smithi153 bash[23852]: cluster 2024-04-06T03:30:46.160680+0000 mgr.y (mgr.24368) 363 : cluster 0 pgmap v369: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-06T03:30:47.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:47.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:47.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:47.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:47.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:48.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:48.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:48.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:48.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:48.286 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:48.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:48.419 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:48.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 49 16 69118 69102 88.1204 76.9375 0.00933102 0.0113433 2024-04-06T03:30:48.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:48.531 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:48 smithi153 bash[23852]: audit 2024-04-06T03:30:48.125891+0000 mon.a (mon.0) 1200 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-04-06T03:30:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:48.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:48.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:48.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:48.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:48 smithi050 bash[21467]: audit 2024-04-06T03:30:48.125891+0000 mon.a (mon.0) 1200 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-04-06T03:30:48.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:48 smithi050 bash[29221]: audit 2024-04-06T03:30:48.125891+0000 mon.a (mon.0) 1200 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-04-06T03:30:48.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:48.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:48.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:30:48.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:48.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:49.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:49.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:49.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:49.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:49.355 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:49.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:49.484 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:49.485 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:30:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:30:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:49.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 50 16 70238 70222 87.7575 70 0.0222772 0.0113905 2024-04-06T03:30:49.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:49.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[21467]: cluster 2024-04-06T03:30:48.161029+0000 mgr.y (mgr.24368) 364 : cluster 0 pgmap v370: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 623 op/s 2024-04-06T03:30:49.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[21467]: audit 2024-04-06T03:30:48.522308+0000 mon.a (mon.0) 1201 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-04-06T03:30:49.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[21467]: cluster 2024-04-06T03:30:48.533285+0000 mon.a (mon.0) 1202 : cluster 0 osdmap e196: 8 total, 8 up, 8 in 2024-04-06T03:30:49.598 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[21467]: audit 2024-04-06T03:30:48.989490+0000 mon.a (mon.0) 1203 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:49.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[29221]: cluster 2024-04-06T03:30:48.161029+0000 mgr.y (mgr.24368) 364 : cluster 0 pgmap v370: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 623 op/s 2024-04-06T03:30:49.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[29221]: audit 2024-04-06T03:30:48.522308+0000 mon.a (mon.0) 1201 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-04-06T03:30:49.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[29221]: cluster 2024-04-06T03:30:48.533285+0000 mon.a (mon.0) 1202 : cluster 0 osdmap e196: 8 total, 8 up, 8 in 2024-04-06T03:30:49.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:49 smithi050 bash[29221]: audit 2024-04-06T03:30:48.989490+0000 mon.a (mon.0) 1203 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:49.624 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:49.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:49.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:30:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:30:49.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:49 smithi153 bash[23852]: cluster 2024-04-06T03:30:48.161029+0000 mgr.y (mgr.24368) 364 : cluster 0 pgmap v370: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 623 op/s 2024-04-06T03:30:49.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:49 smithi153 bash[23852]: audit 2024-04-06T03:30:48.522308+0000 mon.a (mon.0) 1201 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-04-06T03:30:49.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:49 smithi153 bash[23852]: cluster 2024-04-06T03:30:48.533285+0000 mon.a (mon.0) 1202 : cluster 0 osdmap e196: 8 total, 8 up, 8 in 2024-04-06T03:30:49.738 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:49 smithi153 bash[23852]: audit 2024-04-06T03:30:48.989490+0000 mon.a (mon.0) 1203 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:30:49.754 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:49.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:49.886 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:49.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:49.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:50.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:50.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:50.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:50.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:50.269 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:50.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:50.400 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:50.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 51 16 71403 71387 87.4644 72.8125 0.00999743 0.0114276 2024-04-06T03:30:50.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:50.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:50.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:50.661 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:50.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:50.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:50.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:50 smithi153 bash[23852]: audit 2024-04-06T03:30:49.446982+0000 mgr.y (mgr.24368) 365 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:50.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:50 smithi153 bash[23852]: cluster 2024-04-06T03:30:49.539718+0000 mon.a (mon.0) 1204 : cluster 0 osdmap e197: 8 total, 8 up, 8 in 2024-04-06T03:30:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:50.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:50.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:50 smithi050 bash[21467]: audit 2024-04-06T03:30:49.446982+0000 mgr.y (mgr.24368) 365 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:50.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:50 smithi050 bash[21467]: cluster 2024-04-06T03:30:49.539718+0000 mon.a (mon.0) 1204 : cluster 0 osdmap e197: 8 total, 8 up, 8 in 2024-04-06T03:30:50.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:50 smithi050 bash[29221]: audit 2024-04-06T03:30:49.446982+0000 mgr.y (mgr.24368) 365 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:30:50.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:50 smithi050 bash[29221]: cluster 2024-04-06T03:30:49.539718+0000 mon.a (mon.0) 1204 : cluster 0 osdmap e197: 8 total, 8 up, 8 in 2024-04-06T03:30:50.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:50.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:51.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:51.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:51.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:51.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:51.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:51.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:51.396 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:51.398 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:51.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:51.480 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:51.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 52 16 72531 72515 87.138 70.5 0.00793684 0.0114706 2024-04-06T03:30:51.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:51.612 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:51.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:51.727 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:51 smithi050 bash[21467]: cluster 2024-04-06T03:30:50.161715+0000 mgr.y (mgr.24368) 366 : cluster 0 pgmap v373: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-06T03:30:51.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:51 smithi050 bash[29221]: cluster 2024-04-06T03:30:50.161715+0000 mgr.y (mgr.24368) 366 : cluster 0 pgmap v373: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-06T03:30:51.755 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:51.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:51 smithi153 bash[23852]: cluster 2024-04-06T03:30:50.161715+0000 mgr.y (mgr.24368) 366 : cluster 0 pgmap v373: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-06T03:30:51.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:51.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:51.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:51.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:52.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:52.075 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:30:52.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:52.153 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:52.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:52.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:52.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:52.402 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:52.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 53 16 73556 73540 86.7025 64.0625 0.00904254 0.0115289 2024-04-06T03:30:52.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:52.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:52.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:52.678 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:52.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[21467]: cluster 2024-04-06T03:30:50.980879+0000 osd.6 (osd.6) 85 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:52.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[21467]: cluster 2024-04-06T03:30:50.980948+0000 osd.6 (osd.6) 86 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[21467]: cluster 2024-04-06T03:30:51.554110+0000 mon.a (mon.0) 1205 : cluster 0 osdmap e198: 8 total, 8 up, 8 in 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[21467]: cluster 2024-04-06T03:30:51.664470+0000 osd.2 (osd.2) 191 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[21467]: cluster 2024-04-06T03:30:51.664548+0000 osd.2 (osd.2) 192 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[29221]: cluster 2024-04-06T03:30:50.980879+0000 osd.6 (osd.6) 85 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[29221]: cluster 2024-04-06T03:30:50.980948+0000 osd.6 (osd.6) 86 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[29221]: cluster 2024-04-06T03:30:51.554110+0000 mon.a (mon.0) 1205 : cluster 0 osdmap e198: 8 total, 8 up, 8 in 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[29221]: cluster 2024-04-06T03:30:51.664470+0000 osd.2 (osd.2) 191 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:52.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:52 smithi050 bash[29221]: cluster 2024-04-06T03:30:51.664548+0000 osd.2 (osd.2) 192 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:52.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:52.802 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:52.804 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:52 smithi153 bash[23852]: cluster 2024-04-06T03:30:50.980879+0000 osd.6 (osd.6) 85 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:52.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:52 smithi153 bash[23852]: cluster 2024-04-06T03:30:50.980948+0000 osd.6 (osd.6) 86 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:52.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:52 smithi153 bash[23852]: cluster 2024-04-06T03:30:51.554110+0000 mon.a (mon.0) 1205 : cluster 0 osdmap e198: 8 total, 8 up, 8 in 2024-04-06T03:30:52.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:52 smithi153 bash[23852]: cluster 2024-04-06T03:30:51.664470+0000 osd.2 (osd.2) 191 : cluster 0 purged_snaps scrub starts 2024-04-06T03:30:52.805 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:52 smithi153 bash[23852]: cluster 2024-04-06T03:30:51.664548+0000 osd.2 (osd.2) 192 : cluster 0 purged_snaps scrub ok 2024-04-06T03:30:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:52.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:52.936 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:52.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:53.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:53.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:53.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:53.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:53.320 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:53.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:53.425 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:30:53.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:53.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 54 16 74197 74181 85.8389 40.0625 0.0364387 0.0116142 2024-04-06T03:30:53.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:53.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:53.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:53 smithi050 bash[21467]: cluster 2024-04-06T03:30:52.162189+0000 mgr.y (mgr.24368) 367 : cluster 0 pgmap v375: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 901 op/s 2024-04-06T03:30:53.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:53 smithi050 bash[21467]: cluster 2024-04-06T03:30:52.563971+0000 mon.a (mon.0) 1206 : cluster 0 osdmap e199: 8 total, 8 up, 8 in 2024-04-06T03:30:53.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:53 smithi050 bash[29221]: cluster 2024-04-06T03:30:52.162189+0000 mgr.y (mgr.24368) 367 : cluster 0 pgmap v375: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 901 op/s 2024-04-06T03:30:53.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:53 smithi050 bash[29221]: cluster 2024-04-06T03:30:52.563971+0000 mon.a (mon.0) 1206 : cluster 0 osdmap e199: 8 total, 8 up, 8 in 2024-04-06T03:30:53.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:53.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:53 smithi153 bash[23852]: cluster 2024-04-06T03:30:52.162189+0000 mgr.y (mgr.24368) 367 : cluster 0 pgmap v375: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 56 MiB/s wr, 901 op/s 2024-04-06T03:30:53.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:53 smithi153 bash[23852]: cluster 2024-04-06T03:30:52.563971+0000 mon.a (mon.0) 1206 : cluster 0 osdmap e199: 8 total, 8 up, 8 in 2024-04-06T03:30:53.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:53.853 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:53.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:53.983 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:53.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:54.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:54.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:54.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:54.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:54.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:54.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:54.499 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 55 16 75059 75043 85.2577 53.875 0.00309929 0.0116694 2024-04-06T03:30:54.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:54.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:54.621 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:54.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:54.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:54.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:54.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:54.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:55.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:55.143 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:55.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:55.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:55.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:55.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:55.500 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 56 16 76100 76084 84.897 65.0625 0.010433 0.0117733 2024-04-06T03:30:55.519 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:55.544 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:55.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:55.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:55.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:55.792 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:55 smithi153 bash[23852]: cluster 2024-04-06T03:30:54.164010+0000 mgr.y (mgr.24368) 368 : cluster 0 pgmap v377: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 907 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-06T03:30:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:55.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:55.939 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:55 smithi050 bash[29221]: cluster 2024-04-06T03:30:54.164010+0000 mgr.y (mgr.24368) 368 : cluster 0 pgmap v377: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 907 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-06T03:30:55.939 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:55 smithi050 bash[21467]: cluster 2024-04-06T03:30:54.164010+0000 mgr.y (mgr.24368) 368 : cluster 0 pgmap v377: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 907 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-04-06T03:30:55.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:55.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:56.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:56.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:30:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:30:56.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:56.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:56.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:56.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:56.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:56.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:56.500 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 57 15 77206 77191 84.6213 69.1875 0.0104536 0.0118113 2024-04-06T03:30:56.511 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:30:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:30:56.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:56.644 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:56.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:56.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:56.838 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:30:56.838 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14928.0:77550 8.d 8:b59c9f45:::benchmark_data_smithi050_86143_object77549:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:56.829531+0000", 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0084065830000000005, 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0084539100000000002, 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14928", 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 77550 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.839 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829531+0000", 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829531+0000", 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829534+0000", 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0660000000000002e-06 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.840 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829580+0000", 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6660999999999999e-05 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829596+0000", 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6212e-05 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829606+0000", 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0098000000000001e-05 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.841 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829664+0000", 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7998000000000003e-05 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829786+0000", 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121169 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829934+0000", 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014811800000000001 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.842 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:77549 8.5 e199/182 8:a43fff92:::benchmark_data_smithi050_86143_object77548:head v 199'4815, mlcod=199'4813)", 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:56.829954+0000", 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0079842739999999995, 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0080940519999999992, 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.843 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829954+0000", 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829954+0000", 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829955+0000", 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7999999999999993e-07 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.844 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829978+0000", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3357999999999999e-05 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829989+0000", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0681e-05 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.829996+0000", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.238e-06 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:56.845 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833036+0000", 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003039609 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833054+0000", 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8113000000000001e-05 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.14928.0:77555 8.d 8:b52e6dd2:::benchmark_data_smithi050_86143_object77554:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:56.833458+0000", 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0044796819999999996, 2024-04-06T03:30:56.846 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0046310919999999998, 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.14928", 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 77555 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833458+0000", 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:56.847 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833458+0000", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833460+0000", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.378e-06 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833528+0000", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8751000000000006e-05 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833540+0000", 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1885999999999999e-05 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.833550+0000", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1730000000000001e-06 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835023+0000", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0014733299999999999 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835156+0000", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000133607 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:30:56.849 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835310+0000", 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000153289 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop_reply(client.14928.0:77550 8.d e199/182)", 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:56.835155+0000", 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0027831370000000002, 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00298569, 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:56.850 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835155+0000", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835155+0000", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835156+0000", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.08e-06 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:56.851 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835161+0000", 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0649999999999998e-06 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835172+0000", 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.076e-05 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.835179+0000", 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5850000000000002e-06 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.852 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:77554 8.5 e199/182)", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:56.836229+0000", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0017092679999999999, 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001941234, 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.836229+0000", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.836229+0000", 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:30:56.853 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.836230+0000", 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3600000000000003e-07 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.836253+0000", 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3198999999999999e-05 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.836267+0000", 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4334e-05 2024-04-06T03:30:56.854 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:56.836274+0000", 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9600000000000003e-06 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 5 2024-04-06T03:30:56.855 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:30:56.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:56.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:56.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:57.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:57.046 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:57.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:57.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:57.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:57.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.438 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:57.500 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 58 16 78245 78229 84.2808 64.875 0.00595051 0.0118593 2024-04-06T03:30:57.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:57.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:57.696 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:57.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:57 smithi050 bash[21467]: cluster 2024-04-06T03:30:56.164720+0000 mgr.y (mgr.24368) 369 : cluster 0 pgmap v378: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-06T03:30:57.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:57 smithi050 bash[29221]: cluster 2024-04-06T03:30:56.164720+0000 mgr.y (mgr.24368) 369 : cluster 0 pgmap v378: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-06T03:30:57.765 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-06T03:30:57.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:57.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:57.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:57 smithi153 bash[23852]: cluster 2024-04-06T03:30:56.164720+0000 mgr.y (mgr.24368) 369 : cluster 0 pgmap v378: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-06T03:30:57.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:57.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:58.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:58.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:58.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:58.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:30:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:30:58.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:58.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:58.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:30:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:30:58.500 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 59 16 79138 79122 83.7978 55.8125 0.0246421 0.0119278 2024-04-06T03:30:58.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:58.625 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:58.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:58.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:58.852 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:58.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:58.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:58.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:59.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:30:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:30:59.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:59.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:30:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:30:59.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:59.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.397 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:59.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:59.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:30:59.507416+0000 min lat: 0.00122838 max lat: 0.562935 avg lat: 0.012004 2024-04-06T03:30:59.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:30:59.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 60 16 79732 79716 83.0199 37.125 0.00332245 0.012004 2024-04-06T03:30:59.523 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:30:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:30:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:30:59.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:59.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:30:59.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:59.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:30:59 smithi050 bash[21467]: cluster 2024-04-06T03:30:58.169138+0000 mgr.y (mgr.24368) 370 : cluster 0 pgmap v379: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 611 op/s 2024-04-06T03:30:59.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:30:59 smithi050 bash[29221]: cluster 2024-04-06T03:30:58.169138+0000 mgr.y (mgr.24368) 370 : cluster 0 pgmap v379: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 611 op/s 2024-04-06T03:30:59.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:30:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:30:59.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:30:59 smithi153 bash[23852]: cluster 2024-04-06T03:30:58.169138+0000 mgr.y (mgr.24368) 370 : cluster 0 pgmap v379: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 611 op/s 2024-04-06T03:30:59.795 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:30:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:30:59.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:30:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:30:59.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:00.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:00.045 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:00.150 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:00.175 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:00.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:00.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:00.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:00.500 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 61 15 80701 80686 82.6527 60.625 0.0111673 0.0120942 2024-04-06T03:31:00.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:00.570 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:00.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:00.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:00.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:00.830 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:00.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:00 smithi153 bash[23852]: audit 2024-04-06T03:30:59.457279+0000 mgr.y (mgr.24368) 371 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:00.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:00.954 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:00.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:00.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:00 smithi050 bash[21467]: audit 2024-04-06T03:30:59.457279+0000 mgr.y (mgr.24368) 371 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:00.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:00 smithi050 bash[29221]: audit 2024-04-06T03:30:59.457279+0000 mgr.y (mgr.24368) 371 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:01.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:01.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:01.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:01.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:01.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:01.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:01.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:01.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 62 16 81707 81691 82.3321 62.8125 0.019582 0.0121408 2024-04-06T03:31:01.509 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:01.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:01.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:01.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:01 smithi050 bash[21467]: cluster 2024-04-06T03:31:00.170217+0000 mgr.y (mgr.24368) 372 : cluster 0 pgmap v380: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 10 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-06T03:31:01.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:01 smithi050 bash[29221]: cluster 2024-04-06T03:31:00.170217+0000 mgr.y (mgr.24368) 372 : cluster 0 pgmap v380: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 10 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-06T03:31:01.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:01.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:01.747 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:01 smithi153 bash[23852]: cluster 2024-04-06T03:31:00.170217+0000 mgr.y (mgr.24368) 372 : cluster 0 pgmap v380: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 10 KiB/s rd, 69 MiB/s wr, 1.12k op/s 2024-04-06T03:31:01.760 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:01.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:01.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:02.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:02.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:02.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:02.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:02.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:02.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:02.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:02.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 63 16 82615 82599 81.926 56.75 0.0136996 0.0122007 2024-04-06T03:31:02.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:02.561 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:31:02.572 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:02.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:02.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:02.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:02.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:02.840 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:02.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:02.968 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:02.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:03.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:03.100 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:03.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:03.239 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:03.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:03.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:03.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:03.490 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:03.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 64 16 83629 83613 81.636 63.375 0.00699255 0.0122355 2024-04-06T03:31:03.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:03.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:03.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:03.757 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:03.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:03 smithi050 bash[21467]: cluster 2024-04-06T03:31:02.170662+0000 mgr.y (mgr.24368) 373 : cluster 0 pgmap v381: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.8 KiB/s rd, 60 MiB/s wr, 968 op/s 2024-04-06T03:31:03.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:03 smithi050 bash[29221]: cluster 2024-04-06T03:31:02.170662+0000 mgr.y (mgr.24368) 373 : cluster 0 pgmap v381: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.8 KiB/s rd, 60 MiB/s wr, 968 op/s 2024-04-06T03:31:03.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:03.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:03.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:04.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:04.034 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:04.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:04.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:03 smithi153 bash[23852]: cluster 2024-04-06T03:31:02.170662+0000 mgr.y (mgr.24368) 373 : cluster 0 pgmap v381: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 8.8 KiB/s rd, 60 MiB/s wr, 968 op/s 2024-04-06T03:31:04.139 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:04.167 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:04.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:04.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:04.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:04.420 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:04.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 65 16 84301 84285 81.0263 42 0.00751975 0.0123362 2024-04-06T03:31:04.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:04.546 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:04.598 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:31:04.618 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:31:04.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:04.659 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:31:04.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:04 smithi153 bash[23852]: audit 2024-04-06T03:31:04.014009+0000 mon.a (mon.0) 1207 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:04.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:04 smithi153 bash[23852]: audit 2024-04-06T03:31:04.488508+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.50:0/3359244895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:31:04.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:04 smithi153 bash[23852]: audit 2024-04-06T03:31:04.489219+0000 mon.a (mon.0) 1208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:31:04.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:04 smithi153 bash[23852]: audit 2024-04-06T03:31:04.561954+0000 mon.c (mon.2) 62 : audit 0 from='client.? 172.21.15.50:0/1733981820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:31:04.680 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:04.702 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:31:04.739 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:31:04.775 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:31:04.783 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:04.809 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:04.813 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:31:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:04.844 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:31:04.883 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:31:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:31:04.910 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:31:04.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[21467]: audit 2024-04-06T03:31:04.014009+0000 mon.a (mon.0) 1207 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[21467]: audit 2024-04-06T03:31:04.488508+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.50:0/3359244895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[21467]: audit 2024-04-06T03:31:04.489219+0000 mon.a (mon.0) 1208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[21467]: audit 2024-04-06T03:31:04.561954+0000 mon.c (mon.2) 62 : audit 0 from='client.? 172.21.15.50:0/1733981820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[29221]: audit 2024-04-06T03:31:04.014009+0000 mon.a (mon.0) 1207 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[29221]: audit 2024-04-06T03:31:04.488508+0000 mon.c (mon.2) 61 : audit 1 from='client.? 172.21.15.50:0/3359244895' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[29221]: audit 2024-04-06T03:31:04.489219+0000 mon.a (mon.0) 1208 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:31:04.917 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:04 smithi050 bash[29221]: audit 2024-04-06T03:31:04.561954+0000 mon.c (mon.2) 62 : audit 0 from='client.? 172.21.15.50:0/1733981820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:31:04.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:04.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:05.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:05.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:05.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:05.206 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:05.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:05.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:05.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:05.472 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:05.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 66 16 85261 85245 80.7077 60 0.0143604 0.0123858 2024-04-06T03:31:05.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:05.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-06T03:31:05.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:05.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: cluster 2024-04-06T03:31:04.171749+0000 mgr.y (mgr.24368) 374 : cluster 0 pgmap v382: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 23 KiB/s rd, 58 MiB/s wr, 959 op/s 2024-04-06T03:31:05.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: cluster 2024-04-06T03:31:04.595659+0000 mon.a (mon.0) 1209 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:31:05.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: audit 2024-04-06T03:31:04.605339+0000 mon.a (mon.0) 1210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:31:05.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: cluster 2024-04-06T03:31:04.622760+0000 mon.a (mon.0) 1211 : cluster 0 osdmap e200: 8 total, 8 up, 8 in 2024-04-06T03:31:05.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: audit 2024-04-06T03:31:04.959267+0000 mon.b (mon.1) 37 : audit 1 from='client.? 172.21.15.50:0/2876142917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:31:05.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: audit 2024-04-06T03:31:04.963620+0000 mon.a (mon.0) 1212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:31:05.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.018580+0000 osd.2 (osd.2) 193 : cluster 0 2.10 scrub starts 2024-04-06T03:31:05.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:05 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.022303+0000 osd.2 (osd.2) 194 : cluster 0 2.10 scrub ok 2024-04-06T03:31:05.736 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:05.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:05.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:05.863 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:05.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:05.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:05.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: cluster 2024-04-06T03:31:04.171749+0000 mgr.y (mgr.24368) 374 : cluster 0 pgmap v382: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 23 KiB/s rd, 58 MiB/s wr, 959 op/s 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: cluster 2024-04-06T03:31:04.595659+0000 mon.a (mon.0) 1209 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: audit 2024-04-06T03:31:04.605339+0000 mon.a (mon.0) 1210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: cluster 2024-04-06T03:31:04.622760+0000 mon.a (mon.0) 1211 : cluster 0 osdmap e200: 8 total, 8 up, 8 in 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: audit 2024-04-06T03:31:04.959267+0000 mon.b (mon.1) 37 : audit 1 from='client.? 172.21.15.50:0/2876142917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: audit 2024-04-06T03:31:04.963620+0000 mon.a (mon.0) 1212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.018580+0000 osd.2 (osd.2) 193 : cluster 0 2.10 scrub starts 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.022303+0000 osd.2 (osd.2) 194 : cluster 0 2.10 scrub ok 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: cluster 2024-04-06T03:31:04.171749+0000 mgr.y (mgr.24368) 374 : cluster 0 pgmap v382: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 23 KiB/s rd, 58 MiB/s wr, 959 op/s 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: cluster 2024-04-06T03:31:04.595659+0000 mon.a (mon.0) 1209 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: audit 2024-04-06T03:31:04.605339+0000 mon.a (mon.0) 1210 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:31:05.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: cluster 2024-04-06T03:31:04.622760+0000 mon.a (mon.0) 1211 : cluster 0 osdmap e200: 8 total, 8 up, 8 in 2024-04-06T03:31:05.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: audit 2024-04-06T03:31:04.959267+0000 mon.b (mon.1) 37 : audit 1 from='client.? 172.21.15.50:0/2876142917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:31:05.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: audit 2024-04-06T03:31:04.963620+0000 mon.a (mon.0) 1212 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:31:05.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.018580+0000 osd.2 (osd.2) 193 : cluster 0 2.10 scrub starts 2024-04-06T03:31:05.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:05 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.022303+0000 osd.2 (osd.2) 194 : cluster 0 2.10 scrub ok 2024-04-06T03:31:05.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:06.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:06.131 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:06.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:06.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:06.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:06.393 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:06.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:06.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 67 16 86097 86081 80.2829 52.25 0.0031063 0.0124165 2024-04-06T03:31:06.520 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:06.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:06.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:06.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:04.991563+0000 osd.3 (osd.3) 235 : cluster 0 3.7 scrub starts 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.001112+0000 osd.3 (osd.3) 236 : cluster 0 3.7 scrub ok 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.113109+0000 osd.4 (osd.4) 233 : cluster 0 5.c scrub starts 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.115248+0000 osd.4 (osd.4) 234 : cluster 0 5.c scrub ok 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: audit 2024-04-06T03:31:05.609193+0000 mon.a (mon.0) 1213 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.614842+0000 mon.a (mon.0) 1214 : cluster 0 osdmap e201: 8 total, 8 up, 8 in 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.692599+0000 osd.1 (osd.1) 197 : cluster 0 4.2 scrub starts 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.703363+0000 osd.1 (osd.1) 198 : cluster 0 4.2 scrub ok 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.924010+0000 osd.7 (osd.7) 231 : cluster 0 2.5 scrub starts 2024-04-06T03:31:06.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.927249+0000 osd.7 (osd.7) 232 : cluster 0 2.5 scrub ok 2024-04-06T03:31:06.767 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.991610+0000 osd.2 (osd.2) 195 : cluster 0 2.c scrub starts 2024-04-06T03:31:06.767 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:06 smithi153 bash[23852]: cluster 2024-04-06T03:31:05.993241+0000 osd.2 (osd.2) 196 : cluster 0 2.c scrub ok 2024-04-06T03:31:06.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:06.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:06.929 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:06.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:04.991563+0000 osd.3 (osd.3) 235 : cluster 0 3.7 scrub starts 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.001112+0000 osd.3 (osd.3) 236 : cluster 0 3.7 scrub ok 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.113109+0000 osd.4 (osd.4) 233 : cluster 0 5.c scrub starts 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.115248+0000 osd.4 (osd.4) 234 : cluster 0 5.c scrub ok 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: audit 2024-04-06T03:31:05.609193+0000 mon.a (mon.0) 1213 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.614842+0000 mon.a (mon.0) 1214 : cluster 0 osdmap e201: 8 total, 8 up, 8 in 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.692599+0000 osd.1 (osd.1) 197 : cluster 0 4.2 scrub starts 2024-04-06T03:31:06.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.703363+0000 osd.1 (osd.1) 198 : cluster 0 4.2 scrub ok 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.924010+0000 osd.7 (osd.7) 231 : cluster 0 2.5 scrub starts 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.927249+0000 osd.7 (osd.7) 232 : cluster 0 2.5 scrub ok 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.991610+0000 osd.2 (osd.2) 195 : cluster 0 2.c scrub starts 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[21467]: cluster 2024-04-06T03:31:05.993241+0000 osd.2 (osd.2) 196 : cluster 0 2.c scrub ok 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:04.991563+0000 osd.3 (osd.3) 235 : cluster 0 3.7 scrub starts 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.001112+0000 osd.3 (osd.3) 236 : cluster 0 3.7 scrub ok 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.113109+0000 osd.4 (osd.4) 233 : cluster 0 5.c scrub starts 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.115248+0000 osd.4 (osd.4) 234 : cluster 0 5.c scrub ok 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: audit 2024-04-06T03:31:05.609193+0000 mon.a (mon.0) 1213 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.614842+0000 mon.a (mon.0) 1214 : cluster 0 osdmap e201: 8 total, 8 up, 8 in 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.692599+0000 osd.1 (osd.1) 197 : cluster 0 4.2 scrub starts 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.703363+0000 osd.1 (osd.1) 198 : cluster 0 4.2 scrub ok 2024-04-06T03:31:06.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.924010+0000 osd.7 (osd.7) 231 : cluster 0 2.5 scrub starts 2024-04-06T03:31:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.927249+0000 osd.7 (osd.7) 232 : cluster 0 2.5 scrub ok 2024-04-06T03:31:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.991610+0000 osd.2 (osd.2) 195 : cluster 0 2.c scrub starts 2024-04-06T03:31:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:06 smithi050 bash[29221]: cluster 2024-04-06T03:31:05.993241+0000 osd.2 (osd.2) 196 : cluster 0 2.c scrub ok 2024-04-06T03:31:07.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:07.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:07.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:07.080 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:31:07.080 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:31:07.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:07.187 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:07.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:07.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:07.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:07.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 68 16 86983 86967 79.9166 55.375 0.00952513 0.0125079 2024-04-06T03:31:07.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:07.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:07.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:07.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:07 smithi153 bash[23852]: cluster 2024-04-06T03:31:06.150086+0000 osd.4 (osd.4) 235 : cluster 0 3.14 scrub starts 2024-04-06T03:31:07.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:07 smithi153 bash[23852]: cluster 2024-04-06T03:31:06.151785+0000 osd.4 (osd.4) 236 : cluster 0 3.14 scrub ok 2024-04-06T03:31:07.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:07 smithi153 bash[23852]: cluster 2024-04-06T03:31:06.172249+0000 mgr.y (mgr.24368) 375 : cluster 0 pgmap v385: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 55 KiB/s rd, 72 MiB/s wr, 1.24k op/s 2024-04-06T03:31:07.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:07 smithi153 bash[23852]: cluster 2024-04-06T03:31:06.942734+0000 osd.2 (osd.2) 197 : cluster 0 2.3 scrub starts 2024-04-06T03:31:07.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:07 smithi153 bash[23852]: cluster 2024-04-06T03:31:06.950545+0000 osd.2 (osd.2) 198 : cluster 0 2.3 scrub ok 2024-04-06T03:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:07.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:07.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:07.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:07.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[21467]: cluster 2024-04-06T03:31:06.150086+0000 osd.4 (osd.4) 235 : cluster 0 3.14 scrub starts 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[21467]: cluster 2024-04-06T03:31:06.151785+0000 osd.4 (osd.4) 236 : cluster 0 3.14 scrub ok 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[21467]: cluster 2024-04-06T03:31:06.172249+0000 mgr.y (mgr.24368) 375 : cluster 0 pgmap v385: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 55 KiB/s rd, 72 MiB/s wr, 1.24k op/s 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[21467]: cluster 2024-04-06T03:31:06.942734+0000 osd.2 (osd.2) 197 : cluster 0 2.3 scrub starts 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[21467]: cluster 2024-04-06T03:31:06.950545+0000 osd.2 (osd.2) 198 : cluster 0 2.3 scrub ok 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[29221]: cluster 2024-04-06T03:31:06.150086+0000 osd.4 (osd.4) 235 : cluster 0 3.14 scrub starts 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[29221]: cluster 2024-04-06T03:31:06.151785+0000 osd.4 (osd.4) 236 : cluster 0 3.14 scrub ok 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[29221]: cluster 2024-04-06T03:31:06.172249+0000 mgr.y (mgr.24368) 375 : cluster 0 pgmap v385: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 55 KiB/s rd, 72 MiB/s wr, 1.24k op/s 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[29221]: cluster 2024-04-06T03:31:06.942734+0000 osd.2 (osd.2) 197 : cluster 0 2.3 scrub starts 2024-04-06T03:31:07.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:07 smithi050 bash[29221]: cluster 2024-04-06T03:31:06.950545+0000 osd.2 (osd.2) 198 : cluster 0 2.3 scrub ok 2024-04-06T03:31:08.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:08.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-06T03:31:08.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:08.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:08.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:08.276 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:08.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:08.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:08.501 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 69 15 87986 87971 79.6677 62.75 0.0291112 0.0125459 2024-04-06T03:31:08.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:08.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 3 1 2024-04-06T03:31:08.541 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:08.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:08.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:08.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.149352+0000 osd.3 (osd.3) 237 : cluster 0 6.b scrub starts 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.223313+0000 osd.3 (osd.3) 238 : cluster 0 6.b scrub ok 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.704569+0000 osd.1 (osd.1) 199 : cluster 0 6.1 scrub starts 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.728632+0000 osd.1 (osd.1) 200 : cluster 0 6.1 scrub ok 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.905579+0000 osd.2 (osd.2) 199 : cluster 0 3.0 scrub starts 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.910039+0000 osd.2 (osd.2) 200 : cluster 0 3.0 scrub ok 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.911231+0000 osd.7 (osd.7) 233 : cluster 0 3.f scrub starts 2024-04-06T03:31:08.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:08 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.913069+0000 osd.7 (osd.7) 234 : cluster 0 3.f scrub ok 2024-04-06T03:31:08.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:08.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:08.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.149352+0000 osd.3 (osd.3) 237 : cluster 0 6.b scrub starts 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.223313+0000 osd.3 (osd.3) 238 : cluster 0 6.b scrub ok 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.704569+0000 osd.1 (osd.1) 199 : cluster 0 6.1 scrub starts 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.728632+0000 osd.1 (osd.1) 200 : cluster 0 6.1 scrub ok 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.905579+0000 osd.2 (osd.2) 199 : cluster 0 3.0 scrub starts 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.910039+0000 osd.2 (osd.2) 200 : cluster 0 3.0 scrub ok 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.911231+0000 osd.7 (osd.7) 233 : cluster 0 3.f scrub starts 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.913069+0000 osd.7 (osd.7) 234 : cluster 0 3.f scrub ok 2024-04-06T03:31:08.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.149352+0000 osd.3 (osd.3) 237 : cluster 0 6.b scrub starts 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.223313+0000 osd.3 (osd.3) 238 : cluster 0 6.b scrub ok 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.704569+0000 osd.1 (osd.1) 199 : cluster 0 6.1 scrub starts 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.728632+0000 osd.1 (osd.1) 200 : cluster 0 6.1 scrub ok 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.905579+0000 osd.2 (osd.2) 199 : cluster 0 3.0 scrub starts 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.910039+0000 osd.2 (osd.2) 200 : cluster 0 3.0 scrub ok 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.911231+0000 osd.7 (osd.7) 233 : cluster 0 3.f scrub starts 2024-04-06T03:31:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:08 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.913069+0000 osd.7 (osd.7) 234 : cluster 0 3.f scrub ok 2024-04-06T03:31:09.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:09.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:09.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:09.194 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:09.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:09.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:09.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:09.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:09.502 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 70 16 89001 88985 79.4349 63.375 0.0171866 0.0125841 2024-04-06T03:31:09.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:09.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:09.635 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:31:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:31:09.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:09.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:31:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:31:09.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:09 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.964909+0000 osd.0 (osd.0) 202 : cluster 0 3.12 scrub starts 2024-04-06T03:31:09.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:09 smithi153 bash[23852]: cluster 2024-04-06T03:31:07.968669+0000 osd.0 (osd.0) 203 : cluster 0 3.12 scrub ok 2024-04-06T03:31:09.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:09 smithi153 bash[23852]: cluster 2024-04-06T03:31:08.172754+0000 mgr.y (mgr.24368) 376 : cluster 0 pgmap v386: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 44 KiB/s rd, 35 MiB/s wr, 641 op/s 2024-04-06T03:31:09.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:09.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:09.870 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:09.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:09.979 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:09.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.964909+0000 osd.0 (osd.0) 202 : cluster 0 3.12 scrub starts 2024-04-06T03:31:09.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[21467]: cluster 2024-04-06T03:31:07.968669+0000 osd.0 (osd.0) 203 : cluster 0 3.12 scrub ok 2024-04-06T03:31:09.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[21467]: cluster 2024-04-06T03:31:08.172754+0000 mgr.y (mgr.24368) 376 : cluster 0 pgmap v386: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 44 KiB/s rd, 35 MiB/s wr, 641 op/s 2024-04-06T03:31:09.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.964909+0000 osd.0 (osd.0) 202 : cluster 0 3.12 scrub starts 2024-04-06T03:31:09.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[29221]: cluster 2024-04-06T03:31:07.968669+0000 osd.0 (osd.0) 203 : cluster 0 3.12 scrub ok 2024-04-06T03:31:09.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:09 smithi050 bash[29221]: cluster 2024-04-06T03:31:08.172754+0000 mgr.y (mgr.24368) 376 : cluster 0 pgmap v386: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 44 KiB/s rd, 35 MiB/s wr, 641 op/s 2024-04-06T03:31:10.004 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:10.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:10.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:10.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:10.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:10.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:31:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:10.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:10.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:10.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:31:10.503 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 71 16 90065 90049 79.2514 66.5 0.00850908 0.0126121 2024-04-06T03:31:10.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:10.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:10.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:10.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:10.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:10.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[21467]: cluster 2024-04-06T03:31:08.980390+0000 osd.0 (osd.0) 204 : cluster 0 6.9 scrub starts 2024-04-06T03:31:10.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[21467]: cluster 2024-04-06T03:31:09.065237+0000 osd.0 (osd.0) 205 : cluster 0 6.9 scrub ok 2024-04-06T03:31:10.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[21467]: cluster 2024-04-06T03:31:09.193297+0000 osd.3 (osd.3) 239 : cluster 0 5.16 scrub starts 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[21467]: cluster 2024-04-06T03:31:09.200691+0000 osd.3 (osd.3) 240 : cluster 0 5.16 scrub ok 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[21467]: audit 2024-04-06T03:31:09.466987+0000 mgr.y (mgr.24368) 377 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[29221]: cluster 2024-04-06T03:31:08.980390+0000 osd.0 (osd.0) 204 : cluster 0 6.9 scrub starts 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[29221]: cluster 2024-04-06T03:31:09.065237+0000 osd.0 (osd.0) 205 : cluster 0 6.9 scrub ok 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[29221]: cluster 2024-04-06T03:31:09.193297+0000 osd.3 (osd.3) 239 : cluster 0 5.16 scrub starts 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[29221]: cluster 2024-04-06T03:31:09.200691+0000 osd.3 (osd.3) 240 : cluster 0 5.16 scrub ok 2024-04-06T03:31:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:10 smithi050 bash[29221]: audit 2024-04-06T03:31:09.466987+0000 mgr.y (mgr.24368) 377 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:10.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:10.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:10.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:11.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:10 smithi153 bash[23852]: cluster 2024-04-06T03:31:08.980390+0000 osd.0 (osd.0) 204 : cluster 0 6.9 scrub starts 2024-04-06T03:31:11.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:10 smithi153 bash[23852]: cluster 2024-04-06T03:31:09.065237+0000 osd.0 (osd.0) 205 : cluster 0 6.9 scrub ok 2024-04-06T03:31:11.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:10 smithi153 bash[23852]: cluster 2024-04-06T03:31:09.193297+0000 osd.3 (osd.3) 239 : cluster 0 5.16 scrub starts 2024-04-06T03:31:11.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:10 smithi153 bash[23852]: cluster 2024-04-06T03:31:09.200691+0000 osd.3 (osd.3) 240 : cluster 0 5.16 scrub ok 2024-04-06T03:31:11.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:10 smithi153 bash[23852]: audit 2024-04-06T03:31:09.466987+0000 mgr.y (mgr.24368) 377 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:11.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:11.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:11.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:11.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:11.353 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:11.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:11.488 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:11.503 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 72 15 91068 91053 79.0221 62.75 0.0189945 0.01265 2024-04-06T03:31:11.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:11.628 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:11.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:09.942759+0000 osd.0 (osd.0) 206 : cluster 0 5.15 scrub starts 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:09.948820+0000 osd.0 (osd.0) 207 : cluster 0 5.15 scrub ok 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:10.144008+0000 osd.4 (osd.4) 237 : cluster 0 2.1a scrub starts 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:10.147680+0000 osd.4 (osd.4) 238 : cluster 0 2.1a scrub ok 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:10.173658+0000 mgr.y (mgr.24368) 378 : cluster 0 pgmap v387: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 45 KiB/s rd, 72 MiB/s wr, 1.23k op/s 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:10.977924+0000 osd.0 (osd.0) 208 : cluster 0 2.13 scrub starts 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[21467]: cluster 2024-04-06T03:31:10.982635+0000 osd.0 (osd.0) 209 : cluster 0 2.13 scrub ok 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:09.942759+0000 osd.0 (osd.0) 206 : cluster 0 5.15 scrub starts 2024-04-06T03:31:11.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:09.948820+0000 osd.0 (osd.0) 207 : cluster 0 5.15 scrub ok 2024-04-06T03:31:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:10.144008+0000 osd.4 (osd.4) 237 : cluster 0 2.1a scrub starts 2024-04-06T03:31:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:10.147680+0000 osd.4 (osd.4) 238 : cluster 0 2.1a scrub ok 2024-04-06T03:31:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:10.173658+0000 mgr.y (mgr.24368) 378 : cluster 0 pgmap v387: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 45 KiB/s rd, 72 MiB/s wr, 1.23k op/s 2024-04-06T03:31:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:10.977924+0000 osd.0 (osd.0) 208 : cluster 0 2.13 scrub starts 2024-04-06T03:31:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:11 smithi050 bash[29221]: cluster 2024-04-06T03:31:10.982635+0000 osd.0 (osd.0) 209 : cluster 0 2.13 scrub ok 2024-04-06T03:31:11.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:11.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:11.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:11.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:12.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:12.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:09.942759+0000 osd.0 (osd.0) 206 : cluster 0 5.15 scrub starts 2024-04-06T03:31:12.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:09.948820+0000 osd.0 (osd.0) 207 : cluster 0 5.15 scrub ok 2024-04-06T03:31:12.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:10.144008+0000 osd.4 (osd.4) 237 : cluster 0 2.1a scrub starts 2024-04-06T03:31:12.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:10.147680+0000 osd.4 (osd.4) 238 : cluster 0 2.1a scrub ok 2024-04-06T03:31:12.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:10.173658+0000 mgr.y (mgr.24368) 378 : cluster 0 pgmap v387: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 45 KiB/s rd, 72 MiB/s wr, 1.23k op/s 2024-04-06T03:31:12.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:10.977924+0000 osd.0 (osd.0) 208 : cluster 0 2.13 scrub starts 2024-04-06T03:31:12.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:11 smithi153 bash[23852]: cluster 2024-04-06T03:31:10.982635+0000 osd.0 (osd.0) 209 : cluster 0 2.13 scrub ok 2024-04-06T03:31:12.055 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:12.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:12.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:12.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:12.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:12.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:12.436 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:12.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:12.505 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 73 16 92044 92028 78.7725 60.9375 0.0175893 0.0126883 2024-04-06T03:31:12.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:12.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:12.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:12.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:11.703420+0000 osd.1 (osd.1) 201 : cluster 0 8.3 scrub starts 2024-04-06T03:31:12.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:11.892382+0000 osd.7 (osd.7) 235 : cluster 0 6.5 scrub starts 2024-04-06T03:31:12.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:11.919363+0000 osd.7 (osd.7) 236 : cluster 0 6.5 scrub ok 2024-04-06T03:31:12.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:11.999649+0000 osd.2 (osd.2) 201 : cluster 0 3.4 scrub starts 2024-04-06T03:31:12.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.004065+0000 osd.2 (osd.2) 202 : cluster 0 3.4 scrub ok 2024-04-06T03:31:12.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.015715+0000 osd.0 (osd.0) 210 : cluster 0 5.1b scrub starts 2024-04-06T03:31:12.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:12 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.022250+0000 osd.0 (osd.0) 211 : cluster 0 5.1b scrub ok 2024-04-06T03:31:12.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:12.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:31:12.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:12.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:12.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:12.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:11.703420+0000 osd.1 (osd.1) 201 : cluster 0 8.3 scrub starts 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:11.892382+0000 osd.7 (osd.7) 235 : cluster 0 6.5 scrub starts 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:11.919363+0000 osd.7 (osd.7) 236 : cluster 0 6.5 scrub ok 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:11.999649+0000 osd.2 (osd.2) 201 : cluster 0 3.4 scrub starts 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.004065+0000 osd.2 (osd.2) 202 : cluster 0 3.4 scrub ok 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.015715+0000 osd.0 (osd.0) 210 : cluster 0 5.1b scrub starts 2024-04-06T03:31:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.022250+0000 osd.0 (osd.0) 211 : cluster 0 5.1b scrub ok 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:11.703420+0000 osd.1 (osd.1) 201 : cluster 0 8.3 scrub starts 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:11.892382+0000 osd.7 (osd.7) 235 : cluster 0 6.5 scrub starts 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:11.919363+0000 osd.7 (osd.7) 236 : cluster 0 6.5 scrub ok 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:11.999649+0000 osd.2 (osd.2) 201 : cluster 0 3.4 scrub starts 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.004065+0000 osd.2 (osd.2) 202 : cluster 0 3.4 scrub ok 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.015715+0000 osd.0 (osd.0) 210 : cluster 0 5.1b scrub starts 2024-04-06T03:31:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:12 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.022250+0000 osd.0 (osd.0) 211 : cluster 0 5.1b scrub ok 2024-04-06T03:31:13.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:13.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:13.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:13.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.348 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:13.425 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:13.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.484 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:13.505 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 74 16 92998 92982 78.5137 59.625 0.0107543 0.0127289 2024-04-06T03:31:13.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:13.625 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:13.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:13.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.164107+0000 osd.4 (osd.4) 239 : cluster 0 5.f scrub starts 2024-04-06T03:31:13.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.165253+0000 osd.4 (osd.4) 240 : cluster 0 5.f scrub ok 2024-04-06T03:31:13.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.174093+0000 mgr.y (mgr.24368) 379 : cluster 0 pgmap v388: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 23 KiB/s rd, 63 MiB/s wr, 1.05k op/s 2024-04-06T03:31:13.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.183869+0000 osd.3 (osd.3) 241 : cluster 0 3.b scrub starts 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.190747+0000 osd.3 (osd.3) 242 : cluster 0 3.b scrub ok 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.977956+0000 osd.2 (osd.2) 203 : cluster 0 2.e scrub starts 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[21467]: cluster 2024-04-06T03:31:12.980711+0000 osd.2 (osd.2) 204 : cluster 0 2.e scrub ok 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.164107+0000 osd.4 (osd.4) 239 : cluster 0 5.f scrub starts 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.165253+0000 osd.4 (osd.4) 240 : cluster 0 5.f scrub ok 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.174093+0000 mgr.y (mgr.24368) 379 : cluster 0 pgmap v388: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 23 KiB/s rd, 63 MiB/s wr, 1.05k op/s 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.183869+0000 osd.3 (osd.3) 241 : cluster 0 3.b scrub starts 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.190747+0000 osd.3 (osd.3) 242 : cluster 0 3.b scrub ok 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.977956+0000 osd.2 (osd.2) 203 : cluster 0 2.e scrub starts 2024-04-06T03:31:13.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:13 smithi050 bash[29221]: cluster 2024-04-06T03:31:12.980711+0000 osd.2 (osd.2) 204 : cluster 0 2.e scrub ok 2024-04-06T03:31:13.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:13.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.164107+0000 osd.4 (osd.4) 239 : cluster 0 5.f scrub starts 2024-04-06T03:31:13.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.165253+0000 osd.4 (osd.4) 240 : cluster 0 5.f scrub ok 2024-04-06T03:31:13.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.174093+0000 mgr.y (mgr.24368) 379 : cluster 0 pgmap v388: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 23 KiB/s rd, 63 MiB/s wr, 1.05k op/s 2024-04-06T03:31:13.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.183869+0000 osd.3 (osd.3) 241 : cluster 0 3.b scrub starts 2024-04-06T03:31:13.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.190747+0000 osd.3 (osd.3) 242 : cluster 0 3.b scrub ok 2024-04-06T03:31:13.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.977956+0000 osd.2 (osd.2) 203 : cluster 0 2.e scrub starts 2024-04-06T03:31:13.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:13 smithi153 bash[23852]: cluster 2024-04-06T03:31:12.980711+0000 osd.2 (osd.2) 204 : cluster 0 2.e scrub ok 2024-04-06T03:31:13.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.879 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:13.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:13.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:14.003 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:14.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:14.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:14.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:14.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:14.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:14.435 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:14.435 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:14.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:14.505 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 75 16 93622 93606 77.9867 39 0.00916294 0.0128178 2024-04-06T03:31:14.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:14.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:14.690 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:14.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:14 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.247257+0000 osd.1 (osd.1) 202 : cluster 0 8.3 scrub ok 2024-04-06T03:31:14.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:14 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.247257+0000 osd.1 (osd.1) 202 : cluster 0 8.3 scrub ok 2024-04-06T03:31:14.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:14.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:14.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:14.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:14 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.247257+0000 osd.1 (osd.1) 202 : cluster 0 8.3 scrub ok 2024-04-06T03:31:14.963 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:14.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:15.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:15.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:15.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:15.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:15.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:15.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:15.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:15.505 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 76 16 94624 94608 77.7846 62.625 0.00664278 0.0128505 2024-04-06T03:31:15.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:15.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:15.647 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:15.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:15.760 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.154889+0000 osd.4 (osd.4) 241 : cluster 0 5.1f scrub starts 2024-04-06T03:31:15.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.161663+0000 osd.4 (osd.4) 242 : cluster 0 5.1f scrub ok 2024-04-06T03:31:15.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.174780+0000 mgr.y (mgr.24368) 380 : cluster 0 pgmap v389: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 14 KiB/s rd, 56 MiB/s wr, 913 op/s 2024-04-06T03:31:15.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.241633+0000 osd.3 (osd.3) 243 : cluster 0 3.2 scrub starts 2024-04-06T03:31:15.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.245244+0000 osd.3 (osd.3) 244 : cluster 0 3.2 scrub ok 2024-04-06T03:31:15.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.953915+0000 osd.0 (osd.0) 212 : cluster 0 3.1f scrub starts 2024-04-06T03:31:15.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:15 smithi153 bash[23852]: cluster 2024-04-06T03:31:14.963417+0000 osd.0 (osd.0) 213 : cluster 0 3.1f scrub ok 2024-04-06T03:31:15.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:15.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:15.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:15.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:15.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.154889+0000 osd.4 (osd.4) 241 : cluster 0 5.1f scrub starts 2024-04-06T03:31:15.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.161663+0000 osd.4 (osd.4) 242 : cluster 0 5.1f scrub ok 2024-04-06T03:31:15.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.174780+0000 mgr.y (mgr.24368) 380 : cluster 0 pgmap v389: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 14 KiB/s rd, 56 MiB/s wr, 913 op/s 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.241633+0000 osd.3 (osd.3) 243 : cluster 0 3.2 scrub starts 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.245244+0000 osd.3 (osd.3) 244 : cluster 0 3.2 scrub ok 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.953915+0000 osd.0 (osd.0) 212 : cluster 0 3.1f scrub starts 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[21467]: cluster 2024-04-06T03:31:14.963417+0000 osd.0 (osd.0) 213 : cluster 0 3.1f scrub ok 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.154889+0000 osd.4 (osd.4) 241 : cluster 0 5.1f scrub starts 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.161663+0000 osd.4 (osd.4) 242 : cluster 0 5.1f scrub ok 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.174780+0000 mgr.y (mgr.24368) 380 : cluster 0 pgmap v389: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 14 KiB/s rd, 56 MiB/s wr, 913 op/s 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.241633+0000 osd.3 (osd.3) 243 : cluster 0 3.2 scrub starts 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.245244+0000 osd.3 (osd.3) 244 : cluster 0 3.2 scrub ok 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.953915+0000 osd.0 (osd.0) 212 : cluster 0 3.1f scrub starts 2024-04-06T03:31:15.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:15 smithi050 bash[29221]: cluster 2024-04-06T03:31:14.963417+0000 osd.0 (osd.0) 213 : cluster 0 3.1f scrub ok 2024-04-06T03:31:16.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:16.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:16.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:16.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:16.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:16.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:16.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:16.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:16.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:16.505 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 77 16 95156 95140 77.2062 33.25 0.00877062 0.012887 2024-04-06T03:31:16.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:16.547 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:16.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:16.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:16.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:16.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[21467]: cluster 2024-04-06T03:31:15.927952+0000 osd.0 (osd.0) 214 : cluster 0 6.a scrub starts 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[21467]: cluster 2024-04-06T03:31:15.939964+0000 osd.7 (osd.7) 237 : cluster 0 5.10 scrub starts 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[21467]: cluster 2024-04-06T03:31:15.940868+0000 osd.7 (osd.7) 238 : cluster 0 5.10 scrub ok 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[21467]: cluster 2024-04-06T03:31:15.957374+0000 osd.0 (osd.0) 215 : cluster 0 6.a scrub ok 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.574303+0000 osd.1 (osd.1) 203 : cluster 0 2.6 scrub starts 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.576772+0000 osd.1 (osd.1) 204 : cluster 0 2.6 scrub ok 2024-04-06T03:31:16.940 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[29221]: cluster 2024-04-06T03:31:15.927952+0000 osd.0 (osd.0) 214 : cluster 0 6.a scrub starts 2024-04-06T03:31:16.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[29221]: cluster 2024-04-06T03:31:15.939964+0000 osd.7 (osd.7) 237 : cluster 0 5.10 scrub starts 2024-04-06T03:31:16.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[29221]: cluster 2024-04-06T03:31:15.940868+0000 osd.7 (osd.7) 238 : cluster 0 5.10 scrub ok 2024-04-06T03:31:16.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[29221]: cluster 2024-04-06T03:31:15.957374+0000 osd.0 (osd.0) 215 : cluster 0 6.a scrub ok 2024-04-06T03:31:16.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.574303+0000 osd.1 (osd.1) 203 : cluster 0 2.6 scrub starts 2024-04-06T03:31:16.941 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:16 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.576772+0000 osd.1 (osd.1) 204 : cluster 0 2.6 scrub ok 2024-04-06T03:31:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:16.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:17.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:16 smithi153 bash[23852]: cluster 2024-04-06T03:31:15.927952+0000 osd.0 (osd.0) 214 : cluster 0 6.a scrub starts 2024-04-06T03:31:17.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:16 smithi153 bash[23852]: cluster 2024-04-06T03:31:15.939964+0000 osd.7 (osd.7) 237 : cluster 0 5.10 scrub starts 2024-04-06T03:31:17.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:16 smithi153 bash[23852]: cluster 2024-04-06T03:31:15.940868+0000 osd.7 (osd.7) 238 : cluster 0 5.10 scrub ok 2024-04-06T03:31:17.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:16 smithi153 bash[23852]: cluster 2024-04-06T03:31:15.957374+0000 osd.0 (osd.0) 215 : cluster 0 6.a scrub ok 2024-04-06T03:31:17.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:16 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.574303+0000 osd.1 (osd.1) 203 : cluster 0 2.6 scrub starts 2024-04-06T03:31:17.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:16 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.576772+0000 osd.1 (osd.1) 204 : cluster 0 2.6 scrub ok 2024-04-06T03:31:17.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:17.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:17.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:17.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:17.382 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:17.400 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:31:17.400 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:31:17.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:17.507 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 78 16 96069 96053 76.946 57.0625 0.0149281 0.0129909 2024-04-06T03:31:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:17.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:17.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:17.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:17.770 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:17.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:17.902 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:17.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:17.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:17 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.175519+0000 mgr.y (mgr.24368) 381 : cluster 0 pgmap v390: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 969 B/s rd, 55 MiB/s wr, 885 op/s 2024-04-06T03:31:17.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:17 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.984610+0000 osd.7 (osd.7) 239 : cluster 0 3.13 scrub starts 2024-04-06T03:31:17.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:17 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.986495+0000 osd.7 (osd.7) 240 : cluster 0 3.13 scrub ok 2024-04-06T03:31:17.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:17 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.175519+0000 mgr.y (mgr.24368) 381 : cluster 0 pgmap v390: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 969 B/s rd, 55 MiB/s wr, 885 op/s 2024-04-06T03:31:17.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:17 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.984610+0000 osd.7 (osd.7) 239 : cluster 0 3.13 scrub starts 2024-04-06T03:31:17.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:17 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.986495+0000 osd.7 (osd.7) 240 : cluster 0 3.13 scrub ok 2024-04-06T03:31:18.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:18.016 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:17 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.175519+0000 mgr.y (mgr.24368) 381 : cluster 0 pgmap v390: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 969 B/s rd, 55 MiB/s wr, 885 op/s 2024-04-06T03:31:18.016 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:17 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.984610+0000 osd.7 (osd.7) 239 : cluster 0 3.13 scrub starts 2024-04-06T03:31:18.016 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:17 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.986495+0000 osd.7 (osd.7) 240 : cluster 0 3.13 scrub ok 2024-04-06T03:31:18.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:18.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:18.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:18.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:18.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-06T03:31:18.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:18.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:18.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:18.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 79 16 96714 96698 76.4823 40.3125 0.0114736 0.01307 2024-04-06T03:31:18.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:18.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:18.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:18.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:18.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:18.838 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:18.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.949597+0000 osd.0 (osd.0) 216 : cluster 0 6.0 scrub starts 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:16.985405+0000 osd.0 (osd.0) 217 : cluster 0 6.0 scrub ok 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:17.059282+0000 osd.0 (osd.0) 218 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:17.059366+0000 osd.0 (osd.0) 219 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:17.700838+0000 mon.a (mon.0) 1215 : cluster 0 osdmap e202: 8 total, 8 up, 8 in 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:17.993237+0000 osd.7 (osd.7) 241 : cluster 0 5.19 scrub starts 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:17.994679+0000 osd.7 (osd.7) 242 : cluster 0 5.19 scrub ok 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:18.055011+0000 osd.2 (osd.2) 205 : cluster 0 2.17 scrub starts 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:18.056876+0000 osd.2 (osd.2) 206 : cluster 0 2.17 scrub ok 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:18.176041+0000 mgr.y (mgr.24368) 382 : cluster 0 pgmap v392: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 935 op/s 2024-04-06T03:31:18.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:18 smithi153 bash[23852]: cluster 2024-04-06T03:31:18.518010+0000 osd.1 (osd.1) 205 : cluster 0 8.a scrub starts 2024-04-06T03:31:18.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:18.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:18.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.949597+0000 osd.0 (osd.0) 216 : cluster 0 6.0 scrub starts 2024-04-06T03:31:18.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:16.985405+0000 osd.0 (osd.0) 217 : cluster 0 6.0 scrub ok 2024-04-06T03:31:18.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:17.059282+0000 osd.0 (osd.0) 218 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:18.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:17.059366+0000 osd.0 (osd.0) 219 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:17.700838+0000 mon.a (mon.0) 1215 : cluster 0 osdmap e202: 8 total, 8 up, 8 in 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:17.993237+0000 osd.7 (osd.7) 241 : cluster 0 5.19 scrub starts 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:17.994679+0000 osd.7 (osd.7) 242 : cluster 0 5.19 scrub ok 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:18.055011+0000 osd.2 (osd.2) 205 : cluster 0 2.17 scrub starts 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:18.056876+0000 osd.2 (osd.2) 206 : cluster 0 2.17 scrub ok 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:18.176041+0000 mgr.y (mgr.24368) 382 : cluster 0 pgmap v392: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 935 op/s 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[29221]: cluster 2024-04-06T03:31:18.518010+0000 osd.1 (osd.1) 205 : cluster 0 8.a scrub starts 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.949597+0000 osd.0 (osd.0) 216 : cluster 0 6.0 scrub starts 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:16.985405+0000 osd.0 (osd.0) 217 : cluster 0 6.0 scrub ok 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:17.059282+0000 osd.0 (osd.0) 218 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:17.059366+0000 osd.0 (osd.0) 219 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:18.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:17.700838+0000 mon.a (mon.0) 1215 : cluster 0 osdmap e202: 8 total, 8 up, 8 in 2024-04-06T03:31:18.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:17.993237+0000 osd.7 (osd.7) 241 : cluster 0 5.19 scrub starts 2024-04-06T03:31:18.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:17.994679+0000 osd.7 (osd.7) 242 : cluster 0 5.19 scrub ok 2024-04-06T03:31:18.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:18.055011+0000 osd.2 (osd.2) 205 : cluster 0 2.17 scrub starts 2024-04-06T03:31:18.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:18.056876+0000 osd.2 (osd.2) 206 : cluster 0 2.17 scrub ok 2024-04-06T03:31:18.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:18.176041+0000 mgr.y (mgr.24368) 382 : cluster 0 pgmap v392: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 935 op/s 2024-04-06T03:31:18.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:18 smithi050 bash[21467]: cluster 2024-04-06T03:31:18.518010+0000 osd.1 (osd.1) 205 : cluster 0 8.a scrub starts 2024-04-06T03:31:19.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:19.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:19.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:19.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:19.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:19.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:19.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:19.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:19.507 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:31:19.514716+0000 min lat: 0.00122838 max lat: 0.562935 avg lat: 0.0131068 2024-04-06T03:31:19.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:31:19.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 80 16 97646 97630 76.2544 58.25 0.00587191 0.0131068 2024-04-06T03:31:19.567 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:31:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:31:19.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:19.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:19.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:19.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:19.796 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:31:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:31:19.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:19 smithi153 bash[23852]: cluster 2024-04-06T03:31:18.031498+0000 osd.4 (osd.4) 243 : cluster 0 5.18 scrub starts 2024-04-06T03:31:19.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:19 smithi153 bash[23852]: cluster 2024-04-06T03:31:18.032810+0000 osd.4 (osd.4) 244 : cluster 0 5.18 scrub ok 2024-04-06T03:31:19.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:19 smithi153 bash[23852]: audit 2024-04-06T03:31:19.041513+0000 mon.a (mon.0) 1216 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:19.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:19 smithi153 bash[23852]: audit 2024-04-06T03:31:19.475180+0000 mgr.y (mgr.24368) 383 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:19.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[21467]: cluster 2024-04-06T03:31:18.031498+0000 osd.4 (osd.4) 243 : cluster 0 5.18 scrub starts 2024-04-06T03:31:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[21467]: cluster 2024-04-06T03:31:18.032810+0000 osd.4 (osd.4) 244 : cluster 0 5.18 scrub ok 2024-04-06T03:31:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[21467]: audit 2024-04-06T03:31:19.041513+0000 mon.a (mon.0) 1216 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[21467]: audit 2024-04-06T03:31:19.475180+0000 mgr.y (mgr.24368) 383 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:19.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[29221]: cluster 2024-04-06T03:31:18.031498+0000 osd.4 (osd.4) 243 : cluster 0 5.18 scrub starts 2024-04-06T03:31:19.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[29221]: cluster 2024-04-06T03:31:18.032810+0000 osd.4 (osd.4) 244 : cluster 0 5.18 scrub ok 2024-04-06T03:31:19.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[29221]: audit 2024-04-06T03:31:19.041513+0000 mon.a (mon.0) 1216 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:19.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:19 smithi050 bash[29221]: audit 2024-04-06T03:31:19.475180+0000 mgr.y (mgr.24368) 383 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:19.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:19.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:20.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:20.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:20.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:20.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:20.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:20.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:20.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:20.433 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:20.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:20.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 81 16 98567 98551 76.0236 57.5625 0.00464103 0.0131477 2024-04-06T03:31:20.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:20.559 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:20.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:20.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:20.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:20.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:20 smithi050 bash[21467]: cluster 2024-04-06T03:31:20.176980+0000 mgr.y (mgr.24368) 384 : cluster 0 pgmap v393: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 860 op/s 2024-04-06T03:31:20.952 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:20 smithi050 bash[29221]: cluster 2024-04-06T03:31:20.176980+0000 mgr.y (mgr.24368) 384 : cluster 0 pgmap v393: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 860 op/s 2024-04-06T03:31:20.970 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:20.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:21.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:20 smithi153 bash[23852]: cluster 2024-04-06T03:31:20.176980+0000 mgr.y (mgr.24368) 384 : cluster 0 pgmap v393: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 860 op/s 2024-04-06T03:31:21.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:21.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:21.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:21.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.489 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:21.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 82 16 99462 99446 75.7786 55.9375 0.00314286 0.0131744 2024-04-06T03:31:21.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:21.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:21.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:21.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:21.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:21.895 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:21.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:21.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:19.993537+0000 osd.4 (osd.4) 245 : cluster 0 3.1c scrub starts 2024-04-06T03:31:21.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:19.996216+0000 osd.4 (osd.4) 246 : cluster 0 3.1c scrub ok 2024-04-06T03:31:21.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:20.115316+0000 osd.0 (osd.0) 220 : cluster 0 3.10 scrub starts 2024-04-06T03:31:21.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:20.121403+0000 osd.0 (osd.0) 221 : cluster 0 3.10 scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.045358+0000 osd.7 (osd.7) 243 : cluster 0 5.1a scrub starts 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.086109+0000 osd.7 (osd.7) 244 : cluster 0 5.1a scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.106697+0000 osd.1 (osd.1) 206 : cluster 0 8.a scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.611322+0000 osd.1 (osd.1) 207 : cluster 0 2.2 scrub starts 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:19.993537+0000 osd.4 (osd.4) 245 : cluster 0 3.1c scrub starts 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:19.996216+0000 osd.4 (osd.4) 246 : cluster 0 3.1c scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:20.115316+0000 osd.0 (osd.0) 220 : cluster 0 3.10 scrub starts 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:20.121403+0000 osd.0 (osd.0) 221 : cluster 0 3.10 scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.045358+0000 osd.7 (osd.7) 243 : cluster 0 5.1a scrub starts 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.086109+0000 osd.7 (osd.7) 244 : cluster 0 5.1a scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.106697+0000 osd.1 (osd.1) 206 : cluster 0 8.a scrub ok 2024-04-06T03:31:21.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:21 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.611322+0000 osd.1 (osd.1) 207 : cluster 0 2.2 scrub starts 2024-04-06T03:31:21.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:22.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:22.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:19.993537+0000 osd.4 (osd.4) 245 : cluster 0 3.1c scrub starts 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:19.996216+0000 osd.4 (osd.4) 246 : cluster 0 3.1c scrub ok 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:20.115316+0000 osd.0 (osd.0) 220 : cluster 0 3.10 scrub starts 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:20.121403+0000 osd.0 (osd.0) 221 : cluster 0 3.10 scrub ok 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.045358+0000 osd.7 (osd.7) 243 : cluster 0 5.1a scrub starts 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.086109+0000 osd.7 (osd.7) 244 : cluster 0 5.1a scrub ok 2024-04-06T03:31:22.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.106697+0000 osd.1 (osd.1) 206 : cluster 0 8.a scrub ok 2024-04-06T03:31:22.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:21 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.611322+0000 osd.1 (osd.1) 207 : cluster 0 2.2 scrub starts 2024-04-06T03:31:22.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:22.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:22.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:22.300 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:22.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:22.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:22.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:22.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 83 16 100396 100380 75.5689 58.375 0.0121719 0.0132284 2024-04-06T03:31:22.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:22.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:22.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:22.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.269531+0000 osd.3 (osd.3) 245 : cluster 0 3.8 scrub starts 2024-04-06T03:31:22.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.273260+0000 osd.3 (osd.3) 246 : cluster 0 3.8 scrub ok 2024-04-06T03:31:22.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.619370+0000 osd.1 (osd.1) 208 : cluster 0 2.2 scrub ok 2024-04-06T03:31:22.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:21.996068+0000 osd.7 (osd.7) 245 : cluster 0 6.12 scrub starts 2024-04-06T03:31:22.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:22.000551+0000 osd.7 (osd.7) 246 : cluster 0 6.12 scrub ok 2024-04-06T03:31:22.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:22.002106+0000 osd.2 (osd.2) 207 : cluster 0 2.16 scrub starts 2024-04-06T03:31:22.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:22.008740+0000 osd.2 (osd.2) 208 : cluster 0 2.16 scrub ok 2024-04-06T03:31:22.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:22 smithi153 bash[23852]: cluster 2024-04-06T03:31:22.177386+0000 mgr.y (mgr.24368) 385 : cluster 0 pgmap v394: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 860 op/s 2024-04-06T03:31:22.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:22.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:22.935 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:22.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:22.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:22.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.269531+0000 osd.3 (osd.3) 245 : cluster 0 3.8 scrub starts 2024-04-06T03:31:22.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.273260+0000 osd.3 (osd.3) 246 : cluster 0 3.8 scrub ok 2024-04-06T03:31:22.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.619370+0000 osd.1 (osd.1) 208 : cluster 0 2.2 scrub ok 2024-04-06T03:31:22.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:21.996068+0000 osd.7 (osd.7) 245 : cluster 0 6.12 scrub starts 2024-04-06T03:31:22.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:22.000551+0000 osd.7 (osd.7) 246 : cluster 0 6.12 scrub ok 2024-04-06T03:31:22.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:22.002106+0000 osd.2 (osd.2) 207 : cluster 0 2.16 scrub starts 2024-04-06T03:31:22.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:22.008740+0000 osd.2 (osd.2) 208 : cluster 0 2.16 scrub ok 2024-04-06T03:31:22.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[29221]: cluster 2024-04-06T03:31:22.177386+0000 mgr.y (mgr.24368) 385 : cluster 0 pgmap v394: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 860 op/s 2024-04-06T03:31:22.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.269531+0000 osd.3 (osd.3) 245 : cluster 0 3.8 scrub starts 2024-04-06T03:31:22.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.273260+0000 osd.3 (osd.3) 246 : cluster 0 3.8 scrub ok 2024-04-06T03:31:22.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.619370+0000 osd.1 (osd.1) 208 : cluster 0 2.2 scrub ok 2024-04-06T03:31:22.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:21.996068+0000 osd.7 (osd.7) 245 : cluster 0 6.12 scrub starts 2024-04-06T03:31:22.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:22.000551+0000 osd.7 (osd.7) 246 : cluster 0 6.12 scrub ok 2024-04-06T03:31:22.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:22.002106+0000 osd.2 (osd.2) 207 : cluster 0 2.16 scrub starts 2024-04-06T03:31:22.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:22.008740+0000 osd.2 (osd.2) 208 : cluster 0 2.16 scrub ok 2024-04-06T03:31:22.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:22 smithi050 bash[21467]: cluster 2024-04-06T03:31:22.177386+0000 mgr.y (mgr.24368) 385 : cluster 0 pgmap v394: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 54 MiB/s wr, 860 op/s 2024-04-06T03:31:23.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:23.088 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:23.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:23.226 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:23.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:23.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:23.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:23.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:23.508 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 84 16 101324 101308 75.3597 58 0.0152101 0.013265 2024-04-06T03:31:23.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:23.638 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:23.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:23.765 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:23.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:23.894 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:23.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:23.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:24.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:24.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:24.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:24.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:24.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:24.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:24.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:24.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 85 16 102240 102224 75.1448 57.25 0.00799539 0.0133024 2024-04-06T03:31:24.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:24.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[21467]: cluster 2024-04-06T03:31:23.007790+0000 osd.4 (osd.4) 247 : cluster 0 2.11 scrub starts 2024-04-06T03:31:24.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[21467]: cluster 2024-04-06T03:31:23.008707+0000 osd.4 (osd.4) 248 : cluster 0 2.11 scrub ok 2024-04-06T03:31:24.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[21467]: audit 2024-04-06T03:31:23.935972+0000 mon.a (mon.0) 1217 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:24.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[21467]: cluster 2024-04-06T03:31:24.069113+0000 osd.7 (osd.7) 247 : cluster 0 6.3 scrub starts 2024-04-06T03:31:24.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[21467]: cluster 2024-04-06T03:31:24.092461+0000 osd.7 (osd.7) 248 : cluster 0 6.3 scrub ok 2024-04-06T03:31:24.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[29221]: cluster 2024-04-06T03:31:23.007790+0000 osd.4 (osd.4) 247 : cluster 0 2.11 scrub starts 2024-04-06T03:31:24.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[29221]: cluster 2024-04-06T03:31:23.008707+0000 osd.4 (osd.4) 248 : cluster 0 2.11 scrub ok 2024-04-06T03:31:24.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[29221]: audit 2024-04-06T03:31:23.935972+0000 mon.a (mon.0) 1217 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:24.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[29221]: cluster 2024-04-06T03:31:24.069113+0000 osd.7 (osd.7) 247 : cluster 0 6.3 scrub starts 2024-04-06T03:31:24.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:24 smithi050 bash[29221]: cluster 2024-04-06T03:31:24.092461+0000 osd.7 (osd.7) 248 : cluster 0 6.3 scrub ok 2024-04-06T03:31:24.600 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:24.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:24.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:24 smithi153 bash[23852]: cluster 2024-04-06T03:31:23.007790+0000 osd.4 (osd.4) 247 : cluster 0 2.11 scrub starts 2024-04-06T03:31:24.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:24 smithi153 bash[23852]: cluster 2024-04-06T03:31:23.008707+0000 osd.4 (osd.4) 248 : cluster 0 2.11 scrub ok 2024-04-06T03:31:24.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:24 smithi153 bash[23852]: audit 2024-04-06T03:31:23.935972+0000 mon.a (mon.0) 1217 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:31:24.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:24 smithi153 bash[23852]: cluster 2024-04-06T03:31:24.069113+0000 osd.7 (osd.7) 247 : cluster 0 6.3 scrub starts 2024-04-06T03:31:24.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:24 smithi153 bash[23852]: cluster 2024-04-06T03:31:24.092461+0000 osd.7 (osd.7) 248 : cluster 0 6.3 scrub ok 2024-04-06T03:31:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:24.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:24.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:24.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:24.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:25.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:25.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:25.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:25.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:25.510 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 86 16 102906 102890 74.7551 41.625 0.00683713 0.0133705 2024-04-06T03:31:25.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:25.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:25.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: cluster 2024-04-06T03:31:24.178017+0000 mgr.y (mgr.24368) 386 : cluster 0 pgmap v395: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 852 op/s 2024-04-06T03:31:25.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: cluster 2024-04-06T03:31:24.541002+0000 osd.1 (osd.1) 209 : cluster 0 2.9 scrub starts 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: cluster 2024-04-06T03:31:24.548995+0000 osd.1 (osd.1) 210 : cluster 0 2.9 scrub ok 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: audit 2024-04-06T03:31:24.720592+0000 mon.a (mon.0) 1218 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: audit 2024-04-06T03:31:24.728525+0000 mon.a (mon.0) 1219 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: audit 2024-04-06T03:31:25.460609+0000 mon.a (mon.0) 1220 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: audit 2024-04-06T03:31:25.463030+0000 mon.a (mon.0) 1221 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[29221]: audit 2024-04-06T03:31:25.474712+0000 mon.a (mon.0) 1222 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: cluster 2024-04-06T03:31:24.178017+0000 mgr.y (mgr.24368) 386 : cluster 0 pgmap v395: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 852 op/s 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: cluster 2024-04-06T03:31:24.541002+0000 osd.1 (osd.1) 209 : cluster 0 2.9 scrub starts 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: cluster 2024-04-06T03:31:24.548995+0000 osd.1 (osd.1) 210 : cluster 0 2.9 scrub ok 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: audit 2024-04-06T03:31:24.720592+0000 mon.a (mon.0) 1218 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: audit 2024-04-06T03:31:24.728525+0000 mon.a (mon.0) 1219 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: audit 2024-04-06T03:31:25.460609+0000 mon.a (mon.0) 1220 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:25.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: audit 2024-04-06T03:31:25.463030+0000 mon.a (mon.0) 1221 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:25.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:25 smithi050 bash[21467]: audit 2024-04-06T03:31:25.474712+0000 mon.a (mon.0) 1222 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:25.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:25.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: cluster 2024-04-06T03:31:24.178017+0000 mgr.y (mgr.24368) 386 : cluster 0 pgmap v395: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 852 op/s 2024-04-06T03:31:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: cluster 2024-04-06T03:31:24.541002+0000 osd.1 (osd.1) 209 : cluster 0 2.9 scrub starts 2024-04-06T03:31:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: cluster 2024-04-06T03:31:24.548995+0000 osd.1 (osd.1) 210 : cluster 0 2.9 scrub ok 2024-04-06T03:31:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: audit 2024-04-06T03:31:24.720592+0000 mon.a (mon.0) 1218 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: audit 2024-04-06T03:31:24.728525+0000 mon.a (mon.0) 1219 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: audit 2024-04-06T03:31:25.460609+0000 mon.a (mon.0) 1220 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:31:25.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: audit 2024-04-06T03:31:25.463030+0000 mon.a (mon.0) 1221 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:31:25.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:25 smithi153 bash[23852]: audit 2024-04-06T03:31:25.474712+0000 mon.a (mon.0) 1222 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:25.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:25.926 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:25.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:26.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:26.067 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:26.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:26.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:26.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:26.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:26.510 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 87 16 103791 103775 74.5315 55.3125 0.00421793 0.0134122 2024-04-06T03:31:26.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:26.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:26.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:26.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.008672+0000 osd.4 (osd.4) 249 : cluster 0 2.18 scrub starts 2024-04-06T03:31:26.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.010760+0000 osd.4 (osd.4) 250 : cluster 0 2.18 scrub ok 2024-04-06T03:31:26.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.042757+0000 osd.2 (osd.2) 209 : cluster 0 6.14 scrub starts 2024-04-06T03:31:26.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.044779+0000 osd.2 (osd.2) 210 : cluster 0 6.14 scrub ok 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.198635+0000 osd.0 (osd.0) 222 : cluster 0 5.5 scrub starts 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.201834+0000 osd.0 (osd.0) 223 : cluster 0 5.5 scrub ok 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.008672+0000 osd.4 (osd.4) 249 : cluster 0 2.18 scrub starts 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.010760+0000 osd.4 (osd.4) 250 : cluster 0 2.18 scrub ok 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.042757+0000 osd.2 (osd.2) 209 : cluster 0 6.14 scrub starts 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.044779+0000 osd.2 (osd.2) 210 : cluster 0 6.14 scrub ok 2024-04-06T03:31:26.702 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.198635+0000 osd.0 (osd.0) 222 : cluster 0 5.5 scrub starts 2024-04-06T03:31:26.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:26 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.201834+0000 osd.0 (osd.0) 223 : cluster 0 5.5 scrub ok 2024-04-06T03:31:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:26.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:26 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.008672+0000 osd.4 (osd.4) 249 : cluster 0 2.18 scrub starts 2024-04-06T03:31:26.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:26 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.010760+0000 osd.4 (osd.4) 250 : cluster 0 2.18 scrub ok 2024-04-06T03:31:26.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:26 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.042757+0000 osd.2 (osd.2) 209 : cluster 0 6.14 scrub starts 2024-04-06T03:31:26.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:26 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.044779+0000 osd.2 (osd.2) 210 : cluster 0 6.14 scrub ok 2024-04-06T03:31:26.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:26 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.198635+0000 osd.0 (osd.0) 222 : cluster 0 5.5 scrub starts 2024-04-06T03:31:26.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:26 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.201834+0000 osd.0 (osd.0) 223 : cluster 0 5.5 scrub ok 2024-04-06T03:31:26.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:26.839 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:26.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:26.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:26.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:26.992 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:27.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:27.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:27.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:27.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:27.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:27.382 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:27.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:27.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 88 16 104528 104512 74.208 46.0625 0.0133612 0.0134707 2024-04-06T03:31:27.524 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:27.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:27.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:27.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:27 smithi153 bash[23852]: cluster 2024-04-06T03:31:25.994860+0000 osd.4 (osd.4) 251 : cluster 0 6.1a scrub starts 2024-04-06T03:31:27.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:27 smithi153 bash[23852]: cluster 2024-04-06T03:31:26.012616+0000 osd.4 (osd.4) 252 : cluster 0 6.1a scrub ok 2024-04-06T03:31:27.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:27 smithi153 bash[23852]: cluster 2024-04-06T03:31:26.155784+0000 osd.0 (osd.0) 224 : cluster 0 6.10 scrub starts 2024-04-06T03:31:27.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:27 smithi153 bash[23852]: cluster 2024-04-06T03:31:26.177418+0000 osd.0 (osd.0) 225 : cluster 0 6.10 scrub ok 2024-04-06T03:31:27.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:27 smithi153 bash[23852]: cluster 2024-04-06T03:31:26.178725+0000 mgr.y (mgr.24368) 387 : cluster 0 pgmap v396: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 853 op/s 2024-04-06T03:31:27.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:27.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:27.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:27.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:27.921 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:27.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[21467]: cluster 2024-04-06T03:31:25.994860+0000 osd.4 (osd.4) 251 : cluster 0 6.1a scrub starts 2024-04-06T03:31:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[21467]: cluster 2024-04-06T03:31:26.012616+0000 osd.4 (osd.4) 252 : cluster 0 6.1a scrub ok 2024-04-06T03:31:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[21467]: cluster 2024-04-06T03:31:26.155784+0000 osd.0 (osd.0) 224 : cluster 0 6.10 scrub starts 2024-04-06T03:31:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[21467]: cluster 2024-04-06T03:31:26.177418+0000 osd.0 (osd.0) 225 : cluster 0 6.10 scrub ok 2024-04-06T03:31:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[21467]: cluster 2024-04-06T03:31:26.178725+0000 mgr.y (mgr.24368) 387 : cluster 0 pgmap v396: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 853 op/s 2024-04-06T03:31:27.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[29221]: cluster 2024-04-06T03:31:25.994860+0000 osd.4 (osd.4) 251 : cluster 0 6.1a scrub starts 2024-04-06T03:31:27.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[29221]: cluster 2024-04-06T03:31:26.012616+0000 osd.4 (osd.4) 252 : cluster 0 6.1a scrub ok 2024-04-06T03:31:27.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[29221]: cluster 2024-04-06T03:31:26.155784+0000 osd.0 (osd.0) 224 : cluster 0 6.10 scrub starts 2024-04-06T03:31:27.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[29221]: cluster 2024-04-06T03:31:26.177418+0000 osd.0 (osd.0) 225 : cluster 0 6.10 scrub ok 2024-04-06T03:31:27.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:27 smithi050 bash[29221]: cluster 2024-04-06T03:31:26.178725+0000 mgr.y (mgr.24368) 387 : cluster 0 pgmap v396: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 921 B/s rd, 53 MiB/s wr, 853 op/s 2024-04-06T03:31:28.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:28.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:28.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:28.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:28.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:28.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 89 16 105319 105303 73.9297 49.4375 0.0166896 0.0135215 2024-04-06T03:31:28.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:28.572 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:28.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:28.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:28 smithi050 bash[21467]: cluster 2024-04-06T03:31:27.343816+0000 osd.3 (osd.3) 247 : cluster 0 5.17 scrub starts 2024-04-06T03:31:28.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:28 smithi050 bash[21467]: cluster 2024-04-06T03:31:27.375791+0000 osd.3 (osd.3) 248 : cluster 0 5.17 scrub ok 2024-04-06T03:31:28.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:28 smithi050 bash[29221]: cluster 2024-04-06T03:31:27.343816+0000 osd.3 (osd.3) 247 : cluster 0 5.17 scrub starts 2024-04-06T03:31:28.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:28 smithi050 bash[29221]: cluster 2024-04-06T03:31:27.375791+0000 osd.3 (osd.3) 248 : cluster 0 5.17 scrub ok 2024-04-06T03:31:28.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:28 smithi153 bash[23852]: cluster 2024-04-06T03:31:27.343816+0000 osd.3 (osd.3) 247 : cluster 0 5.17 scrub starts 2024-04-06T03:31:28.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:28 smithi153 bash[23852]: cluster 2024-04-06T03:31:27.375791+0000 osd.3 (osd.3) 248 : cluster 0 5.17 scrub ok 2024-04-06T03:31:28.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:28.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:28.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:28.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:28.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:29.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:29.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:29.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:29.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:29.379 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:29.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:29.486 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:31:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:31:29.504 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total time run: 90.0095 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total writes made: 106308 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Write size: 65536 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object size: 65536 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Bandwidth (MB/sec): 73.8172 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Bandwidth: 27.7931 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max bandwidth (MB/sec): 169.688 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min bandwidth (MB/sec): 33.25 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average IOPS: 1181 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev IOPS: 444.689 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max IOPS: 2715 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min IOPS: 532 2024-04-06T03:31:29.511 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average Latency(s): 0.0135428 2024-04-06T03:31:29.512 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Latency(s): 0.0227755 2024-04-06T03:31:29.512 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max latency(s): 0.562935 2024-04-06T03:31:29.512 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min latency(s): 0.00122838 2024-04-06T03:31:29.517 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Cleaning up (deleting benchmark objects) 2024-04-06T03:31:29.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:29.615 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:31:29.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:28.016039+0000 osd.2 (osd.2) 211 : cluster 0 5.d scrub starts 2024-04-06T03:31:29.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:28.018063+0000 osd.2 (osd.2) 212 : cluster 0 5.d scrub ok 2024-04-06T03:31:29.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:28.179108+0000 mgr.y (mgr.24368) 388 : cluster 0 pgmap v397: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 879 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-06T03:31:29.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:28.283259+0000 osd.4 (osd.4) 253 : cluster 0 6.2 scrub starts 2024-04-06T03:31:29.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:28.290985+0000 osd.4 (osd.4) 254 : cluster 0 6.2 scrub ok 2024-04-06T03:31:29.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:29.079782+0000 osd.7 (osd.7) 249 : cluster 0 5.4 scrub starts 2024-04-06T03:31:29.616 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:29 smithi153 bash[23852]: cluster 2024-04-06T03:31:29.080615+0000 osd.7 (osd.7) 250 : cluster 0 5.4 scrub ok 2024-04-06T03:31:29.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:29.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:28.016039+0000 osd.2 (osd.2) 211 : cluster 0 5.d scrub starts 2024-04-06T03:31:29.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:28.018063+0000 osd.2 (osd.2) 212 : cluster 0 5.d scrub ok 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:28.179108+0000 mgr.y (mgr.24368) 388 : cluster 0 pgmap v397: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 879 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:28.283259+0000 osd.4 (osd.4) 253 : cluster 0 6.2 scrub starts 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:28.290985+0000 osd.4 (osd.4) 254 : cluster 0 6.2 scrub ok 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:29.079782+0000 osd.7 (osd.7) 249 : cluster 0 5.4 scrub starts 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[21467]: cluster 2024-04-06T03:31:29.080615+0000 osd.7 (osd.7) 250 : cluster 0 5.4 scrub ok 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:28.016039+0000 osd.2 (osd.2) 211 : cluster 0 5.d scrub starts 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:28.018063+0000 osd.2 (osd.2) 212 : cluster 0 5.d scrub ok 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:28.179108+0000 mgr.y (mgr.24368) 388 : cluster 0 pgmap v397: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 879 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:28.283259+0000 osd.4 (osd.4) 253 : cluster 0 6.2 scrub starts 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:28.290985+0000 osd.4 (osd.4) 254 : cluster 0 6.2 scrub ok 2024-04-06T03:31:29.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:29.079782+0000 osd.7 (osd.7) 249 : cluster 0 5.4 scrub starts 2024-04-06T03:31:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:29 smithi050 bash[29221]: cluster 2024-04-06T03:31:29.080615+0000 osd.7 (osd.7) 250 : cluster 0 5.4 scrub ok 2024-04-06T03:31:29.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:29.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:29.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:29.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:29.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:30.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:30.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:30.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:30.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:30.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:30.309 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:30.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:30.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:30.498 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:31:30.546 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:31:30.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:30.570 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-06T03:31:30.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:30.615 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:31:30.666 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:31:30.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:30.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[21467]: cluster 2024-04-06T03:31:29.055495+0000 osd.4 (osd.4) 255 : cluster 0 8.2 scrub starts 2024-04-06T03:31:30.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[21467]: audit 2024-04-06T03:31:29.482976+0000 mgr.y (mgr.24368) 389 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:30.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[21467]: audit 2024-04-06T03:31:30.392733+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.50:0/3273188930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:31:30.687 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[21467]: audit 2024-04-06T03:31:30.456590+0000 mon.c (mon.2) 63 : audit 0 from='client.? 172.21.15.50:0/1245544662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:31:30.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[29221]: cluster 2024-04-06T03:31:29.055495+0000 osd.4 (osd.4) 255 : cluster 0 8.2 scrub starts 2024-04-06T03:31:30.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[29221]: audit 2024-04-06T03:31:29.482976+0000 mgr.y (mgr.24368) 389 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:30.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[29221]: audit 2024-04-06T03:31:30.392733+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.50:0/3273188930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:31:30.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:30 smithi050 bash[29221]: audit 2024-04-06T03:31:30.456590+0000 mon.c (mon.2) 63 : audit 0 from='client.? 172.21.15.50:0/1245544662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:31:30.697 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:31:30.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:30.747 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:31:30.792 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:31:30.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:30 smithi153 bash[23852]: cluster 2024-04-06T03:31:29.055495+0000 osd.4 (osd.4) 255 : cluster 0 8.2 scrub starts 2024-04-06T03:31:30.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:30 smithi153 bash[23852]: audit 2024-04-06T03:31:29.482976+0000 mgr.y (mgr.24368) 389 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:30.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:30 smithi153 bash[23852]: audit 2024-04-06T03:31:30.392733+0000 mon.a (mon.0) 1223 : audit 1 from='client.? 172.21.15.50:0/3273188930' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:31:30.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:30 smithi153 bash[23852]: audit 2024-04-06T03:31:30.456590+0000 mon.c (mon.2) 63 : audit 0 from='client.? 172.21.15.50:0/1245544662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:31:30.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:30.823 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:31:30.825 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:31:30.825 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:31:30.825 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:31:30.825 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:31:30.826 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:31:30.826 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:31:30.826 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:31:30.826 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:31:30.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:30.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:30.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:30.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:31.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:31.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:31.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:31.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:31.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:31.356 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:31.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:31.485 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:31.572 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:31:31.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.016311+0000 osd.2 (osd.2) 213 : cluster 0 6.f scrub starts 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.119722+0000 osd.2 (osd.2) 214 : cluster 0 6.f scrub ok 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.149540+0000 osd.0 (osd.0) 226 : cluster 0 5.3 scrub starts 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.157106+0000 osd.0 (osd.0) 227 : cluster 0 5.3 scrub ok 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.179918+0000 mgr.y (mgr.24368) 390 : cluster 0 pgmap v398: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 59 MiB/s wr, 946 op/s 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.545080+0000 osd.4 (osd.4) 256 : cluster 0 8.2 scrub ok 2024-04-06T03:31:31.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: audit 2024-04-06T03:31:30.556976+0000 mon.a (mon.0) 1224 : audit 1 from='client.? 172.21.15.50:0/3273188930' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:31:31.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.568594+0000 mon.a (mon.0) 1225 : cluster 0 osdmap e203: 8 total, 8 up, 8 in 2024-04-06T03:31:31.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: cluster 2024-04-06T03:31:30.594252+0000 osd.1 (osd.1) 211 : cluster 0 8.1 scrub starts 2024-04-06T03:31:31.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: audit 2024-04-06T03:31:30.810012+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.50:0/4167575225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:31:31.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:31 smithi153 bash[23852]: audit 2024-04-06T03:31:30.814605+0000 mon.a (mon.0) 1226 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:31:31.610 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:31.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:31.740 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:31.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:31.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:31.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:31.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:31.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.016311+0000 osd.2 (osd.2) 213 : cluster 0 6.f scrub starts 2024-04-06T03:31:31.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.119722+0000 osd.2 (osd.2) 214 : cluster 0 6.f scrub ok 2024-04-06T03:31:31.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.149540+0000 osd.0 (osd.0) 226 : cluster 0 5.3 scrub starts 2024-04-06T03:31:31.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.157106+0000 osd.0 (osd.0) 227 : cluster 0 5.3 scrub ok 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.179918+0000 mgr.y (mgr.24368) 390 : cluster 0 pgmap v398: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 59 MiB/s wr, 946 op/s 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.545080+0000 osd.4 (osd.4) 256 : cluster 0 8.2 scrub ok 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: audit 2024-04-06T03:31:30.556976+0000 mon.a (mon.0) 1224 : audit 1 from='client.? 172.21.15.50:0/3273188930' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.568594+0000 mon.a (mon.0) 1225 : cluster 0 osdmap e203: 8 total, 8 up, 8 in 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: cluster 2024-04-06T03:31:30.594252+0000 osd.1 (osd.1) 211 : cluster 0 8.1 scrub starts 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: audit 2024-04-06T03:31:30.810012+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.50:0/4167575225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[21467]: audit 2024-04-06T03:31:30.814605+0000 mon.a (mon.0) 1226 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.016311+0000 osd.2 (osd.2) 213 : cluster 0 6.f scrub starts 2024-04-06T03:31:31.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.119722+0000 osd.2 (osd.2) 214 : cluster 0 6.f scrub ok 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.149540+0000 osd.0 (osd.0) 226 : cluster 0 5.3 scrub starts 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.157106+0000 osd.0 (osd.0) 227 : cluster 0 5.3 scrub ok 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.179918+0000 mgr.y (mgr.24368) 390 : cluster 0 pgmap v398: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 59 MiB/s wr, 946 op/s 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.545080+0000 osd.4 (osd.4) 256 : cluster 0 8.2 scrub ok 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: audit 2024-04-06T03:31:30.556976+0000 mon.a (mon.0) 1224 : audit 1 from='client.? 172.21.15.50:0/3273188930' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.568594+0000 mon.a (mon.0) 1225 : cluster 0 osdmap e203: 8 total, 8 up, 8 in 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: cluster 2024-04-06T03:31:30.594252+0000 osd.1 (osd.1) 211 : cluster 0 8.1 scrub starts 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: audit 2024-04-06T03:31:30.810012+0000 mon.b (mon.1) 38 : audit 1 from='client.? 172.21.15.50:0/4167575225' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:31:31.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:31 smithi050 bash[29221]: audit 2024-04-06T03:31:30.814605+0000 mon.a (mon.0) 1226 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:31:32.002 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:32.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:32.132 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:32.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:32.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:32.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:32.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:32.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:32.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:32.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:32.675 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:32.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: cluster 2024-04-06T03:31:31.125662+0000 osd.0 (osd.0) 228 : cluster 0 8.9 scrub starts 2024-04-06T03:31:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: cluster 2024-04-06T03:31:31.232971+0000 osd.5 (osd.5) 129 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: cluster 2024-04-06T03:31:31.233033+0000 osd.5 (osd.5) 130 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: cluster 2024-04-06T03:31:31.557392+0000 mon.a (mon.0) 1227 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:31:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: cluster 2024-04-06T03:31:31.557442+0000 mon.a (mon.0) 1228 : cluster 1 Cluster is now healthy 2024-04-06T03:31:32.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: audit 2024-04-06T03:31:31.568112+0000 mon.a (mon.0) 1229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[21467]: cluster 2024-04-06T03:31:31.584474+0000 mon.a (mon.0) 1230 : cluster 0 osdmap e204: 8 total, 8 up, 8 in 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: cluster 2024-04-06T03:31:31.125662+0000 osd.0 (osd.0) 228 : cluster 0 8.9 scrub starts 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: cluster 2024-04-06T03:31:31.232971+0000 osd.5 (osd.5) 129 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: cluster 2024-04-06T03:31:31.233033+0000 osd.5 (osd.5) 130 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: cluster 2024-04-06T03:31:31.557392+0000 mon.a (mon.0) 1227 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: cluster 2024-04-06T03:31:31.557442+0000 mon.a (mon.0) 1228 : cluster 1 Cluster is now healthy 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: audit 2024-04-06T03:31:31.568112+0000 mon.a (mon.0) 1229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:31:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:32 smithi050 bash[29221]: cluster 2024-04-06T03:31:31.584474+0000 mon.a (mon.0) 1230 : cluster 0 osdmap e204: 8 total, 8 up, 8 in 2024-04-06T03:31:32.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:32.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:32.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: cluster 2024-04-06T03:31:31.125662+0000 osd.0 (osd.0) 228 : cluster 0 8.9 scrub starts 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: cluster 2024-04-06T03:31:31.232971+0000 osd.5 (osd.5) 129 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: cluster 2024-04-06T03:31:31.233033+0000 osd.5 (osd.5) 130 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: cluster 2024-04-06T03:31:31.557392+0000 mon.a (mon.0) 1227 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: cluster 2024-04-06T03:31:31.557442+0000 mon.a (mon.0) 1228 : cluster 1 Cluster is now healthy 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: audit 2024-04-06T03:31:31.568112+0000 mon.a (mon.0) 1229 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:31:32.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:32 smithi153 bash[23852]: cluster 2024-04-06T03:31:31.584474+0000 mon.a (mon.0) 1230 : cluster 0 osdmap e204: 8 total, 8 up, 8 in 2024-04-06T03:31:32.926 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:32.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:32.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:33.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:33.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:33.189 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:33.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:33.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:33.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:33.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:33.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:33.574 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:33.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:33.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:33.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:33.755 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:33 smithi050 bash[21467]: cluster 2024-04-06T03:31:32.180389+0000 mgr.y (mgr.24368) 391 : cluster 0 pgmap v401: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 943 op/s 2024-04-06T03:31:33.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:33 smithi050 bash[21467]: cluster 2024-04-06T03:31:32.340057+0000 osd.3 (osd.3) 249 : cluster 0 6.6 scrub starts 2024-04-06T03:31:33.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:33 smithi050 bash[21467]: cluster 2024-04-06T03:31:32.350008+0000 osd.3 (osd.3) 250 : cluster 0 6.6 scrub ok 2024-04-06T03:31:33.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:33 smithi050 bash[29221]: cluster 2024-04-06T03:31:32.180389+0000 mgr.y (mgr.24368) 391 : cluster 0 pgmap v401: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 943 op/s 2024-04-06T03:31:33.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:33 smithi050 bash[29221]: cluster 2024-04-06T03:31:32.340057+0000 osd.3 (osd.3) 249 : cluster 0 6.6 scrub starts 2024-04-06T03:31:33.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:33 smithi050 bash[29221]: cluster 2024-04-06T03:31:32.350008+0000 osd.3 (osd.3) 250 : cluster 0 6.6 scrub ok 2024-04-06T03:31:33.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:33.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:33.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:33.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:33.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:33.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:34.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:33 smithi153 bash[23852]: cluster 2024-04-06T03:31:32.180389+0000 mgr.y (mgr.24368) 391 : cluster 0 pgmap v401: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 943 op/s 2024-04-06T03:31:34.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:33 smithi153 bash[23852]: cluster 2024-04-06T03:31:32.340057+0000 osd.3 (osd.3) 249 : cluster 0 6.6 scrub starts 2024-04-06T03:31:34.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:33 smithi153 bash[23852]: cluster 2024-04-06T03:31:32.350008+0000 osd.3 (osd.3) 250 : cluster 0 6.6 scrub ok 2024-04-06T03:31:34.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:34.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:34.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:34.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:34.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:34.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:34.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:34.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:34.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.294770+0000 osd.3 (osd.3) 251 : cluster 0 6.c scrub starts 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.297036+0000 osd.3 (osd.3) 252 : cluster 0 6.c scrub ok 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.567230+0000 osd.1 (osd.1) 212 : cluster 0 3.19 scrub starts 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.606068+0000 osd.1 (osd.1) 213 : cluster 0 3.19 scrub ok 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.617130+0000 osd.0 (osd.0) 229 : cluster 0 8.9 scrub ok 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: audit 2024-04-06T03:31:34.064596+0000 mon.a (mon.0) 1231 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:34.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[21467]: audit 2024-04-06T03:31:34.065412+0000 mon.a (mon.0) 1232 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.294770+0000 osd.3 (osd.3) 251 : cluster 0 6.c scrub starts 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.297036+0000 osd.3 (osd.3) 252 : cluster 0 6.c scrub ok 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.567230+0000 osd.1 (osd.1) 212 : cluster 0 3.19 scrub starts 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.606068+0000 osd.1 (osd.1) 213 : cluster 0 3.19 scrub ok 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.617130+0000 osd.0 (osd.0) 229 : cluster 0 8.9 scrub ok 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: audit 2024-04-06T03:31:34.064596+0000 mon.a (mon.0) 1231 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:34.717 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:34 smithi050 bash[29221]: audit 2024-04-06T03:31:34.065412+0000 mon.a (mon.0) 1232 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:34.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:34.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:34.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:34.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.294770+0000 osd.3 (osd.3) 251 : cluster 0 6.c scrub starts 2024-04-06T03:31:34.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.297036+0000 osd.3 (osd.3) 252 : cluster 0 6.c scrub ok 2024-04-06T03:31:34.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.567230+0000 osd.1 (osd.1) 212 : cluster 0 3.19 scrub starts 2024-04-06T03:31:34.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.606068+0000 osd.1 (osd.1) 213 : cluster 0 3.19 scrub ok 2024-04-06T03:31:34.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.617130+0000 osd.0 (osd.0) 229 : cluster 0 8.9 scrub ok 2024-04-06T03:31:34.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: audit 2024-04-06T03:31:34.064596+0000 mon.a (mon.0) 1231 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:31:34.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:34 smithi153 bash[23852]: audit 2024-04-06T03:31:34.065412+0000 mon.a (mon.0) 1232 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:34.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:34.890 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:34.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:34.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:35.016 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:35.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:35.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:35.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:35.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:35.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:35.414 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:35.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:35.553 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:35.654 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.862539+0000 osd.3 (osd.3) 253 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:33.862634+0000 osd.3 (osd.3) 254 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:34.180945+0000 mgr.y (mgr.24368) 392 : cluster 0 pgmap v402: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 646 op/s 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:34.476300+0000 osd.1 (osd.1) 214 : cluster 0 8.1 scrub ok 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:34.516287+0000 osd.1 (osd.1) 215 : cluster 0 2.15 scrub starts 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:34.523632+0000 osd.1 (osd.1) 216 : cluster 0 2.15 scrub ok 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:34.586563+0000 mon.a (mon.0) 1233 : cluster 0 osdmap e205: 8 total, 8 up, 8 in 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:35.087524+0000 osd.7 (osd.7) 251 : cluster 0 2.7 scrub starts 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[21467]: cluster 2024-04-06T03:31:35.090549+0000 osd.7 (osd.7) 252 : cluster 0 2.7 scrub ok 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.862539+0000 osd.3 (osd.3) 253 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:35.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:33.862634+0000 osd.3 (osd.3) 254 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:34.180945+0000 mgr.y (mgr.24368) 392 : cluster 0 pgmap v402: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 646 op/s 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:34.476300+0000 osd.1 (osd.1) 214 : cluster 0 8.1 scrub ok 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:34.516287+0000 osd.1 (osd.1) 215 : cluster 0 2.15 scrub starts 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:34.523632+0000 osd.1 (osd.1) 216 : cluster 0 2.15 scrub ok 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:34.586563+0000 mon.a (mon.0) 1233 : cluster 0 osdmap e205: 8 total, 8 up, 8 in 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:35.087524+0000 osd.7 (osd.7) 251 : cluster 0 2.7 scrub starts 2024-04-06T03:31:35.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:35 smithi050 bash[29221]: cluster 2024-04-06T03:31:35.090549+0000 osd.7 (osd.7) 252 : cluster 0 2.7 scrub ok 2024-04-06T03:31:35.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:35.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:35.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:35.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:35.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.862539+0000 osd.3 (osd.3) 253 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:33.862634+0000 osd.3 (osd.3) 254 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:34.180945+0000 mgr.y (mgr.24368) 392 : cluster 0 pgmap v402: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 646 op/s 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:34.476300+0000 osd.1 (osd.1) 214 : cluster 0 8.1 scrub ok 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:34.516287+0000 osd.1 (osd.1) 215 : cluster 0 2.15 scrub starts 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:34.523632+0000 osd.1 (osd.1) 216 : cluster 0 2.15 scrub ok 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:34.586563+0000 mon.a (mon.0) 1233 : cluster 0 osdmap e205: 8 total, 8 up, 8 in 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:35.087524+0000 osd.7 (osd.7) 251 : cluster 0 2.7 scrub starts 2024-04-06T03:31:35.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:35 smithi153 bash[23852]: cluster 2024-04-06T03:31:35.090549+0000 osd.7 (osd.7) 252 : cluster 0 2.7 scrub ok 2024-04-06T03:31:35.954 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:35.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:36.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:36.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:36.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:36.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:36.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:36.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:36.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:36.470 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:36.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:36.602 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:36.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:36.715 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:36 smithi153 bash[23852]: cluster 2024-04-06T03:31:35.382103+0000 osd.3 (osd.3) 255 : cluster 0 5.13 scrub starts 2024-04-06T03:31:36.715 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:36 smithi153 bash[23852]: cluster 2024-04-06T03:31:35.385364+0000 osd.3 (osd.3) 256 : cluster 0 5.13 scrub ok 2024-04-06T03:31:36.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:36.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:36.868 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:36.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:36.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:36.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:36 smithi050 bash[21467]: cluster 2024-04-06T03:31:35.382103+0000 osd.3 (osd.3) 255 : cluster 0 5.13 scrub starts 2024-04-06T03:31:36.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:36 smithi050 bash[21467]: cluster 2024-04-06T03:31:35.385364+0000 osd.3 (osd.3) 256 : cluster 0 5.13 scrub ok 2024-04-06T03:31:36.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:36 smithi050 bash[29221]: cluster 2024-04-06T03:31:35.382103+0000 osd.3 (osd.3) 255 : cluster 0 5.13 scrub starts 2024-04-06T03:31:36.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:36 smithi050 bash[29221]: cluster 2024-04-06T03:31:35.385364+0000 osd.3 (osd.3) 256 : cluster 0 5.13 scrub ok 2024-04-06T03:31:36.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:37.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:37.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:37.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:37.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:37.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:37.390 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:37.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:37.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:37.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 4 1 2024-04-06T03:31:37.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:37.658 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:37.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:37.788 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:37.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:37.900 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:37 smithi153 bash[23852]: cluster 2024-04-06T03:31:36.181633+0000 mgr.y (mgr.24368) 393 : cluster 0 pgmap v404: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 13 MiB/s wr, 874 op/s 2024-04-06T03:31:37.900 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:37 smithi153 bash[23852]: cluster 2024-04-06T03:31:36.375371+0000 osd.0 (osd.0) 230 : cluster 0 2.1f scrub starts 2024-04-06T03:31:37.900 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:37 smithi153 bash[23852]: cluster 2024-04-06T03:31:36.383270+0000 osd.0 (osd.0) 231 : cluster 0 2.1f scrub ok 2024-04-06T03:31:37.900 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:37 smithi153 bash[23852]: cluster 2024-04-06T03:31:37.051561+0000 osd.7 (osd.7) 253 : cluster 0 1.0 scrub starts 2024-04-06T03:31:37.900 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:37 smithi153 bash[23852]: cluster 2024-04-06T03:31:37.056272+0000 osd.7 (osd.7) 254 : cluster 0 1.0 scrub ok 2024-04-06T03:31:37.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:37.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:37.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[21467]: cluster 2024-04-06T03:31:36.181633+0000 mgr.y (mgr.24368) 393 : cluster 0 pgmap v404: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 13 MiB/s wr, 874 op/s 2024-04-06T03:31:37.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[21467]: cluster 2024-04-06T03:31:36.375371+0000 osd.0 (osd.0) 230 : cluster 0 2.1f scrub starts 2024-04-06T03:31:37.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[21467]: cluster 2024-04-06T03:31:36.383270+0000 osd.0 (osd.0) 231 : cluster 0 2.1f scrub ok 2024-04-06T03:31:37.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[21467]: cluster 2024-04-06T03:31:37.051561+0000 osd.7 (osd.7) 253 : cluster 0 1.0 scrub starts 2024-04-06T03:31:37.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[21467]: cluster 2024-04-06T03:31:37.056272+0000 osd.7 (osd.7) 254 : cluster 0 1.0 scrub ok 2024-04-06T03:31:37.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[29221]: cluster 2024-04-06T03:31:36.181633+0000 mgr.y (mgr.24368) 393 : cluster 0 pgmap v404: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.8 KiB/s rd, 13 MiB/s wr, 874 op/s 2024-04-06T03:31:37.978 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[29221]: cluster 2024-04-06T03:31:36.375371+0000 osd.0 (osd.0) 230 : cluster 0 2.1f scrub starts 2024-04-06T03:31:37.978 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[29221]: cluster 2024-04-06T03:31:36.383270+0000 osd.0 (osd.0) 231 : cluster 0 2.1f scrub ok 2024-04-06T03:31:37.978 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[29221]: cluster 2024-04-06T03:31:37.051561+0000 osd.7 (osd.7) 253 : cluster 0 1.0 scrub starts 2024-04-06T03:31:37.978 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:37 smithi050 bash[29221]: cluster 2024-04-06T03:31:37.056272+0000 osd.7 (osd.7) 254 : cluster 0 1.0 scrub ok 2024-04-06T03:31:38.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:38.039 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:38.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:38.164 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113847 8.e 8:76f1f53f:::benchmark_data_smithi050_86143_object7536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.210463+0000", 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.019935010999999999, 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019967487999999999, 2024-04-06T03:31:38.226 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113847 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210463+0000", 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210463+0000", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210464+0000", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2200000000000003e-07 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210467+0000", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1590000000000002e-06 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210473+0000", 2024-04-06T03:31:38.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5939999999999999e-06 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210478+0000", 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.206e-06 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210691+0000", 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021333100000000001 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210821+0000", 2024-04-06T03:31:38.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012977999999999999 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.210949+0000", 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012784 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214454+0000", 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035049959999999998 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214467+0000", 2024-04-06T03:31:38.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3637e-05 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113854 8.9 8:9594b725:::benchmark_data_smithi050_86143_object7543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.212953+0000", 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.017445138999999998, 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017581067999999998, 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.231 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113854 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.212953+0000", 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.212953+0000", 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.212954+0000", 2024-04-06T03:31:38.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.009e-06 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.212959+0000", 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0640000000000001e-06 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.212973+0000", 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3817e-05 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.212982+0000", 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3000000000000007e-06 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.213095+0000", 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011289 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.213226+0000", 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013121300000000001 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.213521+0000", 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029475900000000001 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228812+0000", 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015291124 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113856 8.e 8.a264c40e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.214776+0000", 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.015621998999999999, 2024-04-06T03:31:38.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015844924999999999, 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113856 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214776+0000", 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214776+0000", 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214777+0000", 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0079999999999999e-06 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214783+0000", 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6969999999999998e-06 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214795+0000", 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2456999999999999e-05 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.214804+0000", 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9910000000000008e-06 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14928.0:113854 8.9 e205/182)", 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.215924+0000", 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.014473626, 2024-04-06T03:31:38.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014754176000000001, 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.215924+0000", 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.215924+0000", 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.215925+0000", 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9400000000000005e-07 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.215929+0000", 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1980000000000003e-06 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.215939+0000", 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9939999999999995e-06 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.215946+0000", 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9399999999999996e-06 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14928.0:113855 8.1 e205/182 8:8cf2a991:::benchmark_data_smithi050_86143_object7544:head v 205'7106, mlcod=205'7104)", 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.216467+0000", 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.013930517, 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014267771, 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.216467+0000", 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.216467+0000", 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222621+0000", 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061537110000000001 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222675+0000", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3903000000000002e-05 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222759+0000", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4175e-05 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222770+0000", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0912e-05 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228842+0000", 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060724719999999998 2024-04-06T03:31:38.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228862+0000", 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9123e-05 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113858 8.c 8:33ce41f4:::benchmark_data_smithi050_86143_object7547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.222826+0000", 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0075722109999999997, 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079823189999999999, 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:31:38.244 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113858 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222826+0000", 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222826+0000", 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222827+0000", 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0529999999999999e-06 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222831+0000", 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2610000000000004e-06 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222841+0000", 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8060000000000006e-06 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222849+0000", 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0539999999999995e-06 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222908+0000", 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8925000000000002e-05 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.223018+0000", 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011058200000000001 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.223138+0000", 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119633 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.226187+0000", 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030488569999999999 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113859 8.c 8:3518d439:::benchmark_data_smithi050_86143_object7548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.248 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.222869+0000", 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0075285619999999999, 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080283980000000008, 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113859 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222869+0000", 2024-04-06T03:31:38.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222869+0000", 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222870+0000", 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6699999999999999e-07 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222872+0000", 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.385e-06 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222878+0000", 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2500000000000003e-06 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222885+0000", 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6730000000000003e-06 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:31:38.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.226219+0000", 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033341109999999998 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.226331+0000", 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112176 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.226444+0000", 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113152 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.252 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113862 8.6 8.7001efb6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.222895+0000", 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0075023260000000001, 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080848729999999994, 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113862 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222895+0000", 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222895+0000", 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222896+0000", 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6899999999999998e-07 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.254 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222899+0000", 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9160000000000001e-06 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222906+0000", 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1579999999999999e-06 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222912+0000", 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4099999999999996e-06 2024-04-06T03:31:38.255 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113863 8.c 8:3bb5279a:::benchmark_data_smithi050_86143_object7552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.222921+0000", 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0074764300000000001, 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081207950000000001, 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113863 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222921+0000", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222921+0000", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222922+0000", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3900000000000005e-07 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222925+0000", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6910000000000002e-06 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222933+0000", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6009999999999995e-06 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.222939+0000", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8610000000000003e-06 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.229524+0000", 2024-04-06T03:31:38.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065847579999999996 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.229642+0000", 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118508 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.229759+0000", 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116507 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14928.0:113847 8.e e205/182)", 2024-04-06T03:31:38.259 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.223007+0000", 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.007390359, 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081100500000000006, 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.223007+0000", 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.223007+0000", 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.223008+0000", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4000000000000002e-07 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.223012+0000", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8190000000000002e-06 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.224602+0000", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015906970000000001 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.224612+0000", 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6700000000000006e-06 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.261 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14928.0:113859 8.c e205/182)", 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.228474+0000", 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.001923534, 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026931659999999999, 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228474+0000", 2024-04-06T03:31:38.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228474+0000", 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228475+0000", 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9299999999999996e-07 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228480+0000", 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9590000000000003e-06 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228494+0000", 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4250000000000001e-05 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.228501+0000", 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8059999999999997e-06 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:113865 8.c 8.76097cec (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:31:38.230765+0000", 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9996328, 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000455017, 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 113865 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:31:38.265 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.230765+0000", 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.230765+0000", 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.230766+0000", 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3459999999999999e-06 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.230773+0000", 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4919999999999998e-06 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.230789+0000", 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6242000000000001e-05 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:31:38.230798+0000", 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.482e-06 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:31:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:31:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:31:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 12 2024-04-06T03:31:38.268 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:31:38.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:38.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:38.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:38.421 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:38.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:38.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:38.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:38.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:38 smithi153 bash[23852]: cluster 2024-04-06T03:31:37.180144+0000 osd.4 (osd.4) 257 : cluster 0 5.1c scrub starts 2024-04-06T03:31:38.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:38 smithi153 bash[23852]: cluster 2024-04-06T03:31:37.182120+0000 osd.4 (osd.4) 258 : cluster 0 5.1c scrub ok 2024-04-06T03:31:38.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:38 smithi153 bash[23852]: cluster 2024-04-06T03:31:37.390394+0000 osd.0 (osd.0) 232 : cluster 0 8.e scrub starts 2024-04-06T03:31:38.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:38 smithi153 bash[23852]: cluster 2024-04-06T03:31:38.050260+0000 osd.7 (osd.7) 255 : cluster 0 2.0 scrub starts 2024-04-06T03:31:38.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:38 smithi153 bash[23852]: cluster 2024-04-06T03:31:38.054093+0000 osd.7 (osd.7) 256 : cluster 0 2.0 scrub ok 2024-04-06T03:31:38.704 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:38.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:38.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:38.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[29221]: cluster 2024-04-06T03:31:37.180144+0000 osd.4 (osd.4) 257 : cluster 0 5.1c scrub starts 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[29221]: cluster 2024-04-06T03:31:37.182120+0000 osd.4 (osd.4) 258 : cluster 0 5.1c scrub ok 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[29221]: cluster 2024-04-06T03:31:37.390394+0000 osd.0 (osd.0) 232 : cluster 0 8.e scrub starts 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[29221]: cluster 2024-04-06T03:31:38.050260+0000 osd.7 (osd.7) 255 : cluster 0 2.0 scrub starts 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[29221]: cluster 2024-04-06T03:31:38.054093+0000 osd.7 (osd.7) 256 : cluster 0 2.0 scrub ok 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[21467]: cluster 2024-04-06T03:31:37.180144+0000 osd.4 (osd.4) 257 : cluster 0 5.1c scrub starts 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[21467]: cluster 2024-04-06T03:31:37.182120+0000 osd.4 (osd.4) 258 : cluster 0 5.1c scrub ok 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[21467]: cluster 2024-04-06T03:31:37.390394+0000 osd.0 (osd.0) 232 : cluster 0 8.e scrub starts 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[21467]: cluster 2024-04-06T03:31:38.050260+0000 osd.7 (osd.7) 255 : cluster 0 2.0 scrub starts 2024-04-06T03:31:38.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:38 smithi050 bash[21467]: cluster 2024-04-06T03:31:38.054093+0000 osd.7 (osd.7) 256 : cluster 0 2.0 scrub ok 2024-04-06T03:31:38.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:38.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:39.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:39.092 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:39.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:39.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:39.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:39.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:39.464 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:39.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:39.485 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:31:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:31:39.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:39.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:39.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:31:39.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:39.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:31:39.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:31:39.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[23852]: cluster 2024-04-06T03:31:38.071619+0000 osd.2 (osd.2) 215 : cluster 0 5.b scrub starts 2024-04-06T03:31:39.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[23852]: cluster 2024-04-06T03:31:38.075774+0000 osd.2 (osd.2) 216 : cluster 0 5.b scrub ok 2024-04-06T03:31:39.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[23852]: cluster 2024-04-06T03:31:38.182074+0000 mgr.y (mgr.24368) 394 : cluster 0 pgmap v405: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 806 B/s rd, 840 KiB/s wr, 512 op/s 2024-04-06T03:31:39.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[23852]: cluster 2024-04-06T03:31:38.469440+0000 osd.0 (osd.0) 233 : cluster 0 8.6 deep-scrub starts 2024-04-06T03:31:39.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[23852]: cluster 2024-04-06T03:31:39.067124+0000 osd.7 (osd.7) 257 : cluster 0 3.15 scrub starts 2024-04-06T03:31:39.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:39 smithi153 bash[23852]: cluster 2024-04-06T03:31:39.075694+0000 osd.7 (osd.7) 258 : cluster 0 3.15 scrub ok 2024-04-06T03:31:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:39.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:39.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21467]: cluster 2024-04-06T03:31:38.071619+0000 osd.2 (osd.2) 215 : cluster 0 5.b scrub starts 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21467]: cluster 2024-04-06T03:31:38.075774+0000 osd.2 (osd.2) 216 : cluster 0 5.b scrub ok 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21467]: cluster 2024-04-06T03:31:38.182074+0000 mgr.y (mgr.24368) 394 : cluster 0 pgmap v405: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 806 B/s rd, 840 KiB/s wr, 512 op/s 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21467]: cluster 2024-04-06T03:31:38.469440+0000 osd.0 (osd.0) 233 : cluster 0 8.6 deep-scrub starts 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21467]: cluster 2024-04-06T03:31:39.067124+0000 osd.7 (osd.7) 257 : cluster 0 3.15 scrub starts 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[21467]: cluster 2024-04-06T03:31:39.075694+0000 osd.7 (osd.7) 258 : cluster 0 3.15 scrub ok 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[29221]: cluster 2024-04-06T03:31:38.071619+0000 osd.2 (osd.2) 215 : cluster 0 5.b scrub starts 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[29221]: cluster 2024-04-06T03:31:38.075774+0000 osd.2 (osd.2) 216 : cluster 0 5.b scrub ok 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[29221]: cluster 2024-04-06T03:31:38.182074+0000 mgr.y (mgr.24368) 394 : cluster 0 pgmap v405: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 806 B/s rd, 840 KiB/s wr, 512 op/s 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[29221]: cluster 2024-04-06T03:31:38.469440+0000 osd.0 (osd.0) 233 : cluster 0 8.6 deep-scrub starts 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[29221]: cluster 2024-04-06T03:31:39.067124+0000 osd.7 (osd.7) 257 : cluster 0 3.15 scrub starts 2024-04-06T03:31:39.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:39 smithi050 bash[29221]: cluster 2024-04-06T03:31:39.075694+0000 osd.7 (osd.7) 258 : cluster 0 3.15 scrub ok 2024-04-06T03:31:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:39.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:39.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:40.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:40.144 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:40.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:40.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-06T03:31:40.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:40.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:40.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:40.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:40.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:40.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:40.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:40.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:40 smithi050 bash[21467]: cluster 2024-04-06T03:31:39.457105+0000 osd.3 (osd.3) 257 : cluster 0 3.5 scrub starts 2024-04-06T03:31:40.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:40 smithi050 bash[21467]: cluster 2024-04-06T03:31:39.459035+0000 osd.3 (osd.3) 258 : cluster 0 3.5 scrub ok 2024-04-06T03:31:40.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:40 smithi050 bash[21467]: audit 2024-04-06T03:31:39.492580+0000 mgr.y (mgr.24368) 395 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:40.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:40 smithi050 bash[29221]: cluster 2024-04-06T03:31:39.457105+0000 osd.3 (osd.3) 257 : cluster 0 3.5 scrub starts 2024-04-06T03:31:40.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:40 smithi050 bash[29221]: cluster 2024-04-06T03:31:39.459035+0000 osd.3 (osd.3) 258 : cluster 0 3.5 scrub ok 2024-04-06T03:31:40.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:40 smithi050 bash[29221]: audit 2024-04-06T03:31:39.492580+0000 mgr.y (mgr.24368) 395 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:40.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:40.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:40.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:40 smithi153 bash[23852]: cluster 2024-04-06T03:31:39.457105+0000 osd.3 (osd.3) 257 : cluster 0 3.5 scrub starts 2024-04-06T03:31:40.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:40 smithi153 bash[23852]: cluster 2024-04-06T03:31:39.459035+0000 osd.3 (osd.3) 258 : cluster 0 3.5 scrub ok 2024-04-06T03:31:40.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:40 smithi153 bash[23852]: audit 2024-04-06T03:31:39.492580+0000 mgr.y (mgr.24368) 395 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:40.933 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:40.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:41.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:41.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:41.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:41.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:41.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:41.460 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:41.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:41.596 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:41.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:41.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:41 smithi153 bash[23852]: cluster 2024-04-06T03:31:40.096532+0000 osd.2 (osd.2) 217 : cluster 0 5.8 scrub starts 2024-04-06T03:31:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:41 smithi153 bash[23852]: cluster 2024-04-06T03:31:40.104967+0000 osd.2 (osd.2) 218 : cluster 0 5.8 scrub ok 2024-04-06T03:31:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:41 smithi153 bash[23852]: cluster 2024-04-06T03:31:40.182716+0000 mgr.y (mgr.24368) 396 : cluster 0 pgmap v406: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 713 B/s rd, 743 KiB/s wr, 828 op/s 2024-04-06T03:31:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:41 smithi153 bash[23852]: cluster 2024-04-06T03:31:40.516360+0000 osd.1 (osd.1) 217 : cluster 0 2.d deep-scrub starts 2024-04-06T03:31:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:41 smithi153 bash[23852]: cluster 2024-04-06T03:31:40.521788+0000 osd.1 (osd.1) 218 : cluster 0 2.d deep-scrub ok 2024-04-06T03:31:41.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:41.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:41.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:41.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[21467]: cluster 2024-04-06T03:31:40.096532+0000 osd.2 (osd.2) 217 : cluster 0 5.8 scrub starts 2024-04-06T03:31:41.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[21467]: cluster 2024-04-06T03:31:40.104967+0000 osd.2 (osd.2) 218 : cluster 0 5.8 scrub ok 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[21467]: cluster 2024-04-06T03:31:40.182716+0000 mgr.y (mgr.24368) 396 : cluster 0 pgmap v406: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 713 B/s rd, 743 KiB/s wr, 828 op/s 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[21467]: cluster 2024-04-06T03:31:40.516360+0000 osd.1 (osd.1) 217 : cluster 0 2.d deep-scrub starts 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[21467]: cluster 2024-04-06T03:31:40.521788+0000 osd.1 (osd.1) 218 : cluster 0 2.d deep-scrub ok 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[29221]: cluster 2024-04-06T03:31:40.096532+0000 osd.2 (osd.2) 217 : cluster 0 5.8 scrub starts 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[29221]: cluster 2024-04-06T03:31:40.104967+0000 osd.2 (osd.2) 218 : cluster 0 5.8 scrub ok 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[29221]: cluster 2024-04-06T03:31:40.182716+0000 mgr.y (mgr.24368) 396 : cluster 0 pgmap v406: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 713 B/s rd, 743 KiB/s wr, 828 op/s 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[29221]: cluster 2024-04-06T03:31:40.516360+0000 osd.1 (osd.1) 217 : cluster 0 2.d deep-scrub starts 2024-04-06T03:31:41.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:41 smithi050 bash[29221]: cluster 2024-04-06T03:31:40.521788+0000 osd.1 (osd.1) 218 : cluster 0 2.d deep-scrub ok 2024-04-06T03:31:41.987 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:41.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:42.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:42.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:42.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:42.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:42.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:42.392 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:42.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:42.520 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:42.623 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:42.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:42 smithi153 bash[23852]: cluster 2024-04-06T03:31:41.071501+0000 osd.2 (osd.2) 219 : cluster 0 2.1 scrub starts 2024-04-06T03:31:42.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:42 smithi153 bash[23852]: cluster 2024-04-06T03:31:41.116049+0000 osd.2 (osd.2) 220 : cluster 0 2.1 scrub ok 2024-04-06T03:31:42.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:42.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:42.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:42.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:42.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:42 smithi050 bash[21467]: cluster 2024-04-06T03:31:41.071501+0000 osd.2 (osd.2) 219 : cluster 0 2.1 scrub starts 2024-04-06T03:31:42.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:42 smithi050 bash[21467]: cluster 2024-04-06T03:31:41.116049+0000 osd.2 (osd.2) 220 : cluster 0 2.1 scrub ok 2024-04-06T03:31:42.908 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:42 smithi050 bash[29221]: cluster 2024-04-06T03:31:41.071501+0000 osd.2 (osd.2) 219 : cluster 0 2.1 scrub starts 2024-04-06T03:31:42.908 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:42 smithi050 bash[29221]: cluster 2024-04-06T03:31:41.116049+0000 osd.2 (osd.2) 220 : cluster 0 2.1 scrub ok 2024-04-06T03:31:42.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:42.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:42.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:43.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:43.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:43.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:43.194 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:43.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:43.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:43.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:43.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:43.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:43.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:43.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:43.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:43.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:43 smithi153 bash[23852]: cluster 2024-04-06T03:31:42.085309+0000 osd.2 (osd.2) 221 : cluster 0 5.6 scrub starts 2024-04-06T03:31:43.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:43 smithi153 bash[23852]: cluster 2024-04-06T03:31:42.091203+0000 osd.2 (osd.2) 222 : cluster 0 5.6 scrub ok 2024-04-06T03:31:43.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:43 smithi153 bash[23852]: cluster 2024-04-06T03:31:42.183316+0000 mgr.y (mgr.24368) 397 : cluster 0 pgmap v407: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 640 KiB/s wr, 854 op/s 2024-04-06T03:31:43.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:43 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.039444+0000 osd.7 (osd.7) 259 : cluster 0 2.8 scrub starts 2024-04-06T03:31:43.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:43 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.040835+0000 osd.7 (osd.7) 260 : cluster 0 2.8 scrub ok 2024-04-06T03:31:43.803 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:43.803 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:43.805 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:43.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:43.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[21467]: cluster 2024-04-06T03:31:42.085309+0000 osd.2 (osd.2) 221 : cluster 0 5.6 scrub starts 2024-04-06T03:31:43.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[21467]: cluster 2024-04-06T03:31:42.091203+0000 osd.2 (osd.2) 222 : cluster 0 5.6 scrub ok 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[21467]: cluster 2024-04-06T03:31:42.183316+0000 mgr.y (mgr.24368) 397 : cluster 0 pgmap v407: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 640 KiB/s wr, 854 op/s 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.039444+0000 osd.7 (osd.7) 259 : cluster 0 2.8 scrub starts 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.040835+0000 osd.7 (osd.7) 260 : cluster 0 2.8 scrub ok 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[29221]: cluster 2024-04-06T03:31:42.085309+0000 osd.2 (osd.2) 221 : cluster 0 5.6 scrub starts 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[29221]: cluster 2024-04-06T03:31:42.091203+0000 osd.2 (osd.2) 222 : cluster 0 5.6 scrub ok 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[29221]: cluster 2024-04-06T03:31:42.183316+0000 mgr.y (mgr.24368) 397 : cluster 0 pgmap v407: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 640 KiB/s wr, 854 op/s 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.039444+0000 osd.7 (osd.7) 259 : cluster 0 2.8 scrub starts 2024-04-06T03:31:43.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:43 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.040835+0000 osd.7 (osd.7) 260 : cluster 0 2.8 scrub ok 2024-04-06T03:31:43.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:43.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:43.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:44.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:44.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:44.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:44.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:44.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:44.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:44.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:44.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:44.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:44.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:44.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:44.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:44.822 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:31:44.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.339442+0000 osd.0 (osd.0) 234 : cluster 0 8.c scrub starts 2024-04-06T03:31:44.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.414604+0000 osd.0 (osd.0) 235 : cluster 0 8.e scrub ok 2024-04-06T03:31:44.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.416739+0000 osd.3 (osd.3) 259 : cluster 0 6.7 scrub starts 2024-04-06T03:31:44.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.475379+0000 osd.3 (osd.3) 260 : cluster 0 6.7 scrub ok 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.799546+0000 osd.7 (osd.7) 261 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:43.799624+0000 osd.7 (osd.7) 262 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.068543+0000 osd.7 (osd.7) 263 : cluster 0 8.d scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.084638+0000 osd.2 (osd.2) 223 : cluster 0 5.a scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.089462+0000 osd.2 (osd.2) 224 : cluster 0 5.a scrub ok 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.339442+0000 osd.0 (osd.0) 234 : cluster 0 8.c scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.414604+0000 osd.0 (osd.0) 235 : cluster 0 8.e scrub ok 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.416739+0000 osd.3 (osd.3) 259 : cluster 0 6.7 scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.475379+0000 osd.3 (osd.3) 260 : cluster 0 6.7 scrub ok 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.799546+0000 osd.7 (osd.7) 261 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:43.799624+0000 osd.7 (osd.7) 262 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.068543+0000 osd.7 (osd.7) 263 : cluster 0 8.d scrub starts 2024-04-06T03:31:44.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.084638+0000 osd.2 (osd.2) 223 : cluster 0 5.a scrub starts 2024-04-06T03:31:44.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:44 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.089462+0000 osd.2 (osd.2) 224 : cluster 0 5.a scrub ok 2024-04-06T03:31:44.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:44.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:44.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:45.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:45.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.339442+0000 osd.0 (osd.0) 234 : cluster 0 8.c scrub starts 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.414604+0000 osd.0 (osd.0) 235 : cluster 0 8.e scrub ok 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.416739+0000 osd.3 (osd.3) 259 : cluster 0 6.7 scrub starts 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.475379+0000 osd.3 (osd.3) 260 : cluster 0 6.7 scrub ok 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.799546+0000 osd.7 (osd.7) 261 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:43.799624+0000 osd.7 (osd.7) 262 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.068543+0000 osd.7 (osd.7) 263 : cluster 0 8.d scrub starts 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.084638+0000 osd.2 (osd.2) 223 : cluster 0 5.a scrub starts 2024-04-06T03:31:45.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:44 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.089462+0000 osd.2 (osd.2) 224 : cluster 0 5.a scrub ok 2024-04-06T03:31:45.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:45.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:45.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:45.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:45.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:45.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:45.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:45.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:45.582 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:45.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:45.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:45.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:45.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:45.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:45.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:45.960 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.183963+0000 mgr.y (mgr.24368) 398 : cluster 0 pgmap v408: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.433305+0000 osd.1 (osd.1) 219 : cluster 0 5.12 scrub starts 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.434274+0000 osd.1 (osd.1) 220 : cluster 0 5.12 scrub ok 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.436564+0000 osd.3 (osd.3) 261 : cluster 0 5.0 scrub starts 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.439069+0000 osd.3 (osd.3) 262 : cluster 0 5.0 scrub ok 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.640091+0000 mon.a (mon.0) 1234 : cluster 0 osdmap e206: 8 total, 8 up, 8 in 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.095718+0000 osd.7 (osd.7) 264 : cluster 0 2.1c scrub starts 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.099631+0000 osd.7 (osd.7) 265 : cluster 0 2.1c scrub ok 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.183963+0000 mgr.y (mgr.24368) 398 : cluster 0 pgmap v408: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.433305+0000 osd.1 (osd.1) 219 : cluster 0 5.12 scrub starts 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.434274+0000 osd.1 (osd.1) 220 : cluster 0 5.12 scrub ok 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.436564+0000 osd.3 (osd.3) 261 : cluster 0 5.0 scrub starts 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.439069+0000 osd.3 (osd.3) 262 : cluster 0 5.0 scrub ok 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.640091+0000 mon.a (mon.0) 1234 : cluster 0 osdmap e206: 8 total, 8 up, 8 in 2024-04-06T03:31:45.961 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.095718+0000 osd.7 (osd.7) 264 : cluster 0 2.1c scrub starts 2024-04-06T03:31:45.962 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:45 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.099631+0000 osd.7 (osd.7) 265 : cluster 0 2.1c scrub ok 2024-04-06T03:31:45.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:45.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:46.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.183963+0000 mgr.y (mgr.24368) 398 : cluster 0 pgmap v408: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.433305+0000 osd.1 (osd.1) 219 : cluster 0 5.12 scrub starts 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.434274+0000 osd.1 (osd.1) 220 : cluster 0 5.12 scrub ok 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.436564+0000 osd.3 (osd.3) 261 : cluster 0 5.0 scrub starts 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.439069+0000 osd.3 (osd.3) 262 : cluster 0 5.0 scrub ok 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.640091+0000 mon.a (mon.0) 1234 : cluster 0 osdmap e206: 8 total, 8 up, 8 in 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.095718+0000 osd.7 (osd.7) 264 : cluster 0 2.1c scrub starts 2024-04-06T03:31:46.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:45 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.099631+0000 osd.7 (osd.7) 265 : cluster 0 2.1c scrub ok 2024-04-06T03:31:46.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:46.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:46.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:46.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:46.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:46.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:46.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:46.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:46.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:46.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:46.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:46.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[21467]: cluster 2024-04-06T03:31:44.824621+0000 osd.0 (osd.0) 236 : cluster 0 8.c scrub ok 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.481773+0000 osd.1 (osd.1) 221 : cluster 0 5.1d scrub starts 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.486196+0000 osd.1 (osd.1) 222 : cluster 0 5.1d scrub ok 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.487885+0000 osd.0 (osd.0) 237 : cluster 0 3.17 scrub starts 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.495524+0000 osd.0 (osd.0) 238 : cluster 0 3.17 scrub ok 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[21467]: cluster 2024-04-06T03:31:45.597445+0000 osd.7 (osd.7) 266 : cluster 0 8.d scrub ok 2024-04-06T03:31:46.873 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[29221]: cluster 2024-04-06T03:31:44.824621+0000 osd.0 (osd.0) 236 : cluster 0 8.c scrub ok 2024-04-06T03:31:46.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.481773+0000 osd.1 (osd.1) 221 : cluster 0 5.1d scrub starts 2024-04-06T03:31:46.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.486196+0000 osd.1 (osd.1) 222 : cluster 0 5.1d scrub ok 2024-04-06T03:31:46.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.487885+0000 osd.0 (osd.0) 237 : cluster 0 3.17 scrub starts 2024-04-06T03:31:46.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.495524+0000 osd.0 (osd.0) 238 : cluster 0 3.17 scrub ok 2024-04-06T03:31:46.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:46 smithi050 bash[29221]: cluster 2024-04-06T03:31:45.597445+0000 osd.7 (osd.7) 266 : cluster 0 8.d scrub ok 2024-04-06T03:31:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:46.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:46.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:47.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:47.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:46 smithi153 bash[23852]: cluster 2024-04-06T03:31:44.824621+0000 osd.0 (osd.0) 236 : cluster 0 8.c scrub ok 2024-04-06T03:31:47.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:46 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.481773+0000 osd.1 (osd.1) 221 : cluster 0 5.1d scrub starts 2024-04-06T03:31:47.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:46 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.486196+0000 osd.1 (osd.1) 222 : cluster 0 5.1d scrub ok 2024-04-06T03:31:47.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:46 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.487885+0000 osd.0 (osd.0) 237 : cluster 0 3.17 scrub starts 2024-04-06T03:31:47.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:46 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.495524+0000 osd.0 (osd.0) 238 : cluster 0 3.17 scrub ok 2024-04-06T03:31:47.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:46 smithi153 bash[23852]: cluster 2024-04-06T03:31:45.597445+0000 osd.7 (osd.7) 266 : cluster 0 8.d scrub ok 2024-04-06T03:31:47.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:47.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:47.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:47.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:47.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:47.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:47.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:47.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:47.657 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:47.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:47.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:47 smithi050 bash[21467]: cluster 2024-04-06T03:31:46.184682+0000 mgr.y (mgr.24368) 399 : cluster 0 pgmap v410: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 891 op/s 2024-04-06T03:31:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:47 smithi050 bash[21467]: cluster 2024-04-06T03:31:46.540323+0000 osd.0 (osd.0) 239 : cluster 0 6.18 scrub starts 2024-04-06T03:31:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:47 smithi050 bash[21467]: cluster 2024-04-06T03:31:46.585852+0000 osd.0 (osd.0) 240 : cluster 0 6.18 scrub ok 2024-04-06T03:31:47.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:47 smithi050 bash[29221]: cluster 2024-04-06T03:31:46.184682+0000 mgr.y (mgr.24368) 399 : cluster 0 pgmap v410: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 891 op/s 2024-04-06T03:31:47.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:47 smithi050 bash[29221]: cluster 2024-04-06T03:31:46.540323+0000 osd.0 (osd.0) 239 : cluster 0 6.18 scrub starts 2024-04-06T03:31:47.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:47 smithi050 bash[29221]: cluster 2024-04-06T03:31:46.585852+0000 osd.0 (osd.0) 240 : cluster 0 6.18 scrub ok 2024-04-06T03:31:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:47.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:47.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:47 smithi153 bash[23852]: cluster 2024-04-06T03:31:46.184682+0000 mgr.y (mgr.24368) 399 : cluster 0 pgmap v410: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 891 op/s 2024-04-06T03:31:47.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:47 smithi153 bash[23852]: cluster 2024-04-06T03:31:46.540323+0000 osd.0 (osd.0) 239 : cluster 0 6.18 scrub starts 2024-04-06T03:31:47.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:47 smithi153 bash[23852]: cluster 2024-04-06T03:31:46.585852+0000 osd.0 (osd.0) 240 : cluster 0 6.18 scrub ok 2024-04-06T03:31:47.920 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:47.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:48.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:48.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:48.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:48.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:48.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:48.309 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:48.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:48.429 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:48.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:48.575 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:48.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:48.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:48.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:48 smithi153 bash[23852]: cluster 2024-04-06T03:31:47.481305+0000 osd.1 (osd.1) 223 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:31:48.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:48 smithi153 bash[23852]: cluster 2024-04-06T03:31:47.498212+0000 osd.1 (osd.1) 224 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:31:48.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:48 smithi050 bash[29221]: cluster 2024-04-06T03:31:47.481305+0000 osd.1 (osd.1) 223 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:31:48.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:48 smithi050 bash[29221]: cluster 2024-04-06T03:31:47.498212+0000 osd.1 (osd.1) 224 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:31:48.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:48.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:48.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:48.984 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:48.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:48 smithi050 bash[21467]: cluster 2024-04-06T03:31:47.481305+0000 osd.1 (osd.1) 223 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:31:48.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:48 smithi050 bash[21467]: cluster 2024-04-06T03:31:47.498212+0000 osd.1 (osd.1) 224 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:31:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:48.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:49.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.117 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:49.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:49.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:49.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:49.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:49.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:49.647 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:31:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:31:49.661 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:49.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:49.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:49 smithi153 bash[23852]: cluster 2024-04-06T03:31:48.101676+0000 osd.4 (osd.4) 259 : cluster 0 8.5 scrub starts 2024-04-06T03:31:49.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:49 smithi153 bash[23852]: cluster 2024-04-06T03:31:48.185056+0000 mgr.y (mgr.24368) 400 : cluster 0 pgmap v411: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 891 op/s 2024-04-06T03:31:49.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:49 smithi153 bash[23852]: cluster 2024-04-06T03:31:48.495661+0000 osd.0 (osd.0) 241 : cluster 0 3.6 scrub starts 2024-04-06T03:31:49.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:49 smithi153 bash[23852]: cluster 2024-04-06T03:31:48.505054+0000 osd.0 (osd.0) 242 : cluster 0 3.6 scrub ok 2024-04-06T03:31:49.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:49 smithi153 bash[23852]: audit 2024-04-06T03:31:49.077627+0000 mon.a (mon.0) 1235 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:49.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:31:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:31:49.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:49.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.931 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:49.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:49.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[21467]: cluster 2024-04-06T03:31:48.101676+0000 osd.4 (osd.4) 259 : cluster 0 8.5 scrub starts 2024-04-06T03:31:49.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[21467]: cluster 2024-04-06T03:31:48.185056+0000 mgr.y (mgr.24368) 400 : cluster 0 pgmap v411: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 891 op/s 2024-04-06T03:31:49.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[21467]: cluster 2024-04-06T03:31:48.495661+0000 osd.0 (osd.0) 241 : cluster 0 3.6 scrub starts 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[21467]: cluster 2024-04-06T03:31:48.505054+0000 osd.0 (osd.0) 242 : cluster 0 3.6 scrub ok 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[21467]: audit 2024-04-06T03:31:49.077627+0000 mon.a (mon.0) 1235 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[29221]: cluster 2024-04-06T03:31:48.101676+0000 osd.4 (osd.4) 259 : cluster 0 8.5 scrub starts 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[29221]: cluster 2024-04-06T03:31:48.185056+0000 mgr.y (mgr.24368) 400 : cluster 0 pgmap v411: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 891 op/s 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[29221]: cluster 2024-04-06T03:31:48.495661+0000 osd.0 (osd.0) 241 : cluster 0 3.6 scrub starts 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[29221]: cluster 2024-04-06T03:31:48.505054+0000 osd.0 (osd.0) 242 : cluster 0 3.6 scrub ok 2024-04-06T03:31:49.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:49 smithi050 bash[29221]: audit 2024-04-06T03:31:49.077627+0000 mon.a (mon.0) 1235 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:31:50.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:50.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:50.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:50.189 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:50.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:50.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:50.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:50.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:50.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:50.576 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:50.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:50.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:50 smithi153 bash[23852]: cluster 2024-04-06T03:31:49.270765+0000 osd.4 (osd.4) 260 : cluster 0 8.5 scrub ok 2024-04-06T03:31:50.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:50 smithi153 bash[23852]: cluster 2024-04-06T03:31:49.379118+0000 osd.1 (osd.1) 225 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:31:50.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:50 smithi153 bash[23852]: cluster 2024-04-06T03:31:49.387386+0000 osd.1 (osd.1) 226 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:31:50.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:50 smithi153 bash[23852]: audit 2024-04-06T03:31:49.503080+0000 mgr.y (mgr.24368) 401 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:50.707 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:50.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:50.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[21467]: cluster 2024-04-06T03:31:49.270765+0000 osd.4 (osd.4) 260 : cluster 0 8.5 scrub ok 2024-04-06T03:31:50.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[21467]: cluster 2024-04-06T03:31:49.379118+0000 osd.1 (osd.1) 225 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:31:50.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[21467]: cluster 2024-04-06T03:31:49.387386+0000 osd.1 (osd.1) 226 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:31:50.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[21467]: audit 2024-04-06T03:31:49.503080+0000 mgr.y (mgr.24368) 401 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:50.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[29221]: cluster 2024-04-06T03:31:49.270765+0000 osd.4 (osd.4) 260 : cluster 0 8.5 scrub ok 2024-04-06T03:31:50.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[29221]: cluster 2024-04-06T03:31:49.379118+0000 osd.1 (osd.1) 225 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:31:50.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[29221]: cluster 2024-04-06T03:31:49.387386+0000 osd.1 (osd.1) 226 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:31:50.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:50 smithi050 bash[29221]: audit 2024-04-06T03:31:49.503080+0000 mgr.y (mgr.24368) 401 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:31:50.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:50.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:50.964 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:50.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:51.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:51.099 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:51.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:51.236 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:51.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:51.369 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:51.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:51.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:51.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:51.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.185795+0000 mgr.y (mgr.24368) 402 : cluster 0 pgmap v412: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.463480+0000 osd.1 (osd.1) 227 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.463547+0000 osd.1 (osd.1) 228 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.549749+0000 osd.0 (osd.0) 243 : cluster 0 6.16 scrub starts 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.670225+0000 mon.a (mon.0) 1236 : cluster 0 osdmap e207: 8 total, 8 up, 8 in 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.701166+0000 osd.0 (osd.0) 244 : cluster 0 6.16 scrub ok 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.896557+0000 osd.7 (osd.7) 267 : cluster 0 3.16 scrub starts 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[21467]: cluster 2024-04-06T03:31:50.901641+0000 osd.7 (osd.7) 268 : cluster 0 3.16 scrub ok 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.185795+0000 mgr.y (mgr.24368) 402 : cluster 0 pgmap v412: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.463480+0000 osd.1 (osd.1) 227 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.463547+0000 osd.1 (osd.1) 228 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.549749+0000 osd.0 (osd.0) 243 : cluster 0 6.16 scrub starts 2024-04-06T03:31:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.670225+0000 mon.a (mon.0) 1236 : cluster 0 osdmap e207: 8 total, 8 up, 8 in 2024-04-06T03:31:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.701166+0000 osd.0 (osd.0) 244 : cluster 0 6.16 scrub ok 2024-04-06T03:31:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.896557+0000 osd.7 (osd.7) 267 : cluster 0 3.16 scrub starts 2024-04-06T03:31:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:51 smithi050 bash[29221]: cluster 2024-04-06T03:31:50.901641+0000 osd.7 (osd.7) 268 : cluster 0 3.16 scrub ok 2024-04-06T03:31:51.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:51.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:51.921 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.185795+0000 mgr.y (mgr.24368) 402 : cluster 0 pgmap v412: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.463480+0000 osd.1 (osd.1) 227 : cluster 0 purged_snaps scrub starts 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.463547+0000 osd.1 (osd.1) 228 : cluster 0 purged_snaps scrub ok 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.549749+0000 osd.0 (osd.0) 243 : cluster 0 6.16 scrub starts 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.670225+0000 mon.a (mon.0) 1236 : cluster 0 osdmap e207: 8 total, 8 up, 8 in 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.701166+0000 osd.0 (osd.0) 244 : cluster 0 6.16 scrub ok 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.896557+0000 osd.7 (osd.7) 267 : cluster 0 3.16 scrub starts 2024-04-06T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:51 smithi153 bash[23852]: cluster 2024-04-06T03:31:50.901641+0000 osd.7 (osd.7) 268 : cluster 0 3.16 scrub ok 2024-04-06T03:31:51.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:51.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:52.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:52.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:52.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:52.208 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:52.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:52.339 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:52.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:52.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:52.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:52.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:52.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:52.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:52.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:52.738 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:52.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:52.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:52.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:52.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.056832+0000 osd.4 (osd.4) 261 : cluster 0 3.a scrub starts 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.059103+0000 osd.4 (osd.4) 262 : cluster 0 3.a scrub ok 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.322901+0000 osd.1 (osd.1) 229 : cluster 0 6.19 scrub starts 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.353702+0000 osd.1 (osd.1) 230 : cluster 0 6.19 scrub ok 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.566194+0000 osd.0 (osd.0) 245 : cluster 0 6.d scrub starts 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.619857+0000 osd.0 (osd.0) 246 : cluster 0 6.d scrub ok 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.900367+0000 osd.7 (osd.7) 269 : cluster 0 6.1c deep-scrub starts 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[21467]: cluster 2024-04-06T03:31:51.905339+0000 osd.7 (osd.7) 270 : cluster 0 6.1c deep-scrub ok 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.056832+0000 osd.4 (osd.4) 261 : cluster 0 3.a scrub starts 2024-04-06T03:31:52.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.059103+0000 osd.4 (osd.4) 262 : cluster 0 3.a scrub ok 2024-04-06T03:31:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.322901+0000 osd.1 (osd.1) 229 : cluster 0 6.19 scrub starts 2024-04-06T03:31:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.353702+0000 osd.1 (osd.1) 230 : cluster 0 6.19 scrub ok 2024-04-06T03:31:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.566194+0000 osd.0 (osd.0) 245 : cluster 0 6.d scrub starts 2024-04-06T03:31:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.619857+0000 osd.0 (osd.0) 246 : cluster 0 6.d scrub ok 2024-04-06T03:31:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.900367+0000 osd.7 (osd.7) 269 : cluster 0 6.1c deep-scrub starts 2024-04-06T03:31:52.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:52 smithi050 bash[29221]: cluster 2024-04-06T03:31:51.905339+0000 osd.7 (osd.7) 270 : cluster 0 6.1c deep-scrub ok 2024-04-06T03:31:52.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.056832+0000 osd.4 (osd.4) 261 : cluster 0 3.a scrub starts 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.059103+0000 osd.4 (osd.4) 262 : cluster 0 3.a scrub ok 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.322901+0000 osd.1 (osd.1) 229 : cluster 0 6.19 scrub starts 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.353702+0000 osd.1 (osd.1) 230 : cluster 0 6.19 scrub ok 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.566194+0000 osd.0 (osd.0) 245 : cluster 0 6.d scrub starts 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.619857+0000 osd.0 (osd.0) 246 : cluster 0 6.d scrub ok 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.900367+0000 osd.7 (osd.7) 269 : cluster 0 6.1c deep-scrub starts 2024-04-06T03:31:53.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:52 smithi153 bash[23852]: cluster 2024-04-06T03:31:51.905339+0000 osd.7 (osd.7) 270 : cluster 0 6.1c deep-scrub ok 2024-04-06T03:31:53.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:53.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:53.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:53.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:53.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:53.388 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:53.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:53.521 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:53.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:53.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:53.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:53.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:53.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:53 smithi153 bash[23852]: cluster 2024-04-06T03:31:52.186140+0000 mgr.y (mgr.24368) 403 : cluster 0 pgmap v414: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s 2024-04-06T03:31:53.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:53 smithi153 bash[23852]: cluster 2024-04-06T03:31:52.584253+0000 osd.0 (osd.0) 247 : cluster 0 5.1e scrub starts 2024-04-06T03:31:53.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:53 smithi153 bash[23852]: cluster 2024-04-06T03:31:52.591666+0000 osd.0 (osd.0) 248 : cluster 0 5.1e scrub ok 2024-04-06T03:31:53.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:53 smithi153 bash[23852]: cluster 2024-04-06T03:31:52.933166+0000 osd.7 (osd.7) 271 : cluster 0 3.d scrub starts 2024-04-06T03:31:53.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:53 smithi153 bash[23852]: cluster 2024-04-06T03:31:52.935652+0000 osd.7 (osd.7) 272 : cluster 0 3.d scrub ok 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[21467]: cluster 2024-04-06T03:31:52.186140+0000 mgr.y (mgr.24368) 403 : cluster 0 pgmap v414: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[21467]: cluster 2024-04-06T03:31:52.584253+0000 osd.0 (osd.0) 247 : cluster 0 5.1e scrub starts 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[21467]: cluster 2024-04-06T03:31:52.591666+0000 osd.0 (osd.0) 248 : cluster 0 5.1e scrub ok 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[21467]: cluster 2024-04-06T03:31:52.933166+0000 osd.7 (osd.7) 271 : cluster 0 3.d scrub starts 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[21467]: cluster 2024-04-06T03:31:52.935652+0000 osd.7 (osd.7) 272 : cluster 0 3.d scrub ok 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[29221]: cluster 2024-04-06T03:31:52.186140+0000 mgr.y (mgr.24368) 403 : cluster 0 pgmap v414: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 929 op/s 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[29221]: cluster 2024-04-06T03:31:52.584253+0000 osd.0 (osd.0) 247 : cluster 0 5.1e scrub starts 2024-04-06T03:31:53.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[29221]: cluster 2024-04-06T03:31:52.591666+0000 osd.0 (osd.0) 248 : cluster 0 5.1e scrub ok 2024-04-06T03:31:53.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[29221]: cluster 2024-04-06T03:31:52.933166+0000 osd.7 (osd.7) 271 : cluster 0 3.d scrub starts 2024-04-06T03:31:53.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:53 smithi050 bash[29221]: cluster 2024-04-06T03:31:52.935652+0000 osd.7 (osd.7) 272 : cluster 0 3.d scrub ok 2024-04-06T03:31:53.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:53.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:53.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:54.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:54.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:54.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:54.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:54.327 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:54.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:54.468 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:54.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:54.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:54.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:54.726 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:54 smithi050 bash[21467]: cluster 2024-04-06T03:31:53.145750+0000 osd.4 (osd.4) 263 : cluster 0 8.4 scrub starts 2024-04-06T03:31:54.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:54 smithi050 bash[29221]: cluster 2024-04-06T03:31:53.145750+0000 osd.4 (osd.4) 263 : cluster 0 8.4 scrub starts 2024-04-06T03:31:54.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:54.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:54.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:54.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:54.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:55.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:55.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:54 smithi153 bash[23852]: cluster 2024-04-06T03:31:53.145750+0000 osd.4 (osd.4) 263 : cluster 0 8.4 scrub starts 2024-04-06T03:31:55.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:55.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:55.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:55.266 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:55.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:55.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:55.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:55.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:55.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:55.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:55.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:55.676 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:55.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:55.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:55.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:55 smithi153 bash[23852]: cluster 2024-04-06T03:31:54.186699+0000 mgr.y (mgr.24368) 404 : cluster 0 pgmap v415: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:31:55.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:55 smithi153 bash[23852]: cluster 2024-04-06T03:31:54.372097+0000 osd.4 (osd.4) 264 : cluster 0 8.4 scrub ok 2024-04-06T03:31:55.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:55 smithi153 bash[23852]: cluster 2024-04-06T03:31:54.536802+0000 osd.0 (osd.0) 249 : cluster 0 2.4 scrub starts 2024-04-06T03:31:55.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:55 smithi153 bash[23852]: cluster 2024-04-06T03:31:54.539055+0000 osd.0 (osd.0) 250 : cluster 0 2.4 scrub ok 2024-04-06T03:31:55.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:55 smithi153 bash[23852]: cluster 2024-04-06T03:31:54.577986+0000 osd.3 (osd.3) 263 : cluster 0 2.1b scrub starts 2024-04-06T03:31:55.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:55 smithi153 bash[23852]: cluster 2024-04-06T03:31:54.579722+0000 osd.3 (osd.3) 264 : cluster 0 2.1b scrub ok 2024-04-06T03:31:55.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:55.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[21467]: cluster 2024-04-06T03:31:54.186699+0000 mgr.y (mgr.24368) 404 : cluster 0 pgmap v415: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:31:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[21467]: cluster 2024-04-06T03:31:54.372097+0000 osd.4 (osd.4) 264 : cluster 0 8.4 scrub ok 2024-04-06T03:31:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[21467]: cluster 2024-04-06T03:31:54.536802+0000 osd.0 (osd.0) 249 : cluster 0 2.4 scrub starts 2024-04-06T03:31:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[21467]: cluster 2024-04-06T03:31:54.539055+0000 osd.0 (osd.0) 250 : cluster 0 2.4 scrub ok 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[21467]: cluster 2024-04-06T03:31:54.577986+0000 osd.3 (osd.3) 263 : cluster 0 2.1b scrub starts 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[21467]: cluster 2024-04-06T03:31:54.579722+0000 osd.3 (osd.3) 264 : cluster 0 2.1b scrub ok 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[29221]: cluster 2024-04-06T03:31:54.186699+0000 mgr.y (mgr.24368) 404 : cluster 0 pgmap v415: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[29221]: cluster 2024-04-06T03:31:54.372097+0000 osd.4 (osd.4) 264 : cluster 0 8.4 scrub ok 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[29221]: cluster 2024-04-06T03:31:54.536802+0000 osd.0 (osd.0) 249 : cluster 0 2.4 scrub starts 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[29221]: cluster 2024-04-06T03:31:54.539055+0000 osd.0 (osd.0) 250 : cluster 0 2.4 scrub ok 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[29221]: cluster 2024-04-06T03:31:54.577986+0000 osd.3 (osd.3) 263 : cluster 0 2.1b scrub starts 2024-04-06T03:31:55.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:55 smithi050 bash[29221]: cluster 2024-04-06T03:31:54.579722+0000 osd.3 (osd.3) 264 : cluster 0 2.1b scrub ok 2024-04-06T03:31:56.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:56.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:56.165 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:56.185 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:56.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:56.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:56.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:56.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:56.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:56.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:56.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:56.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:56 smithi050 bash[21467]: cluster 2024-04-06T03:31:55.622587+0000 osd.0 (osd.0) 251 : cluster 0 3.1b scrub starts 2024-04-06T03:31:56.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:56 smithi050 bash[21467]: cluster 2024-04-06T03:31:55.624537+0000 osd.0 (osd.0) 252 : cluster 0 3.1b scrub ok 2024-04-06T03:31:56.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:56 smithi050 bash[29221]: cluster 2024-04-06T03:31:55.622587+0000 osd.0 (osd.0) 251 : cluster 0 3.1b scrub starts 2024-04-06T03:31:56.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:56 smithi050 bash[29221]: cluster 2024-04-06T03:31:55.624537+0000 osd.0 (osd.0) 252 : cluster 0 3.1b scrub ok 2024-04-06T03:31:56.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:56.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:56.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:56.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:56 smithi153 bash[23852]: cluster 2024-04-06T03:31:55.622587+0000 osd.0 (osd.0) 251 : cluster 0 3.1b scrub starts 2024-04-06T03:31:56.950 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:56 smithi153 bash[23852]: cluster 2024-04-06T03:31:55.624537+0000 osd.0 (osd.0) 252 : cluster 0 3.1b scrub ok 2024-04-06T03:31:56.964 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:56.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:57.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:57.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:57.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:57.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:57.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:57.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:57.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:57.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:57.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:57.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:57.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:57.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[29221]: cluster 2024-04-06T03:31:56.187288+0000 mgr.y (mgr.24368) 405 : cluster 0 pgmap v416: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 830 op/s 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[29221]: cluster 2024-04-06T03:31:56.605699+0000 osd.0 (osd.0) 253 : cluster 0 3.1 scrub starts 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[29221]: cluster 2024-04-06T03:31:56.607446+0000 osd.0 (osd.0) 254 : cluster 0 3.1 scrub ok 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[29221]: cluster 2024-04-06T03:31:56.832692+0000 osd.7 (osd.7) 273 : cluster 0 3.e deep-scrub starts 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[29221]: cluster 2024-04-06T03:31:56.847232+0000 osd.7 (osd.7) 274 : cluster 0 3.e deep-scrub ok 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[21467]: cluster 2024-04-06T03:31:56.187288+0000 mgr.y (mgr.24368) 405 : cluster 0 pgmap v416: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 830 op/s 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[21467]: cluster 2024-04-06T03:31:56.605699+0000 osd.0 (osd.0) 253 : cluster 0 3.1 scrub starts 2024-04-06T03:31:57.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[21467]: cluster 2024-04-06T03:31:56.607446+0000 osd.0 (osd.0) 254 : cluster 0 3.1 scrub ok 2024-04-06T03:31:57.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[21467]: cluster 2024-04-06T03:31:56.832692+0000 osd.7 (osd.7) 273 : cluster 0 3.e deep-scrub starts 2024-04-06T03:31:57.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:57 smithi050 bash[21467]: cluster 2024-04-06T03:31:56.847232+0000 osd.7 (osd.7) 274 : cluster 0 3.e deep-scrub ok 2024-04-06T03:31:57.766 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:57.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:57.898 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:57.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:58.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:58.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:57 smithi153 bash[23852]: cluster 2024-04-06T03:31:56.187288+0000 mgr.y (mgr.24368) 405 : cluster 0 pgmap v416: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 830 op/s 2024-04-06T03:31:58.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:57 smithi153 bash[23852]: cluster 2024-04-06T03:31:56.605699+0000 osd.0 (osd.0) 253 : cluster 0 3.1 scrub starts 2024-04-06T03:31:58.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:57 smithi153 bash[23852]: cluster 2024-04-06T03:31:56.607446+0000 osd.0 (osd.0) 254 : cluster 0 3.1 scrub ok 2024-04-06T03:31:58.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:57 smithi153 bash[23852]: cluster 2024-04-06T03:31:56.832692+0000 osd.7 (osd.7) 273 : cluster 0 3.e deep-scrub starts 2024-04-06T03:31:58.012 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:57 smithi153 bash[23852]: cluster 2024-04-06T03:31:56.847232+0000 osd.7 (osd.7) 274 : cluster 0 3.e deep-scrub ok 2024-04-06T03:31:58.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:58.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:58.163 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:58.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:58.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:58.284 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:31:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:31:58.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:58.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:58.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:58.546 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:58.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:58.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:58.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:58.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:58 smithi153 bash[23852]: cluster 2024-04-06T03:31:57.557199+0000 osd.3 (osd.3) 265 : cluster 0 3.c scrub starts 2024-04-06T03:31:58.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:58 smithi153 bash[23852]: cluster 2024-04-06T03:31:57.560372+0000 osd.3 (osd.3) 266 : cluster 0 3.c scrub ok 2024-04-06T03:31:58.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:58 smithi153 bash[23852]: cluster 2024-04-06T03:31:57.564908+0000 osd.0 (osd.0) 255 : cluster 0 5.2 scrub starts 2024-04-06T03:31:58.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:58 smithi153 bash[23852]: cluster 2024-04-06T03:31:57.572370+0000 osd.0 (osd.0) 256 : cluster 0 5.2 scrub ok 2024-04-06T03:31:58.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:58 smithi153 bash[23852]: cluster 2024-04-06T03:31:57.838071+0000 osd.7 (osd.7) 275 : cluster 0 6.15 scrub starts 2024-04-06T03:31:58.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:58 smithi153 bash[23852]: cluster 2024-04-06T03:31:57.853312+0000 osd.7 (osd.7) 276 : cluster 0 6.15 scrub ok 2024-04-06T03:31:58.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:58.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:58.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:58.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:59.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:59.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:31:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[21467]: cluster 2024-04-06T03:31:57.557199+0000 osd.3 (osd.3) 265 : cluster 0 3.c scrub starts 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[21467]: cluster 2024-04-06T03:31:57.560372+0000 osd.3 (osd.3) 266 : cluster 0 3.c scrub ok 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[21467]: cluster 2024-04-06T03:31:57.564908+0000 osd.0 (osd.0) 255 : cluster 0 5.2 scrub starts 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[21467]: cluster 2024-04-06T03:31:57.572370+0000 osd.0 (osd.0) 256 : cluster 0 5.2 scrub ok 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[21467]: cluster 2024-04-06T03:31:57.838071+0000 osd.7 (osd.7) 275 : cluster 0 6.15 scrub starts 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[21467]: cluster 2024-04-06T03:31:57.853312+0000 osd.7 (osd.7) 276 : cluster 0 6.15 scrub ok 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[29221]: cluster 2024-04-06T03:31:57.557199+0000 osd.3 (osd.3) 265 : cluster 0 3.c scrub starts 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[29221]: cluster 2024-04-06T03:31:57.560372+0000 osd.3 (osd.3) 266 : cluster 0 3.c scrub ok 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[29221]: cluster 2024-04-06T03:31:57.564908+0000 osd.0 (osd.0) 255 : cluster 0 5.2 scrub starts 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[29221]: cluster 2024-04-06T03:31:57.572370+0000 osd.0 (osd.0) 256 : cluster 0 5.2 scrub ok 2024-04-06T03:31:59.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[29221]: cluster 2024-04-06T03:31:57.838071+0000 osd.7 (osd.7) 275 : cluster 0 6.15 scrub starts 2024-04-06T03:31:59.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:58 smithi050 bash[29221]: cluster 2024-04-06T03:31:57.853312+0000 osd.7 (osd.7) 276 : cluster 0 6.15 scrub ok 2024-04-06T03:31:59.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:59.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:31:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:31:59.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:59.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:31:59.342 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:31:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:31:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:31:59.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:59.466 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:31:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:31:59.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:59.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:31:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:31:59.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:59.702 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:31:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:31:59.730 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:59.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:59 smithi153 bash[23852]: cluster 2024-04-06T03:31:58.187696+0000 mgr.y (mgr.24368) 406 : cluster 0 pgmap v417: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 830 op/s 2024-04-06T03:31:59.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:59 smithi153 bash[23852]: cluster 2024-04-06T03:31:58.595678+0000 osd.3 (osd.3) 267 : cluster 0 2.19 scrub starts 2024-04-06T03:31:59.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:31:59 smithi153 bash[23852]: cluster 2024-04-06T03:31:58.602925+0000 osd.3 (osd.3) 268 : cluster 0 2.19 scrub ok 2024-04-06T03:31:59.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:59.854 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:31:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:31:59.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:59.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:31:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:31:59.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:31:59.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[21467]: cluster 2024-04-06T03:31:58.187696+0000 mgr.y (mgr.24368) 406 : cluster 0 pgmap v417: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 830 op/s 2024-04-06T03:31:59.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[21467]: cluster 2024-04-06T03:31:58.595678+0000 osd.3 (osd.3) 267 : cluster 0 2.19 scrub starts 2024-04-06T03:31:59.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[21467]: cluster 2024-04-06T03:31:58.602925+0000 osd.3 (osd.3) 268 : cluster 0 2.19 scrub ok 2024-04-06T03:31:59.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[29221]: cluster 2024-04-06T03:31:58.187696+0000 mgr.y (mgr.24368) 406 : cluster 0 pgmap v417: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 830 op/s 2024-04-06T03:31:59.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[29221]: cluster 2024-04-06T03:31:58.595678+0000 osd.3 (osd.3) 267 : cluster 0 2.19 scrub starts 2024-04-06T03:31:59.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:31:59 smithi050 bash[29221]: cluster 2024-04-06T03:31:58.602925+0000 osd.3 (osd.3) 268 : cluster 0 2.19 scrub ok 2024-04-06T03:32:00.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:00.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:00.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:00.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:00.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:00.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:00.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:00.490 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:00.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:00.621 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:00.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[21467]: audit 2024-04-06T03:31:59.511071+0000 mgr.y (mgr.24368) 407 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:00.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[21467]: cluster 2024-04-06T03:31:59.849665+0000 osd.7 (osd.7) 277 : cluster 0 6.8 scrub starts 2024-04-06T03:32:00.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[21467]: cluster 2024-04-06T03:31:59.850522+0000 osd.7 (osd.7) 278 : cluster 0 6.8 scrub ok 2024-04-06T03:32:00.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[21467]: cluster 2024-04-06T03:32:00.189050+0000 mgr.y (mgr.24368) 408 : cluster 0 pgmap v418: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 841 op/s 2024-04-06T03:32:00.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[29221]: audit 2024-04-06T03:31:59.511071+0000 mgr.y (mgr.24368) 407 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:00.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[29221]: cluster 2024-04-06T03:31:59.849665+0000 osd.7 (osd.7) 277 : cluster 0 6.8 scrub starts 2024-04-06T03:32:00.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[29221]: cluster 2024-04-06T03:31:59.850522+0000 osd.7 (osd.7) 278 : cluster 0 6.8 scrub ok 2024-04-06T03:32:00.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:00 smithi050 bash[29221]: cluster 2024-04-06T03:32:00.189050+0000 mgr.y (mgr.24368) 408 : cluster 0 pgmap v418: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 841 op/s 2024-04-06T03:32:00.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:00.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:00.870 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:00 smithi153 bash[23852]: audit 2024-04-06T03:31:59.511071+0000 mgr.y (mgr.24368) 407 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:00.870 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:00 smithi153 bash[23852]: cluster 2024-04-06T03:31:59.849665+0000 osd.7 (osd.7) 277 : cluster 0 6.8 scrub starts 2024-04-06T03:32:00.870 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:00 smithi153 bash[23852]: cluster 2024-04-06T03:31:59.850522+0000 osd.7 (osd.7) 278 : cluster 0 6.8 scrub ok 2024-04-06T03:32:00.870 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:00 smithi153 bash[23852]: cluster 2024-04-06T03:32:00.189050+0000 mgr.y (mgr.24368) 408 : cluster 0 pgmap v418: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 841 op/s 2024-04-06T03:32:00.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:00.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:00.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:01.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:01.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:01.156 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:01.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:01.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:01.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:01.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:01.401 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:01.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:01.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:01.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:01.663 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:01.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:01.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:01.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:01.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:01 smithi153 bash[23852]: cluster 2024-04-06T03:32:00.105509+0000 osd.4 (osd.4) 265 : cluster 0 5.1 scrub starts 2024-04-06T03:32:01.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:01 smithi153 bash[23852]: cluster 2024-04-06T03:32:00.108949+0000 osd.4 (osd.4) 266 : cluster 0 5.1 scrub ok 2024-04-06T03:32:01.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:01 smithi153 bash[23852]: cluster 2024-04-06T03:32:00.826652+0000 osd.7 (osd.7) 279 : cluster 0 2.1d scrub starts 2024-04-06T03:32:01.907 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:01 smithi153 bash[23852]: cluster 2024-04-06T03:32:00.828247+0000 osd.7 (osd.7) 280 : cluster 0 2.1d scrub ok 2024-04-06T03:32:01.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:01.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:02.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:02.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:02.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:02.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[21467]: cluster 2024-04-06T03:32:00.105509+0000 osd.4 (osd.4) 265 : cluster 0 5.1 scrub starts 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[21467]: cluster 2024-04-06T03:32:00.108949+0000 osd.4 (osd.4) 266 : cluster 0 5.1 scrub ok 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[21467]: cluster 2024-04-06T03:32:00.826652+0000 osd.7 (osd.7) 279 : cluster 0 2.1d scrub starts 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[21467]: cluster 2024-04-06T03:32:00.828247+0000 osd.7 (osd.7) 280 : cluster 0 2.1d scrub ok 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[29221]: cluster 2024-04-06T03:32:00.105509+0000 osd.4 (osd.4) 265 : cluster 0 5.1 scrub starts 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[29221]: cluster 2024-04-06T03:32:00.108949+0000 osd.4 (osd.4) 266 : cluster 0 5.1 scrub ok 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[29221]: cluster 2024-04-06T03:32:00.826652+0000 osd.7 (osd.7) 279 : cluster 0 2.1d scrub starts 2024-04-06T03:32:02.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:01 smithi050 bash[29221]: cluster 2024-04-06T03:32:00.828247+0000 osd.7 (osd.7) 280 : cluster 0 2.1d scrub ok 2024-04-06T03:32:02.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:02.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:02.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:02.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:02.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:02.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:02.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:02.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:02.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:02.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:02.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:02 smithi050 bash[21467]: cluster 2024-04-06T03:32:02.189677+0000 mgr.y (mgr.24368) 409 : cluster 0 pgmap v419: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 888 B/s rd, 0 B/s wr, 713 op/s 2024-04-06T03:32:02.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:02 smithi050 bash[29221]: cluster 2024-04-06T03:32:02.189677+0000 mgr.y (mgr.24368) 409 : cluster 0 pgmap v419: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 888 B/s rd, 0 B/s wr, 713 op/s 2024-04-06T03:32:03.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:02 smithi153 bash[23852]: cluster 2024-04-06T03:32:02.189677+0000 mgr.y (mgr.24368) 409 : cluster 0 pgmap v419: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 888 B/s rd, 0 B/s wr, 713 op/s 2024-04-06T03:32:03.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:03.087 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:03.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:03.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:03.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:03.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:03.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:03.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:03.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:03.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:03.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:03.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:03.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:03.838 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:03.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:03.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:03.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:04.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:03 smithi153 bash[23852]: cluster 2024-04-06T03:32:02.650832+0000 osd.3 (osd.3) 269 : cluster 0 6.11 scrub starts 2024-04-06T03:32:04.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:03 smithi153 bash[23852]: cluster 2024-04-06T03:32:02.669320+0000 osd.3 (osd.3) 270 : cluster 0 6.11 scrub ok 2024-04-06T03:32:04.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:03 smithi153 bash[23852]: cluster 2024-04-06T03:32:02.955904+0000 osd.6 (osd.6) 87 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:04.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:03 smithi153 bash[23852]: cluster 2024-04-06T03:32:02.955939+0000 osd.6 (osd.6) 88 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:04.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:04.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:04.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:04.229 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:04.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[21467]: cluster 2024-04-06T03:32:02.650832+0000 osd.3 (osd.3) 269 : cluster 0 6.11 scrub starts 2024-04-06T03:32:04.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[21467]: cluster 2024-04-06T03:32:02.669320+0000 osd.3 (osd.3) 270 : cluster 0 6.11 scrub ok 2024-04-06T03:32:04.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[21467]: cluster 2024-04-06T03:32:02.955904+0000 osd.6 (osd.6) 87 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:04.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[21467]: cluster 2024-04-06T03:32:02.955939+0000 osd.6 (osd.6) 88 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:04.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[29221]: cluster 2024-04-06T03:32:02.650832+0000 osd.3 (osd.3) 269 : cluster 0 6.11 scrub starts 2024-04-06T03:32:04.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[29221]: cluster 2024-04-06T03:32:02.669320+0000 osd.3 (osd.3) 270 : cluster 0 6.11 scrub ok 2024-04-06T03:32:04.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[29221]: cluster 2024-04-06T03:32:02.955904+0000 osd.6 (osd.6) 87 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:04.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:03 smithi050 bash[29221]: cluster 2024-04-06T03:32:02.955939+0000 osd.6 (osd.6) 88 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:04.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.360 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:04.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:04.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:04.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:04.628 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:32:04.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:04.675 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:32:04.721 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:32:04.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:04.764 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:32:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:04.802 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:32:04.829 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:32:04.864 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:32:04.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:04.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:04.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:04.906 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:32:04.909 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:32:04.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[21467]: cluster 2024-04-06T03:32:03.210011+0000 osd.4 (osd.4) 267 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[21467]: cluster 2024-04-06T03:32:03.210102+0000 osd.4 (osd.4) 268 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[21467]: cluster 2024-04-06T03:32:03.723487+0000 mon.a (mon.0) 1237 : cluster 0 osdmap e208: 8 total, 8 up, 8 in 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[21467]: audit 2024-04-06T03:32:04.095559+0000 mon.a (mon.0) 1238 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[21467]: cluster 2024-04-06T03:32:04.190341+0000 mgr.y (mgr.24368) 410 : cluster 0 pgmap v421: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 795 op/s 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[21467]: audit 2024-04-06T03:32:04.588576+0000 mon.b (mon.1) 39 : audit 0 from='client.? 172.21.15.50:0/467597755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[29221]: cluster 2024-04-06T03:32:03.210011+0000 osd.4 (osd.4) 267 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[29221]: cluster 2024-04-06T03:32:03.210102+0000 osd.4 (osd.4) 268 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[29221]: cluster 2024-04-06T03:32:03.723487+0000 mon.a (mon.0) 1237 : cluster 0 osdmap e208: 8 total, 8 up, 8 in 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[29221]: audit 2024-04-06T03:32:04.095559+0000 mon.a (mon.0) 1238 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:04.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[29221]: cluster 2024-04-06T03:32:04.190341+0000 mgr.y (mgr.24368) 410 : cluster 0 pgmap v421: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 795 op/s 2024-04-06T03:32:04.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:04 smithi050 bash[29221]: audit 2024-04-06T03:32:04.588576+0000 mon.b (mon.1) 39 : audit 0 from='client.? 172.21.15.50:0/467597755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:32:04.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:05.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:04 smithi153 bash[23852]: cluster 2024-04-06T03:32:03.210011+0000 osd.4 (osd.4) 267 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:05.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:04 smithi153 bash[23852]: cluster 2024-04-06T03:32:03.210102+0000 osd.4 (osd.4) 268 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:05.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:04 smithi153 bash[23852]: cluster 2024-04-06T03:32:03.723487+0000 mon.a (mon.0) 1237 : cluster 0 osdmap e208: 8 total, 8 up, 8 in 2024-04-06T03:32:05.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:04 smithi153 bash[23852]: audit 2024-04-06T03:32:04.095559+0000 mon.a (mon.0) 1238 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:05.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:04 smithi153 bash[23852]: cluster 2024-04-06T03:32:04.190341+0000 mgr.y (mgr.24368) 410 : cluster 0 pgmap v421: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 795 op/s 2024-04-06T03:32:05.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:04 smithi153 bash[23852]: audit 2024-04-06T03:32:04.588576+0000 mon.b (mon.1) 39 : audit 0 from='client.? 172.21.15.50:0/467597755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:32:05.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:05.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:05.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:05.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:05.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:05.568 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:05.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:05.756 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-06T03:32:05.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:05.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:05 smithi050 bash[21467]: audit 2024-04-06T03:32:05.585576+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.50:0/1873401708' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:32:05.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:05 smithi050 bash[21467]: audit 2024-04-06T03:32:05.586049+0000 mon.a (mon.0) 1239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:32:05.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:05 smithi050 bash[29221]: audit 2024-04-06T03:32:05.585576+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.50:0/1873401708' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:32:05.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:05 smithi050 bash[29221]: audit 2024-04-06T03:32:05.586049+0000 mon.a (mon.0) 1239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:32:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:05.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:05.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:06.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:05 smithi153 bash[23852]: audit 2024-04-06T03:32:05.585576+0000 mon.c (mon.2) 64 : audit 1 from='client.? 172.21.15.50:0/1873401708' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:32:06.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:05 smithi153 bash[23852]: audit 2024-04-06T03:32:05.586049+0000 mon.a (mon.0) 1239 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:32:06.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:06.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:06.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:06.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:06.397 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:06.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:06.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:06.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:06.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:06.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:06.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:06.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[21467]: audit 2024-04-06T03:32:05.745918+0000 mon.a (mon.0) 1240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:32:06.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[21467]: cluster 2024-04-06T03:32:05.750678+0000 mon.a (mon.0) 1241 : cluster 0 osdmap e209: 8 total, 8 up, 8 in 2024-04-06T03:32:06.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[21467]: cluster 2024-04-06T03:32:05.752375+0000 osd.7 (osd.7) 281 : cluster 0 3.11 scrub starts 2024-04-06T03:32:06.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[21467]: cluster 2024-04-06T03:32:05.753977+0000 osd.7 (osd.7) 282 : cluster 0 3.11 scrub ok 2024-04-06T03:32:06.916 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[21467]: cluster 2024-04-06T03:32:06.190885+0000 mgr.y (mgr.24368) 411 : cluster 0 pgmap v423: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-04-06T03:32:06.916 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[29221]: audit 2024-04-06T03:32:05.745918+0000 mon.a (mon.0) 1240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:32:06.916 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[29221]: cluster 2024-04-06T03:32:05.750678+0000 mon.a (mon.0) 1241 : cluster 0 osdmap e209: 8 total, 8 up, 8 in 2024-04-06T03:32:06.916 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[29221]: cluster 2024-04-06T03:32:05.752375+0000 osd.7 (osd.7) 281 : cluster 0 3.11 scrub starts 2024-04-06T03:32:06.916 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[29221]: cluster 2024-04-06T03:32:05.753977+0000 osd.7 (osd.7) 282 : cluster 0 3.11 scrub ok 2024-04-06T03:32:06.916 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:06 smithi050 bash[29221]: cluster 2024-04-06T03:32:06.190885+0000 mgr.y (mgr.24368) 411 : cluster 0 pgmap v423: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-04-06T03:32:06.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:06.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:07.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:07.043 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:06 smithi153 bash[23852]: audit 2024-04-06T03:32:05.745918+0000 mon.a (mon.0) 1240 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:32:07.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:06 smithi153 bash[23852]: cluster 2024-04-06T03:32:05.750678+0000 mon.a (mon.0) 1241 : cluster 0 osdmap e209: 8 total, 8 up, 8 in 2024-04-06T03:32:07.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:06 smithi153 bash[23852]: cluster 2024-04-06T03:32:05.752375+0000 osd.7 (osd.7) 281 : cluster 0 3.11 scrub starts 2024-04-06T03:32:07.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:06 smithi153 bash[23852]: cluster 2024-04-06T03:32:05.753977+0000 osd.7 (osd.7) 282 : cluster 0 3.11 scrub ok 2024-04-06T03:32:07.044 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:06 smithi153 bash[23852]: cluster 2024-04-06T03:32:06.190885+0000 mgr.y (mgr.24368) 411 : cluster 0 pgmap v423: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-04-06T03:32:07.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:07.126 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:32:07.126 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:32:07.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:07.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:07.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:07.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:07.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:07.434 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:07.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:07.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:07.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:07.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:07.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:07.775 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:32:07.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:07.840 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:07.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:07.964 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:07.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:07.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:07 smithi050 bash[29221]: cluster 2024-04-06T03:32:06.802935+0000 osd.7 (osd.7) 283 : cluster 0 5.9 deep-scrub starts 2024-04-06T03:32:07.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:07 smithi050 bash[29221]: cluster 2024-04-06T03:32:06.805179+0000 osd.7 (osd.7) 284 : cluster 0 5.9 deep-scrub ok 2024-04-06T03:32:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:07 smithi050 bash[29221]: audit 2024-04-06T03:32:06.996584+0000 mon.a (mon.0) 1242 : audit 1 from='client.? 172.21.15.50:0/528407040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:32:07.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:07 smithi050 bash[21467]: cluster 2024-04-06T03:32:06.802935+0000 osd.7 (osd.7) 283 : cluster 0 5.9 deep-scrub starts 2024-04-06T03:32:07.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:07 smithi050 bash[21467]: cluster 2024-04-06T03:32:06.805179+0000 osd.7 (osd.7) 284 : cluster 0 5.9 deep-scrub ok 2024-04-06T03:32:07.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:07 smithi050 bash[21467]: audit 2024-04-06T03:32:06.996584+0000 mon.a (mon.0) 1242 : audit 1 from='client.? 172.21.15.50:0/528407040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:32:08.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:07 smithi153 bash[23852]: cluster 2024-04-06T03:32:06.802935+0000 osd.7 (osd.7) 283 : cluster 0 5.9 deep-scrub starts 2024-04-06T03:32:08.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:07 smithi153 bash[23852]: cluster 2024-04-06T03:32:06.805179+0000 osd.7 (osd.7) 284 : cluster 0 5.9 deep-scrub ok 2024-04-06T03:32:08.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:07 smithi153 bash[23852]: audit 2024-04-06T03:32:06.996584+0000 mon.a (mon.0) 1242 : audit 1 from='client.? 172.21.15.50:0/528407040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:32:08.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:08.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:08.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:08.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:08.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:08.485 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:08.509 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:08.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:08.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:08.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:08.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:08.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:08.902 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:08.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:06.975819+0000 osd.4 (osd.4) 269 : cluster 0 6.e scrub starts 2024-04-06T03:32:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:06.991776+0000 osd.4 (osd.4) 270 : cluster 0 6.e scrub ok 2024-04-06T03:32:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:07.709471+0000 osd.3 (osd.3) 271 : cluster 0 6.13 scrub starts 2024-04-06T03:32:08.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:07.755295+0000 osd.3 (osd.3) 272 : cluster 0 6.13 scrub ok 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:07.758681+0000 mon.a (mon.0) 1243 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: audit 2024-04-06T03:32:07.769459+0000 mon.a (mon.0) 1244 : audit 1 from='client.? 172.21.15.50:0/528407040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:07.777109+0000 mon.a (mon.0) 1245 : cluster 0 osdmap e210: 8 total, 8 up, 8 in 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[21467]: cluster 2024-04-06T03:32:08.191357+0000 mgr.y (mgr.24368) 412 : cluster 0 pgmap v425: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 554 op/s 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:06.975819+0000 osd.4 (osd.4) 269 : cluster 0 6.e scrub starts 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:06.991776+0000 osd.4 (osd.4) 270 : cluster 0 6.e scrub ok 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:07.709471+0000 osd.3 (osd.3) 271 : cluster 0 6.13 scrub starts 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:07.755295+0000 osd.3 (osd.3) 272 : cluster 0 6.13 scrub ok 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:07.758681+0000 mon.a (mon.0) 1243 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: audit 2024-04-06T03:32:07.769459+0000 mon.a (mon.0) 1244 : audit 1 from='client.? 172.21.15.50:0/528407040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:07.777109+0000 mon.a (mon.0) 1245 : cluster 0 osdmap e210: 8 total, 8 up, 8 in 2024-04-06T03:32:08.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:08 smithi050 bash[29221]: cluster 2024-04-06T03:32:08.191357+0000 mgr.y (mgr.24368) 412 : cluster 0 pgmap v425: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 554 op/s 2024-04-06T03:32:09.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:09.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:09.045 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:06.975819+0000 osd.4 (osd.4) 269 : cluster 0 6.e scrub starts 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:06.991776+0000 osd.4 (osd.4) 270 : cluster 0 6.e scrub ok 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:07.709471+0000 osd.3 (osd.3) 271 : cluster 0 6.13 scrub starts 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:07.755295+0000 osd.3 (osd.3) 272 : cluster 0 6.13 scrub ok 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:07.758681+0000 mon.a (mon.0) 1243 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: audit 2024-04-06T03:32:07.769459+0000 mon.a (mon.0) 1244 : audit 1 from='client.? 172.21.15.50:0/528407040' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:07.777109+0000 mon.a (mon.0) 1245 : cluster 0 osdmap e210: 8 total, 8 up, 8 in 2024-04-06T03:32:09.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:08 smithi153 bash[23852]: cluster 2024-04-06T03:32:08.191357+0000 mgr.y (mgr.24368) 412 : cluster 0 pgmap v425: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 554 op/s 2024-04-06T03:32:09.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:09.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:09.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:09.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:09.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:09.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:09.485 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:32:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:32:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:32:09.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:09.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:09.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:09.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:32:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:32:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:09.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:09.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:09.921 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:09.932 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:09 smithi153 bash[23852]: audit 2024-04-06T03:32:09.519164+0000 mgr.y (mgr.24368) 413 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:09.945 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:09.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:09.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:09 smithi050 bash[21467]: audit 2024-04-06T03:32:09.519164+0000 mgr.y (mgr.24368) 413 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:09.966 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:09 smithi050 bash[29221]: audit 2024-04-06T03:32:09.519164+0000 mgr.y (mgr.24368) 413 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:10.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:10.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:10.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:10.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:10.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:10.326 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:10.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:10.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:10.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:10.581 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:10.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:10.715 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:10.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:10.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 5 1 2024-04-06T03:32:10.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:10.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:10.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:10 smithi153 bash[23852]: cluster 2024-04-06T03:32:10.194424+0000 mgr.y (mgr.24368) 414 : cluster 0 pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 790 B/s rd, 0 B/s wr, 898 op/s 2024-04-06T03:32:10.974 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:10.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:11.080 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:11.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:11.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:11.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:10 smithi050 bash[21467]: cluster 2024-04-06T03:32:10.194424+0000 mgr.y (mgr.24368) 414 : cluster 0 pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 790 B/s rd, 0 B/s wr, 898 op/s 2024-04-06T03:32:11.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:10 smithi050 bash[29221]: cluster 2024-04-06T03:32:10.194424+0000 mgr.y (mgr.24368) 414 : cluster 0 pgmap v426: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 790 B/s rd, 0 B/s wr, 898 op/s 2024-04-06T03:32:11.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:11.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:11.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:11.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:11.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:11.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:11.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:32:11.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-06T03:32:11.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:11.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:11.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:11.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:11.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:12.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:12.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:12.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:12.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:12.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:12.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:12.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:12.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:12.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:12.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:12.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:12.726 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:12.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:12.838 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[21467]: cluster 2024-04-06T03:32:11.726453+0000 osd.2 (osd.2) 225 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:12.838 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[21467]: cluster 2024-04-06T03:32:11.726527+0000 osd.2 (osd.2) 226 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:12.838 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[21467]: cluster 2024-04-06T03:32:11.802040+0000 mon.a (mon.0) 1246 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-04-06T03:32:12.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[21467]: cluster 2024-04-06T03:32:12.195067+0000 mgr.y (mgr.24368) 415 : cluster 0 pgmap v428: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 562 op/s 2024-04-06T03:32:12.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[29221]: cluster 2024-04-06T03:32:11.726453+0000 osd.2 (osd.2) 225 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:12.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[29221]: cluster 2024-04-06T03:32:11.726527+0000 osd.2 (osd.2) 226 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:12.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[29221]: cluster 2024-04-06T03:32:11.802040+0000 mon.a (mon.0) 1246 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-04-06T03:32:12.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:12 smithi050 bash[29221]: cluster 2024-04-06T03:32:12.195067+0000 mgr.y (mgr.24368) 415 : cluster 0 pgmap v428: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 562 op/s 2024-04-06T03:32:12.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:12.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:12.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:12.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:12 smithi153 bash[23852]: cluster 2024-04-06T03:32:11.726453+0000 osd.2 (osd.2) 225 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:12.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:12 smithi153 bash[23852]: cluster 2024-04-06T03:32:11.726527+0000 osd.2 (osd.2) 226 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:12.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:12 smithi153 bash[23852]: cluster 2024-04-06T03:32:11.802040+0000 mon.a (mon.0) 1246 : cluster 0 osdmap e211: 8 total, 8 up, 8 in 2024-04-06T03:32:12.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:12 smithi153 bash[23852]: cluster 2024-04-06T03:32:12.195067+0000 mgr.y (mgr.24368) 415 : cluster 0 pgmap v428: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 794 B/s rd, 0 B/s wr, 562 op/s 2024-04-06T03:32:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:12.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:13.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:13.100 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:13.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:13.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:13.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:13.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:13.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:13.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:13.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:13.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:32:13.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:13.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:13.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:13.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:13.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:13.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:14.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:14.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:14.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:14.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:14.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:14.310 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:14.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:14.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:14.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:14.570 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:14.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:14.706 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:14.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:14 smithi050 bash[21467]: cluster 2024-04-06T03:32:12.953404+0000 osd.0 (osd.0) 257 : cluster 0 8.6 deep-scrub ok 2024-04-06T03:32:14.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:14 smithi050 bash[29221]: cluster 2024-04-06T03:32:12.953404+0000 osd.0 (osd.0) 257 : cluster 0 8.6 deep-scrub ok 2024-04-06T03:32:14.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:14 smithi153 bash[23852]: cluster 2024-04-06T03:32:12.953404+0000 osd.0 (osd.0) 257 : cluster 0 8.6 deep-scrub ok 2024-04-06T03:32:14.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:14.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:14.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:14.967 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:14.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:15.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:15.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:15.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:15.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:15.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:15.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.630 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:15.709 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:15.710 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:15.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:15.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:15 smithi050 bash[21467]: cluster 2024-04-06T03:32:14.195618+0000 mgr.y (mgr.24368) 416 : cluster 0 pgmap v429: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-04-06T03:32:15.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:15 smithi050 bash[29221]: cluster 2024-04-06T03:32:14.195618+0000 mgr.y (mgr.24368) 416 : cluster 0 pgmap v429: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-04-06T03:32:15.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:15 smithi153 bash[23852]: cluster 2024-04-06T03:32:14.195618+0000 mgr.y (mgr.24368) 416 : cluster 0 pgmap v429: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-04-06T03:32:15.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:15.910 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:16.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:16.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:16.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:16.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:16.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:16.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:16.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:16.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:16.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:16.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:16.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:16.719 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:16.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:16.857 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:16.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:16.986 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:16.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:17.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:17.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:17.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:17.262 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:17.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:17.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:17.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:17.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:17.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:17.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:17.646 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:17 smithi050 bash[21467]: cluster 2024-04-06T03:32:16.196554+0000 mgr.y (mgr.24368) 417 : cluster 0 pgmap v430: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 964 op/s 2024-04-06T03:32:17.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:17 smithi050 bash[29221]: cluster 2024-04-06T03:32:16.196554+0000 mgr.y (mgr.24368) 417 : cluster 0 pgmap v430: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 964 op/s 2024-04-06T03:32:17.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:17.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:17.739 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:17.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:17.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:17 smithi153 bash[23852]: cluster 2024-04-06T03:32:16.196554+0000 mgr.y (mgr.24368) 417 : cluster 0 pgmap v430: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 964 op/s 2024-04-06T03:32:17.801 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:17.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:17.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:17.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:18.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:18.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:18.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:18.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:18.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:18.330 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:18.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:18.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:18.591 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:18.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:18.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:18.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:18.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:18.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:18.840 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:18.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:18.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:18.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:19.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:19.100 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:19.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:19.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:19.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:19.360 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:19.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:19.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:19.533 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:32:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:32:19.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:19.611 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:32:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:32:19.611 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:19 smithi153 bash[23852]: cluster 2024-04-06T03:32:18.196905+0000 mgr.y (mgr.24368) 418 : cluster 0 pgmap v431: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 812 op/s 2024-04-06T03:32:19.611 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:19 smithi153 bash[23852]: audit 2024-04-06T03:32:19.112844+0000 mon.a (mon.0) 1247 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:19.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:19 smithi153 bash[23852]: audit 2024-04-06T03:32:19.114959+0000 mon.a (mon.0) 1248 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:19.621 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:19.696 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[29221]: cluster 2024-04-06T03:32:18.196905+0000 mgr.y (mgr.24368) 418 : cluster 0 pgmap v431: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 812 op/s 2024-04-06T03:32:19.696 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[29221]: audit 2024-04-06T03:32:19.112844+0000 mon.a (mon.0) 1247 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:19.696 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[29221]: audit 2024-04-06T03:32:19.114959+0000 mon.a (mon.0) 1248 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:19.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:19.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:19.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:19.872 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[21467]: cluster 2024-04-06T03:32:18.196905+0000 mgr.y (mgr.24368) 418 : cluster 0 pgmap v431: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 812 op/s 2024-04-06T03:32:19.872 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[21467]: audit 2024-04-06T03:32:19.112844+0000 mon.a (mon.0) 1247 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:19.872 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:19 smithi050 bash[21467]: audit 2024-04-06T03:32:19.114959+0000 mon.a (mon.0) 1248 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:19.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:19.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:19.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:20.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:20.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:20.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:20.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:20.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:20.270 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:20.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:20.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:20.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:20.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:20.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:20.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:20.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:20 smithi050 bash[21467]: audit 2024-04-06T03:32:19.527237+0000 mgr.y (mgr.24368) 419 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:20.649 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:20 smithi050 bash[29221]: audit 2024-04-06T03:32:19.527237+0000 mgr.y (mgr.24368) 419 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:20.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:20.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:20.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:20 smithi153 bash[23852]: audit 2024-04-06T03:32:19.527237+0000 mgr.y (mgr.24368) 419 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:20.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:20.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:20.943 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:20.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:21.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:21.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:21.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:21.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:21.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:21.340 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:21.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:21.465 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:21.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:21.577 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:21 smithi153 bash[23852]: cluster 2024-04-06T03:32:20.197545+0000 mgr.y (mgr.24368) 420 : cluster 0 pgmap v432: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-04-06T03:32:21.590 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:21.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:21.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:21.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:21.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:21 smithi050 bash[21467]: cluster 2024-04-06T03:32:20.197545+0000 mgr.y (mgr.24368) 420 : cluster 0 pgmap v432: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-04-06T03:32:21.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:21 smithi050 bash[29221]: cluster 2024-04-06T03:32:20.197545+0000 mgr.y (mgr.24368) 420 : cluster 0 pgmap v432: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 919 op/s 2024-04-06T03:32:21.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:21.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:21.982 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:21.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:22.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:22.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:22.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:22.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:22.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:22.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:22.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:22.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:22.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:22.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:22.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:22.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:22.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:22.944 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:22.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:23.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:23.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:23.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:23.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:23.194 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:23.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:23.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:23.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:23.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:23.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:23.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:23.595 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:23.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:23.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:23.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:23.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:23 smithi050 bash[21467]: cluster 2024-04-06T03:32:22.198095+0000 mgr.y (mgr.24368) 421 : cluster 0 pgmap v433: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 984 B/s rd, 0 B/s wr, 922 op/s 2024-04-06T03:32:23.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:23 smithi050 bash[29221]: cluster 2024-04-06T03:32:22.198095+0000 mgr.y (mgr.24368) 421 : cluster 0 pgmap v433: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 984 B/s rd, 0 B/s wr, 922 op/s 2024-04-06T03:32:23.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:23.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:23.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:24.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:23 smithi153 bash[23852]: cluster 2024-04-06T03:32:22.198095+0000 mgr.y (mgr.24368) 421 : cluster 0 pgmap v433: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 984 B/s rd, 0 B/s wr, 922 op/s 2024-04-06T03:32:24.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:24.123 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:24.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:24.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:24.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:24.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:24.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:24.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:24.639 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:24.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:24.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:24.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:24.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:24.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:24.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:25.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:25.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:25.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:25.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:25.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:25.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:25.310 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:25.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:25.440 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:25.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:25.573 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:25.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:25.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:25.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:25 smithi050 bash[21467]: cluster 2024-04-06T03:32:24.198732+0000 mgr.y (mgr.24368) 422 : cluster 0 pgmap v434: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:32:25.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:25 smithi050 bash[21467]: audit 2024-04-06T03:32:25.539754+0000 mon.a (mon.0) 1249 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:25.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:25 smithi050 bash[29221]: cluster 2024-04-06T03:32:24.198732+0000 mgr.y (mgr.24368) 422 : cluster 0 pgmap v434: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:32:25.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:25 smithi050 bash[29221]: audit 2024-04-06T03:32:25.539754+0000 mon.a (mon.0) 1249 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:25.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:25.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:25.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:25.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:26.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:25 smithi153 bash[23852]: cluster 2024-04-06T03:32:24.198732+0000 mgr.y (mgr.24368) 422 : cluster 0 pgmap v434: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:32:26.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:25 smithi153 bash[23852]: audit 2024-04-06T03:32:25.539754+0000 mon.a (mon.0) 1249 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:32:26.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:26.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:26.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:26.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:26.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:26.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:26.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:26.506 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:26.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:26.621 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.250815+0000 mon.a (mon.0) 1250 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.621 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.258958+0000 mon.a (mon.0) 1251 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.264550+0000 mon.a (mon.0) 1252 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.277605+0000 mon.a (mon.0) 1253 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.298935+0000 mon.a (mon.0) 1254 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.357271+0000 mon.a (mon.0) 1255 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.359125+0000 mon.a (mon.0) 1256 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.360906+0000 mon.a (mon.0) 1257 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.362654+0000 mon.a (mon.0) 1258 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.375706+0000 mon.a (mon.0) 1259 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.379958+0000 mon.a (mon.0) 1260 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.380901+0000 mon.a (mon.0) 1261 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:26.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:26 smithi153 bash[23852]: audit 2024-04-06T03:32:26.387253+0000 mon.a (mon.0) 1262 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:26.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:26.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:26.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:26.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:26.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.250815+0000 mon.a (mon.0) 1250 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.258958+0000 mon.a (mon.0) 1251 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.264550+0000 mon.a (mon.0) 1252 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.277605+0000 mon.a (mon.0) 1253 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.298935+0000 mon.a (mon.0) 1254 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.357271+0000 mon.a (mon.0) 1255 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.359125+0000 mon.a (mon.0) 1256 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.360906+0000 mon.a (mon.0) 1257 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.362654+0000 mon.a (mon.0) 1258 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.375706+0000 mon.a (mon.0) 1259 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.379958+0000 mon.a (mon.0) 1260 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.380901+0000 mon.a (mon.0) 1261 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[21467]: audit 2024-04-06T03:32:26.387253+0000 mon.a (mon.0) 1262 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.250815+0000 mon.a (mon.0) 1250 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.258958+0000 mon.a (mon.0) 1251 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.264550+0000 mon.a (mon.0) 1252 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.277605+0000 mon.a (mon.0) 1253 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.298935+0000 mon.a (mon.0) 1254 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.357271+0000 mon.a (mon.0) 1255 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.359125+0000 mon.a (mon.0) 1256 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.360906+0000 mon.a (mon.0) 1257 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.362654+0000 mon.a (mon.0) 1258 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-06T03:32:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.375706+0000 mon.a (mon.0) 1259 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.379958+0000 mon.a (mon.0) 1260 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:32:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.380901+0000 mon.a (mon.0) 1261 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:32:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:26 smithi050 bash[29221]: audit 2024-04-06T03:32:26.387253+0000 mon.a (mon.0) 1262 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:32:26.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:27.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:27.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:27.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:27.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:27.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:27.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:27.422 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:27.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:27.563 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:27.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:27.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:27 smithi153 bash[23852]: cluster 2024-04-06T03:32:26.199494+0000 mgr.y (mgr.24368) 423 : cluster 0 pgmap v435: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-04-06T03:32:27.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:27 smithi153 bash[23852]: cephadm 2024-04-06T03:32:26.290663+0000 mgr.y (mgr.24368) 424 : cephadm 1 Adjusting osd_memory_target on smithi153 to 3292M 2024-04-06T03:32:27.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:27 smithi153 bash[23852]: cephadm 2024-04-06T03:32:26.364102+0000 mgr.y (mgr.24368) 425 : cephadm 1 Adjusting osd_memory_target on smithi050 to 3293M 2024-04-06T03:32:27.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:27 smithi153 bash[23852]: cluster 2024-04-06T03:32:26.785043+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:27.677 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:27 smithi153 bash[23852]: cluster 2024-04-06T03:32:26.785111+0000 osd.0 (osd.0) 259 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:27.703 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:27.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[21467]: cluster 2024-04-06T03:32:26.199494+0000 mgr.y (mgr.24368) 423 : cluster 0 pgmap v435: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-04-06T03:32:27.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[21467]: cephadm 2024-04-06T03:32:26.290663+0000 mgr.y (mgr.24368) 424 : cephadm 1 Adjusting osd_memory_target on smithi153 to 3292M 2024-04-06T03:32:27.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[21467]: cephadm 2024-04-06T03:32:26.364102+0000 mgr.y (mgr.24368) 425 : cephadm 1 Adjusting osd_memory_target on smithi050 to 3293M 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[21467]: cluster 2024-04-06T03:32:26.785043+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[21467]: cluster 2024-04-06T03:32:26.785111+0000 osd.0 (osd.0) 259 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[29221]: cluster 2024-04-06T03:32:26.199494+0000 mgr.y (mgr.24368) 423 : cluster 0 pgmap v435: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[29221]: cephadm 2024-04-06T03:32:26.290663+0000 mgr.y (mgr.24368) 424 : cephadm 1 Adjusting osd_memory_target on smithi153 to 3292M 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[29221]: cephadm 2024-04-06T03:32:26.364102+0000 mgr.y (mgr.24368) 425 : cephadm 1 Adjusting osd_memory_target on smithi050 to 3293M 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[29221]: cluster 2024-04-06T03:32:26.785043+0000 osd.0 (osd.0) 258 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:27.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:27 smithi050 bash[29221]: cluster 2024-04-06T03:32:26.785111+0000 osd.0 (osd.0) 259 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:27.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:27.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:27.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:27.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:27.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:28.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:28.087 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:28.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:28.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:28.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:28.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:28.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:28.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:28.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:28.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:28.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:28.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:28 smithi050 bash[21467]: cluster 2024-04-06T03:32:27.582781+0000 mon.a (mon.0) 1263 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-04-06T03:32:28.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:28 smithi050 bash[29221]: cluster 2024-04-06T03:32:27.582781+0000 mon.a (mon.0) 1263 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-04-06T03:32:28.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:28.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:28.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:28.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:28.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:28.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:29.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:29.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:28 smithi153 bash[23852]: cluster 2024-04-06T03:32:27.582781+0000 mon.a (mon.0) 1263 : cluster 0 osdmap e212: 8 total, 8 up, 8 in 2024-04-06T03:32:29.130 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:29.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:29.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:29.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:29.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:29.416 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:29.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:29.529 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:32:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:32:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:32:29.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:29.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:29.661 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:32:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:32:29.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:29 smithi153 bash[23852]: cluster 2024-04-06T03:32:28.201780+0000 mgr.y (mgr.24368) 426 : cluster 0 pgmap v437: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 935 op/s 2024-04-06T03:32:29.678 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:29.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:29.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:29 smithi050 bash[21467]: cluster 2024-04-06T03:32:28.201780+0000 mgr.y (mgr.24368) 426 : cluster 0 pgmap v437: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 935 op/s 2024-04-06T03:32:29.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:29 smithi050 bash[29221]: cluster 2024-04-06T03:32:28.201780+0000 mgr.y (mgr.24368) 426 : cluster 0 pgmap v437: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 935 op/s 2024-04-06T03:32:29.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:29.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:29.946 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:29.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:30.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:30.073 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:30.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:30.198 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:30.300 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:30.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:30.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:30.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:30.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:30.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:30.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:30 smithi050 bash[21467]: audit 2024-04-06T03:32:29.535239+0000 mgr.y (mgr.24368) 427 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:30.722 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:30 smithi050 bash[29221]: audit 2024-04-06T03:32:29.535239+0000 mgr.y (mgr.24368) 427 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:30.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:30.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:30.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:30.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:30.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:31.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:31.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:30 smithi153 bash[23852]: audit 2024-04-06T03:32:29.535239+0000 mgr.y (mgr.24368) 427 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:31.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:31.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:31.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:31.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:31.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:31.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:31.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:31.538 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:31.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:31.669 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:31.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:31 smithi050 bash[21467]: cluster 2024-04-06T03:32:30.202974+0000 mgr.y (mgr.24368) 428 : cluster 0 pgmap v438: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 758 op/s 2024-04-06T03:32:31.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:31 smithi050 bash[29221]: cluster 2024-04-06T03:32:30.202974+0000 mgr.y (mgr.24368) 428 : cluster 0 pgmap v438: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 758 op/s 2024-04-06T03:32:31.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:31.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:31 smithi153 bash[23852]: cluster 2024-04-06T03:32:30.202974+0000 mgr.y (mgr.24368) 428 : cluster 0 pgmap v438: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 758 op/s 2024-04-06T03:32:31.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:31.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:31.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:31.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:32.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:32.057 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:32.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:32.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:32.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:32.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:32.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:32.449 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:32.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:32.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:32.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:32.733 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:32.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:32.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:32.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:32.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:32.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:33.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:33.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:33.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:33.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:33.267 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:33.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:33.498 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:33.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:33.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:33 smithi050 bash[21467]: cluster 2024-04-06T03:32:32.205868+0000 mgr.y (mgr.24368) 429 : cluster 0 pgmap v439: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 849 op/s 2024-04-06T03:32:33.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:33 smithi050 bash[29221]: cluster 2024-04-06T03:32:32.205868+0000 mgr.y (mgr.24368) 429 : cluster 0 pgmap v439: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 849 op/s 2024-04-06T03:32:33.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:33.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:33.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:33 smithi153 bash[23852]: cluster 2024-04-06T03:32:32.205868+0000 mgr.y (mgr.24368) 429 : cluster 0 pgmap v439: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 849 op/s 2024-04-06T03:32:33.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:33.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:33.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:33.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:34.011 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:32:34.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:34.060 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:32:34.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:34.116 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:32:34.163 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:32:34.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:34.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:34.231 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:32:34.272 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:32:34.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:34.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:34.343 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:32:34.422 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:32:34.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:32:34.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:32:34.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:32:34.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:32:34.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:32:34.430 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:32:34.431 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:32:34.431 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:32:34.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:34.466 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:31542 7.f 7:f0d617c7:::benchmark_data_smithi050_68188_object31541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.833422+0000", 2024-04-06T03:32:34.489 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.65398115199997, 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48967959, 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 31542 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833422+0000", 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833422+0000", 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833423+0000", 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4309999999999999e-06 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833448+0000", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4504000000000001e-05 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833461+0000", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3033e-05 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833469+0000", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4689999999999997e-06 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.833530+0000", 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.084e-05 2024-04-06T03:32:34.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835450+0000", 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019200529999999999 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835585+0000", 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013447099999999999 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840488+0000", 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049033389999999996 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840501+0000", 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2962000000000001e-05 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.322814+0000", 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48231300700000002 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.322981+0000", 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016749399999999999 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.323101+0000", 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119987 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:31547 7.f 7:f9048c42:::benchmark_data_smithi050_68188_object31546:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.835101+0000", 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.65230181099997, 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48834317799999999, 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.495 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 31547 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835101+0000", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835101+0000", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835103+0000", 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6270000000000001e-06 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835124+0000", 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1393e-05 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835133+0000", 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1279999999999997e-06 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.835140+0000", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5309999999999998e-06 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.837139+0000", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001999298 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.837305+0000", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016571500000000001 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.837529+0000", 2024-04-06T03:32:34.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022405600000000001 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316846+0000", 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47931763999999999 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316943+0000", 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6683999999999998e-05 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.323210+0000", 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062666750000000002 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.323345+0000", 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013552900000000001 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.323444+0000", 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8901999999999996e-05 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14745.0:31553 7.e e136/127 7:7e9d210e:::benchmark_data_smithi050_68188_object31552:head v 136'1991, mlcod=136'1989)", 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.838643+0000", 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.64875952, 2024-04-06T03:32:34.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47878783000000003, 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838643+0000", 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838643+0000", 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838644+0000", 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5199999999999996e-07 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838669+0000", 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4587e-05 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838678+0000", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6099999999999995e-06 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838685+0000", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7100000000000001e-06 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839019+0000", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033391200000000001 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839039+0000", 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0480999999999999e-05 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317340+0000", 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47830087599999999 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317431+0000", 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0902000000000005e-05 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14745.0:31547 7.f e136/127 ondisk, result = 0)", 2024-04-06T03:32:34.504 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.838960+0000", 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.64844258699998, 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47793719299999998, 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838960+0000", 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838960+0000", 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838961+0000", 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3799999999999997e-07 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838965+0000", 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0740000000000003e-06 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838973+0000", 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6650000000000003e-06 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.838979+0000", 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.241e-06 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840526+0000", 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001547475 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316809+0000", 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47628254399999997 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316898+0000", 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8555999999999997e-05 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:31556 7.6 7:6c877756:::benchmark_data_smithi050_68188_object31555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.839019+0000", 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.64838359599997, 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49157616399999998, 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.508 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 31556 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839019+0000", 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839019+0000", 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839020+0000", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.95e-07 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839039+0000", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8369e-05 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839047+0000", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.276e-06 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839053+0000", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5879999999999997e-06 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839093+0000", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0321999999999998e-05 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839210+0000", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117358 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.839368+0000", 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000158262 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317307+0000", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47793893799999998 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.320807+0000", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034999409999999999 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.330429+0000", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0096218359999999999 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.330522+0000", 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2952e-05 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.330595+0000", 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3326999999999998e-05 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:31558 7.6 7:669bf480:::benchmark_data_smithi050_68188_object31557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.840581+0000", 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.646821434, 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49024183599999999, 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 31558 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840581+0000", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840581+0000", 2024-04-06T03:32:34.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840582+0000", 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.017e-06 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.840608+0000", 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5041999999999998e-05 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316924+0000", 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47631602699999998 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316959+0000", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5913e-05 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317418+0000", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045902200000000001 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317701+0000", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028252799999999999 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317962+0000", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026082299999999998 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.328980+0000", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011018577 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.329015+0000", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5042000000000001e-05 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.330665+0000", 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001649314 2024-04-06T03:32:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.330748+0000", 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3500999999999999e-05 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.330823+0000", 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5030000000000005e-05 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14745.0:31556 7.6 e136/127 ondisk, result = 0)", 2024-04-06T03:32:34.518 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:25:55.842034+0000", 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 398.64536888700002, 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47877996299999998, 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.842034+0000", 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.842034+0000", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.842035+0000", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7800000000000001e-07 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:55.842039+0000", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2520000000000001e-06 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316809+0000", 2024-04-06T03:32:34.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47476989400000003 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.316844+0000", 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4693000000000001e-05 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317278+0000", 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043484900000000001 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.317297+0000", 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8490000000000001e-05 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:25:56.320814+0000", 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035171069999999998 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:82343 7.9 7:9734688e:::benchmark_data_smithi050_68188_object82342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:47.209898+0000", 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 347.27750511900001, 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42609865400000002, 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.522 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82343 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.209898+0000", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.209898+0000", 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.209899+0000", 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.265e-06 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.209920+0000", 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0869e-05 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.209932+0000", 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2084e-05 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.209941+0000", 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8319999999999995e-06 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.210000+0000", 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8727000000000002e-05 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.210122+0000", 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000122173 2024-04-06T03:32:34.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.210247+0000", 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012547099999999999 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.213764+0000", 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035166339999999998 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.215000+0000", 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00123616 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635879+0000", 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42087897499999999 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635944+0000", 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5313000000000003e-05 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635996+0000", 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2151000000000003e-05 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:82348 7.9 7:9bf363a2:::benchmark_data_smithi050_68188_object82347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:47.218293+0000", 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 347.26910988999998, 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42432732099999998, 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82348 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218293+0000", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218293+0000", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218294+0000", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7000000000000003e-07 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218336+0000", 2024-04-06T03:32:34.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1576999999999999e-05 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218412+0000", 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6439000000000005e-05 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218420+0000", 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8599999999999993e-06 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218487+0000", 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7528999999999997e-05 2024-04-06T03:32:34.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218611+0000", 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000124072 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.625757+0000", 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40714509100000001 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.634934+0000", 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0091773940000000002 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.636238+0000", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013041769999999999 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.642504+0000", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062659559999999996 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.642567+0000", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2842000000000006e-05 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.642620+0000", 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3414000000000001e-05 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:82349 7.6 7:68236487:::benchmark_data_smithi050_68188_object82348:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:47.218453+0000", 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 347.26894979100001, 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42491638199999998, 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.533 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82349 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218453+0000", 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218453+0000", 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218454+0000", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1600000000000001e-07 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218478+0000", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4629000000000001e-05 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218487+0000", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3000000000000002e-06 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218493+0000", 2024-04-06T03:32:34.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6290000000000004e-06 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218536+0000", 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3195000000000002e-05 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218622+0000", 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5618000000000006e-05 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.625800+0000", 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40717767300000002 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.640229+0000", 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014429632 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.640292+0000", 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2458000000000004e-05 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.643236+0000", 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002944136 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.643294+0000", 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8226999999999997e-05 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.643369+0000", 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5269000000000001e-05 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:82350 7.6 7:66ea6689:::benchmark_data_smithi050_68188_object82349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:47.218507+0000", 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 347.26889615300001, 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.432632817, 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82350 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218507+0000", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218507+0000", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218507+0000", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.58e-07 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218524+0000", 2024-04-06T03:32:34.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6765999999999998e-05 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218529+0000", 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.045e-06 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218533+0000", 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8010000000000001e-06 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635202+0000", 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.41666937799999998 2024-04-06T03:32:34.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635335+0000", 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013254400000000001 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635439+0000", 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104483 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.643194+0000", 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077550359999999999 2024-04-06T03:32:34.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.649773+0000", 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065788679999999999 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.651015+0000", 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012417940000000001 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.651083+0000", 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7966000000000006e-05 2024-04-06T03:32:34.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.651140+0000", 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6678000000000002e-05 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:82351 7.9 7:91ecfae0:::benchmark_data_smithi050_68188_object82350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:26:47.218540+0000", 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 347.26886274700001, 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42852217500000001, 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 82351 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218540+0000", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218540+0000", 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218540+0000", 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.34e-07 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218557+0000", 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.607e-05 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218561+0000", 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3449999999999999e-06 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.218564+0000", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.551e-06 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.628751+0000", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.41018685100000002 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.631371+0000", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026195799999999998 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,4", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.631482+0000", 2024-04-06T03:32:34.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111195 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.635183+0000", 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037006809999999999 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.641087+0000", 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059040730000000001 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.646930+0000", 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058429110000000001 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.646989+0000", 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9092999999999998e-05 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:26:47.647062+0000", 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3491000000000005e-05 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:163115 7.6 7:661af92f:::benchmark_data_smithi050_68188_object66365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:28:27.819918+0000", 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 246.66748490699999, 2024-04-06T03:32:34.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.447578005, 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 163115 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.819918+0000", 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.819918+0000", 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.819920+0000", 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.809e-06 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.819928+0000", 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3429999999999996e-06 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.819945+0000", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7241999999999999e-05 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.819957+0000", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1155000000000001e-05 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.820030+0000", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3070000000000003e-05 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.820161+0000", 2024-04-06T03:32:34.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013119100000000001 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.820288+0000", 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000127219 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.825382+0000", 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005093841 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.826871+0000", 2024-04-06T03:32:34.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014891699999999999 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.267136+0000", 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.44026507300000001 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.267369+0000", 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023338499999999999 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.267496+0000", 2024-04-06T03:32:34.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012650699999999999 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:163126 7.f 7:f77e7c5f:::benchmark_data_smithi050_68188_object66376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:28:27.828676+0000", 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 246.658726809, 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45241259, 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:32:34.555 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 163126 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.828676+0000", 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.828676+0000", 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.828677+0000", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8800000000000001e-07 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.828680+0000", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.326e-06 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.991712+0000", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.16303188900000001 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.991744+0000", 2024-04-06T03:32:34.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1661000000000001e-05 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.991889+0000", 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000144881 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.992133+0000", 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024382700000000001 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-06T03:32:34.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.992205+0000", 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2519000000000001e-05 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.995946+0000", 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037409819999999999 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.996104+0000", 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015769800000000001 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.280947+0000", 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.28484349799999997 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.281020+0000", 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2451000000000001e-05 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.281089+0000", 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8969999999999999e-05 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5610 8.e 8:73268da6:::benchmark_data_smithi050_86143_object5609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.647345+0000", 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 152.84005814099999, 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55471053100000001, 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5610 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647345+0000", 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647345+0000", 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647345+0000", 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647367+0000", 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1223e-05 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647375+0000", 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5699999999999993e-06 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647382+0000", 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.861e-06 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647439+0000", 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7111999999999998e-05 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647552+0000", 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011257299999999999 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647702+0000", 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000150607 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648788+0000", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010853340000000001 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648799+0000", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1338000000000001e-05 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.201929+0000", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55312996199999998 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202000+0000", 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1260000000000006e-05 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202055+0000", 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5056000000000003e-05 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5616 8.e 8:7ca7657a:::benchmark_data_smithi050_86143_object5615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.651119+0000", 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 152.836283888, 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.551267905, 2024-04-06T03:32:34.566 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5616 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651119+0000", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651119+0000", 2024-04-06T03:32:34.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651120+0000", 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4900000000000005e-07 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651140+0000", 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0208999999999999e-05 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651149+0000", 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4640000000000006e-06 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651154+0000", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3159999999999996e-06 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651282+0000", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012765699999999999 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651475+0000", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019336200000000001 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651659+0000", 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000184339 2024-04-06T03:32:34.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652177+0000", 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051728899999999999 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.653010+0000", 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083326500000000003 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202158+0000", 2024-04-06T03:32:34.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54914820799999997 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202327+0000", 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016948899999999999 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202387+0000", 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9457999999999999e-05 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74150 8.e 8:75d6fd5a:::benchmark_data_smithi050_86143_object74149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.034619+0000", 2024-04-06T03:32:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.452784315, 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42533740599999997, 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74150 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034619+0000", 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034619+0000", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034620+0000", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3039999999999999e-06 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034647+0000", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6968e-05 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034655+0000", 2024-04-06T03:32:34.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9929999999999995e-06 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034660+0000", 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3630000000000002e-06 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034737+0000", 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6669999999999996e-05 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034823+0000", 2024-04-06T03:32:34.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6012000000000003e-05 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034903+0000", 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0284999999999994e-05 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.042936+0000", 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080326440000000002 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.045702+0000", 2024-04-06T03:32:34.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027660229999999998 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.459849+0000", 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.41414750299999997 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.459916+0000", 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7033999999999995e-05 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.459956+0000", 2024-04-06T03:32:34.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9607000000000002e-05 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74166 8.9 8:9a57aa0f:::benchmark_data_smithi050_86143_object74165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.051856+0000", 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.43554661500001, 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43935837799999999, 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74166 2024-04-06T03:32:34.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051856+0000", 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051856+0000", 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051857+0000", 2024-04-06T03:32:34.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0589999999999999e-06 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051881+0000", 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3932000000000001e-05 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051892+0000", 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0609e-05 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051900+0000", 2024-04-06T03:32:34.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.242e-06 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051967+0000", 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7236999999999993e-05 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.052101+0000", 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013316900000000001 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.052257+0000", 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015600299999999999 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.054653+0000", 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023967429999999998 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.057456+0000", 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028024019999999998 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491145+0000", 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43368966799999997 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491184+0000", 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8547999999999998e-05 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491215+0000", 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0766000000000003e-05 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74172 8.9 8:9f0adc0c:::benchmark_data_smithi050_86143_object74171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.060899+0000", 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.426503682, 2024-04-06T03:32:34.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43049049900000003, 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74172 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060899+0000", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060899+0000", 2024-04-06T03:32:34.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060900+0000", 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060931+0000", 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0071999999999999e-05 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060945+0000", 2024-04-06T03:32:34.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4661000000000001e-05 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060956+0000", 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1314999999999999e-05 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.061030+0000", 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3783999999999997e-05 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.061205+0000", 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017448400000000001 2024-04-06T03:32:34.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.061380+0000", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017525 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.064573+0000", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031926400000000001 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.064685+0000", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011252300000000001 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491276+0000", 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.426590786 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491343+0000", 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6563000000000006e-05 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491390+0000", 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7209999999999997e-05 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:134289 8.e 8:7bc21049:::benchmark_data_smithi050_86143_object27978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:03.585004+0000", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.902398822999999, 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42752563999999998, 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 134289 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585004+0000", 2024-04-06T03:32:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585004+0000", 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585005+0000", 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.33e-06 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585011+0000", 2024-04-06T03:32:34.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6500000000000001e-06 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585022+0000", 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0664e-05 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585030+0000", 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8800000000000008e-06 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.982776+0000", 2024-04-06T03:32:34.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39774652999999999 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.982892+0000", 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116047 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.983016+0000", 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123681 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.991834+0000", 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088182160000000002 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.997793+0000", 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059593930000000003 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:04.012230+0000", 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014436818000000001 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:04.012475+0000", 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024445599999999999 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:32:34.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:04.012530+0000", 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4975000000000001e-05 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:32:34.594 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:32:34.631 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:32:34.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:34.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:34 smithi050 bash[21467]: audit 2024-04-06T03:32:33.961612+0000 mon.c (mon.2) 65 : audit 0 from='client.? 172.21.15.50:0/1290258968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:32:34.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:34 smithi050 bash[21467]: audit 2024-04-06T03:32:34.139539+0000 mon.a (mon.0) 1264 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:34.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:34 smithi050 bash[21467]: audit 2024-04-06T03:32:34.574921+0000 mon.a (mon.0) 1265 : audit 1 from='client.? 172.21.15.50:0/3934761476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:32:34.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:34 smithi050 bash[29221]: audit 2024-04-06T03:32:33.961612+0000 mon.c (mon.2) 65 : audit 0 from='client.? 172.21.15.50:0/1290258968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:32:34.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:34 smithi050 bash[29221]: audit 2024-04-06T03:32:34.139539+0000 mon.a (mon.0) 1264 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:34.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:34 smithi050 bash[29221]: audit 2024-04-06T03:32:34.574921+0000 mon.a (mon.0) 1265 : audit 1 from='client.? 172.21.15.50:0/3934761476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:32:34.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:34.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:34.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:34.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:34.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:35.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:35.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:35.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:34 smithi153 bash[23852]: audit 2024-04-06T03:32:33.961612+0000 mon.c (mon.2) 65 : audit 0 from='client.? 172.21.15.50:0/1290258968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:32:35.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:34 smithi153 bash[23852]: audit 2024-04-06T03:32:34.139539+0000 mon.a (mon.0) 1264 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:35.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:34 smithi153 bash[23852]: audit 2024-04-06T03:32:34.574921+0000 mon.a (mon.0) 1265 : audit 1 from='client.? 172.21.15.50:0/3934761476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:32:35.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:35.159 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:35.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:35.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:35.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:35.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:35.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:35.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:35.635 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-04-06T03:32:35.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[21467]: cluster 2024-04-06T03:32:34.206559+0000 mgr.y (mgr.24368) 430 : cluster 0 pgmap v440: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s 2024-04-06T03:32:35.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[21467]: cluster 2024-04-06T03:32:34.615841+0000 mon.a (mon.0) 1266 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:32:35.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[21467]: audit 2024-04-06T03:32:34.628353+0000 mon.a (mon.0) 1267 : audit 1 from='client.? 172.21.15.50:0/3934761476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[21467]: cluster 2024-04-06T03:32:34.643121+0000 mon.a (mon.0) 1268 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[21467]: audit 2024-04-06T03:32:34.933677+0000 mon.a (mon.0) 1269 : audit 1 from='client.? 172.21.15.50:0/235863419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[29221]: cluster 2024-04-06T03:32:34.206559+0000 mgr.y (mgr.24368) 430 : cluster 0 pgmap v440: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[29221]: cluster 2024-04-06T03:32:34.615841+0000 mon.a (mon.0) 1266 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[29221]: audit 2024-04-06T03:32:34.628353+0000 mon.a (mon.0) 1267 : audit 1 from='client.? 172.21.15.50:0/3934761476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[29221]: cluster 2024-04-06T03:32:34.643121+0000 mon.a (mon.0) 1268 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-04-06T03:32:35.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:35 smithi050 bash[29221]: audit 2024-04-06T03:32:34.933677+0000 mon.a (mon.0) 1269 : audit 1 from='client.? 172.21.15.50:0/235863419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:32:35.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:35.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:35.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:35.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:35.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:35.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:35.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:36.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:35 smithi153 bash[23852]: cluster 2024-04-06T03:32:34.206559+0000 mgr.y (mgr.24368) 430 : cluster 0 pgmap v440: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 851 op/s 2024-04-06T03:32:36.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:35 smithi153 bash[23852]: cluster 2024-04-06T03:32:34.615841+0000 mon.a (mon.0) 1266 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:32:36.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:35 smithi153 bash[23852]: audit 2024-04-06T03:32:34.628353+0000 mon.a (mon.0) 1267 : audit 1 from='client.? 172.21.15.50:0/3934761476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:32:36.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:35 smithi153 bash[23852]: cluster 2024-04-06T03:32:34.643121+0000 mon.a (mon.0) 1268 : cluster 0 osdmap e213: 8 total, 8 up, 8 in 2024-04-06T03:32:36.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:35 smithi153 bash[23852]: audit 2024-04-06T03:32:34.933677+0000 mon.a (mon.0) 1269 : audit 1 from='client.? 172.21.15.50:0/235863419' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:32:36.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:36.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:36.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:36.216 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:36.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:36.342 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:36.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:36.471 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:36.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:36.594 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:36.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:36.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:36 smithi050 bash[21467]: audit 2024-04-06T03:32:35.629853+0000 mon.a (mon.0) 1270 : audit 1 from='client.? 172.21.15.50:0/235863419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:32:36.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:36 smithi050 bash[21467]: cluster 2024-04-06T03:32:35.635968+0000 mon.a (mon.0) 1271 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-04-06T03:32:36.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:36 smithi050 bash[29221]: audit 2024-04-06T03:32:35.629853+0000 mon.a (mon.0) 1270 : audit 1 from='client.? 172.21.15.50:0/235863419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:32:36.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:36 smithi050 bash[29221]: cluster 2024-04-06T03:32:35.635968+0000 mon.a (mon.0) 1271 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-04-06T03:32:36.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:36.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:36.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:36 smithi153 bash[23852]: audit 2024-04-06T03:32:35.629853+0000 mon.a (mon.0) 1270 : audit 1 from='client.? 172.21.15.50:0/235863419' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:32:36.839 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:36 smithi153 bash[23852]: cluster 2024-04-06T03:32:35.635968+0000 mon.a (mon.0) 1271 : cluster 0 osdmap e214: 8 total, 8 up, 8 in 2024-04-06T03:32:36.850 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:36.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:36.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:36.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:37.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:37.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:37.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:37.228 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:37.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:37.354 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:37.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:37.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:37.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:37.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:37.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:37.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:37.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:37 smithi153 bash[23852]: cluster 2024-04-06T03:32:36.207058+0000 mgr.y (mgr.24368) 431 : cluster 0 pgmap v443: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:32:37.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:37.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:37.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:37 smithi050 bash[21467]: cluster 2024-04-06T03:32:36.207058+0000 mgr.y (mgr.24368) 431 : cluster 0 pgmap v443: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:32:37.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:37 smithi050 bash[29221]: cluster 2024-04-06T03:32:36.207058+0000 mgr.y (mgr.24368) 431 : cluster 0 pgmap v443: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:32:37.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:38.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:38.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:38.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:38.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:38.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:38.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:38.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:38.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:38.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:38.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:38.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:38.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:38.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:38.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:38.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:38.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:39.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:39.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:39.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:39.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:39.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:39.332 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:39.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:39.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-06T03:32:39.431 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:32:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:32:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:32:39.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:39.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:39.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:39.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:32:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:32:39.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:39.688 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:39 smithi050 bash[21467]: cluster 2024-04-06T03:32:38.207486+0000 mgr.y (mgr.24368) 432 : cluster 0 pgmap v444: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:32:39.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:39 smithi050 bash[29221]: cluster 2024-04-06T03:32:38.207486+0000 mgr.y (mgr.24368) 432 : cluster 0 pgmap v444: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:32:39.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 6 1 2024-04-06T03:32:39.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:39.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:39 smithi153 bash[23852]: cluster 2024-04-06T03:32:38.207486+0000 mgr.y (mgr.24368) 432 : cluster 0 pgmap v444: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:32:39.813 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:39.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:39.942 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:39.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:40.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:40.087 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:40.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:40.229 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:40.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:40.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:40.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:40.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:40.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:40.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:40.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:40.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:40 smithi050 bash[21467]: audit 2024-04-06T03:32:39.543315+0000 mgr.y (mgr.24368) 433 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:40.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:40 smithi050 bash[29221]: audit 2024-04-06T03:32:39.543315+0000 mgr.y (mgr.24368) 433 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:40.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:40.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:40.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:40.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:41.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:41.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:32:41.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:41.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:40 smithi153 bash[23852]: audit 2024-04-06T03:32:39.543315+0000 mgr.y (mgr.24368) 433 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:41.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:41.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:41.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:41.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:41.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:32:41.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:41.465 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:41.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:41.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:41.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:41.724 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:41 smithi153 bash[23852]: cluster 2024-04-06T03:32:40.208221+0000 mgr.y (mgr.24368) 434 : cluster 0 pgmap v445: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 717 op/s 2024-04-06T03:32:41.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:41 smithi050 bash[21467]: cluster 2024-04-06T03:32:40.208221+0000 mgr.y (mgr.24368) 434 : cluster 0 pgmap v445: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 717 op/s 2024-04-06T03:32:41.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:41 smithi050 bash[29221]: cluster 2024-04-06T03:32:40.208221+0000 mgr.y (mgr.24368) 434 : cluster 0 pgmap v445: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 717 op/s 2024-04-06T03:32:41.746 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:41.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:41.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:41.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:42.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:42.143 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:42.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:42.269 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:42.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:42.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:42.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:42.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:42.661 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:42.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:42.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:42.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:42.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:42.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:43.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:43.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:43.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:43.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:43.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:43.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:43.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:43.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:43.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:43.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:43.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:43.746 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:43 smithi153 bash[23852]: cluster 2024-04-06T03:32:42.208895+0000 mgr.y (mgr.24368) 435 : cluster 0 pgmap v446: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s 2024-04-06T03:32:43.761 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:43.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:43.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:43.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:43.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:43 smithi050 bash[21467]: cluster 2024-04-06T03:32:42.208895+0000 mgr.y (mgr.24368) 435 : cluster 0 pgmap v446: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s 2024-04-06T03:32:43.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:43 smithi050 bash[29221]: cluster 2024-04-06T03:32:42.208895+0000 mgr.y (mgr.24368) 435 : cluster 0 pgmap v446: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 877 op/s 2024-04-06T03:32:43.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:44.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:44.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:44.143 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:44.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:44.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:44.311 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:44.312 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:44.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:44.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:44.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:44.543 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:44.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:44.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:44.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:44.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:44.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:44.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:45.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:45.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:45.198 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:45.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:45.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:45.330 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:45.331 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:32:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:45.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:45.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:45.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:45.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:45.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:45.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:45.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:45 smithi153 bash[23852]: cluster 2024-04-06T03:32:44.209831+0000 mgr.y (mgr.24368) 436 : cluster 0 pgmap v447: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 872 op/s 2024-04-06T03:32:45.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:45.857 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:45.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:45.983 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:45.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:45.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:45 smithi050 bash[21467]: cluster 2024-04-06T03:32:44.209831+0000 mgr.y (mgr.24368) 436 : cluster 0 pgmap v447: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 872 op/s 2024-04-06T03:32:45.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:45 smithi050 bash[29221]: cluster 2024-04-06T03:32:44.209831+0000 mgr.y (mgr.24368) 436 : cluster 0 pgmap v447: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 872 op/s 2024-04-06T03:32:46.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:46.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:46.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:46.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:46.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:46.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:46.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:46.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:46.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:46.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:46.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:46.759 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:46.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:46.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:46.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:46.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:47.033 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:47.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:47.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:47.294 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:47.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:47.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:47.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:47.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:47.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:47.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:47.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:47.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:47 smithi050 bash[21467]: cluster 2024-04-06T03:32:46.212741+0000 mgr.y (mgr.24368) 437 : cluster 0 pgmap v448: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 991 op/s 2024-04-06T03:32:47.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:47 smithi050 bash[29221]: cluster 2024-04-06T03:32:46.212741+0000 mgr.y (mgr.24368) 437 : cluster 0 pgmap v448: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 991 op/s 2024-04-06T03:32:47.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:47.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:47.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:48.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:47 smithi153 bash[23852]: cluster 2024-04-06T03:32:46.212741+0000 mgr.y (mgr.24368) 437 : cluster 0 pgmap v448: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 991 op/s 2024-04-06T03:32:48.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:48.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:48.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.225 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:48.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:48.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:48.354 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:48.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:48.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:48.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:48.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:48.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:48.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:48.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:48.871 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:48.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:48.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.005 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:49.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:49.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:49.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:49.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:49.250 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:49.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:49.376 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:49.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:49.485 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:32:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:32:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:32:49.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:49.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:49.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:49.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:49.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:32:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:32:49.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:49 smithi153 bash[23852]: cluster 2024-04-06T03:32:48.213173+0000 mgr.y (mgr.24368) 438 : cluster 0 pgmap v449: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 771 op/s 2024-04-06T03:32:49.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:49 smithi153 bash[23852]: audit 2024-04-06T03:32:49.161985+0000 mon.a (mon.0) 1272 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:49.787 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:49.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:49.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:49 smithi050 bash[21467]: cluster 2024-04-06T03:32:48.213173+0000 mgr.y (mgr.24368) 438 : cluster 0 pgmap v449: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 771 op/s 2024-04-06T03:32:49.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:49 smithi050 bash[21467]: audit 2024-04-06T03:32:49.161985+0000 mon.a (mon.0) 1272 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:49.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:49 smithi050 bash[29221]: cluster 2024-04-06T03:32:48.213173+0000 mgr.y (mgr.24368) 438 : cluster 0 pgmap v449: 148 pgs: 148 active+clean; 2.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 771 op/s 2024-04-06T03:32:49.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:49 smithi050 bash[29221]: audit 2024-04-06T03:32:49.161985+0000 mon.a (mon.0) 1272 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:32:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:49.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:49.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:50.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:50.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:50.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:50.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:50.195 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:50.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:50.328 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:50.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:50.457 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:50.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:50.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:50.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:50.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:50.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:50 smithi050 bash[21467]: audit 2024-04-06T03:32:49.551480+0000 mgr.y (mgr.24368) 439 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:50.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:50 smithi050 bash[29221]: audit 2024-04-06T03:32:49.551480+0000 mgr.y (mgr.24368) 439 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:50.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:50.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:50.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:50.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:51.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:51.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:50 smithi153 bash[23852]: audit 2024-04-06T03:32:49.551480+0000 mgr.y (mgr.24368) 439 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:32:51.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.145 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:51.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:51.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:51.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:51.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:51.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:51.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:51.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:51.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[21467]: cluster 2024-04-06T03:32:50.213806+0000 mgr.y (mgr.24368) 440 : cluster 0 pgmap v450: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 949 op/s 2024-04-06T03:32:51.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[21467]: cluster 2024-04-06T03:32:50.309945+0000 osd.5 (osd.5) 131 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:51.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[21467]: cluster 2024-04-06T03:32:50.310002+0000 osd.5 (osd.5) 132 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:51.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[21467]: cluster 2024-04-06T03:32:50.707433+0000 mon.a (mon.0) 1273 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-04-06T03:32:51.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[29221]: cluster 2024-04-06T03:32:50.213806+0000 mgr.y (mgr.24368) 440 : cluster 0 pgmap v450: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 949 op/s 2024-04-06T03:32:51.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[29221]: cluster 2024-04-06T03:32:50.309945+0000 osd.5 (osd.5) 131 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:51.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[29221]: cluster 2024-04-06T03:32:50.310002+0000 osd.5 (osd.5) 132 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:51.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:51 smithi050 bash[29221]: cluster 2024-04-06T03:32:50.707433+0000 mon.a (mon.0) 1273 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-04-06T03:32:51.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:51.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:51 smithi153 bash[23852]: cluster 2024-04-06T03:32:50.213806+0000 mgr.y (mgr.24368) 440 : cluster 0 pgmap v450: 148 pgs: 148 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 949 op/s 2024-04-06T03:32:51.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:51 smithi153 bash[23852]: cluster 2024-04-06T03:32:50.309945+0000 osd.5 (osd.5) 131 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:51.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:51 smithi153 bash[23852]: cluster 2024-04-06T03:32:50.310002+0000 osd.5 (osd.5) 132 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:51.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:51 smithi153 bash[23852]: cluster 2024-04-06T03:32:50.707433+0000 mon.a (mon.0) 1273 : cluster 0 osdmap e215: 8 total, 8 up, 8 in 2024-04-06T03:32:51.929 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:51.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:51.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:52.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:52.067 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:52.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:52.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:52.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:52.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:52.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:52.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:52.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:52.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:52.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:52.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:52.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:52.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:52.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:52.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:53.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:53.094 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:53.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:53.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:53.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:53.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:53.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:53.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:53.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:53.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:53.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:53.740 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:53.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:53.872 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:53.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:53.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:53 smithi153 bash[23852]: cluster 2024-04-06T03:32:52.214257+0000 mgr.y (mgr.24368) 441 : cluster 0 pgmap v452: 148 pgs: 148 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:32:53.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:53 smithi153 bash[23852]: cluster 2024-04-06T03:32:52.854900+0000 osd.3 (osd.3) 273 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:53.983 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:53 smithi153 bash[23852]: cluster 2024-04-06T03:32:52.854973+0000 osd.3 (osd.3) 274 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:53.994 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:53.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:54.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:54.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:54.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:53 smithi050 bash[21467]: cluster 2024-04-06T03:32:52.214257+0000 mgr.y (mgr.24368) 441 : cluster 0 pgmap v452: 148 pgs: 148 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:32:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:53 smithi050 bash[21467]: cluster 2024-04-06T03:32:52.854900+0000 osd.3 (osd.3) 273 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:54.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:53 smithi050 bash[21467]: cluster 2024-04-06T03:32:52.854973+0000 osd.3 (osd.3) 274 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:54.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:53 smithi050 bash[29221]: cluster 2024-04-06T03:32:52.214257+0000 mgr.y (mgr.24368) 441 : cluster 0 pgmap v452: 148 pgs: 148 active+clean; 2.3 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:32:54.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:53 smithi050 bash[29221]: cluster 2024-04-06T03:32:52.854900+0000 osd.3 (osd.3) 273 : cluster 0 purged_snaps scrub starts 2024-04-06T03:32:54.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:53 smithi050 bash[29221]: cluster 2024-04-06T03:32:52.854973+0000 osd.3 (osd.3) 274 : cluster 0 purged_snaps scrub ok 2024-04-06T03:32:54.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:54.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:54.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:54.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:54.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:54.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:54.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:54.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:54.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:54.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:54.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:54.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:55.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:55.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:55.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:54 smithi153 bash[23852]: cluster 2024-04-06T03:32:53.716387+0000 mon.a (mon.0) 1274 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-04-06T03:32:55.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:54 smithi153 bash[23852]: cluster 2024-04-06T03:32:54.215064+0000 mgr.y (mgr.24368) 442 : cluster 0 pgmap v454: 148 pgs: 148 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:32:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:55.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:54 smithi050 bash[21467]: cluster 2024-04-06T03:32:53.716387+0000 mon.a (mon.0) 1274 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-04-06T03:32:55.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:54 smithi050 bash[21467]: cluster 2024-04-06T03:32:54.215064+0000 mgr.y (mgr.24368) 442 : cluster 0 pgmap v454: 148 pgs: 148 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:32:55.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:54 smithi050 bash[29221]: cluster 2024-04-06T03:32:53.716387+0000 mon.a (mon.0) 1274 : cluster 0 osdmap e216: 8 total, 8 up, 8 in 2024-04-06T03:32:55.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:54 smithi050 bash[29221]: cluster 2024-04-06T03:32:54.215064+0000 mgr.y (mgr.24368) 442 : cluster 0 pgmap v454: 148 pgs: 148 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:32:55.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:55.173 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:55.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:55.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:55.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:55.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:55.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:55.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:55.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:55.712 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:55.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:55.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:55.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:55.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:55.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:56.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:56.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:56.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:56.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:56.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:56.376 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:56.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:56.502 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:56.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:56.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:56.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:56.766 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:32:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:32:56.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:56.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:56.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:57.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:57.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:57.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:57.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:57.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:57.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:57 smithi050 bash[21467]: cluster 2024-04-06T03:32:56.215986+0000 mgr.y (mgr.24368) 443 : cluster 0 pgmap v455: 148 pgs: 148 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-04-06T03:32:57.291 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:57 smithi050 bash[29221]: cluster 2024-04-06T03:32:56.215986+0000 mgr.y (mgr.24368) 443 : cluster 0 pgmap v455: 148 pgs: 148 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-04-06T03:32:57.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:32:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:32:57.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:57.439 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:57.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:57.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:57 smithi153 bash[23852]: cluster 2024-04-06T03:32:56.215986+0000 mgr.y (mgr.24368) 443 : cluster 0 pgmap v455: 148 pgs: 148 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-04-06T03:32:57.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:57.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:57.696 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:57.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:57.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:57.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:57.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:57.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:58.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:58.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:58.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:58.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:58.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:58.343 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:58.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:58.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:58.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:58.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:32:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:32:58.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:58.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:32:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:32:58.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:58.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:58.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:32:58.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:58.995 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:59.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:59.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:59.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:59.262 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:59.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:59.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:59.487 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:32:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:32:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:32:59.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:32:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:32:59.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:32:59 smithi050 bash[21467]: cluster 2024-04-06T03:32:58.216459+0000 mgr.y (mgr.24368) 444 : cluster 0 pgmap v456: 148 pgs: 148 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s 2024-04-06T03:32:59.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:32:59 smithi050 bash[29221]: cluster 2024-04-06T03:32:58.216459+0000 mgr.y (mgr.24368) 444 : cluster 0 pgmap v456: 148 pgs: 148 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s 2024-04-06T03:32:59.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:59.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:32:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:32:59.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:59.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:32:59.787 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:32:59 smithi153 bash[23852]: cluster 2024-04-06T03:32:58.216459+0000 mgr.y (mgr.24368) 444 : cluster 0 pgmap v456: 148 pgs: 148 active+clean; 2.0 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s 2024-04-06T03:32:59.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:32:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:32:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:32:59.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:59.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:32:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:32:59.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:00.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:00.046 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:00.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:00.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:00.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:00.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:00.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:00.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:00.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:00.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:00 smithi153 bash[23852]: audit 2024-04-06T03:32:59.562204+0000 mgr.y (mgr.24368) 445 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:00.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:00.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:00.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:00.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:00 smithi050 bash[21467]: audit 2024-04-06T03:32:59.562204+0000 mgr.y (mgr.24368) 445 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:00.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:00 smithi050 bash[29221]: audit 2024-04-06T03:32:59.562204+0000 mgr.y (mgr.24368) 445 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:00.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:00.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:00.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:00.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:00.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:01.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:01.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:01.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:01.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:01.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:01.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:01.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:01.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:01.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:01.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:01.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:01.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:01.797 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:33:01 smithi153 bash[51781]: logger=cleanup t=2024-04-06T03:33:01.490405658Z level=info msg="Completed cleanup jobs" duration=330.9508ms 2024-04-06T03:33:01.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:01 smithi153 bash[23852]: cluster 2024-04-06T03:33:00.217208+0000 mgr.y (mgr.24368) 446 : cluster 0 pgmap v457: 148 pgs: 148 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 835 op/s 2024-04-06T03:33:01.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:01.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:01.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:01.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:01 smithi050 bash[21467]: cluster 2024-04-06T03:33:00.217208+0000 mgr.y (mgr.24368) 446 : cluster 0 pgmap v457: 148 pgs: 148 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 835 op/s 2024-04-06T03:33:01.948 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:01 smithi050 bash[29221]: cluster 2024-04-06T03:33:00.217208+0000 mgr.y (mgr.24368) 446 : cluster 0 pgmap v457: 148 pgs: 148 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 835 op/s 2024-04-06T03:33:01.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:02.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:02.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:02.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:02.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:02.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:02.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:02.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:02.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:02.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:02 smithi153 bash[23852]: cluster 2024-04-06T03:33:01.631404+0000 osd.7 (osd.7) 285 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:02.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:02 smithi153 bash[23852]: cluster 2024-04-06T03:33:01.631481+0000 osd.7 (osd.7) 286 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:02.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:02 smithi050 bash[21467]: cluster 2024-04-06T03:33:01.631404+0000 osd.7 (osd.7) 285 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:02.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:02 smithi050 bash[21467]: cluster 2024-04-06T03:33:01.631481+0000 osd.7 (osd.7) 286 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:02.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:02 smithi050 bash[29221]: cluster 2024-04-06T03:33:01.631404+0000 osd.7 (osd.7) 285 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:02.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:02 smithi050 bash[29221]: cluster 2024-04-06T03:33:01.631481+0000 osd.7 (osd.7) 286 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:02.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:02.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:02.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:02.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:03.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:03.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:03.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:03.216 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:03.318 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:03.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:03.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:03.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:03.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:03.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:03.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:03.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:03 smithi050 bash[29221]: cluster 2024-04-06T03:33:02.217723+0000 mgr.y (mgr.24368) 447 : cluster 0 pgmap v458: 148 pgs: 148 active+clean; 1.8 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 885 op/s 2024-04-06T03:33:03.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:03 smithi050 bash[29221]: cluster 2024-04-06T03:33:02.555910+0000 mon.a (mon.0) 1275 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-04-06T03:33:03.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:03 smithi050 bash[21467]: cluster 2024-04-06T03:33:02.217723+0000 mgr.y (mgr.24368) 447 : cluster 0 pgmap v458: 148 pgs: 148 active+clean; 1.8 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 885 op/s 2024-04-06T03:33:03.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:03 smithi050 bash[21467]: cluster 2024-04-06T03:33:02.555910+0000 mon.a (mon.0) 1275 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-04-06T03:33:03.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:03.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:03 smithi153 bash[23852]: cluster 2024-04-06T03:33:02.217723+0000 mgr.y (mgr.24368) 447 : cluster 0 pgmap v458: 148 pgs: 148 active+clean; 1.8 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 885 op/s 2024-04-06T03:33:03.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:03 smithi153 bash[23852]: cluster 2024-04-06T03:33:02.555910+0000 mon.a (mon.0) 1275 : cluster 0 osdmap e217: 8 total, 8 up, 8 in 2024-04-06T03:33:03.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:03.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:03.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:03.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:03.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:04.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:04.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:04.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:04.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:04.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:04.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:04.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:04.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:04.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:04.623 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:04.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:04.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:04 smithi050 bash[21467]: audit 2024-04-06T03:33:04.180289+0000 mon.a (mon.0) 1276 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:04.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:04 smithi050 bash[29221]: audit 2024-04-06T03:33:04.180289+0000 mon.a (mon.0) 1276 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:04.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:04.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:04 smithi153 bash[23852]: audit 2024-04-06T03:33:04.180289+0000 mon.a (mon.0) 1276 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:04.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:04.886 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:04.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:04.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:05.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:05.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:187884 8.1 8:864ae9b1:::benchmark_data_smithi050_86143_object81573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.122227+0000", 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.017673300999999999, 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017701250000000002, 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:05.135 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 187884 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122227+0000", 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122227+0000", 2024-04-06T03:33:05.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122227+0000", 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2699999999999999e-07 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122232+0000", 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3189999999999996e-06 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122242+0000", 2024-04-06T03:33:05.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0267e-05 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122249+0000", 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4309999999999997e-06 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122309+0000", 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9296999999999998e-05 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:05.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122445+0000", 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136701 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.122574+0000", 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012868999999999999 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129545+0000", 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069713960000000004 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:05.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.132498+0000", 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002952503 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:187886 8.a 8:53ab2794:::benchmark_data_smithi050_86143_object81575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.126804+0000", 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.013096233000000001, 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013184391, 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:05.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 187886 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126804+0000", 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126804+0000", 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126805+0000", 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6249999999999999e-06 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126812+0000", 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3799999999999999e-06 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126829+0000", 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7431e-05 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126839+0000", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5610000000000004e-06 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.126956+0000", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117031 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.127187+0000", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000231079 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:05.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.127313+0000", 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000126327 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14928.0:187874 8.a e217/182)", 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.129025+0000", 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010874695, 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011005281, 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129025+0000", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129025+0000", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129026+0000", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.049e-06 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129032+0000", 2024-04-06T03:33:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.874e-06 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129049+0000", 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7274000000000001e-05 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.129059+0000", 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4739999999999994e-06 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:187893 8.a 8.7b0ddcfa (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.130994+0000", 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0089054500000000005, 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090643660000000008, 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 187893 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.130994+0000", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.130994+0000", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.130996+0000", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.42e-06 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.131002+0000", 2024-04-06T03:33:05.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7790000000000001e-06 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.131017+0000", 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5119999999999999e-05 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.131026+0000", 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4739999999999994e-06 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:187895 8.a 8.1e3e5cca (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-04-06T03:33:05.149 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.133227+0000", 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0066733649999999997, 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068649820000000004, 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 187895 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.133227+0000", 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.133227+0000", 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.133228+0000", 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.066e-06 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.133232+0000", 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0709999999999996e-06 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.133248+0000", 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6174000000000001e-05 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.133257+0000", 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7760000000000003e-06 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.152 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.14928.0:187886 8.a e217/182)", 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.134483+0000", 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0054172029999999998, 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056385860000000001, 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.134483+0000", 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.134483+0000", 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.134483+0000", 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9299999999999997e-07 2024-04-06T03:33:05.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.134488+0000", 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8790000000000002e-06 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.134504+0000", 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5274999999999999e-05 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.134512+0000", 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8630000000000008e-06 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.155 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14928.0:187897 8.b e217/182 8:d4a9e3cc:::benchmark_data_smithi050_86143_object81586:head v 217'11689, mlcod=217'11687)", 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.136898+0000", 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0030014970000000001, 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032519910000000001, 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136898+0000", 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136898+0000", 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136899+0000", 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6099999999999999e-07 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136907+0000", 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5630000000000003e-06 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136925+0000", 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7856e-05 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136934+0000", 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7569999999999993e-06 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.137264+0000", 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032999599999999999 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.137285+0000", 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1661000000000001e-05 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:187899 8.3 8:cfed7b6d:::benchmark_data_smithi050_86143_object81588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.136993+0000", 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0029069500000000002, 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031928820000000002, 2024-04-06T03:33:05.159 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.160 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.160 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:33:05.160 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:05.160 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 187899 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136993+0000", 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136993+0000", 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136994+0000", 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.341e-06 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.136999+0000", 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7269999999999998e-06 2024-04-06T03:33:05.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.137011+0000", 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.163e-05 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.137020+0000", 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5219999999999996e-06 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.137993+0000", 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097238700000000001 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.138117+0000", 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123972 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.138376+0000", 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025957399999999999 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:187874 8.a 8:5f86e86a:::benchmark_data_smithi050_86143_object81563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:33:05.111513+0000", 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.028387306000000001, 2024-04-06T03:33:05.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028725627, 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 187874 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111513+0000", 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111513+0000", 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111514+0000", 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.316e-06 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111520+0000", 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2639999999999997e-06 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111534+0000", 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4019000000000001e-05 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111543+0000", 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7350000000000002e-06 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111724+0000", 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018145600000000001 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.111882+0000", 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000157248 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.112027+0000", 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014527799999999999 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:33:05.119857+0000", 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078298959999999994 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:05.168 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:33:05.169 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 9 2024-04-06T03:33:05.169 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:33:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:05.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:05.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:05.289 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:33:05.307 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:05.336 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:33:05.374 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:33:05.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:05.435 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:33:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:05.466 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:33:05.498 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:33:05.534 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:33:05.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:05.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:05.576 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:33:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:05.580 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-06T03:33:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:33:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:33:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:33:05.586 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:33:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:33:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:33:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:33:05.587 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:33:05.592 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:33:05.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:05.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[21467]: cluster 2024-04-06T03:33:04.218462+0000 mgr.y (mgr.24368) 448 : cluster 0 pgmap v460: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[21467]: audit 2024-04-06T03:33:04.857091+0000 mon.a (mon.0) 1277 : audit 1 from='client.? 172.21.15.50:0/2909728504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[21467]: audit 2024-04-06T03:33:05.194388+0000 mon.c (mon.2) 66 : audit 1 from='client.? 172.21.15.50:0/3092302029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[21467]: audit 2024-04-06T03:33:05.195251+0000 mon.a (mon.0) 1278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[21467]: audit 2024-04-06T03:33:05.209134+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.50:0/1430789005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[29221]: cluster 2024-04-06T03:33:04.218462+0000 mgr.y (mgr.24368) 448 : cluster 0 pgmap v460: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[29221]: audit 2024-04-06T03:33:04.857091+0000 mon.a (mon.0) 1277 : audit 1 from='client.? 172.21.15.50:0/2909728504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[29221]: audit 2024-04-06T03:33:05.194388+0000 mon.c (mon.2) 66 : audit 1 from='client.? 172.21.15.50:0/3092302029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[29221]: audit 2024-04-06T03:33:05.195251+0000 mon.a (mon.0) 1278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:33:05.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:05 smithi050 bash[29221]: audit 2024-04-06T03:33:05.209134+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.50:0/1430789005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:33:05.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:05.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:05 smithi153 bash[23852]: cluster 2024-04-06T03:33:04.218462+0000 mgr.y (mgr.24368) 448 : cluster 0 pgmap v460: 148 pgs: 148 active+clean; 1.7 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:33:05.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:05 smithi153 bash[23852]: audit 2024-04-06T03:33:04.857091+0000 mon.a (mon.0) 1277 : audit 1 from='client.? 172.21.15.50:0/2909728504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:33:05.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:05 smithi153 bash[23852]: audit 2024-04-06T03:33:05.194388+0000 mon.c (mon.2) 66 : audit 1 from='client.? 172.21.15.50:0/3092302029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:33:05.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:05 smithi153 bash[23852]: audit 2024-04-06T03:33:05.195251+0000 mon.a (mon.0) 1278 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:33:05.825 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:05 smithi153 bash[23852]: audit 2024-04-06T03:33:05.209134+0000 mon.c (mon.2) 67 : audit 0 from='client.? 172.21.15.50:0/1430789005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:33:05.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:05.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:05.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:05.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:06.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:06.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:06.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:06.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:06.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:06.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:06.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:06.483 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:06.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:06.611 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:06.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[21467]: cluster 2024-04-06T03:33:05.460360+0000 osd.1 (osd.1) 231 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:06.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[21467]: cluster 2024-04-06T03:33:05.460539+0000 osd.1 (osd.1) 232 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:06.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[21467]: cluster 2024-04-06T03:33:05.562882+0000 mon.a (mon.0) 1279 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:33:06.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:06.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:06 smithi153 bash[23852]: cluster 2024-04-06T03:33:05.460360+0000 osd.1 (osd.1) 231 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:06.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:06 smithi153 bash[23852]: cluster 2024-04-06T03:33:05.460539+0000 osd.1 (osd.1) 232 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:06.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:06 smithi153 bash[23852]: cluster 2024-04-06T03:33:05.562882+0000 mon.a (mon.0) 1279 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:33:06.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:06 smithi153 bash[23852]: audit 2024-04-06T03:33:05.577085+0000 mon.a (mon.0) 1280 : audit 1 from='client.? 172.21.15.50:0/2909728504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:33:06.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:06 smithi153 bash[23852]: audit 2024-04-06T03:33:05.577399+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:33:06.723 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:06 smithi153 bash[23852]: cluster 2024-04-06T03:33:05.589548+0000 mon.a (mon.0) 1282 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-04-06T03:33:06.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:06.850 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:06.881 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:06.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:06.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[21467]: audit 2024-04-06T03:33:05.577085+0000 mon.a (mon.0) 1280 : audit 1 from='client.? 172.21.15.50:0/2909728504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[21467]: audit 2024-04-06T03:33:05.577399+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[21467]: cluster 2024-04-06T03:33:05.589548+0000 mon.a (mon.0) 1282 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[29221]: cluster 2024-04-06T03:33:05.460360+0000 osd.1 (osd.1) 231 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[29221]: cluster 2024-04-06T03:33:05.460539+0000 osd.1 (osd.1) 232 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[29221]: cluster 2024-04-06T03:33:05.562882+0000 mon.a (mon.0) 1279 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[29221]: audit 2024-04-06T03:33:05.577085+0000 mon.a (mon.0) 1280 : audit 1 from='client.? 172.21.15.50:0/2909728504' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[29221]: audit 2024-04-06T03:33:05.577399+0000 mon.a (mon.0) 1281 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:33:06.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:06 smithi050 bash[29221]: cluster 2024-04-06T03:33:05.589548+0000 mon.a (mon.0) 1282 : cluster 0 osdmap e218: 8 total, 8 up, 8 in 2024-04-06T03:33:07.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:07.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-06T03:33:07.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.155 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:07.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:07.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:07.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:07.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:07.417 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:07.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:07.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:07.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:07.668 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:07 smithi153 bash[23852]: cluster 2024-04-06T03:33:06.219385+0000 mgr.y (mgr.24368) 449 : cluster 0 pgmap v462: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:33:07.668 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:07 smithi153 bash[23852]: cluster 2024-04-06T03:33:06.478160+0000 osd.7 (osd.7) 287 : cluster 0 2.b scrub starts 2024-04-06T03:33:07.668 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:07 smithi153 bash[23852]: cluster 2024-04-06T03:33:06.482495+0000 osd.7 (osd.7) 288 : cluster 0 2.b scrub ok 2024-04-06T03:33:07.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:07 smithi050 bash[21467]: cluster 2024-04-06T03:33:06.219385+0000 mgr.y (mgr.24368) 449 : cluster 0 pgmap v462: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:33:07.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:07 smithi050 bash[21467]: cluster 2024-04-06T03:33:06.478160+0000 osd.7 (osd.7) 287 : cluster 0 2.b scrub starts 2024-04-06T03:33:07.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:07 smithi050 bash[21467]: cluster 2024-04-06T03:33:06.482495+0000 osd.7 (osd.7) 288 : cluster 0 2.b scrub ok 2024-04-06T03:33:07.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:07.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:07.810 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:07.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:07.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:07.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:07 smithi050 bash[29221]: cluster 2024-04-06T03:33:06.219385+0000 mgr.y (mgr.24368) 449 : cluster 0 pgmap v462: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:33:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:07 smithi050 bash[29221]: cluster 2024-04-06T03:33:06.478160+0000 osd.7 (osd.7) 287 : cluster 0 2.b scrub starts 2024-04-06T03:33:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:07 smithi050 bash[29221]: cluster 2024-04-06T03:33:06.482495+0000 osd.7 (osd.7) 288 : cluster 0 2.b scrub ok 2024-04-06T03:33:08.039 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:08.067 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:08.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:08.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:08.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:08.329 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:08.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:08.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:08.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:08.594 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:08.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:08.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:08.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:08 smithi050 bash[21467]: cluster 2024-04-06T03:33:07.510758+0000 osd.7 (osd.7) 289 : cluster 0 2.5 scrub starts 2024-04-06T03:33:08.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:08 smithi050 bash[21467]: cluster 2024-04-06T03:33:07.517179+0000 osd.7 (osd.7) 290 : cluster 0 2.5 scrub ok 2024-04-06T03:33:08.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:08 smithi050 bash[29221]: cluster 2024-04-06T03:33:07.510758+0000 osd.7 (osd.7) 289 : cluster 0 2.5 scrub starts 2024-04-06T03:33:08.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:08 smithi050 bash[29221]: cluster 2024-04-06T03:33:07.517179+0000 osd.7 (osd.7) 290 : cluster 0 2.5 scrub ok 2024-04-06T03:33:08.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:08.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:08.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:08 smithi153 bash[23852]: cluster 2024-04-06T03:33:07.510758+0000 osd.7 (osd.7) 289 : cluster 0 2.5 scrub starts 2024-04-06T03:33:08.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:08 smithi153 bash[23852]: cluster 2024-04-06T03:33:07.517179+0000 osd.7 (osd.7) 290 : cluster 0 2.5 scrub ok 2024-04-06T03:33:08.853 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:08.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:08.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:08.995 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:09.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:09.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:09.244 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:09.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:09.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:09.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:09.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:09.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:09.586 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:33:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:33:09.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:09.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:09 smithi153 bash[23852]: cluster 2024-04-06T03:33:08.219798+0000 mgr.y (mgr.24368) 450 : cluster 0 pgmap v463: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:33:09.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:09 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.011581+0000 osd.3 (osd.3) 275 : cluster 0 5.0 scrub starts 2024-04-06T03:33:09.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:09 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.022693+0000 osd.3 (osd.3) 276 : cluster 0 5.0 scrub ok 2024-04-06T03:33:09.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:09 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.314405+0000 osd.1 (osd.1) 233 : cluster 0 6.1 scrub starts 2024-04-06T03:33:09.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:09 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.336831+0000 osd.1 (osd.1) 234 : cluster 0 6.1 scrub ok 2024-04-06T03:33:09.642 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:33:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:33:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:09.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 7 1 2024-04-06T03:33:09.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:09.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:09.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[21467]: cluster 2024-04-06T03:33:08.219798+0000 mgr.y (mgr.24368) 450 : cluster 0 pgmap v463: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.011581+0000 osd.3 (osd.3) 275 : cluster 0 5.0 scrub starts 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.022693+0000 osd.3 (osd.3) 276 : cluster 0 5.0 scrub ok 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.314405+0000 osd.1 (osd.1) 233 : cluster 0 6.1 scrub starts 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.336831+0000 osd.1 (osd.1) 234 : cluster 0 6.1 scrub ok 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[29221]: cluster 2024-04-06T03:33:08.219798+0000 mgr.y (mgr.24368) 450 : cluster 0 pgmap v463: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:33:09.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.011581+0000 osd.3 (osd.3) 275 : cluster 0 5.0 scrub starts 2024-04-06T03:33:09.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.022693+0000 osd.3 (osd.3) 276 : cluster 0 5.0 scrub ok 2024-04-06T03:33:09.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.314405+0000 osd.1 (osd.1) 233 : cluster 0 6.1 scrub starts 2024-04-06T03:33:09.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:09 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.336831+0000 osd.1 (osd.1) 234 : cluster 0 6.1 scrub ok 2024-04-06T03:33:09.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:09.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:10.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:10.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:10.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:10.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:10.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:10.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:10.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:10.466 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:10.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:10.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:10.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:10.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.435700+0000 osd.7 (osd.7) 291 : cluster 0 6.5 scrub starts 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.452864+0000 osd.7 (osd.7) 292 : cluster 0 6.5 scrub ok 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.479232+0000 osd.4 (osd.4) 271 : cluster 0 3.1d scrub starts 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.484396+0000 osd.4 (osd.4) 272 : cluster 0 3.1d scrub ok 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.501601+0000 osd.0 (osd.0) 260 : cluster 0 3.12 scrub starts 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:09.505404+0000 osd.0 (osd.0) 261 : cluster 0 3.12 scrub ok 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: audit 2024-04-06T03:33:09.571486+0000 mgr.y (mgr.24368) 451 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:10.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.042884+0000 osd.3 (osd.3) 277 : cluster 0 6.1b scrub starts 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.115337+0000 osd.3 (osd.3) 278 : cluster 0 6.1b scrub ok 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.435700+0000 osd.7 (osd.7) 291 : cluster 0 6.5 scrub starts 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.452864+0000 osd.7 (osd.7) 292 : cluster 0 6.5 scrub ok 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.479232+0000 osd.4 (osd.4) 271 : cluster 0 3.1d scrub starts 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.484396+0000 osd.4 (osd.4) 272 : cluster 0 3.1d scrub ok 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.501601+0000 osd.0 (osd.0) 260 : cluster 0 3.12 scrub starts 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:09.505404+0000 osd.0 (osd.0) 261 : cluster 0 3.12 scrub ok 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: audit 2024-04-06T03:33:09.571486+0000 mgr.y (mgr.24368) 451 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.042884+0000 osd.3 (osd.3) 277 : cluster 0 6.1b scrub starts 2024-04-06T03:33:10.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:10 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.115337+0000 osd.3 (osd.3) 278 : cluster 0 6.1b scrub ok 2024-04-06T03:33:10.869 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:10.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:10.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:11.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.435700+0000 osd.7 (osd.7) 291 : cluster 0 6.5 scrub starts 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.452864+0000 osd.7 (osd.7) 292 : cluster 0 6.5 scrub ok 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.479232+0000 osd.4 (osd.4) 271 : cluster 0 3.1d scrub starts 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.484396+0000 osd.4 (osd.4) 272 : cluster 0 3.1d scrub ok 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.501601+0000 osd.0 (osd.0) 260 : cluster 0 3.12 scrub starts 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:09.505404+0000 osd.0 (osd.0) 261 : cluster 0 3.12 scrub ok 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: audit 2024-04-06T03:33:09.571486+0000 mgr.y (mgr.24368) 451 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.042884+0000 osd.3 (osd.3) 277 : cluster 0 6.1b scrub starts 2024-04-06T03:33:11.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:10 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.115337+0000 osd.3 (osd.3) 278 : cluster 0 6.1b scrub ok 2024-04-06T03:33:11.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:11.137 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:11.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:11.262 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:11.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:11.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:11.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:11.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:11.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.220387+0000 mgr.y (mgr.24368) 452 : cluster 0 pgmap v464: 148 pgs: 148 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s 2024-04-06T03:33:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.357432+0000 osd.2 (osd.2) 227 : cluster 0 2.c scrub starts 2024-04-06T03:33:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.358780+0000 osd.2 (osd.2) 228 : cluster 0 2.c scrub ok 2024-04-06T03:33:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.422423+0000 osd.7 (osd.7) 293 : cluster 0 3.f scrub starts 2024-04-06T03:33:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.424668+0000 osd.7 (osd.7) 294 : cluster 0 3.f scrub ok 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.494580+0000 osd.4 (osd.4) 273 : cluster 0 5.e scrub starts 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[21467]: cluster 2024-04-06T03:33:10.496044+0000 osd.4 (osd.4) 274 : cluster 0 5.e scrub ok 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.220387+0000 mgr.y (mgr.24368) 452 : cluster 0 pgmap v464: 148 pgs: 148 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.357432+0000 osd.2 (osd.2) 227 : cluster 0 2.c scrub starts 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.358780+0000 osd.2 (osd.2) 228 : cluster 0 2.c scrub ok 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.422423+0000 osd.7 (osd.7) 293 : cluster 0 3.f scrub starts 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.424668+0000 osd.7 (osd.7) 294 : cluster 0 3.f scrub ok 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.494580+0000 osd.4 (osd.4) 273 : cluster 0 5.e scrub starts 2024-04-06T03:33:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:11 smithi050 bash[29221]: cluster 2024-04-06T03:33:10.496044+0000 osd.4 (osd.4) 274 : cluster 0 5.e scrub ok 2024-04-06T03:33:11.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:11.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:11.775 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:11.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:11.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.220387+0000 mgr.y (mgr.24368) 452 : cluster 0 pgmap v464: 148 pgs: 148 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s 2024-04-06T03:33:11.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.357432+0000 osd.2 (osd.2) 227 : cluster 0 2.c scrub starts 2024-04-06T03:33:11.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.358780+0000 osd.2 (osd.2) 228 : cluster 0 2.c scrub ok 2024-04-06T03:33:11.888 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.422423+0000 osd.7 (osd.7) 293 : cluster 0 3.f scrub starts 2024-04-06T03:33:11.889 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.424668+0000 osd.7 (osd.7) 294 : cluster 0 3.f scrub ok 2024-04-06T03:33:11.889 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.494580+0000 osd.4 (osd.4) 273 : cluster 0 5.e scrub starts 2024-04-06T03:33:11.889 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:11 smithi153 bash[23852]: cluster 2024-04-06T03:33:10.496044+0000 osd.4 (osd.4) 274 : cluster 0 5.e scrub ok 2024-04-06T03:33:11.902 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:11.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:12.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:12.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:12.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:33:12.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:33:12.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:12.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:12.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:12.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:12.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:12.455 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:12.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:12.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:12.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:12.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:12 smithi153 bash[23852]: cluster 2024-04-06T03:33:11.508889+0000 osd.0 (osd.0) 262 : cluster 0 6.9 scrub starts 2024-04-06T03:33:12.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:12 smithi153 bash[23852]: cluster 2024-04-06T03:33:11.522452+0000 osd.4 (osd.4) 275 : cluster 0 3.9 scrub starts 2024-04-06T03:33:12.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:12 smithi153 bash[23852]: cluster 2024-04-06T03:33:11.524283+0000 osd.4 (osd.4) 276 : cluster 0 3.9 scrub ok 2024-04-06T03:33:12.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:12 smithi153 bash[23852]: cluster 2024-04-06T03:33:11.574709+0000 osd.0 (osd.0) 263 : cluster 0 6.9 scrub ok 2024-04-06T03:33:12.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:12 smithi153 bash[23852]: cluster 2024-04-06T03:33:12.045140+0000 osd.3 (osd.3) 279 : cluster 0 2.a scrub starts 2024-04-06T03:33:12.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:12 smithi153 bash[23852]: cluster 2024-04-06T03:33:12.048728+0000 osd.3 (osd.3) 280 : cluster 0 2.a scrub ok 2024-04-06T03:33:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:12.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:12.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:12.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:12.882 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:12.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[21467]: cluster 2024-04-06T03:33:11.508889+0000 osd.0 (osd.0) 262 : cluster 0 6.9 scrub starts 2024-04-06T03:33:12.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[21467]: cluster 2024-04-06T03:33:11.522452+0000 osd.4 (osd.4) 275 : cluster 0 3.9 scrub starts 2024-04-06T03:33:12.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[21467]: cluster 2024-04-06T03:33:11.524283+0000 osd.4 (osd.4) 276 : cluster 0 3.9 scrub ok 2024-04-06T03:33:12.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[21467]: cluster 2024-04-06T03:33:11.574709+0000 osd.0 (osd.0) 263 : cluster 0 6.9 scrub ok 2024-04-06T03:33:12.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[21467]: cluster 2024-04-06T03:33:12.045140+0000 osd.3 (osd.3) 279 : cluster 0 2.a scrub starts 2024-04-06T03:33:12.910 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[21467]: cluster 2024-04-06T03:33:12.048728+0000 osd.3 (osd.3) 280 : cluster 0 2.a scrub ok 2024-04-06T03:33:12.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[29221]: cluster 2024-04-06T03:33:11.508889+0000 osd.0 (osd.0) 262 : cluster 0 6.9 scrub starts 2024-04-06T03:33:12.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[29221]: cluster 2024-04-06T03:33:11.522452+0000 osd.4 (osd.4) 275 : cluster 0 3.9 scrub starts 2024-04-06T03:33:12.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[29221]: cluster 2024-04-06T03:33:11.524283+0000 osd.4 (osd.4) 276 : cluster 0 3.9 scrub ok 2024-04-06T03:33:12.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[29221]: cluster 2024-04-06T03:33:11.574709+0000 osd.0 (osd.0) 263 : cluster 0 6.9 scrub ok 2024-04-06T03:33:12.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[29221]: cluster 2024-04-06T03:33:12.045140+0000 osd.3 (osd.3) 279 : cluster 0 2.a scrub starts 2024-04-06T03:33:12.911 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:12 smithi050 bash[29221]: cluster 2024-04-06T03:33:12.048728+0000 osd.3 (osd.3) 280 : cluster 0 2.a scrub ok 2024-04-06T03:33:12.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:12.987 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:12.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:13.092 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:13.117 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:13.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:13.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:13.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:13.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:13.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:13.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:13.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:13.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:13.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:13.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:13.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:13.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:13 smithi050 bash[21467]: cluster 2024-04-06T03:33:12.224977+0000 mgr.y (mgr.24368) 453 : cluster 0 pgmap v465: 148 pgs: 148 active+clean; 1.3 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 807 op/s 2024-04-06T03:33:13.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:13 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.379231+0000 osd.1 (osd.1) 235 : cluster 0 8.a scrub starts 2024-04-06T03:33:13.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:13 smithi050 bash[29221]: cluster 2024-04-06T03:33:12.224977+0000 mgr.y (mgr.24368) 453 : cluster 0 pgmap v465: 148 pgs: 148 active+clean; 1.3 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 807 op/s 2024-04-06T03:33:13.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:13 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.379231+0000 osd.1 (osd.1) 235 : cluster 0 8.a scrub starts 2024-04-06T03:33:13.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:13.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:13.918 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:14.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:14.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:14.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:13 smithi153 bash[23852]: cluster 2024-04-06T03:33:12.224977+0000 mgr.y (mgr.24368) 453 : cluster 0 pgmap v465: 148 pgs: 148 active+clean; 1.3 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 807 op/s 2024-04-06T03:33:14.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:13 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.379231+0000 osd.1 (osd.1) 235 : cluster 0 8.a scrub starts 2024-04-06T03:33:14.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:14.153 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:14.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:14.281 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:14.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:14.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:14.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:14.543 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:14.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.284801+0000 osd.2 (osd.2) 229 : cluster 0 2.10 scrub starts 2024-04-06T03:33:14.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.288873+0000 osd.2 (osd.2) 230 : cluster 0 2.10 scrub ok 2024-04-06T03:33:14.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.374969+0000 osd.7 (osd.7) 295 : cluster 0 5.10 scrub starts 2024-04-06T03:33:14.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.378604+0000 osd.7 (osd.7) 296 : cluster 0 5.10 scrub ok 2024-04-06T03:33:14.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.530490+0000 osd.0 (osd.0) 264 : cluster 0 5.1b scrub starts 2024-04-06T03:33:14.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.537700+0000 osd.0 (osd.0) 265 : cluster 0 5.1b scrub ok 2024-04-06T03:33:14.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.557577+0000 osd.4 (osd.4) 277 : cluster 0 3.3 scrub starts 2024-04-06T03:33:14.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.560809+0000 osd.4 (osd.4) 278 : cluster 0 3.3 scrub ok 2024-04-06T03:33:14.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.873379+0000 osd.1 (osd.1) 236 : cluster 0 8.a scrub ok 2024-04-06T03:33:14.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.062889+0000 osd.3 (osd.3) 281 : cluster 0 8.7 scrub starts 2024-04-06T03:33:14.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.186994+0000 osd.3 (osd.3) 282 : cluster 0 8.7 scrub ok 2024-04-06T03:33:14.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:14.680 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.284801+0000 osd.2 (osd.2) 229 : cluster 0 2.10 scrub starts 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.288873+0000 osd.2 (osd.2) 230 : cluster 0 2.10 scrub ok 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.374969+0000 osd.7 (osd.7) 295 : cluster 0 5.10 scrub starts 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.378604+0000 osd.7 (osd.7) 296 : cluster 0 5.10 scrub ok 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.530490+0000 osd.0 (osd.0) 264 : cluster 0 5.1b scrub starts 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.537700+0000 osd.0 (osd.0) 265 : cluster 0 5.1b scrub ok 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.557577+0000 osd.4 (osd.4) 277 : cluster 0 3.3 scrub starts 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.560809+0000 osd.4 (osd.4) 278 : cluster 0 3.3 scrub ok 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.873379+0000 osd.1 (osd.1) 236 : cluster 0 8.a scrub ok 2024-04-06T03:33:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.062889+0000 osd.3 (osd.3) 281 : cluster 0 8.7 scrub starts 2024-04-06T03:33:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.186994+0000 osd.3 (osd.3) 282 : cluster 0 8.7 scrub ok 2024-04-06T03:33:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.391000+0000 osd.1 (osd.1) 237 : cluster 0 4.2 scrub starts 2024-04-06T03:33:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:14 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.391000+0000 osd.1 (osd.1) 237 : cluster 0 4.2 scrub starts 2024-04-06T03:33:14.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:14.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.284801+0000 osd.2 (osd.2) 229 : cluster 0 2.10 scrub starts 2024-04-06T03:33:14.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.288873+0000 osd.2 (osd.2) 230 : cluster 0 2.10 scrub ok 2024-04-06T03:33:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.374969+0000 osd.7 (osd.7) 295 : cluster 0 5.10 scrub starts 2024-04-06T03:33:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.378604+0000 osd.7 (osd.7) 296 : cluster 0 5.10 scrub ok 2024-04-06T03:33:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.530490+0000 osd.0 (osd.0) 264 : cluster 0 5.1b scrub starts 2024-04-06T03:33:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.537700+0000 osd.0 (osd.0) 265 : cluster 0 5.1b scrub ok 2024-04-06T03:33:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.557577+0000 osd.4 (osd.4) 277 : cluster 0 3.3 scrub starts 2024-04-06T03:33:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.560809+0000 osd.4 (osd.4) 278 : cluster 0 3.3 scrub ok 2024-04-06T03:33:14.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.873379+0000 osd.1 (osd.1) 236 : cluster 0 8.a scrub ok 2024-04-06T03:33:14.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.062889+0000 osd.3 (osd.3) 281 : cluster 0 8.7 scrub starts 2024-04-06T03:33:14.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.186994+0000 osd.3 (osd.3) 282 : cluster 0 8.7 scrub ok 2024-04-06T03:33:14.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:14 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.391000+0000 osd.1 (osd.1) 237 : cluster 0 4.2 scrub starts 2024-04-06T03:33:14.819 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:14.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:14.948 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:14.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:15.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:15.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:15.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:15.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:15.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:15.343 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:15.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:15.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:15.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:15.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:15.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:15.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.784330+0000 osd.6 (osd.6) 89 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:13.784395+0000 osd.6 (osd.6) 90 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.225941+0000 mgr.y (mgr.24368) 454 : cluster 0 pgmap v466: 148 pgs: 148 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.415996+0000 osd.1 (osd.1) 238 : cluster 0 4.2 scrub ok 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.579435+0000 osd.4 (osd.4) 279 : cluster 0 2.f scrub starts 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.584194+0000 osd.4 (osd.4) 280 : cluster 0 2.f scrub ok 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:14.651739+0000 mon.a (mon.0) 1283 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.349654+0000 osd.1 (osd.1) 239 : cluster 0 2.6 scrub starts 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.352204+0000 osd.1 (osd.1) 240 : cluster 0 2.6 scrub ok 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.784330+0000 osd.6 (osd.6) 89 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:15.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:13.784395+0000 osd.6 (osd.6) 90 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.225941+0000 mgr.y (mgr.24368) 454 : cluster 0 pgmap v466: 148 pgs: 148 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.415996+0000 osd.1 (osd.1) 238 : cluster 0 4.2 scrub ok 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.579435+0000 osd.4 (osd.4) 279 : cluster 0 2.f scrub starts 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.584194+0000 osd.4 (osd.4) 280 : cluster 0 2.f scrub ok 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:14.651739+0000 mon.a (mon.0) 1283 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.349654+0000 osd.1 (osd.1) 239 : cluster 0 2.6 scrub starts 2024-04-06T03:33:15.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:15 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.352204+0000 osd.1 (osd.1) 240 : cluster 0 2.6 scrub ok 2024-04-06T03:33:15.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:15.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:15.959 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:15.973 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.784330+0000 osd.6 (osd.6) 89 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:13.784395+0000 osd.6 (osd.6) 90 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.225941+0000 mgr.y (mgr.24368) 454 : cluster 0 pgmap v466: 148 pgs: 148 active+clean; 1.2 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 806 op/s 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.415996+0000 osd.1 (osd.1) 238 : cluster 0 4.2 scrub ok 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.579435+0000 osd.4 (osd.4) 279 : cluster 0 2.f scrub starts 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.584194+0000 osd.4 (osd.4) 280 : cluster 0 2.f scrub ok 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:14.651739+0000 mon.a (mon.0) 1283 : cluster 0 osdmap e219: 8 total, 8 up, 8 in 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.349654+0000 osd.1 (osd.1) 239 : cluster 0 2.6 scrub starts 2024-04-06T03:33:15.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:15 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.352204+0000 osd.1 (osd.1) 240 : cluster 0 2.6 scrub ok 2024-04-06T03:33:15.985 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:15.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:16.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:16.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:16.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:16.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:16.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:16.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:16.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:16.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:16.613 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:16.639 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:16.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:16.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:16.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.329833+0000 osd.2 (osd.2) 231 : cluster 0 2.17 scrub starts 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.332586+0000 osd.2 (osd.2) 232 : cluster 0 2.17 scrub ok 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.373111+0000 osd.7 (osd.7) 297 : cluster 0 3.13 scrub starts 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.375384+0000 osd.7 (osd.7) 298 : cluster 0 3.13 scrub ok 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.541902+0000 osd.0 (osd.0) 266 : cluster 0 3.10 scrub starts 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.547642+0000 osd.0 (osd.0) 267 : cluster 0 3.10 scrub ok 2024-04-06T03:33:16.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.586901+0000 osd.4 (osd.4) 281 : cluster 0 6.1e scrub starts 2024-04-06T03:33:16.881 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:16 smithi153 bash[23852]: cluster 2024-04-06T03:33:15.601991+0000 osd.4 (osd.4) 282 : cluster 0 6.1e scrub ok 2024-04-06T03:33:16.901 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:16.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.329833+0000 osd.2 (osd.2) 231 : cluster 0 2.17 scrub starts 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.332586+0000 osd.2 (osd.2) 232 : cluster 0 2.17 scrub ok 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.373111+0000 osd.7 (osd.7) 297 : cluster 0 3.13 scrub starts 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.375384+0000 osd.7 (osd.7) 298 : cluster 0 3.13 scrub ok 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.541902+0000 osd.0 (osd.0) 266 : cluster 0 3.10 scrub starts 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.547642+0000 osd.0 (osd.0) 267 : cluster 0 3.10 scrub ok 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.586901+0000 osd.4 (osd.4) 281 : cluster 0 6.1e scrub starts 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[21467]: cluster 2024-04-06T03:33:15.601991+0000 osd.4 (osd.4) 282 : cluster 0 6.1e scrub ok 2024-04-06T03:33:16.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.329833+0000 osd.2 (osd.2) 231 : cluster 0 2.17 scrub starts 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.332586+0000 osd.2 (osd.2) 232 : cluster 0 2.17 scrub ok 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.373111+0000 osd.7 (osd.7) 297 : cluster 0 3.13 scrub starts 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.375384+0000 osd.7 (osd.7) 298 : cluster 0 3.13 scrub ok 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.541902+0000 osd.0 (osd.0) 266 : cluster 0 3.10 scrub starts 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.547642+0000 osd.0 (osd.0) 267 : cluster 0 3.10 scrub ok 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.586901+0000 osd.4 (osd.4) 281 : cluster 0 6.1e scrub starts 2024-04-06T03:33:16.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:16 smithi050 bash[29221]: cluster 2024-04-06T03:33:15.601991+0000 osd.4 (osd.4) 282 : cluster 0 6.1e scrub ok 2024-04-06T03:33:17.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:17.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:17.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:17.167 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:17.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:17.302 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:17.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:17.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:17.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:17.562 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:17.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:17.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:17 smithi153 bash[23852]: cluster 2024-04-06T03:33:16.226544+0000 mgr.y (mgr.24368) 455 : cluster 0 pgmap v468: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 834 op/s 2024-04-06T03:33:17.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:17 smithi153 bash[23852]: cluster 2024-04-06T03:33:16.344390+0000 osd.7 (osd.7) 299 : cluster 0 5.4 scrub starts 2024-04-06T03:33:17.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:17 smithi153 bash[23852]: cluster 2024-04-06T03:33:16.345556+0000 osd.7 (osd.7) 300 : cluster 0 5.4 scrub ok 2024-04-06T03:33:17.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:17 smithi153 bash[23852]: cluster 2024-04-06T03:33:17.046843+0000 osd.3 (osd.3) 283 : cluster 0 8.0 scrub starts 2024-04-06T03:33:17.703 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:17.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:17.837 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:17.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:17.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[21467]: cluster 2024-04-06T03:33:16.226544+0000 mgr.y (mgr.24368) 455 : cluster 0 pgmap v468: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 834 op/s 2024-04-06T03:33:17.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[21467]: cluster 2024-04-06T03:33:16.344390+0000 osd.7 (osd.7) 299 : cluster 0 5.4 scrub starts 2024-04-06T03:33:17.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[21467]: cluster 2024-04-06T03:33:16.345556+0000 osd.7 (osd.7) 300 : cluster 0 5.4 scrub ok 2024-04-06T03:33:17.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[21467]: cluster 2024-04-06T03:33:17.046843+0000 osd.3 (osd.3) 283 : cluster 0 8.0 scrub starts 2024-04-06T03:33:17.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[29221]: cluster 2024-04-06T03:33:16.226544+0000 mgr.y (mgr.24368) 455 : cluster 0 pgmap v468: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 834 op/s 2024-04-06T03:33:17.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[29221]: cluster 2024-04-06T03:33:16.344390+0000 osd.7 (osd.7) 299 : cluster 0 5.4 scrub starts 2024-04-06T03:33:17.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[29221]: cluster 2024-04-06T03:33:16.345556+0000 osd.7 (osd.7) 300 : cluster 0 5.4 scrub ok 2024-04-06T03:33:17.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:17 smithi050 bash[29221]: cluster 2024-04-06T03:33:17.046843+0000 osd.3 (osd.3) 283 : cluster 0 8.0 scrub starts 2024-04-06T03:33:17.968 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:17.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:18.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:18.094 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:18.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:18.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:18.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:18.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:18.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:18.502 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:18.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:18.634 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:18.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:18.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:18.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:18 smithi153 bash[23852]: cluster 2024-04-06T03:33:17.342317+0000 osd.2 (osd.2) 233 : cluster 0 3.4 scrub starts 2024-04-06T03:33:18.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:18 smithi153 bash[23852]: cluster 2024-04-06T03:33:17.349148+0000 osd.2 (osd.2) 234 : cluster 0 3.4 scrub ok 2024-04-06T03:33:18.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:18 smithi153 bash[23852]: cluster 2024-04-06T03:33:17.539159+0000 osd.3 (osd.3) 284 : cluster 0 8.0 scrub ok 2024-04-06T03:33:18.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:18 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.023367+0000 osd.3 (osd.3) 285 : cluster 0 3.1e scrub starts 2024-04-06T03:33:18.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:18 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.029634+0000 osd.3 (osd.3) 286 : cluster 0 3.1e scrub ok 2024-04-06T03:33:18.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:18 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.377200+0000 osd.1 (osd.1) 241 : cluster 0 8.1 scrub starts 2024-04-06T03:33:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:18.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[21467]: cluster 2024-04-06T03:33:17.342317+0000 osd.2 (osd.2) 233 : cluster 0 3.4 scrub starts 2024-04-06T03:33:18.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[21467]: cluster 2024-04-06T03:33:17.349148+0000 osd.2 (osd.2) 234 : cluster 0 3.4 scrub ok 2024-04-06T03:33:18.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[21467]: cluster 2024-04-06T03:33:17.539159+0000 osd.3 (osd.3) 284 : cluster 0 8.0 scrub ok 2024-04-06T03:33:18.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.023367+0000 osd.3 (osd.3) 285 : cluster 0 3.1e scrub starts 2024-04-06T03:33:18.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.029634+0000 osd.3 (osd.3) 286 : cluster 0 3.1e scrub ok 2024-04-06T03:33:18.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.377200+0000 osd.1 (osd.1) 241 : cluster 0 8.1 scrub starts 2024-04-06T03:33:18.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:18.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:18.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:19.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:19.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:19.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:19.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[29221]: cluster 2024-04-06T03:33:17.342317+0000 osd.2 (osd.2) 233 : cluster 0 3.4 scrub starts 2024-04-06T03:33:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[29221]: cluster 2024-04-06T03:33:17.349148+0000 osd.2 (osd.2) 234 : cluster 0 3.4 scrub ok 2024-04-06T03:33:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[29221]: cluster 2024-04-06T03:33:17.539159+0000 osd.3 (osd.3) 284 : cluster 0 8.0 scrub ok 2024-04-06T03:33:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.023367+0000 osd.3 (osd.3) 285 : cluster 0 3.1e scrub starts 2024-04-06T03:33:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.029634+0000 osd.3 (osd.3) 286 : cluster 0 3.1e scrub ok 2024-04-06T03:33:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:18 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.377200+0000 osd.1 (osd.1) 241 : cluster 0 8.1 scrub starts 2024-04-06T03:33:19.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:19.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:19.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:19.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:19.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:19.616 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:33:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:33:19.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:19.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:19.753 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:33:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:33:19.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:19 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.226899+0000 mgr.y (mgr.24368) 456 : cluster 0 pgmap v469: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 834 op/s 2024-04-06T03:33:19.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:19 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.394785+0000 osd.2 (osd.2) 235 : cluster 0 2.e scrub starts 2024-04-06T03:33:19.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:19 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.398977+0000 osd.2 (osd.2) 236 : cluster 0 2.e scrub ok 2024-04-06T03:33:19.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:19 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.764935+0000 osd.1 (osd.1) 242 : cluster 0 8.1 scrub ok 2024-04-06T03:33:19.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:19 smithi153 bash[23852]: audit 2024-04-06T03:33:19.204237+0000 mon.a (mon.0) 1284 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:19.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:19.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.226899+0000 mgr.y (mgr.24368) 456 : cluster 0 pgmap v469: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 834 op/s 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.394785+0000 osd.2 (osd.2) 235 : cluster 0 2.e scrub starts 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.398977+0000 osd.2 (osd.2) 236 : cluster 0 2.e scrub ok 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.764935+0000 osd.1 (osd.1) 242 : cluster 0 8.1 scrub ok 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[21467]: audit 2024-04-06T03:33:19.204237+0000 mon.a (mon.0) 1284 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.226899+0000 mgr.y (mgr.24368) 456 : cluster 0 pgmap v469: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.0 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 834 op/s 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.394785+0000 osd.2 (osd.2) 235 : cluster 0 2.e scrub starts 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.398977+0000 osd.2 (osd.2) 236 : cluster 0 2.e scrub ok 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.764935+0000 osd.1 (osd.1) 242 : cluster 0 8.1 scrub ok 2024-04-06T03:33:19.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:19 smithi050 bash[29221]: audit 2024-04-06T03:33:19.204237+0000 mon.a (mon.0) 1284 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:19.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:19.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:20.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:20.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:20.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:20.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:20.175 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:20.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:20.305 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:20.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:20.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:20.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:20.562 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:20.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:20.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:20.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.983327+0000 osd.4 (osd.4) 283 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:20.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:18.983425+0000 osd.4 (osd.4) 284 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: audit 2024-04-06T03:33:19.580697+0000 mgr.y (mgr.24368) 457 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:19.664447+0000 mon.a (mon.0) 1285 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:19.688214+0000 osd.0 (osd.0) 268 : cluster 0 5.15 scrub starts 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:19.693895+0000 osd.0 (osd.0) 269 : cluster 0 5.15 scrub ok 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.104996+0000 osd.3 (osd.3) 287 : cluster 0 2.14 scrub starts 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.106504+0000 osd.3 (osd.3) 288 : cluster 0 2.14 scrub ok 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.363462+0000 osd.1 (osd.1) 243 : cluster 0 8.3 scrub starts 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.983327+0000 osd.4 (osd.4) 283 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:18.983425+0000 osd.4 (osd.4) 284 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: audit 2024-04-06T03:33:19.580697+0000 mgr.y (mgr.24368) 457 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:19.664447+0000 mon.a (mon.0) 1285 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:19.688214+0000 osd.0 (osd.0) 268 : cluster 0 5.15 scrub starts 2024-04-06T03:33:20.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:19.693895+0000 osd.0 (osd.0) 269 : cluster 0 5.15 scrub ok 2024-04-06T03:33:20.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.104996+0000 osd.3 (osd.3) 287 : cluster 0 2.14 scrub starts 2024-04-06T03:33:20.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.106504+0000 osd.3 (osd.3) 288 : cluster 0 2.14 scrub ok 2024-04-06T03:33:20.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:20 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.363462+0000 osd.1 (osd.1) 243 : cluster 0 8.3 scrub starts 2024-04-06T03:33:20.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.983327+0000 osd.4 (osd.4) 283 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:20.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:18.983425+0000 osd.4 (osd.4) 284 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:20.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: audit 2024-04-06T03:33:19.580697+0000 mgr.y (mgr.24368) 457 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:20.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:19.664447+0000 mon.a (mon.0) 1285 : cluster 0 osdmap e220: 8 total, 8 up, 8 in 2024-04-06T03:33:20.796 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:19.688214+0000 osd.0 (osd.0) 268 : cluster 0 5.15 scrub starts 2024-04-06T03:33:20.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:19.693895+0000 osd.0 (osd.0) 269 : cluster 0 5.15 scrub ok 2024-04-06T03:33:20.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.104996+0000 osd.3 (osd.3) 287 : cluster 0 2.14 scrub starts 2024-04-06T03:33:20.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.106504+0000 osd.3 (osd.3) 288 : cluster 0 2.14 scrub ok 2024-04-06T03:33:20.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:20 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.363462+0000 osd.1 (osd.1) 243 : cluster 0 8.3 scrub starts 2024-04-06T03:33:20.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:20.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:20.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:20.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:20.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:21.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:21.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:21.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:21.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:21.350 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:21.380 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:21.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:21.506 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:21.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:21.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:21.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:21.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:21.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.227432+0000 mgr.y (mgr.24368) 458 : cluster 0 pgmap v471: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 917 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 799 op/s 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.472094+0000 osd.2 (osd.2) 237 : cluster 0 2.16 scrub starts 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.475701+0000 osd.2 (osd.2) 238 : cluster 0 2.16 scrub ok 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:20.764051+0000 osd.1 (osd.1) 244 : cluster 0 8.3 scrub ok 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.153429+0000 osd.3 (osd.3) 289 : cluster 0 8.8 scrub starts 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.218849+0000 osd.3 (osd.3) 290 : cluster 0 8.8 scrub ok 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.370416+0000 osd.1 (osd.1) 245 : cluster 0 2.9 scrub starts 2024-04-06T03:33:21.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.373538+0000 osd.1 (osd.1) 246 : cluster 0 2.9 scrub ok 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.227432+0000 mgr.y (mgr.24368) 458 : cluster 0 pgmap v471: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 917 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 799 op/s 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.472094+0000 osd.2 (osd.2) 237 : cluster 0 2.16 scrub starts 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.475701+0000 osd.2 (osd.2) 238 : cluster 0 2.16 scrub ok 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:20.764051+0000 osd.1 (osd.1) 244 : cluster 0 8.3 scrub ok 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.153429+0000 osd.3 (osd.3) 289 : cluster 0 8.8 scrub starts 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.218849+0000 osd.3 (osd.3) 290 : cluster 0 8.8 scrub ok 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.370416+0000 osd.1 (osd.1) 245 : cluster 0 2.9 scrub starts 2024-04-06T03:33:21.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:21 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.373538+0000 osd.1 (osd.1) 246 : cluster 0 2.9 scrub ok 2024-04-06T03:33:21.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:21.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:22.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:22.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:22.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.227432+0000 mgr.y (mgr.24368) 458 : cluster 0 pgmap v471: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 917 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 799 op/s 2024-04-06T03:33:22.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.472094+0000 osd.2 (osd.2) 237 : cluster 0 2.16 scrub starts 2024-04-06T03:33:22.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.475701+0000 osd.2 (osd.2) 238 : cluster 0 2.16 scrub ok 2024-04-06T03:33:22.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:20.764051+0000 osd.1 (osd.1) 244 : cluster 0 8.3 scrub ok 2024-04-06T03:33:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.153429+0000 osd.3 (osd.3) 289 : cluster 0 8.8 scrub starts 2024-04-06T03:33:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.218849+0000 osd.3 (osd.3) 290 : cluster 0 8.8 scrub ok 2024-04-06T03:33:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.370416+0000 osd.1 (osd.1) 245 : cluster 0 2.9 scrub starts 2024-04-06T03:33:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:21 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.373538+0000 osd.1 (osd.1) 246 : cluster 0 2.9 scrub ok 2024-04-06T03:33:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:22.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:22.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:22.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:22.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:22.439 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:22.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:22.566 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:22.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:22.696 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:22.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:22.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:22.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:22.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.327167+0000 osd.7 (osd.7) 301 : cluster 0 6.12 scrub starts 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.329347+0000 osd.2 (osd.2) 239 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.329484+0000 osd.2 (osd.2) 240 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.336124+0000 osd.7 (osd.7) 302 : cluster 0 6.12 scrub ok 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.446986+0000 osd.4 (osd.4) 285 : cluster 0 3.1a scrub starts 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.449356+0000 osd.4 (osd.4) 286 : cluster 0 3.1a scrub ok 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:21.749949+0000 mon.a (mon.0) 1286 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:22.227901+0000 mgr.y (mgr.24368) 459 : cluster 0 pgmap v473: 148 pgs: 148 active+clean; 758 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:22.353112+0000 osd.1 (osd.1) 247 : cluster 0 3.19 scrub starts 2024-04-06T03:33:23.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:22 smithi153 bash[23852]: cluster 2024-04-06T03:33:22.362130+0000 osd.1 (osd.1) 248 : cluster 0 3.19 scrub ok 2024-04-06T03:33:23.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:23.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.327167+0000 osd.7 (osd.7) 301 : cluster 0 6.12 scrub starts 2024-04-06T03:33:23.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.329347+0000 osd.2 (osd.2) 239 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:23.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.329484+0000 osd.2 (osd.2) 240 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:23.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.336124+0000 osd.7 (osd.7) 302 : cluster 0 6.12 scrub ok 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.446986+0000 osd.4 (osd.4) 285 : cluster 0 3.1a scrub starts 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.449356+0000 osd.4 (osd.4) 286 : cluster 0 3.1a scrub ok 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:21.749949+0000 mon.a (mon.0) 1286 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:22.227901+0000 mgr.y (mgr.24368) 459 : cluster 0 pgmap v473: 148 pgs: 148 active+clean; 758 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:22.353112+0000 osd.1 (osd.1) 247 : cluster 0 3.19 scrub starts 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[21467]: cluster 2024-04-06T03:33:22.362130+0000 osd.1 (osd.1) 248 : cluster 0 3.19 scrub ok 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.327167+0000 osd.7 (osd.7) 301 : cluster 0 6.12 scrub starts 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.329347+0000 osd.2 (osd.2) 239 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.329484+0000 osd.2 (osd.2) 240 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.336124+0000 osd.7 (osd.7) 302 : cluster 0 6.12 scrub ok 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.446986+0000 osd.4 (osd.4) 285 : cluster 0 3.1a scrub starts 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.449356+0000 osd.4 (osd.4) 286 : cluster 0 3.1a scrub ok 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:21.749949+0000 mon.a (mon.0) 1286 : cluster 0 osdmap e221: 8 total, 8 up, 8 in 2024-04-06T03:33:23.173 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:22.227901+0000 mgr.y (mgr.24368) 459 : cluster 0 pgmap v473: 148 pgs: 148 active+clean; 758 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:33:23.174 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:22.353112+0000 osd.1 (osd.1) 247 : cluster 0 3.19 scrub starts 2024-04-06T03:33:23.174 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:22 smithi050 bash[29221]: cluster 2024-04-06T03:33:22.362130+0000 osd.1 (osd.1) 248 : cluster 0 3.19 scrub ok 2024-04-06T03:33:23.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:23.218 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:23.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:23.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:23.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:23.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:23.492 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:23.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:23.625 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:23.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:23.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:23.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:23.881 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:23.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:23.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:23.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:23 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.161534+0000 osd.3 (osd.3) 291 : cluster 0 8.f scrub starts 2024-04-06T03:33:23.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:23 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.361692+0000 osd.1 (osd.1) 249 : cluster 0 2.2 scrub starts 2024-04-06T03:33:23.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:23 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.364855+0000 osd.1 (osd.1) 250 : cluster 0 2.2 scrub ok 2024-04-06T03:33:23.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:23 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.161534+0000 osd.3 (osd.3) 291 : cluster 0 8.f scrub starts 2024-04-06T03:33:23.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:23 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.361692+0000 osd.1 (osd.1) 249 : cluster 0 2.2 scrub starts 2024-04-06T03:33:23.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:23 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.364855+0000 osd.1 (osd.1) 250 : cluster 0 2.2 scrub ok 2024-04-06T03:33:24.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:24.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:23 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.161534+0000 osd.3 (osd.3) 291 : cluster 0 8.f scrub starts 2024-04-06T03:33:24.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:23 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.361692+0000 osd.1 (osd.1) 249 : cluster 0 2.2 scrub starts 2024-04-06T03:33:24.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:23 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.364855+0000 osd.1 (osd.1) 250 : cluster 0 2.2 scrub ok 2024-04-06T03:33:24.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:24.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:24.250 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:24.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:24.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:24.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:24.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:24.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:24.666 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:24.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.499378+0000 osd.4 (osd.4) 287 : cluster 0 6.1d scrub starts 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.512648+0000 osd.3 (osd.3) 292 : cluster 0 8.f scrub ok 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.526301+0000 osd.4 (osd.4) 288 : cluster 0 6.1d scrub ok 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.768020+0000 osd.0 (osd.0) 270 : cluster 0 2.13 scrub starts 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:23.775400+0000 osd.0 (osd.0) 271 : cluster 0 2.13 scrub ok 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.228591+0000 mgr.y (mgr.24368) 460 : cluster 0 pgmap v474: 148 pgs: 148 active+clean; 683 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 692 op/s 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.360216+0000 osd.1 (osd.1) 251 : cluster 0 5.12 scrub starts 2024-04-06T03:33:24.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:24 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.363056+0000 osd.1 (osd.1) 252 : cluster 0 5.12 scrub ok 2024-04-06T03:33:24.797 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:24.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:24.925 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:24.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.499378+0000 osd.4 (osd.4) 287 : cluster 0 6.1d scrub starts 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.512648+0000 osd.3 (osd.3) 292 : cluster 0 8.f scrub ok 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.526301+0000 osd.4 (osd.4) 288 : cluster 0 6.1d scrub ok 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.768020+0000 osd.0 (osd.0) 270 : cluster 0 2.13 scrub starts 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:23.775400+0000 osd.0 (osd.0) 271 : cluster 0 2.13 scrub ok 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.228591+0000 mgr.y (mgr.24368) 460 : cluster 0 pgmap v474: 148 pgs: 148 active+clean; 683 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 692 op/s 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.360216+0000 osd.1 (osd.1) 251 : cluster 0 5.12 scrub starts 2024-04-06T03:33:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.363056+0000 osd.1 (osd.1) 252 : cluster 0 5.12 scrub ok 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.499378+0000 osd.4 (osd.4) 287 : cluster 0 6.1d scrub starts 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.512648+0000 osd.3 (osd.3) 292 : cluster 0 8.f scrub ok 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.526301+0000 osd.4 (osd.4) 288 : cluster 0 6.1d scrub ok 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.768020+0000 osd.0 (osd.0) 270 : cluster 0 2.13 scrub starts 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:23.775400+0000 osd.0 (osd.0) 271 : cluster 0 2.13 scrub ok 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.228591+0000 mgr.y (mgr.24368) 460 : cluster 0 pgmap v474: 148 pgs: 148 active+clean; 683 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 692 op/s 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.360216+0000 osd.1 (osd.1) 251 : cluster 0 5.12 scrub starts 2024-04-06T03:33:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:24 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.363056+0000 osd.1 (osd.1) 252 : cluster 0 5.12 scrub ok 2024-04-06T03:33:25.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:25.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:25.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:25.184 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:25.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:25.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:25.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:25.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:25.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:25.585 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:25.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:25.708 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:25.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:25.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:25.978 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:25.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.343055+0000 osd.7 (osd.7) 303 : cluster 0 5.19 scrub starts 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.344082+0000 osd.7 (osd.7) 304 : cluster 0 5.19 scrub ok 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.504115+0000 osd.2 (osd.2) 241 : cluster 0 3.0 scrub starts 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.513659+0000 osd.2 (osd.2) 242 : cluster 0 3.0 scrub ok 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:24.809907+0000 osd.0 (osd.0) 272 : cluster 0 8.9 scrub starts 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:25.084861+0000 osd.0 (osd.0) 273 : cluster 0 8.9 scrub ok 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:25.349698+0000 osd.1 (osd.1) 253 : cluster 0 5.1d scrub starts 2024-04-06T03:33:26.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:25 smithi153 bash[23852]: cluster 2024-04-06T03:33:25.355257+0000 osd.1 (osd.1) 254 : cluster 0 5.1d scrub ok 2024-04-06T03:33:26.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:26.106 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:26.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.343055+0000 osd.7 (osd.7) 303 : cluster 0 5.19 scrub starts 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.344082+0000 osd.7 (osd.7) 304 : cluster 0 5.19 scrub ok 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.504115+0000 osd.2 (osd.2) 241 : cluster 0 3.0 scrub starts 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.513659+0000 osd.2 (osd.2) 242 : cluster 0 3.0 scrub ok 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:24.809907+0000 osd.0 (osd.0) 272 : cluster 0 8.9 scrub starts 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:25.084861+0000 osd.0 (osd.0) 273 : cluster 0 8.9 scrub ok 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:25.349698+0000 osd.1 (osd.1) 253 : cluster 0 5.1d scrub starts 2024-04-06T03:33:26.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:25 smithi050 bash[29221]: cluster 2024-04-06T03:33:25.355257+0000 osd.1 (osd.1) 254 : cluster 0 5.1d scrub ok 2024-04-06T03:33:26.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:26.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:26.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:26.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.343055+0000 osd.7 (osd.7) 303 : cluster 0 5.19 scrub starts 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.344082+0000 osd.7 (osd.7) 304 : cluster 0 5.19 scrub ok 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.504115+0000 osd.2 (osd.2) 241 : cluster 0 3.0 scrub starts 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.513659+0000 osd.2 (osd.2) 242 : cluster 0 3.0 scrub ok 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:24.809907+0000 osd.0 (osd.0) 272 : cluster 0 8.9 scrub starts 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:25.084861+0000 osd.0 (osd.0) 273 : cluster 0 8.9 scrub ok 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:25.349698+0000 osd.1 (osd.1) 253 : cluster 0 5.1d scrub starts 2024-04-06T03:33:26.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:25.355257+0000 osd.1 (osd.1) 254 : cluster 0 5.1d scrub ok 2024-04-06T03:33:26.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:26.512 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:26.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:26.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:26.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:26.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:26.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:26.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:26.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:27.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:27.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:27.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:26.229192+0000 mgr.y (mgr.24368) 461 : cluster 0 pgmap v475: 148 pgs: 148 active+clean; 463 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-04-06T03:33:27.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:26.356972+0000 osd.1 (osd.1) 255 : cluster 0 6.19 scrub starts 2024-04-06T03:33:27.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: cluster 2024-04-06T03:33:26.400643+0000 osd.1 (osd.1) 256 : cluster 0 6.19 scrub ok 2024-04-06T03:33:27.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[21467]: audit 2024-04-06T03:33:26.434793+0000 mon.a (mon.0) 1287 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:33:27.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[29221]: cluster 2024-04-06T03:33:26.229192+0000 mgr.y (mgr.24368) 461 : cluster 0 pgmap v475: 148 pgs: 148 active+clean; 463 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-04-06T03:33:27.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[29221]: cluster 2024-04-06T03:33:26.356972+0000 osd.1 (osd.1) 255 : cluster 0 6.19 scrub starts 2024-04-06T03:33:27.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[29221]: cluster 2024-04-06T03:33:26.400643+0000 osd.1 (osd.1) 256 : cluster 0 6.19 scrub ok 2024-04-06T03:33:27.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:26 smithi050 bash[29221]: audit 2024-04-06T03:33:26.434793+0000 mon.a (mon.0) 1287 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:33:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:27.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:26 smithi153 bash[23852]: cluster 2024-04-06T03:33:26.229192+0000 mgr.y (mgr.24368) 461 : cluster 0 pgmap v475: 148 pgs: 148 active+clean; 463 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-04-06T03:33:27.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:26 smithi153 bash[23852]: cluster 2024-04-06T03:33:26.356972+0000 osd.1 (osd.1) 255 : cluster 0 6.19 scrub starts 2024-04-06T03:33:27.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:26 smithi153 bash[23852]: cluster 2024-04-06T03:33:26.400643+0000 osd.1 (osd.1) 256 : cluster 0 6.19 scrub ok 2024-04-06T03:33:27.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:26 smithi153 bash[23852]: audit 2024-04-06T03:33:26.434793+0000 mon.a (mon.0) 1287 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:33:27.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:27.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:27.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:27.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:27.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:27.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:27.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:27.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:27.678 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:27.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:27.807 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:27.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:27.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[21467]: cluster 2024-04-06T03:33:26.450666+0000 osd.4 (osd.4) 289 : cluster 0 4.1 scrub starts 2024-04-06T03:33:27.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[21467]: cluster 2024-04-06T03:33:26.451627+0000 osd.4 (osd.4) 290 : cluster 0 4.1 scrub ok 2024-04-06T03:33:27.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[21467]: audit 2024-04-06T03:33:27.019845+0000 mon.a (mon.0) 1288 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:27.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[21467]: audit 2024-04-06T03:33:27.026165+0000 mon.a (mon.0) 1289 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:27.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[29221]: cluster 2024-04-06T03:33:26.450666+0000 osd.4 (osd.4) 289 : cluster 0 4.1 scrub starts 2024-04-06T03:33:27.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[29221]: cluster 2024-04-06T03:33:26.451627+0000 osd.4 (osd.4) 290 : cluster 0 4.1 scrub ok 2024-04-06T03:33:27.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[29221]: audit 2024-04-06T03:33:27.019845+0000 mon.a (mon.0) 1288 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:27.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:27 smithi050 bash[29221]: audit 2024-04-06T03:33:27.026165+0000 mon.a (mon.0) 1289 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:27.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:27.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:28.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:27 smithi153 bash[23852]: cluster 2024-04-06T03:33:26.450666+0000 osd.4 (osd.4) 289 : cluster 0 4.1 scrub starts 2024-04-06T03:33:28.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:27 smithi153 bash[23852]: cluster 2024-04-06T03:33:26.451627+0000 osd.4 (osd.4) 290 : cluster 0 4.1 scrub ok 2024-04-06T03:33:28.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:27 smithi153 bash[23852]: audit 2024-04-06T03:33:27.019845+0000 mon.a (mon.0) 1288 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:27 smithi153 bash[23852]: audit 2024-04-06T03:33:27.026165+0000 mon.a (mon.0) 1289 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:28.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:28.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:28.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:28.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:28.318 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:28.345 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:28.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:28.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:28.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:28.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:28.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:28.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:28.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:28.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:28.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:28.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:28 smithi050 bash[21467]: cluster 2024-04-06T03:33:27.709724+0000 osd.2 (osd.2) 243 : cluster 0 2.3 scrub starts 2024-04-06T03:33:28.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:28 smithi050 bash[21467]: cluster 2024-04-06T03:33:27.712139+0000 osd.2 (osd.2) 244 : cluster 0 2.3 scrub ok 2024-04-06T03:33:28.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:28 smithi050 bash[21467]: cluster 2024-04-06T03:33:28.230130+0000 mgr.y (mgr.24368) 462 : cluster 0 pgmap v476: 148 pgs: 148 active+clean; 463 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-04-06T03:33:28.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:28 smithi050 bash[29221]: cluster 2024-04-06T03:33:27.709724+0000 osd.2 (osd.2) 243 : cluster 0 2.3 scrub starts 2024-04-06T03:33:28.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:28 smithi050 bash[29221]: cluster 2024-04-06T03:33:27.712139+0000 osd.2 (osd.2) 244 : cluster 0 2.3 scrub ok 2024-04-06T03:33:28.978 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:28 smithi050 bash[29221]: cluster 2024-04-06T03:33:28.230130+0000 mgr.y (mgr.24368) 462 : cluster 0 pgmap v476: 148 pgs: 148 active+clean; 463 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-04-06T03:33:29.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:29.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:28 smithi153 bash[23852]: cluster 2024-04-06T03:33:27.709724+0000 osd.2 (osd.2) 243 : cluster 0 2.3 scrub starts 2024-04-06T03:33:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:28 smithi153 bash[23852]: cluster 2024-04-06T03:33:27.712139+0000 osd.2 (osd.2) 244 : cluster 0 2.3 scrub ok 2024-04-06T03:33:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:28 smithi153 bash[23852]: cluster 2024-04-06T03:33:28.230130+0000 mgr.y (mgr.24368) 462 : cluster 0 pgmap v476: 148 pgs: 148 active+clean; 463 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 933 op/s 2024-04-06T03:33:29.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:29.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:29.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:29.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:29.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:29.422 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:29.485 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:33:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:33:29.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:29.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:29.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:29.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:29.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:29.778 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:33:29.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:29.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:29.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:29.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[23852]: cluster 2024-04-06T03:33:28.306724+0000 osd.7 (osd.7) 305 : cluster 0 6.3 scrub starts 2024-04-06T03:33:29.928 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[23852]: cluster 2024-04-06T03:33:28.341392+0000 osd.7 (osd.7) 306 : cluster 0 6.3 scrub ok 2024-04-06T03:33:29.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.024671+0000 osd.0 (osd.0) 274 : cluster 0 6.a scrub starts 2024-04-06T03:33:29.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.059607+0000 osd.0 (osd.0) 275 : cluster 0 6.a scrub ok 2024-04-06T03:33:29.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.199882+0000 osd.3 (osd.3) 293 : cluster 0 8.b scrub starts 2024-04-06T03:33:29.929 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:29 smithi153 bash[23852]: audit 2024-04-06T03:33:29.587875+0000 mgr.y (mgr.24368) 463 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:29.943 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:29.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:30.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:30.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:30.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21467]: cluster 2024-04-06T03:33:28.306724+0000 osd.7 (osd.7) 305 : cluster 0 6.3 scrub starts 2024-04-06T03:33:30.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21467]: cluster 2024-04-06T03:33:28.341392+0000 osd.7 (osd.7) 306 : cluster 0 6.3 scrub ok 2024-04-06T03:33:30.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.024671+0000 osd.0 (osd.0) 274 : cluster 0 6.a scrub starts 2024-04-06T03:33:30.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.059607+0000 osd.0 (osd.0) 275 : cluster 0 6.a scrub ok 2024-04-06T03:33:30.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.199882+0000 osd.3 (osd.3) 293 : cluster 0 8.b scrub starts 2024-04-06T03:33:30.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[21467]: audit 2024-04-06T03:33:29.587875+0000 mgr.y (mgr.24368) 463 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:30.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[29221]: cluster 2024-04-06T03:33:28.306724+0000 osd.7 (osd.7) 305 : cluster 0 6.3 scrub starts 2024-04-06T03:33:30.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[29221]: cluster 2024-04-06T03:33:28.341392+0000 osd.7 (osd.7) 306 : cluster 0 6.3 scrub ok 2024-04-06T03:33:30.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.024671+0000 osd.0 (osd.0) 274 : cluster 0 6.a scrub starts 2024-04-06T03:33:30.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.059607+0000 osd.0 (osd.0) 275 : cluster 0 6.a scrub ok 2024-04-06T03:33:30.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.199882+0000 osd.3 (osd.3) 293 : cluster 0 8.b scrub starts 2024-04-06T03:33:30.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:29 smithi050 bash[29221]: audit 2024-04-06T03:33:29.587875+0000 mgr.y (mgr.24368) 463 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:30.174 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:30.204 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:30.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:30.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:30.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:30.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:30.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:30.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:30.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:30.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:30.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:30.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.325146+0000 osd.7 (osd.7) 307 : cluster 0 5.1a scrub starts 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.327134+0000 osd.7 (osd.7) 308 : cluster 0 5.1a scrub ok 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.385367+0000 osd.3 (osd.3) 294 : cluster 0 8.b scrub ok 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:29.749724+0000 osd.2 (osd.2) 245 : cluster 0 6.f scrub starts 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.230942+0000 mgr.y (mgr.24368) 464 : cluster 0 pgmap v477: 148 pgs: 148 active+clean; 350 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.253900+0000 osd.3 (osd.3) 295 : cluster 0 3.18 scrub starts 2024-04-06T03:33:30.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:30 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.273217+0000 osd.3 (osd.3) 296 : cluster 0 3.18 scrub ok 2024-04-06T03:33:31.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:31.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:31.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.325146+0000 osd.7 (osd.7) 307 : cluster 0 5.1a scrub starts 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.327134+0000 osd.7 (osd.7) 308 : cluster 0 5.1a scrub ok 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.385367+0000 osd.3 (osd.3) 294 : cluster 0 8.b scrub ok 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:29.749724+0000 osd.2 (osd.2) 245 : cluster 0 6.f scrub starts 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.230942+0000 mgr.y (mgr.24368) 464 : cluster 0 pgmap v477: 148 pgs: 148 active+clean; 350 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.253900+0000 osd.3 (osd.3) 295 : cluster 0 3.18 scrub starts 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.273217+0000 osd.3 (osd.3) 296 : cluster 0 3.18 scrub ok 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.325146+0000 osd.7 (osd.7) 307 : cluster 0 5.1a scrub starts 2024-04-06T03:33:31.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.327134+0000 osd.7 (osd.7) 308 : cluster 0 5.1a scrub ok 2024-04-06T03:33:31.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.385367+0000 osd.3 (osd.3) 294 : cluster 0 8.b scrub ok 2024-04-06T03:33:31.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:29.749724+0000 osd.2 (osd.2) 245 : cluster 0 6.f scrub starts 2024-04-06T03:33:31.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.230942+0000 mgr.y (mgr.24368) 464 : cluster 0 pgmap v477: 148 pgs: 148 active+clean; 350 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:33:31.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.253900+0000 osd.3 (osd.3) 295 : cluster 0 3.18 scrub starts 2024-04-06T03:33:31.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:30 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.273217+0000 osd.3 (osd.3) 296 : cluster 0 3.18 scrub ok 2024-04-06T03:33:31.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:31.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:31.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:31.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:31.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:31.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:31.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:31.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:31.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:31.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:31.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:31.909 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:31.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:32.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:32.039 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:32.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.117638+0000 osd.2 (osd.2) 246 : cluster 0 6.f scrub ok 2024-04-06T03:33:32.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.456723+0000 osd.1 (osd.1) 257 : cluster 0 2.15 scrub starts 2024-04-06T03:33:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.460422+0000 osd.1 (osd.1) 258 : cluster 0 2.15 scrub ok 2024-04-06T03:33:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.758967+0000 osd.2 (osd.2) 247 : cluster 0 5.d scrub starts 2024-04-06T03:33:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:30.761303+0000 osd.2 (osd.2) 248 : cluster 0 5.d scrub ok 2024-04-06T03:33:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:31.111258+0000 osd.0 (osd.0) 276 : cluster 0 6.0 scrub starts 2024-04-06T03:33:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:31 smithi153 bash[23852]: cluster 2024-04-06T03:33:31.120063+0000 osd.0 (osd.0) 277 : cluster 0 6.0 scrub ok 2024-04-06T03:33:32.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.117638+0000 osd.2 (osd.2) 246 : cluster 0 6.f scrub ok 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.456723+0000 osd.1 (osd.1) 257 : cluster 0 2.15 scrub starts 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.460422+0000 osd.1 (osd.1) 258 : cluster 0 2.15 scrub ok 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.758967+0000 osd.2 (osd.2) 247 : cluster 0 5.d scrub starts 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:30.761303+0000 osd.2 (osd.2) 248 : cluster 0 5.d scrub ok 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:31.111258+0000 osd.0 (osd.0) 276 : cluster 0 6.0 scrub starts 2024-04-06T03:33:32.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[21467]: cluster 2024-04-06T03:33:31.120063+0000 osd.0 (osd.0) 277 : cluster 0 6.0 scrub ok 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.117638+0000 osd.2 (osd.2) 246 : cluster 0 6.f scrub ok 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.456723+0000 osd.1 (osd.1) 257 : cluster 0 2.15 scrub starts 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.460422+0000 osd.1 (osd.1) 258 : cluster 0 2.15 scrub ok 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.758967+0000 osd.2 (osd.2) 247 : cluster 0 5.d scrub starts 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:30.761303+0000 osd.2 (osd.2) 248 : cluster 0 5.d scrub ok 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:31.111258+0000 osd.0 (osd.0) 276 : cluster 0 6.0 scrub starts 2024-04-06T03:33:32.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:31 smithi050 bash[29221]: cluster 2024-04-06T03:33:31.120063+0000 osd.0 (osd.0) 277 : cluster 0 6.0 scrub ok 2024-04-06T03:33:32.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:32.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:32.307 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:32.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:32.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:32.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:32.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:32.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:32.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:32.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:32.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:32.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:32.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:32.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:33.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:32 smithi153 bash[23852]: cluster 2024-04-06T03:33:31.417762+0000 osd.3 (osd.3) 297 : cluster 0 5.14 scrub starts 2024-04-06T03:33:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:32 smithi153 bash[23852]: cluster 2024-04-06T03:33:31.421271+0000 osd.3 (osd.3) 298 : cluster 0 5.14 scrub ok 2024-04-06T03:33:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:32 smithi153 bash[23852]: cluster 2024-04-06T03:33:32.231576+0000 mgr.y (mgr.24368) 465 : cluster 0 pgmap v478: 148 pgs: 148 active+clean; 208 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 976 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:33:33.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:33.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:33.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:32 smithi050 bash[21467]: cluster 2024-04-06T03:33:31.417762+0000 osd.3 (osd.3) 297 : cluster 0 5.14 scrub starts 2024-04-06T03:33:33.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:32 smithi050 bash[21467]: cluster 2024-04-06T03:33:31.421271+0000 osd.3 (osd.3) 298 : cluster 0 5.14 scrub ok 2024-04-06T03:33:33.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:32 smithi050 bash[21467]: cluster 2024-04-06T03:33:32.231576+0000 mgr.y (mgr.24368) 465 : cluster 0 pgmap v478: 148 pgs: 148 active+clean; 208 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 976 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:33:33.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:32 smithi050 bash[29221]: cluster 2024-04-06T03:33:31.417762+0000 osd.3 (osd.3) 297 : cluster 0 5.14 scrub starts 2024-04-06T03:33:33.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:32 smithi050 bash[29221]: cluster 2024-04-06T03:33:31.421271+0000 osd.3 (osd.3) 298 : cluster 0 5.14 scrub ok 2024-04-06T03:33:33.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:32 smithi050 bash[29221]: cluster 2024-04-06T03:33:32.231576+0000 mgr.y (mgr.24368) 465 : cluster 0 pgmap v478: 148 pgs: 148 active+clean; 208 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 976 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:33:33.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:33.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:33.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:33.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:33.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:33.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:33.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:33.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:33.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:33.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:33.855 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:33.892 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:33.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:33.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:34.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:34.033 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[29221]: cluster 2024-04-06T03:33:32.331607+0000 osd.7 (osd.7) 309 : cluster 0 2.7 scrub starts 2024-04-06T03:33:34.033 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[29221]: cluster 2024-04-06T03:33:32.334321+0000 osd.7 (osd.7) 310 : cluster 0 2.7 scrub ok 2024-04-06T03:33:34.033 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[29221]: cluster 2024-04-06T03:33:32.783712+0000 osd.2 (osd.2) 249 : cluster 0 6.14 scrub starts 2024-04-06T03:33:34.033 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[29221]: cluster 2024-04-06T03:33:32.787293+0000 osd.2 (osd.2) 250 : cluster 0 6.14 scrub ok 2024-04-06T03:33:34.033 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[29221]: audit 2024-04-06T03:33:33.271528+0000 mon.a (mon.0) 1290 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.033 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[29221]: audit 2024-04-06T03:33:33.278877+0000 mon.a (mon.0) 1291 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:34.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[21467]: cluster 2024-04-06T03:33:32.331607+0000 osd.7 (osd.7) 309 : cluster 0 2.7 scrub starts 2024-04-06T03:33:34.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[21467]: cluster 2024-04-06T03:33:32.334321+0000 osd.7 (osd.7) 310 : cluster 0 2.7 scrub ok 2024-04-06T03:33:34.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[21467]: cluster 2024-04-06T03:33:32.783712+0000 osd.2 (osd.2) 249 : cluster 0 6.14 scrub starts 2024-04-06T03:33:34.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[21467]: cluster 2024-04-06T03:33:32.787293+0000 osd.2 (osd.2) 250 : cluster 0 6.14 scrub ok 2024-04-06T03:33:34.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[21467]: audit 2024-04-06T03:33:33.271528+0000 mon.a (mon.0) 1290 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:33 smithi050 bash[21467]: audit 2024-04-06T03:33:33.278877+0000 mon.a (mon.0) 1291 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:34.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:34.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:34.277 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:33 smithi153 bash[23852]: cluster 2024-04-06T03:33:32.331607+0000 osd.7 (osd.7) 309 : cluster 0 2.7 scrub starts 2024-04-06T03:33:34.277 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:33 smithi153 bash[23852]: cluster 2024-04-06T03:33:32.334321+0000 osd.7 (osd.7) 310 : cluster 0 2.7 scrub ok 2024-04-06T03:33:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:33 smithi153 bash[23852]: cluster 2024-04-06T03:33:32.783712+0000 osd.2 (osd.2) 249 : cluster 0 6.14 scrub starts 2024-04-06T03:33:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:33 smithi153 bash[23852]: cluster 2024-04-06T03:33:32.787293+0000 osd.2 (osd.2) 250 : cluster 0 6.14 scrub ok 2024-04-06T03:33:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:33 smithi153 bash[23852]: audit 2024-04-06T03:33:33.271528+0000 mon.a (mon.0) 1290 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:33 smithi153 bash[23852]: audit 2024-04-06T03:33:33.278877+0000 mon.a (mon.0) 1291 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.297 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:34.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:34.422 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:34.438 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:33:34.438 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:33:34.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:34.543 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Removed 106308 objects 2024-04-06T03:33:34.543 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Clean up completed and total clean up time :125.025 2024-04-06T03:33:34.546 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:34.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:34.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:34.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:34.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:34.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:34.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:34.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:34.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: cluster 2024-04-06T03:33:33.331035+0000 osd.7 (osd.7) 311 : cluster 0 1.0 scrub starts 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: cluster 2024-04-06T03:33:33.335950+0000 osd.7 (osd.7) 312 : cluster 0 1.0 scrub ok 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.106211+0000 osd.0 (osd.0) 278 : cluster 0 3.1f scrub starts 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.113188+0000 osd.0 (osd.0) 279 : cluster 0 3.1f scrub ok 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: audit 2024-04-06T03:33:34.227037+0000 mon.a (mon.0) 1292 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.235709+0000 mgr.y (mgr.24368) 466 : cluster 0 pgmap v479: 148 pgs: 148 active+clean; 149 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 812 op/s 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: audit 2024-04-06T03:33:34.654787+0000 mon.a (mon.0) 1293 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[29221]: audit 2024-04-06T03:33:34.660277+0000 mon.a (mon.0) 1294 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: cluster 2024-04-06T03:33:33.331035+0000 osd.7 (osd.7) 311 : cluster 0 1.0 scrub starts 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: cluster 2024-04-06T03:33:33.335950+0000 osd.7 (osd.7) 312 : cluster 0 1.0 scrub ok 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.106211+0000 osd.0 (osd.0) 278 : cluster 0 3.1f scrub starts 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.113188+0000 osd.0 (osd.0) 279 : cluster 0 3.1f scrub ok 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: audit 2024-04-06T03:33:34.227037+0000 mon.a (mon.0) 1292 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.235709+0000 mgr.y (mgr.24368) 466 : cluster 0 pgmap v479: 148 pgs: 148 active+clean; 149 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 812 op/s 2024-04-06T03:33:34.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: audit 2024-04-06T03:33:34.654787+0000 mon.a (mon.0) 1293 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:34.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:34 smithi050 bash[21467]: audit 2024-04-06T03:33:34.660277+0000 mon.a (mon.0) 1294 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:35.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:35.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:35.165 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:35.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: cluster 2024-04-06T03:33:33.331035+0000 osd.7 (osd.7) 311 : cluster 0 1.0 scrub starts 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: cluster 2024-04-06T03:33:33.335950+0000 osd.7 (osd.7) 312 : cluster 0 1.0 scrub ok 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.106211+0000 osd.0 (osd.0) 278 : cluster 0 3.1f scrub starts 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.113188+0000 osd.0 (osd.0) 279 : cluster 0 3.1f scrub ok 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: audit 2024-04-06T03:33:34.227037+0000 mon.a (mon.0) 1292 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.235709+0000 mgr.y (mgr.24368) 466 : cluster 0 pgmap v479: 148 pgs: 148 active+clean; 149 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 812 op/s 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: audit 2024-04-06T03:33:34.654787+0000 mon.a (mon.0) 1293 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:35.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:34 smithi153 bash[23852]: audit 2024-04-06T03:33:34.660277+0000 mon.a (mon.0) 1294 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:35.293 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:35.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:35.422 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:35.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:35.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:35.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:35.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:35.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:35.814 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-06T03:33:35.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:35.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:35.931 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.319389+0000 osd.7 (osd.7) 313 : cluster 0 2.1c scrub starts 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.322397+0000 osd.7 (osd.7) 314 : cluster 0 2.1c scrub ok 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.758967+0000 osd.2 (osd.2) 251 : cluster 0 5.b scrub starts 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:34.760310+0000 osd.2 (osd.2) 252 : cluster 0 5.b scrub ok 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.117206+0000 osd.0 (osd.0) 280 : cluster 0 5.5 scrub starts 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.118852+0000 osd.0 (osd.0) 281 : cluster 0 5.5 scrub ok 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: audit 2024-04-06T03:33:35.132717+0000 mon.a (mon.0) 1295 : audit 1 from='client.? 172.21.15.50:0/3731676675' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: audit 2024-04-06T03:33:35.231242+0000 mon.a (mon.0) 1296 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: audit 2024-04-06T03:33:35.232284+0000 mon.a (mon.0) 1297 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: audit 2024-04-06T03:33:35.239101+0000 mon.a (mon.0) 1298 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.483472+0000 osd.3 (osd.3) 299 : cluster 0 4.0 scrub starts 2024-04-06T03:33:35.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.485598+0000 osd.3 (osd.3) 300 : cluster 0 4.0 scrub ok 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.319389+0000 osd.7 (osd.7) 313 : cluster 0 2.1c scrub starts 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.322397+0000 osd.7 (osd.7) 314 : cluster 0 2.1c scrub ok 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.758967+0000 osd.2 (osd.2) 251 : cluster 0 5.b scrub starts 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:34.760310+0000 osd.2 (osd.2) 252 : cluster 0 5.b scrub ok 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.117206+0000 osd.0 (osd.0) 280 : cluster 0 5.5 scrub starts 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.118852+0000 osd.0 (osd.0) 281 : cluster 0 5.5 scrub ok 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: audit 2024-04-06T03:33:35.132717+0000 mon.a (mon.0) 1295 : audit 1 from='client.? 172.21.15.50:0/3731676675' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: audit 2024-04-06T03:33:35.231242+0000 mon.a (mon.0) 1296 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: audit 2024-04-06T03:33:35.232284+0000 mon.a (mon.0) 1297 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: audit 2024-04-06T03:33:35.239101+0000 mon.a (mon.0) 1298 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.483472+0000 osd.3 (osd.3) 299 : cluster 0 4.0 scrub starts 2024-04-06T03:33:35.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:35 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.485598+0000 osd.3 (osd.3) 300 : cluster 0 4.0 scrub ok 2024-04-06T03:33:35.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:35.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:36.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:36.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.319389+0000 osd.7 (osd.7) 313 : cluster 0 2.1c scrub starts 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.322397+0000 osd.7 (osd.7) 314 : cluster 0 2.1c scrub ok 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.758967+0000 osd.2 (osd.2) 251 : cluster 0 5.b scrub starts 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:34.760310+0000 osd.2 (osd.2) 252 : cluster 0 5.b scrub ok 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.117206+0000 osd.0 (osd.0) 280 : cluster 0 5.5 scrub starts 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.118852+0000 osd.0 (osd.0) 281 : cluster 0 5.5 scrub ok 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: audit 2024-04-06T03:33:35.132717+0000 mon.a (mon.0) 1295 : audit 1 from='client.? 172.21.15.50:0/3731676675' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: audit 2024-04-06T03:33:35.231242+0000 mon.a (mon.0) 1296 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:33:36.076 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: audit 2024-04-06T03:33:35.232284+0000 mon.a (mon.0) 1297 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:33:36.077 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: audit 2024-04-06T03:33:35.239101+0000 mon.a (mon.0) 1298 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:36.077 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.483472+0000 osd.3 (osd.3) 299 : cluster 0 4.0 scrub starts 2024-04-06T03:33:36.077 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:35 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.485598+0000 osd.3 (osd.3) 300 : cluster 0 4.0 scrub ok 2024-04-06T03:33:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:36.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:36.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:36.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:36.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:36.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:36.465 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:36.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:36.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:36.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:36.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:36.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:36.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:36.858 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:36.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.358522+0000 osd.7 (osd.7) 315 : cluster 0 3.e scrub starts 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.363005+0000 osd.7 (osd.7) 316 : cluster 0 3.e scrub ok 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.765208+0000 osd.2 (osd.2) 253 : cluster 0 5.8 scrub starts 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.767598+0000 osd.2 (osd.2) 254 : cluster 0 5.8 scrub ok 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: audit 2024-04-06T03:33:35.812520+0000 mon.a (mon.0) 1299 : audit 1 from='client.? 172.21.15.50:0/3731676675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:35.816863+0000 mon.a (mon.0) 1300 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.098591+0000 osd.0 (osd.0) 282 : cluster 0 2.1f scrub starts 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.100479+0000 osd.0 (osd.0) 283 : cluster 0 2.1f scrub ok 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.236096+0000 mgr.y (mgr.24368) 467 : cluster 0 pgmap v481: 148 pgs: 148 active+clean; 5.1 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 733 op/s 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.373014+0000 osd.1 (osd.1) 259 : cluster 0 2.d scrub starts 2024-04-06T03:33:36.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.374771+0000 osd.1 (osd.1) 260 : cluster 0 2.d scrub ok 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.735317+0000 osd.2 (osd.2) 255 : cluster 0 5.6 scrub starts 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[21467]: cluster 2024-04-06T03:33:36.737075+0000 osd.2 (osd.2) 256 : cluster 0 5.6 scrub ok 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.358522+0000 osd.7 (osd.7) 315 : cluster 0 3.e scrub starts 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.363005+0000 osd.7 (osd.7) 316 : cluster 0 3.e scrub ok 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.765208+0000 osd.2 (osd.2) 253 : cluster 0 5.8 scrub starts 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.767598+0000 osd.2 (osd.2) 254 : cluster 0 5.8 scrub ok 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: audit 2024-04-06T03:33:35.812520+0000 mon.a (mon.0) 1299 : audit 1 from='client.? 172.21.15.50:0/3731676675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:35.816863+0000 mon.a (mon.0) 1300 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.098591+0000 osd.0 (osd.0) 282 : cluster 0 2.1f scrub starts 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.100479+0000 osd.0 (osd.0) 283 : cluster 0 2.1f scrub ok 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.236096+0000 mgr.y (mgr.24368) 467 : cluster 0 pgmap v481: 148 pgs: 148 active+clean; 5.1 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 733 op/s 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.373014+0000 osd.1 (osd.1) 259 : cluster 0 2.d scrub starts 2024-04-06T03:33:36.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.374771+0000 osd.1 (osd.1) 260 : cluster 0 2.d scrub ok 2024-04-06T03:33:36.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.735317+0000 osd.2 (osd.2) 255 : cluster 0 5.6 scrub starts 2024-04-06T03:33:36.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:36 smithi050 bash[29221]: cluster 2024-04-06T03:33:36.737075+0000 osd.2 (osd.2) 256 : cluster 0 5.6 scrub ok 2024-04-06T03:33:36.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:36.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:37.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.358522+0000 osd.7 (osd.7) 315 : cluster 0 3.e scrub starts 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.363005+0000 osd.7 (osd.7) 316 : cluster 0 3.e scrub ok 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.765208+0000 osd.2 (osd.2) 253 : cluster 0 5.8 scrub starts 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.767598+0000 osd.2 (osd.2) 254 : cluster 0 5.8 scrub ok 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: audit 2024-04-06T03:33:35.812520+0000 mon.a (mon.0) 1299 : audit 1 from='client.? 172.21.15.50:0/3731676675' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:35.816863+0000 mon.a (mon.0) 1300 : cluster 0 osdmap e222: 8 total, 8 up, 8 in 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.098591+0000 osd.0 (osd.0) 282 : cluster 0 2.1f scrub starts 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.100479+0000 osd.0 (osd.0) 283 : cluster 0 2.1f scrub ok 2024-04-06T03:33:37.104 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.236096+0000 mgr.y (mgr.24368) 467 : cluster 0 pgmap v481: 148 pgs: 148 active+clean; 5.1 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 733 op/s 2024-04-06T03:33:37.105 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.373014+0000 osd.1 (osd.1) 259 : cluster 0 2.d scrub starts 2024-04-06T03:33:37.105 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.374771+0000 osd.1 (osd.1) 260 : cluster 0 2.d scrub ok 2024-04-06T03:33:37.105 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.735317+0000 osd.2 (osd.2) 255 : cluster 0 5.6 scrub starts 2024-04-06T03:33:37.105 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:36 smithi153 bash[23852]: cluster 2024-04-06T03:33:36.737075+0000 osd.2 (osd.2) 256 : cluster 0 5.6 scrub ok 2024-04-06T03:33:37.117 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:37.192 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:33:37.215 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:33:37.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:37.237 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:33:37.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:37.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:37.258 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:33:37.281 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:33:37.304 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:33:37.328 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:33:37.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:37.353 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:33:37.355 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:33:37.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:37.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:37.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:37.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:37.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:37.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-06T03:33:37.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:37.837 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:33:37.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:37.868 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.122539+0000 osd.0 (osd.0) 284 : cluster 0 3.17 scrub starts 2024-04-06T03:33:37.868 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.125023+0000 osd.0 (osd.0) 285 : cluster 0 3.17 scrub ok 2024-04-06T03:33:37.868 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: audit 2024-04-06T03:33:37.176678+0000 mon.a (mon.0) 1301 : audit 0 from='client.? 172.21.15.50:0/1748387850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:33:37.868 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.383810+0000 osd.1 (osd.1) 261 : cluster 0 6.4 scrub starts 2024-04-06T03:33:37.868 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.385284+0000 osd.1 (osd.1) 262 : cluster 0 6.4 scrub ok 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: audit 2024-04-06T03:33:37.755639+0000 mon.a (mon.0) 1302 : audit 1 from='client.? 172.21.15.50:0/467352730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.771686+0000 osd.2 (osd.2) 257 : cluster 0 2.1 scrub starts 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.772665+0000 osd.2 (osd.2) 258 : cluster 0 2.1 scrub ok 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.122539+0000 osd.0 (osd.0) 284 : cluster 0 3.17 scrub starts 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.125023+0000 osd.0 (osd.0) 285 : cluster 0 3.17 scrub ok 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: audit 2024-04-06T03:33:37.176678+0000 mon.a (mon.0) 1301 : audit 0 from='client.? 172.21.15.50:0/1748387850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.383810+0000 osd.1 (osd.1) 261 : cluster 0 6.4 scrub starts 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.385284+0000 osd.1 (osd.1) 262 : cluster 0 6.4 scrub ok 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: audit 2024-04-06T03:33:37.755639+0000 mon.a (mon.0) 1302 : audit 1 from='client.? 172.21.15.50:0/467352730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.771686+0000 osd.2 (osd.2) 257 : cluster 0 2.1 scrub starts 2024-04-06T03:33:37.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:37 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.772665+0000 osd.2 (osd.2) 258 : cluster 0 2.1 scrub ok 2024-04-06T03:33:37.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:37.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:37.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:38.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:38.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:38.129 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:38.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.122539+0000 osd.0 (osd.0) 284 : cluster 0 3.17 scrub starts 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.125023+0000 osd.0 (osd.0) 285 : cluster 0 3.17 scrub ok 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: audit 2024-04-06T03:33:37.176678+0000 mon.a (mon.0) 1301 : audit 0 from='client.? 172.21.15.50:0/1748387850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.383810+0000 osd.1 (osd.1) 261 : cluster 0 6.4 scrub starts 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.385284+0000 osd.1 (osd.1) 262 : cluster 0 6.4 scrub ok 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: audit 2024-04-06T03:33:37.755639+0000 mon.a (mon.0) 1302 : audit 1 from='client.? 172.21.15.50:0/467352730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.771686+0000 osd.2 (osd.2) 257 : cluster 0 2.1 scrub starts 2024-04-06T03:33:38.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:37 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.772665+0000 osd.2 (osd.2) 258 : cluster 0 2.1 scrub ok 2024-04-06T03:33:38.258 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:38.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:38.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:38.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:38.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:38.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:38.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:38.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:38.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:38.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:38.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:38.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:38.963 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-06T03:33:38.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:33:39.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:39.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.425099+0000 osd.4 (osd.4) 291 : cluster 0 6.17 scrub starts 2024-04-06T03:33:39.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.480540+0000 osd.4 (osd.4) 292 : cluster 0 6.17 scrub ok 2024-04-06T03:33:39.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.507580+0000 osd.3 (osd.3) 301 : cluster 0 6.1f scrub starts 2024-04-06T03:33:39.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.570976+0000 osd.3 (osd.3) 302 : cluster 0 6.1f scrub ok 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.831490+0000 mon.a (mon.0) 1303 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.831552+0000 mon.a (mon.0) 1304 : cluster 1 Cluster is now healthy 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: audit 2024-04-06T03:33:37.833467+0000 mon.a (mon.0) 1305 : audit 1 from='client.? 172.21.15.50:0/467352730' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:37.838288+0000 mon.a (mon.0) 1306 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:38.175956+0000 osd.0 (osd.0) 286 : cluster 0 6.10 scrub starts 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:38.193260+0000 osd.0 (osd.0) 287 : cluster 0 6.10 scrub ok 2024-04-06T03:33:39.019 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:38 smithi153 bash[23852]: cluster 2024-04-06T03:33:38.236786+0000 mgr.y (mgr.24368) 468 : cluster 0 pgmap v483: 148 pgs: 148 active+clean; 5.1 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 690 op/s 2024-04-06T03:33:39.034 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:39.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.425099+0000 osd.4 (osd.4) 291 : cluster 0 6.17 scrub starts 2024-04-06T03:33:39.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.480540+0000 osd.4 (osd.4) 292 : cluster 0 6.17 scrub ok 2024-04-06T03:33:39.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.507580+0000 osd.3 (osd.3) 301 : cluster 0 6.1f scrub starts 2024-04-06T03:33:39.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.570976+0000 osd.3 (osd.3) 302 : cluster 0 6.1f scrub ok 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.831490+0000 mon.a (mon.0) 1303 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.831552+0000 mon.a (mon.0) 1304 : cluster 1 Cluster is now healthy 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: audit 2024-04-06T03:33:37.833467+0000 mon.a (mon.0) 1305 : audit 1 from='client.? 172.21.15.50:0/467352730' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:37.838288+0000 mon.a (mon.0) 1306 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:38.175956+0000 osd.0 (osd.0) 286 : cluster 0 6.10 scrub starts 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:38.193260+0000 osd.0 (osd.0) 287 : cluster 0 6.10 scrub ok 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[21467]: cluster 2024-04-06T03:33:38.236786+0000 mgr.y (mgr.24368) 468 : cluster 0 pgmap v483: 148 pgs: 148 active+clean; 5.1 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 690 op/s 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.425099+0000 osd.4 (osd.4) 291 : cluster 0 6.17 scrub starts 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.480540+0000 osd.4 (osd.4) 292 : cluster 0 6.17 scrub ok 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.507580+0000 osd.3 (osd.3) 301 : cluster 0 6.1f scrub starts 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.570976+0000 osd.3 (osd.3) 302 : cluster 0 6.1f scrub ok 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.831490+0000 mon.a (mon.0) 1303 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.831552+0000 mon.a (mon.0) 1304 : cluster 1 Cluster is now healthy 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: audit 2024-04-06T03:33:37.833467+0000 mon.a (mon.0) 1305 : audit 1 from='client.? 172.21.15.50:0/467352730' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:33:39.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:37.838288+0000 mon.a (mon.0) 1306 : cluster 0 osdmap e223: 8 total, 8 up, 8 in 2024-04-06T03:33:39.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:38.175956+0000 osd.0 (osd.0) 286 : cluster 0 6.10 scrub starts 2024-04-06T03:33:39.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:38.193260+0000 osd.0 (osd.0) 287 : cluster 0 6.10 scrub ok 2024-04-06T03:33:39.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:38 smithi050 bash[29221]: cluster 2024-04-06T03:33:38.236786+0000 mgr.y (mgr.24368) 468 : cluster 0 pgmap v483: 148 pgs: 148 active+clean; 5.1 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 690 op/s 2024-04-06T03:33:39.086 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-04-06T03:33:39.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-04-06T03:33:39.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:39.163 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:39.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:39.307 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:39.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:39.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:39.486 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:33:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:33:39.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:39.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:39.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:39.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:39.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:39.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:33:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:33:39.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:39.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:39.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:39.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:39.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:40.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:40.087 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:40.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:40.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[21467]: cluster 2024-04-06T03:33:39.217946+0000 osd.0 (osd.0) 288 : cluster 0 6.18 scrub starts 2024-04-06T03:33:40.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[21467]: cluster 2024-04-06T03:33:39.245342+0000 osd.0 (osd.0) 289 : cluster 0 6.18 scrub ok 2024-04-06T03:33:40.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[21467]: audit 2024-04-06T03:33:39.596007+0000 mgr.y (mgr.24368) 469 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:40.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[29221]: cluster 2024-04-06T03:33:39.217946+0000 osd.0 (osd.0) 288 : cluster 0 6.18 scrub starts 2024-04-06T03:33:40.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[29221]: cluster 2024-04-06T03:33:39.245342+0000 osd.0 (osd.0) 289 : cluster 0 6.18 scrub ok 2024-04-06T03:33:40.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:39 smithi050 bash[29221]: audit 2024-04-06T03:33:39.596007+0000 mgr.y (mgr.24368) 469 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:40.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:40.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:39 smithi153 bash[23852]: cluster 2024-04-06T03:33:39.217946+0000 osd.0 (osd.0) 288 : cluster 0 6.18 scrub starts 2024-04-06T03:33:40.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:39 smithi153 bash[23852]: cluster 2024-04-06T03:33:39.245342+0000 osd.0 (osd.0) 289 : cluster 0 6.18 scrub ok 2024-04-06T03:33:40.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:39 smithi153 bash[23852]: audit 2024-04-06T03:33:39.596007+0000 mgr.y (mgr.24368) 469 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:40.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:40.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:40.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:40.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:40.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:40.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:40.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:40.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:40.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:40.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:40.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:40.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:40.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:40.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:40.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:40 smithi050 bash[29221]: cluster 2024-04-06T03:33:39.513766+0000 osd.3 (osd.3) 303 : cluster 0 2.1e scrub starts 2024-04-06T03:33:40.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:40 smithi050 bash[29221]: cluster 2024-04-06T03:33:39.516212+0000 osd.3 (osd.3) 304 : cluster 0 2.1e scrub ok 2024-04-06T03:33:40.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:40 smithi050 bash[29221]: cluster 2024-04-06T03:33:40.237709+0000 mgr.y (mgr.24368) 470 : cluster 0 pgmap v484: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 416 op/s 2024-04-06T03:33:40.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:40 smithi050 bash[21467]: cluster 2024-04-06T03:33:39.513766+0000 osd.3 (osd.3) 303 : cluster 0 2.1e scrub starts 2024-04-06T03:33:40.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:40 smithi050 bash[21467]: cluster 2024-04-06T03:33:39.516212+0000 osd.3 (osd.3) 304 : cluster 0 2.1e scrub ok 2024-04-06T03:33:40.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:40 smithi050 bash[21467]: cluster 2024-04-06T03:33:40.237709+0000 mgr.y (mgr.24368) 470 : cluster 0 pgmap v484: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 416 op/s 2024-04-06T03:33:40.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:41.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:41.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:40 smithi153 bash[23852]: cluster 2024-04-06T03:33:39.513766+0000 osd.3 (osd.3) 303 : cluster 0 2.1e scrub starts 2024-04-06T03:33:41.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:40 smithi153 bash[23852]: cluster 2024-04-06T03:33:39.516212+0000 osd.3 (osd.3) 304 : cluster 0 2.1e scrub ok 2024-04-06T03:33:41.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:40 smithi153 bash[23852]: cluster 2024-04-06T03:33:40.237709+0000 mgr.y (mgr.24368) 470 : cluster 0 pgmap v484: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 416 op/s 2024-04-06T03:33:41.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:41.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:41.257 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:41.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:41.387 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:41.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:41.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:41.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:41.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:41.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:41.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:41.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:41.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:41.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:42.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:42.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:41 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.239476+0000 osd.0 (osd.0) 290 : cluster 0 5.3 scrub starts 2024-04-06T03:33:42.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:41 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.242493+0000 osd.0 (osd.0) 291 : cluster 0 5.3 scrub ok 2024-04-06T03:33:42.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:41 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.393246+0000 osd.1 (osd.1) 263 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:33:42.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:41 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.412733+0000 osd.1 (osd.1) 264 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:33:42.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:41 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.787159+0000 osd.2 (osd.2) 259 : cluster 0 5.a scrub starts 2024-04-06T03:33:42.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:41 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.788279+0000 osd.2 (osd.2) 260 : cluster 0 5.a scrub ok 2024-04-06T03:33:42.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:42.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.239476+0000 osd.0 (osd.0) 290 : cluster 0 5.3 scrub starts 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.242493+0000 osd.0 (osd.0) 291 : cluster 0 5.3 scrub ok 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.393246+0000 osd.1 (osd.1) 263 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.412733+0000 osd.1 (osd.1) 264 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.787159+0000 osd.2 (osd.2) 259 : cluster 0 5.a scrub starts 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.788279+0000 osd.2 (osd.2) 260 : cluster 0 5.a scrub ok 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.239476+0000 osd.0 (osd.0) 290 : cluster 0 5.3 scrub starts 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.242493+0000 osd.0 (osd.0) 291 : cluster 0 5.3 scrub ok 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.393246+0000 osd.1 (osd.1) 263 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.412733+0000 osd.1 (osd.1) 264 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.787159+0000 osd.2 (osd.2) 259 : cluster 0 5.a scrub starts 2024-04-06T03:33:42.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:41 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.788279+0000 osd.2 (osd.2) 260 : cluster 0 5.a scrub ok 2024-04-06T03:33:42.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:33:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:42.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:33:42.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:42.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:42.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:42.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:42.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:42.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:42.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:42.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:42.688 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:42.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:42.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:42.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.279539+0000 osd.7 (osd.7) 317 : cluster 0 2.0 scrub starts 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.281904+0000 osd.7 (osd.7) 318 : cluster 0 2.0 scrub ok 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.622616+0000 osd.0 (osd.0) 292 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.622687+0000 osd.0 (osd.0) 293 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[21467]: cluster 2024-04-06T03:33:41.886978+0000 mon.a (mon.0) 1307 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[21467]: cluster 2024-04-06T03:33:42.238472+0000 mgr.y (mgr.24368) 471 : cluster 0 pgmap v486: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 12 op/s 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.279539+0000 osd.7 (osd.7) 317 : cluster 0 2.0 scrub starts 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.281904+0000 osd.7 (osd.7) 318 : cluster 0 2.0 scrub ok 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.622616+0000 osd.0 (osd.0) 292 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.622687+0000 osd.0 (osd.0) 293 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[29221]: cluster 2024-04-06T03:33:41.886978+0000 mon.a (mon.0) 1307 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-04-06T03:33:42.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:42 smithi050 bash[29221]: cluster 2024-04-06T03:33:42.238472+0000 mgr.y (mgr.24368) 471 : cluster 0 pgmap v486: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 12 op/s 2024-04-06T03:33:42.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:42.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:43.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:43.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:43.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:43.186 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:42 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.279539+0000 osd.7 (osd.7) 317 : cluster 0 2.0 scrub starts 2024-04-06T03:33:43.186 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:42 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.281904+0000 osd.7 (osd.7) 318 : cluster 0 2.0 scrub ok 2024-04-06T03:33:43.186 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:42 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.622616+0000 osd.0 (osd.0) 292 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:43.186 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:42 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.622687+0000 osd.0 (osd.0) 293 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:43.186 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:42 smithi153 bash[23852]: cluster 2024-04-06T03:33:41.886978+0000 mon.a (mon.0) 1307 : cluster 0 osdmap e224: 8 total, 8 up, 8 in 2024-04-06T03:33:43.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:42 smithi153 bash[23852]: cluster 2024-04-06T03:33:42.238472+0000 mgr.y (mgr.24368) 471 : cluster 0 pgmap v486: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 796 B/s rd, 0 B/s wr, 12 op/s 2024-04-06T03:33:43.200 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:43.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:43.331 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:43.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:43.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:43.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:43.593 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:43.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:43.724 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:43.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:43.774 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:43.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:43.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:43.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:43.965 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:43 smithi050 bash[21467]: cluster 2024-04-06T03:33:42.514125+0000 osd.3 (osd.3) 305 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:33:43.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:43 smithi050 bash[21467]: cluster 2024-04-06T03:33:42.516362+0000 osd.3 (osd.3) 306 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:33:43.966 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:43 smithi050 bash[29221]: cluster 2024-04-06T03:33:42.514125+0000 osd.3 (osd.3) 305 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:33:43.966 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:43 smithi050 bash[29221]: cluster 2024-04-06T03:33:42.516362+0000 osd.3 (osd.3) 306 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:33:43.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:43.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:44.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:44.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:44.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:44.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:44.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:43 smithi153 bash[23852]: cluster 2024-04-06T03:33:42.514125+0000 osd.3 (osd.3) 305 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:33:44.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:43 smithi153 bash[23852]: cluster 2024-04-06T03:33:42.516362+0000 osd.3 (osd.3) 306 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:33:44.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:44.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:44.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:44.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:44.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:44.638 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:44.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:44.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:44.791 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:44.792 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:44.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:44.898 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:44.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:45.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:45.012 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:44 smithi050 bash[21467]: cluster 2024-04-06T03:33:44.239227+0000 mgr.y (mgr.24368) 472 : cluster 0 pgmap v487: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 9 op/s 2024-04-06T03:33:45.012 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:44 smithi050 bash[29221]: cluster 2024-04-06T03:33:44.239227+0000 mgr.y (mgr.24368) 472 : cluster 0 pgmap v487: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 9 op/s 2024-04-06T03:33:45.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:45.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:45.142 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:44 smithi153 bash[23852]: cluster 2024-04-06T03:33:44.239227+0000 mgr.y (mgr.24368) 472 : cluster 0 pgmap v487: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 9 op/s 2024-04-06T03:33:45.159 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:45.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:45.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:45.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:45.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:45.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:45.551 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:45.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:45.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:45.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:45.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:45.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:45.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:45.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:46.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[21467]: cluster 2024-04-06T03:33:44.220705+0000 osd.7 (osd.7) 319 : cluster 0 3.15 scrub starts 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[21467]: cluster 2024-04-06T03:33:44.226322+0000 osd.7 (osd.7) 320 : cluster 0 3.15 scrub ok 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[21467]: cluster 2024-04-06T03:33:45.379140+0000 osd.4 (osd.4) 293 : cluster 0 5.11 scrub starts 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[21467]: cluster 2024-04-06T03:33:45.381422+0000 osd.4 (osd.4) 294 : cluster 0 5.11 scrub ok 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[29221]: cluster 2024-04-06T03:33:44.220705+0000 osd.7 (osd.7) 319 : cluster 0 3.15 scrub starts 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[29221]: cluster 2024-04-06T03:33:44.226322+0000 osd.7 (osd.7) 320 : cluster 0 3.15 scrub ok 2024-04-06T03:33:46.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[29221]: cluster 2024-04-06T03:33:45.379140+0000 osd.4 (osd.4) 293 : cluster 0 5.11 scrub starts 2024-04-06T03:33:46.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:45 smithi050 bash[29221]: cluster 2024-04-06T03:33:45.381422+0000 osd.4 (osd.4) 294 : cluster 0 5.11 scrub ok 2024-04-06T03:33:46.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:46.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:46.191 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:45 smithi153 bash[23852]: cluster 2024-04-06T03:33:44.220705+0000 osd.7 (osd.7) 319 : cluster 0 3.15 scrub starts 2024-04-06T03:33:46.191 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:45 smithi153 bash[23852]: cluster 2024-04-06T03:33:44.226322+0000 osd.7 (osd.7) 320 : cluster 0 3.15 scrub ok 2024-04-06T03:33:46.191 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:45 smithi153 bash[23852]: cluster 2024-04-06T03:33:45.379140+0000 osd.4 (osd.4) 293 : cluster 0 5.11 scrub starts 2024-04-06T03:33:46.191 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:45 smithi153 bash[23852]: cluster 2024-04-06T03:33:45.381422+0000 osd.4 (osd.4) 294 : cluster 0 5.11 scrub ok 2024-04-06T03:33:46.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:46.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:46.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:46.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:46.466 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:46.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:46.595 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:46.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:46.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:46.824 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:46.825 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:33:46.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:46.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:46.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:46.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:46.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:47.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:47.112 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:47.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:47.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[21467]: cluster 2024-04-06T03:33:46.159876+0000 osd.7 (osd.7) 321 : cluster 0 6.1c scrub starts 2024-04-06T03:33:47.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[21467]: cluster 2024-04-06T03:33:46.163333+0000 osd.7 (osd.7) 322 : cluster 0 6.1c scrub ok 2024-04-06T03:33:47.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[21467]: cluster 2024-04-06T03:33:46.240541+0000 mgr.y (mgr.24368) 473 : cluster 0 pgmap v488: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T03:33:47.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[29221]: cluster 2024-04-06T03:33:46.159876+0000 osd.7 (osd.7) 321 : cluster 0 6.1c scrub starts 2024-04-06T03:33:47.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[29221]: cluster 2024-04-06T03:33:46.163333+0000 osd.7 (osd.7) 322 : cluster 0 6.1c scrub ok 2024-04-06T03:33:47.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[29221]: cluster 2024-04-06T03:33:46.240541+0000 mgr.y (mgr.24368) 473 : cluster 0 pgmap v488: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T03:33:47.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:47.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:46 smithi153 bash[23852]: cluster 2024-04-06T03:33:46.159876+0000 osd.7 (osd.7) 321 : cluster 0 6.1c scrub starts 2024-04-06T03:33:47.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:46 smithi153 bash[23852]: cluster 2024-04-06T03:33:46.163333+0000 osd.7 (osd.7) 322 : cluster 0 6.1c scrub ok 2024-04-06T03:33:47.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:46 smithi153 bash[23852]: cluster 2024-04-06T03:33:46.240541+0000 mgr.y (mgr.24368) 473 : cluster 0 pgmap v488: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-06T03:33:47.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:47.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:47.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:47.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:47.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:47.634 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:47.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:47.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:47.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:47.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:47.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:48.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:47 smithi153 bash[23852]: cluster 2024-04-06T03:33:46.538862+0000 osd.3 (osd.3) 307 : cluster 0 5.16 scrub starts 2024-04-06T03:33:48.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:47 smithi153 bash[23852]: cluster 2024-04-06T03:33:46.541218+0000 osd.3 (osd.3) 308 : cluster 0 5.16 scrub ok 2024-04-06T03:33:48.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:48.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:48.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:48.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[21467]: cluster 2024-04-06T03:33:46.538862+0000 osd.3 (osd.3) 307 : cluster 0 5.16 scrub starts 2024-04-06T03:33:48.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[21467]: cluster 2024-04-06T03:33:46.541218+0000 osd.3 (osd.3) 308 : cluster 0 5.16 scrub ok 2024-04-06T03:33:48.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[29221]: cluster 2024-04-06T03:33:46.538862+0000 osd.3 (osd.3) 307 : cluster 0 5.16 scrub starts 2024-04-06T03:33:48.169 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:47 smithi050 bash[29221]: cluster 2024-04-06T03:33:46.541218+0000 osd.3 (osd.3) 308 : cluster 0 5.16 scrub ok 2024-04-06T03:33:48.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.282 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:48.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:48.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:48.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:48.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:48.544 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:48.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:48.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:48.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:48.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:48.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:48.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:49.041 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.067 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:49.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:48 smithi050 bash[21467]: cluster 2024-04-06T03:33:48.095083+0000 osd.7 (osd.7) 323 : cluster 0 2.1d scrub starts 2024-04-06T03:33:49.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:48 smithi050 bash[21467]: cluster 2024-04-06T03:33:48.097587+0000 osd.7 (osd.7) 324 : cluster 0 2.1d scrub ok 2024-04-06T03:33:49.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:48 smithi050 bash[21467]: cluster 2024-04-06T03:33:48.241185+0000 mgr.y (mgr.24368) 474 : cluster 0 pgmap v489: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 8 op/s 2024-04-06T03:33:49.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:48 smithi050 bash[29221]: cluster 2024-04-06T03:33:48.095083+0000 osd.7 (osd.7) 323 : cluster 0 2.1d scrub starts 2024-04-06T03:33:49.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:48 smithi050 bash[29221]: cluster 2024-04-06T03:33:48.097587+0000 osd.7 (osd.7) 324 : cluster 0 2.1d scrub ok 2024-04-06T03:33:49.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:48 smithi050 bash[29221]: cluster 2024-04-06T03:33:48.241185+0000 mgr.y (mgr.24368) 474 : cluster 0 pgmap v489: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 8 op/s 2024-04-06T03:33:49.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.181 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:48 smithi153 bash[23852]: cluster 2024-04-06T03:33:48.095083+0000 osd.7 (osd.7) 323 : cluster 0 2.1d scrub starts 2024-04-06T03:33:49.181 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:48 smithi153 bash[23852]: cluster 2024-04-06T03:33:48.097587+0000 osd.7 (osd.7) 324 : cluster 0 2.1d scrub ok 2024-04-06T03:33:49.181 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:48 smithi153 bash[23852]: cluster 2024-04-06T03:33:48.241185+0000 mgr.y (mgr.24368) 474 : cluster 0 pgmap v489: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 8 op/s 2024-04-06T03:33:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:49.301 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:49.328 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:49.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:49.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:49.559 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:33:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:33:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:33:49.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:49.583 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:49.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.696 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:33:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:33:49.712 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:49.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:49.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:49.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:49.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:49.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:50.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:50.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:50.227 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:50.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:50.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:50 smithi050 bash[21467]: audit 2024-04-06T03:33:49.256386+0000 mon.a (mon.0) 1308 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:50.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:50 smithi050 bash[21467]: audit 2024-04-06T03:33:49.258804+0000 mon.a (mon.0) 1309 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:50.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:50 smithi050 bash[21467]: audit 2024-04-06T03:33:49.604325+0000 mgr.y (mgr.24368) 475 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:50.429 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:50 smithi050 bash[29221]: audit 2024-04-06T03:33:49.256386+0000 mon.a (mon.0) 1308 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:50.429 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:50 smithi050 bash[29221]: audit 2024-04-06T03:33:49.258804+0000 mon.a (mon.0) 1309 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:50.429 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:50 smithi050 bash[29221]: audit 2024-04-06T03:33:49.604325+0000 mgr.y (mgr.24368) 475 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:50.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:50.486 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:50 smithi153 bash[23852]: audit 2024-04-06T03:33:49.256386+0000 mon.a (mon.0) 1308 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:33:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:50 smithi153 bash[23852]: audit 2024-04-06T03:33:49.258804+0000 mon.a (mon.0) 1309 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:33:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:50 smithi153 bash[23852]: audit 2024-04-06T03:33:49.604325+0000 mgr.y (mgr.24368) 475 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:33:50.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:50.616 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:50.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:50.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:50.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:50.878 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:50.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:50.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:51.007 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:51.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:51.139 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:51.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:51.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:51.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:51.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:51.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:49.506342+0000 osd.3 (osd.3) 309 : cluster 0 3.7 scrub starts 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:49.512380+0000 osd.3 (osd.3) 310 : cluster 0 3.7 scrub ok 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:50.242246+0000 mgr.y (mgr.24368) 476 : cluster 0 pgmap v490: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:50.479913+0000 osd.3 (osd.3) 311 : cluster 0 3.2 deep-scrub starts 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:50.491193+0000 osd.3 (osd.3) 312 : cluster 0 3.2 deep-scrub ok 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.055115+0000 osd.7 (osd.7) 325 : cluster 0 3.d scrub starts 2024-04-06T03:33:51.515 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:51 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.057488+0000 osd.7 (osd.7) 326 : cluster 0 3.d scrub ok 2024-04-06T03:33:51.528 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:51.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:49.506342+0000 osd.3 (osd.3) 309 : cluster 0 3.7 scrub starts 2024-04-06T03:33:51.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:49.512380+0000 osd.3 (osd.3) 310 : cluster 0 3.7 scrub ok 2024-04-06T03:33:51.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:50.242246+0000 mgr.y (mgr.24368) 476 : cluster 0 pgmap v490: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:33:51.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:50.479913+0000 osd.3 (osd.3) 311 : cluster 0 3.2 deep-scrub starts 2024-04-06T03:33:51.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:50.491193+0000 osd.3 (osd.3) 312 : cluster 0 3.2 deep-scrub ok 2024-04-06T03:33:51.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.055115+0000 osd.7 (osd.7) 325 : cluster 0 3.d scrub starts 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.057488+0000 osd.7 (osd.7) 326 : cluster 0 3.d scrub ok 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:49.506342+0000 osd.3 (osd.3) 309 : cluster 0 3.7 scrub starts 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:49.512380+0000 osd.3 (osd.3) 310 : cluster 0 3.7 scrub ok 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:50.242246+0000 mgr.y (mgr.24368) 476 : cluster 0 pgmap v490: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:50.479913+0000 osd.3 (osd.3) 311 : cluster 0 3.2 deep-scrub starts 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:50.491193+0000 osd.3 (osd.3) 312 : cluster 0 3.2 deep-scrub ok 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.055115+0000 osd.7 (osd.7) 325 : cluster 0 3.d scrub starts 2024-04-06T03:33:51.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:51 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.057488+0000 osd.7 (osd.7) 326 : cluster 0 3.d scrub ok 2024-04-06T03:33:51.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:51.658 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:51.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:51.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:51.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:51.918 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:51.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:52.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:52.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:52.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:52.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:52.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:52.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:52 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.160697+0000 osd.0 (osd.0) 294 : cluster 0 3.1b deep-scrub starts 2024-04-06T03:33:52.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:52 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.165051+0000 osd.0 (osd.0) 295 : cluster 0 3.1b deep-scrub ok 2024-04-06T03:33:52.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:52 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.359881+0000 osd.4 (osd.4) 295 : cluster 0 5.c scrub starts 2024-04-06T03:33:52.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:52 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.362239+0000 osd.4 (osd.4) 296 : cluster 0 5.c scrub ok 2024-04-06T03:33:52.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:52 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.484545+0000 osd.3 (osd.3) 313 : cluster 0 3.b scrub starts 2024-04-06T03:33:52.426 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:52 smithi153 bash[23852]: cluster 2024-04-06T03:33:51.487041+0000 osd.3 (osd.3) 314 : cluster 0 3.b scrub ok 2024-04-06T03:33:52.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:52.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:52.573 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:52.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:52.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.160697+0000 osd.0 (osd.0) 294 : cluster 0 3.1b deep-scrub starts 2024-04-06T03:33:52.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.165051+0000 osd.0 (osd.0) 295 : cluster 0 3.1b deep-scrub ok 2024-04-06T03:33:52.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.359881+0000 osd.4 (osd.4) 295 : cluster 0 5.c scrub starts 2024-04-06T03:33:52.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.362239+0000 osd.4 (osd.4) 296 : cluster 0 5.c scrub ok 2024-04-06T03:33:52.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.484545+0000 osd.3 (osd.3) 313 : cluster 0 3.b scrub starts 2024-04-06T03:33:52.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[29221]: cluster 2024-04-06T03:33:51.487041+0000 osd.3 (osd.3) 314 : cluster 0 3.b scrub ok 2024-04-06T03:33:52.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.160697+0000 osd.0 (osd.0) 294 : cluster 0 3.1b deep-scrub starts 2024-04-06T03:33:52.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.165051+0000 osd.0 (osd.0) 295 : cluster 0 3.1b deep-scrub ok 2024-04-06T03:33:52.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.359881+0000 osd.4 (osd.4) 295 : cluster 0 5.c scrub starts 2024-04-06T03:33:52.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.362239+0000 osd.4 (osd.4) 296 : cluster 0 5.c scrub ok 2024-04-06T03:33:52.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.484545+0000 osd.3 (osd.3) 313 : cluster 0 3.b scrub starts 2024-04-06T03:33:52.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:52 smithi050 bash[21467]: cluster 2024-04-06T03:33:51.487041+0000 osd.3 (osd.3) 314 : cluster 0 3.b scrub ok 2024-04-06T03:33:52.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:52.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:52.831 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:52.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:52.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:52.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:53.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:53.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:53.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:53.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:53.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:53.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:53.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:53.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:53 smithi153 bash[23852]: cluster 2024-04-06T03:33:52.243294+0000 mgr.y (mgr.24368) 477 : cluster 0 pgmap v491: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-06T03:33:53.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:53 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.045910+0000 osd.7 (osd.7) 327 : cluster 0 2.8 scrub starts 2024-04-06T03:33:53.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:53 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.048547+0000 osd.7 (osd.7) 328 : cluster 0 2.8 scrub ok 2024-04-06T03:33:53.485 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:53.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:53.615 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:53.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:53.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:53 smithi050 bash[21467]: cluster 2024-04-06T03:33:52.243294+0000 mgr.y (mgr.24368) 477 : cluster 0 pgmap v491: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-06T03:33:53.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:53 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.045910+0000 osd.7 (osd.7) 327 : cluster 0 2.8 scrub starts 2024-04-06T03:33:53.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:53 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.048547+0000 osd.7 (osd.7) 328 : cluster 0 2.8 scrub ok 2024-04-06T03:33:53.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:53 smithi050 bash[29221]: cluster 2024-04-06T03:33:52.243294+0000 mgr.y (mgr.24368) 477 : cluster 0 pgmap v491: 148 pgs: 148 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-04-06T03:33:53.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:53 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.045910+0000 osd.7 (osd.7) 327 : cluster 0 2.8 scrub starts 2024-04-06T03:33:53.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:53 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.048547+0000 osd.7 (osd.7) 328 : cluster 0 2.8 scrub ok 2024-04-06T03:33:53.746 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:53.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:53.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:53.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:53.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:54.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:54.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:54.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:54.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:54.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:54.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:54.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:54.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:54.521 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:54.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:54.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.211779+0000 osd.0 (osd.0) 296 : cluster 0 5.2 scrub starts 2024-04-06T03:33:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.214378+0000 osd.0 (osd.0) 297 : cluster 0 5.2 scrub ok 2024-04-06T03:33:54.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.324377+0000 osd.4 (osd.4) 297 : cluster 0 3.14 scrub starts 2024-04-06T03:33:54.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.326881+0000 osd.4 (osd.4) 298 : cluster 0 3.14 scrub ok 2024-04-06T03:33:54.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.509128+0000 osd.3 (osd.3) 315 : cluster 0 3.8 scrub starts 2024-04-06T03:33:54.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[21467]: cluster 2024-04-06T03:33:53.514473+0000 osd.3 (osd.3) 316 : cluster 0 3.8 scrub ok 2024-04-06T03:33:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.211779+0000 osd.0 (osd.0) 296 : cluster 0 5.2 scrub starts 2024-04-06T03:33:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.214378+0000 osd.0 (osd.0) 297 : cluster 0 5.2 scrub ok 2024-04-06T03:33:54.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.324377+0000 osd.4 (osd.4) 297 : cluster 0 3.14 scrub starts 2024-04-06T03:33:54.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.326881+0000 osd.4 (osd.4) 298 : cluster 0 3.14 scrub ok 2024-04-06T03:33:54.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.509128+0000 osd.3 (osd.3) 315 : cluster 0 3.8 scrub starts 2024-04-06T03:33:54.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:54 smithi050 bash[29221]: cluster 2024-04-06T03:33:53.514473+0000 osd.3 (osd.3) 316 : cluster 0 3.8 scrub ok 2024-04-06T03:33:54.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:54.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:54.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:54.909 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:54.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:55.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:55.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:54 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.211779+0000 osd.0 (osd.0) 296 : cluster 0 5.2 scrub starts 2024-04-06T03:33:55.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:54 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.214378+0000 osd.0 (osd.0) 297 : cluster 0 5.2 scrub ok 2024-04-06T03:33:55.023 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:54 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.324377+0000 osd.4 (osd.4) 297 : cluster 0 3.14 scrub starts 2024-04-06T03:33:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:54 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.326881+0000 osd.4 (osd.4) 298 : cluster 0 3.14 scrub ok 2024-04-06T03:33:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:54 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.509128+0000 osd.3 (osd.3) 315 : cluster 0 3.8 scrub starts 2024-04-06T03:33:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:54 smithi153 bash[23852]: cluster 2024-04-06T03:33:53.514473+0000 osd.3 (osd.3) 316 : cluster 0 3.8 scrub ok 2024-04-06T03:33:55.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:55.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:55.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:55.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:55.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:55.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:55.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:55.428 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:55.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:55.559 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:55.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:55.691 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:55.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:54.243950+0000 mgr.y (mgr.24368) 478 : cluster 0 pgmap v492: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:33:55.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:54.302913+0000 osd.4 (osd.4) 299 : cluster 0 5.18 scrub starts 2024-04-06T03:33:55.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:54.305135+0000 osd.4 (osd.4) 300 : cluster 0 5.18 scrub ok 2024-04-06T03:33:55.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:54.589590+0000 osd.5 (osd.5) 133 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:55.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:54.589736+0000 osd.5 (osd.5) 134 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:55.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:55.091869+0000 osd.7 (osd.7) 329 : cluster 0 6.15 scrub starts 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[21467]: cluster 2024-04-06T03:33:55.107426+0000 osd.7 (osd.7) 330 : cluster 0 6.15 scrub ok 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:54.243950+0000 mgr.y (mgr.24368) 478 : cluster 0 pgmap v492: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:54.302913+0000 osd.4 (osd.4) 299 : cluster 0 5.18 scrub starts 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:54.305135+0000 osd.4 (osd.4) 300 : cluster 0 5.18 scrub ok 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:54.589590+0000 osd.5 (osd.5) 133 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:54.589736+0000 osd.5 (osd.5) 134 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:55.091869+0000 osd.7 (osd.7) 329 : cluster 0 6.15 scrub starts 2024-04-06T03:33:55.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:55 smithi050 bash[29221]: cluster 2024-04-06T03:33:55.107426+0000 osd.7 (osd.7) 330 : cluster 0 6.15 scrub ok 2024-04-06T03:33:55.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:54.243950+0000 mgr.y (mgr.24368) 478 : cluster 0 pgmap v492: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:33:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:54.302913+0000 osd.4 (osd.4) 299 : cluster 0 5.18 scrub starts 2024-04-06T03:33:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:54.305135+0000 osd.4 (osd.4) 300 : cluster 0 5.18 scrub ok 2024-04-06T03:33:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:54.589590+0000 osd.5 (osd.5) 133 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:54.589736+0000 osd.5 (osd.5) 134 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:55.091869+0000 osd.7 (osd.7) 329 : cluster 0 6.15 scrub starts 2024-04-06T03:33:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:55 smithi153 bash[23852]: cluster 2024-04-06T03:33:55.107426+0000 osd.7 (osd.7) 330 : cluster 0 6.15 scrub ok 2024-04-06T03:33:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:55.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:55.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:56.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:56.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:56.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:56.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:56.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:56.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:56.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.601 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:56.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.715 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:56 smithi153 bash[23852]: cluster 2024-04-06T03:33:55.301151+0000 osd.4 (osd.4) 301 : cluster 0 3.1c scrub starts 2024-04-06T03:33:56.715 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:56 smithi153 bash[23852]: cluster 2024-04-06T03:33:55.303971+0000 osd.4 (osd.4) 302 : cluster 0 3.1c scrub ok 2024-04-06T03:33:56.715 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:56 smithi153 bash[23852]: cluster 2024-04-06T03:33:55.570374+0000 mon.a (mon.0) 1310 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-04-06T03:33:56.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:56.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:56 smithi050 bash[21467]: cluster 2024-04-06T03:33:55.301151+0000 osd.4 (osd.4) 301 : cluster 0 3.1c scrub starts 2024-04-06T03:33:56.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:56 smithi050 bash[21467]: cluster 2024-04-06T03:33:55.303971+0000 osd.4 (osd.4) 302 : cluster 0 3.1c scrub ok 2024-04-06T03:33:56.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:56 smithi050 bash[21467]: cluster 2024-04-06T03:33:55.570374+0000 mon.a (mon.0) 1310 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-04-06T03:33:56.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:56 smithi050 bash[29221]: cluster 2024-04-06T03:33:55.301151+0000 osd.4 (osd.4) 301 : cluster 0 3.1c scrub starts 2024-04-06T03:33:56.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:56 smithi050 bash[29221]: cluster 2024-04-06T03:33:55.303971+0000 osd.4 (osd.4) 302 : cluster 0 3.1c scrub ok 2024-04-06T03:33:56.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:56 smithi050 bash[29221]: cluster 2024-04-06T03:33:55.570374+0000 mon.a (mon.0) 1310 : cluster 0 osdmap e225: 8 total, 8 up, 8 in 2024-04-06T03:33:56.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:56.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:56.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:56.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:57.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:57.120 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:57.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:57.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:57.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:57.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:57.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:57.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:57.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:57.638 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:33:57.638 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:33:57.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:33:57.638 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:33:57.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.638 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5604 8.a 8:5f202c4e:::benchmark_data_smithi050_86143_object5603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.647625+0000", 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.99371678, 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55522835500000001, 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5604 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647625+0000", 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647625+0000", 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647627+0000", 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6050000000000001e-06 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647650+0000", 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2572999999999999e-05 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647661+0000", 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.097e-05 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647668+0000", 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.824e-06 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647761+0000", 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3063e-05 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647903+0000", 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014144900000000001 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648042+0000", 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013927199999999999 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648856+0000", 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081367899999999999 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.649403+0000", 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054749300000000002 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202325+0000", 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55292173600000005 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202731+0000", 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040563799999999999 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.202854+0000", 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123053 2024-04-06T03:33:57.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5608 8.1 8:82e7b879:::benchmark_data_smithi050_86143_object5607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.648850+0000", 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.99249254599999, 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55644181199999998, 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5608 2024-04-06T03:33:57.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648850+0000", 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648850+0000", 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648851+0000", 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7199999999999997e-07 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648873+0000", 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2717e-05 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648882+0000", 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0620000000000007e-06 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648888+0000", 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6579999999999999e-06 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648938+0000", 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0096000000000003e-05 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.649051+0000", 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011275799999999999 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.649189+0000", 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137758 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.650471+0000", 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001282748 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.651187+0000", 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071593000000000004 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.204780+0000", 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55359255900000004 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.205151+0000", 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000370582 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.205291+0000", 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140972 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5620 8.a 8:56cec1de:::benchmark_data_smithi050_86143_object5619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.652575+0000", 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.98876769399999, 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55106059100000004, 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5620 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652575+0000", 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652575+0000", 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652576+0000", 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.238e-06 2024-04-06T03:33:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652596+0000", 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9755999999999999e-05 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652606+0000", 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414000000000001e-05 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652614+0000", 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6420000000000007e-06 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652669+0000", 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5590000000000001e-05 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652781+0000", 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112327 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652920+0000", 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138928 2024-04-06T03:33:57.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.653469+0000", 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054899099999999998 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.653771+0000", 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030111800000000001 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203003+0000", 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54923284000000006 2024-04-06T03:33:57.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203525+0000", 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052203700000000004 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203635+0000", 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010971 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5621 8.1 8:89ed0621:::benchmark_data_smithi050_86143_object5620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.656 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.652769+0000", 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.98857322699999, 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55405263100000002, 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5621 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652769+0000", 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652769+0000", 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652770+0000", 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1100000000000005e-07 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652791+0000", 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0774000000000001e-05 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652800+0000", 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5100000000000004e-06 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652807+0000", 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.126e-06 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652850+0000", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3183000000000003e-05 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.652952+0000", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010125599999999999 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.653082+0000", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013049700000000001 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.653598+0000", 2024-04-06T03:33:57.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051616499999999996 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.653942+0000", 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034353900000000002 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.205419+0000", 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55147670100000001 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.205486+0000", 2024-04-06T03:33:57.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7644999999999994e-05 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.206822+0000", 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001335424 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5626 8.1 8:831a2bd0:::benchmark_data_smithi050_86143_object5625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.658965+0000", 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.98237729100001, 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54879892299999999, 2024-04-06T03:33:57.662 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5626 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.658965+0000", 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.658965+0000", 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.658966+0000", 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7900000000000007e-07 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.658985+0000", 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8847000000000001e-05 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.658995+0000", 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0729e-05 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.659004+0000", 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3620000000000006e-06 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.659047+0000", 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2917000000000003e-05 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.659132+0000", 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4944000000000005e-05 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.659221+0000", 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9439999999999997e-05 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.659711+0000", 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049006899999999999 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.659916+0000", 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020526100000000001 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.207520+0000", 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54760367300000001 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.207658+0000", 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013788599999999999 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.207764+0000", 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105816 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:7504 8.a 8:51c54339:::benchmark_data_smithi050_86143_object7503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:02.929628+0000", 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.71171460599999, 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40198023599999999, 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7504 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929628+0000", 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929628+0000", 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929629+0000", 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.145e-06 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929651+0000", 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2291999999999999e-05 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929662+0000", 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0787e-05 2024-04-06T03:33:57.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929670+0000", 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2339999999999994e-06 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.929918+0000", 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024745600000000001 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.930052+0000", 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000134451 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.930183+0000", 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131187 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932469+0000", 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022855739999999999 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932632+0000", 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016341000000000001 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.331476+0000", 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39884380800000002 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.331546+0000", 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9981000000000006e-05 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.331608+0000", 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1910999999999997e-05 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.14928.0:7505 8.b e188/182 8:df89fdf2:::benchmark_data_smithi050_86143_object7504:head v 188'451, mlcod=188'448)", 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:02.932589+0000", 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.708753586, 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40075290099999999, 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932589+0000", 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932589+0000", 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932589+0000", 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8800000000000002e-07 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932615+0000", 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5247000000000001e-05 2024-04-06T03:33:57.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932623+0000", 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6300000000000004e-06 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932628+0000", 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.741e-06 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932659+0000", 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1022000000000002e-05 2024-04-06T03:33:57.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.932671+0000", 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1746000000000001e-05 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.333297+0000", 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40062606099999998 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.333342+0000", 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4666000000000002e-05 2024-04-06T03:33:57.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:7510 8.a 8:551cb490:::benchmark_data_smithi050_86143_object7509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:02.933337+0000", 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.708004897, 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40076557099999999, 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.678 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7510 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933337+0000", 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933337+0000", 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933338+0000", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4e-07 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933371+0000", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2495e-05 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933378+0000", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5519999999999999e-06 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933383+0000", 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7809999999999996e-06 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933419+0000", 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5642999999999997e-05 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933497+0000", 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8499999999999997e-05 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933614+0000", 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011695400000000001 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.331210+0000", 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39759577600000001 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.333381+0000", 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002171199 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.333948+0000", 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056736000000000004 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.334042+0000", 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3640000000000005e-05 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.334103+0000", 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0831000000000001e-05 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.683 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:7512 8.3 8:c035ca53:::benchmark_data_smithi050_86143_object7511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:02.933891+0000", 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 234.70745129400001, 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40169340100000001, 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 7512 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.684 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933891+0000", 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933891+0000", 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933892+0000", 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.78e-07 2024-04-06T03:33:57.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933916+0000", 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3696000000000001e-05 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933924+0000", 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7250000000000003e-06 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933930+0000", 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4790000000000004e-06 2024-04-06T03:33:57.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.933977+0000", 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7397e-05 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.331326+0000", 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39734867699999998 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.331583+0000", 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000256851 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.333385+0000", 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018020219999999999 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.335390+0000", 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020053419999999998 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.335445+0000", 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4769999999999999e-05 2024-04-06T03:33:57.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.335525+0000", 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0180999999999997e-05 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:03.335584+0000", 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9283000000000001e-05 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.689 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:15952 8.1 8:82b4c6ea:::benchmark_data_smithi050_86143_object15951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:07.074422+0000", 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 230.56691974200001, 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.41042590200000001, 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 15952 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.074422+0000", 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.074422+0000", 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.074424+0000", 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1200000000000001e-06 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.074446+0000", 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2129000000000001e-05 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.074454+0000", 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7399999999999993e-06 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.074461+0000", 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9870000000000002e-06 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.076079+0000", 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016179199999999999 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.076223+0000", 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014395900000000001 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.076416+0000", 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019313699999999999 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.078225+0000", 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001808231 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.078848+0000", 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062352000000000002 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.484685+0000", 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40583676800000001 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.484778+0000", 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2869999999999997e-05 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.484848+0000", 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0520999999999998e-05 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:15967 8.1 8:8dcc4555:::benchmark_data_smithi050_86143_object15966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:07.079171+0000", 2024-04-06T03:33:57.695 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 230.56217136999999, 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40634268299999998, 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 15967 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079171+0000", 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079171+0000", 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079171+0000", 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5200000000000002e-07 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079192+0000", 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1030999999999999e-05 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079202+0000", 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1819999999999995e-06 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079207+0000", 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9850000000000003e-06 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079414+0000", 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000206042 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079533+0000", 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119663 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.079678+0000", 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014441399999999999 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.081440+0000", 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00176256 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.082939+0000", 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014990450000000001 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.485301+0000", 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.40236224700000001 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.485399+0000", 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7703999999999998e-05 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:07.485514+0000", 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011435800000000001 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74162 8.a 8:58c2a27e:::benchmark_data_smithi050_86143_object74161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.047999+0000", 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 184.59334321599999, 2024-04-06T03:33:57.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.44849708599999999, 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74162 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.047999+0000", 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.047999+0000", 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048000+0000", 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2330000000000001e-06 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048042+0000", 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2166999999999998e-05 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048053+0000", 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0721e-05 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048061+0000", 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4610000000000002e-06 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.053075+0000", 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050145909999999997 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.053192+0000", 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011639200000000001 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.053293+0000", 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010117 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.059960+0000", 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066670089999999998 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060780+0000", 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000820008 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.496345+0000", 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43556495899999997 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.496437+0000", 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2115999999999997e-05 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.496496+0000", 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9259000000000003e-05 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74163 8.a 8:5789611e:::benchmark_data_smithi050_86143_object74162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.048073+0000", 2024-04-06T03:33:57.708 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 184.593269674, 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45383480900000001, 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74163 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048073+0000", 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048073+0000", 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048073+0000", 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9900000000000001e-07 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048092+0000", 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8627999999999999e-05 2024-04-06T03:33:57.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048099+0000", 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8800000000000002e-06 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.048104+0000", 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.468e-06 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.056281+0000", 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081767520000000007 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.056446+0000", 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016565699999999999 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.056606+0000", 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015986700000000001 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.059965+0000", 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033583229999999999 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.062999+0000", 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030345960000000001 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.501685+0000", 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43868547200000002 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.501838+0000", 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015354600000000001 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.501907+0000", 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9121000000000004e-05 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74174 8.a 8:5d180c2b:::benchmark_data_smithi050_86143_object74173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.068132+0000", 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 184.573210303, 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.434005688, 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.714 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74174 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068132+0000", 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068132+0000", 2024-04-06T03:33:57.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068133+0000", 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1650000000000001e-06 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068152+0000", 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8984999999999999e-05 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068166+0000", 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3837e-05 2024-04-06T03:33:57.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068174+0000", 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3070000000000003e-06 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068221+0000", 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6879999999999998e-05 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068331+0000", 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110367 2024-04-06T03:33:57.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.068452+0000", 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120209 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.070554+0000", 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002102488 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.072345+0000", 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017909880000000001 2024-04-06T03:33:57.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.501963+0000", 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42961803500000001 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.502086+0000", 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000122984 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.502138+0000", 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1443000000000002e-05 2024-04-06T03:33:57.719 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160890 8.a 8:5b1f70e8:::benchmark_data_smithi050_86143_object54579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.768023+0000", 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 82.873319140999996, 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.51791149199999997, 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.720 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160890 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768023+0000", 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768023+0000", 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768025+0000", 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.767e-06 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768030+0000", 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0699999999999997e-06 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768041+0000", 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1253999999999999e-05 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768050+0000", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1230000000000006e-06 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768167+0000", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116978 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768293+0000", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012565100000000001 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768440+0000", 2024-04-06T03:33:57.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014731299999999999 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.771365+0000", 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029250420000000001 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.778760+0000", 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073947250000000004 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.285594+0000", 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50683423500000002 2024-04-06T03:33:57.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.285825+0000", 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000230714 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.285935+0000", 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010962 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160892 8.1 8:83a6418b:::benchmark_data_smithi050_86143_object54581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.768881+0000", 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 82.872461181000006, 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.51831987999999996, 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160892 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768881+0000", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768881+0000", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768882+0000", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7599999999999996e-07 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768886+0000", 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5430000000000004e-06 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768898+0000", 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1088e-05 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768906+0000", 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1489999999999994e-06 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.768960+0000", 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4339000000000003e-05 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769069+0000", 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010929600000000001 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769178+0000", 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109153 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.774788+0000", 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056097839999999996 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781791+0000", 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070028490000000002 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.286928+0000", 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50513662599999998 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.730 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.287105+0000", 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017708700000000001 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.287201+0000", 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6089999999999996e-05 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160893 8.a 8:5958c4f7:::benchmark_data_smithi050_86143_object54582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.769980+0000", 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 82.871362429000001, 2024-04-06T03:33:57.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53515883799999997, 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160893 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769980+0000", 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769980+0000", 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769981+0000", 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6099999999999999e-07 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769986+0000", 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1880000000000001e-06 2024-04-06T03:33:57.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.769997+0000", 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1060000000000001e-05 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.770005+0000", 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3089999999999996e-06 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.771395+0000", 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001389957 2024-04-06T03:33:57.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.771610+0000", 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000214866 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.772460+0000", 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00084938099999999996 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.774823+0000", 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023638999999999999 2024-04-06T03:33:57.735 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.779864+0000", 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050404710000000004 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.304980+0000", 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52511662400000003 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.305070+0000", 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9007999999999999e-05 2024-04-06T03:33:57.736 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.305139+0000", 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9113e-05 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160898 8.1 8:82243751:::benchmark_data_smithi050_86143_object54587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.776124+0000", 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 82.865218353000003, 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.51138741700000001, 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.737 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160898 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776124+0000", 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.738 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776124+0000", 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776125+0000", 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.16e-07 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776129+0000", 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6539999999999998e-06 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.739 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776138+0000", 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0310000000000004e-06 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776144+0000", 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7339999999999996e-06 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776193+0000", 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8467999999999997e-05 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.740 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776265+0000", 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2327000000000001e-05 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776332+0000", 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6713999999999999e-05 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.779781+0000", 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034489410000000001 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.741 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.784567+0000", 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047861420000000002 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.287338+0000", 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50277079400000002 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.287433+0000", 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5678000000000001e-05 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.742 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.287511+0000", 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8017999999999997e-05 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160903 8.a 8:5a4740ae:::benchmark_data_smithi050_86143_object54592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.781625+0000", 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 82.859717196999995, 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52370451200000001, 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.743 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160903 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781625+0000", 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781625+0000", 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.744 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781626+0000", 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4789999999999999e-06 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781632+0000", 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2220000000000001e-06 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781643+0000", 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1768e-05 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.745 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781649+0000", 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8889999999999999e-06 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781769+0000", 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000119911 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781887+0000", 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117871 2024-04-06T03:33:57.746 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.781957+0000", 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9413999999999996e-05 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.784516+0000", 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025592100000000001 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.787175+0000", 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026588150000000001 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.747 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.305201+0000", 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.51802595799999995 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.305273+0000", 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2143000000000004e-05 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.305330+0000", 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6832000000000001e-05 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.748 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160917 8.1 8:8a4af09e:::benchmark_data_smithi050_86143_object54606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.802266+0000", 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 82.839076224999999, 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48944896599999999, 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160917 2024-04-06T03:33:57.749 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802266+0000", 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802266+0000", 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802267+0000", 2024-04-06T03:33:57.750 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2920000000000001e-06 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802272+0000", 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9830000000000004e-06 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802285+0000", 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3028e-05 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802291+0000", 2024-04-06T03:33:57.751 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0209999999999996e-06 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802348+0000", 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6613999999999997e-05 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802485+0000", 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013724700000000001 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.802620+0000", 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013473599999999999 2024-04-06T03:33:57.752 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.804800+0000", 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021805040000000002 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.805242+0000", 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044143699999999998 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.291150+0000", 2024-04-06T03:33:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48590789299999998 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.291512+0000", 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036236599999999998 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.291715+0000", 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000202845 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:33:57.754 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:33:57.755 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:33:57.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:57.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[21467]: cluster 2024-04-06T03:33:56.245022+0000 mgr.y (mgr.24368) 479 : cluster 0 pgmap v494: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:33:57.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[21467]: cluster 2024-04-06T03:33:56.337137+0000 osd.4 (osd.4) 303 : cluster 0 8.2 deep-scrub starts 2024-04-06T03:33:57.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[21467]: cluster 2024-04-06T03:33:56.365608+0000 osd.4 (osd.4) 304 : cluster 0 8.2 deep-scrub ok 2024-04-06T03:33:57.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[21467]: cluster 2024-04-06T03:33:57.470017+0000 osd.3 (osd.3) 317 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:57.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[21467]: cluster 2024-04-06T03:33:57.470132+0000 osd.3 (osd.3) 318 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:57.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[29221]: cluster 2024-04-06T03:33:56.245022+0000 mgr.y (mgr.24368) 479 : cluster 0 pgmap v494: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:33:57.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[29221]: cluster 2024-04-06T03:33:56.337137+0000 osd.4 (osd.4) 303 : cluster 0 8.2 deep-scrub starts 2024-04-06T03:33:57.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[29221]: cluster 2024-04-06T03:33:56.365608+0000 osd.4 (osd.4) 304 : cluster 0 8.2 deep-scrub ok 2024-04-06T03:33:57.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[29221]: cluster 2024-04-06T03:33:57.470017+0000 osd.3 (osd.3) 317 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:57.768 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:57 smithi050 bash[29221]: cluster 2024-04-06T03:33:57.470132+0000 osd.3 (osd.3) 318 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:57.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:57.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:57.895 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:57 smithi153 bash[23852]: cluster 2024-04-06T03:33:56.245022+0000 mgr.y (mgr.24368) 479 : cluster 0 pgmap v494: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:33:57.895 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:57 smithi153 bash[23852]: cluster 2024-04-06T03:33:56.337137+0000 osd.4 (osd.4) 303 : cluster 0 8.2 deep-scrub starts 2024-04-06T03:33:57.896 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:57 smithi153 bash[23852]: cluster 2024-04-06T03:33:56.365608+0000 osd.4 (osd.4) 304 : cluster 0 8.2 deep-scrub ok 2024-04-06T03:33:57.896 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:57 smithi153 bash[23852]: cluster 2024-04-06T03:33:57.470017+0000 osd.3 (osd.3) 317 : cluster 0 purged_snaps scrub starts 2024-04-06T03:33:57.896 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:57 smithi153 bash[23852]: cluster 2024-04-06T03:33:57.470132+0000 osd.3 (osd.3) 318 : cluster 0 purged_snaps scrub ok 2024-04-06T03:33:57.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:33:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:57.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:33:58.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:58.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:58.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:58.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:58.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:58.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:58.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:58.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:33:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:33:58.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:58.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:58 smithi153 bash[23852]: cluster 2024-04-06T03:33:57.582809+0000 mon.a (mon.0) 1311 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-04-06T03:33:58.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:58 smithi153 bash[23852]: cluster 2024-04-06T03:33:58.065515+0000 osd.7 (osd.7) 331 : cluster 0 3.11 scrub starts 2024-04-06T03:33:58.683 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:58 smithi153 bash[23852]: cluster 2024-04-06T03:33:58.071989+0000 osd.7 (osd.7) 332 : cluster 0 3.11 scrub ok 2024-04-06T03:33:58.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:33:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:33:58.725 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:33:58.731 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:33:58.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:58 smithi050 bash[29221]: cluster 2024-04-06T03:33:57.582809+0000 mon.a (mon.0) 1311 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-04-06T03:33:58.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:58 smithi050 bash[29221]: cluster 2024-04-06T03:33:58.065515+0000 osd.7 (osd.7) 331 : cluster 0 3.11 scrub starts 2024-04-06T03:33:58.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:58 smithi050 bash[29221]: cluster 2024-04-06T03:33:58.071989+0000 osd.7 (osd.7) 332 : cluster 0 3.11 scrub ok 2024-04-06T03:33:58.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:58 smithi050 bash[21467]: cluster 2024-04-06T03:33:57.582809+0000 mon.a (mon.0) 1311 : cluster 0 osdmap e226: 8 total, 8 up, 8 in 2024-04-06T03:33:58.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:58 smithi050 bash[21467]: cluster 2024-04-06T03:33:58.065515+0000 osd.7 (osd.7) 331 : cluster 0 3.11 scrub starts 2024-04-06T03:33:58.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:33:58 smithi050 bash[21467]: cluster 2024-04-06T03:33:58.071989+0000 osd.7 (osd.7) 332 : cluster 0 3.11 scrub ok 2024-04-06T03:33:58.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:58.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:58.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:58.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:58.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:59.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:59.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:59.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:59.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:33:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:33:59.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:59.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:59.416 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:33:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:33:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:33:59.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:59.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:33:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:33:59.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:59.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:33:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:33:59.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:59.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:33:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:33:59.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:59.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:33:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:33:59.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:33:59.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:00.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:00.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:33:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:59 smithi153 bash[23852]: cluster 2024-04-06T03:33:58.245708+0000 mgr.y (mgr.24368) 480 : cluster 0 pgmap v496: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:59 smithi153 bash[23852]: cluster 2024-04-06T03:33:58.294145+0000 osd.4 (osd.4) 305 : cluster 0 5.1f scrub starts 2024-04-06T03:34:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:59 smithi153 bash[23852]: cluster 2024-04-06T03:33:58.297250+0000 osd.4 (osd.4) 306 : cluster 0 5.1f scrub ok 2024-04-06T03:34:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:33:59 smithi153 bash[23852]: audit 2024-04-06T03:33:58.709393+0000 mgr.y (mgr.24368) 481 : audit 0 from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:00.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:00.143 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:00.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:59 smithi050 bash[29221]: cluster 2024-04-06T03:33:58.245708+0000 mgr.y (mgr.24368) 480 : cluster 0 pgmap v496: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:00.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:59 smithi050 bash[29221]: cluster 2024-04-06T03:33:58.294145+0000 osd.4 (osd.4) 305 : cluster 0 5.1f scrub starts 2024-04-06T03:34:00.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:59 smithi050 bash[29221]: cluster 2024-04-06T03:33:58.297250+0000 osd.4 (osd.4) 306 : cluster 0 5.1f scrub ok 2024-04-06T03:34:00.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:33:59 smithi050 bash[29221]: audit 2024-04-06T03:33:58.709393+0000 mgr.y (mgr.24368) 481 : audit 0 from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:00.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:00.274 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:00.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:00.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:00.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: cluster 2024-04-06T03:33:58.245708+0000 mgr.y (mgr.24368) 480 : cluster 0 pgmap v496: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:00.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: cluster 2024-04-06T03:33:58.294145+0000 osd.4 (osd.4) 305 : cluster 0 5.1f scrub starts 2024-04-06T03:34:00.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: cluster 2024-04-06T03:33:58.297250+0000 osd.4 (osd.4) 306 : cluster 0 5.1f scrub ok 2024-04-06T03:34:00.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: audit 2024-04-06T03:33:58.709393+0000 mgr.y (mgr.24368) 481 : audit 0 from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:00.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:00.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:00.612 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_1' removed 2024-04-06T03:34:00.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:00.666 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:00.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[29221]: audit 2024-04-06T03:33:59.611877+0000 mgr.y (mgr.24368) 482 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[29221]: audit 2024-04-06T03:33:59.832939+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.50:0/3750775211' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[29221]: audit 2024-04-06T03:33:59.833282+0000 mon.a (mon.0) 1312 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[29221]: cluster 2024-04-06T03:34:00.336450+0000 osd.0 (osd.0) 298 : cluster 0 3.1 scrub starts 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[29221]: cluster 2024-04-06T03:34:00.341044+0000 osd.0 (osd.0) 299 : cluster 0 3.1 scrub ok 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: audit 2024-04-06T03:33:59.611877+0000 mgr.y (mgr.24368) 482 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: audit 2024-04-06T03:33:59.832939+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.50:0/3750775211' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:34:00.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: audit 2024-04-06T03:33:59.833282+0000 mon.a (mon.0) 1312 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:34:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: cluster 2024-04-06T03:34:00.336450+0000 osd.0 (osd.0) 298 : cluster 0 3.1 scrub starts 2024-04-06T03:34:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:00 smithi050 bash[21467]: cluster 2024-04-06T03:34:00.341044+0000 osd.0 (osd.0) 299 : cluster 0 3.1 scrub ok 2024-04-06T03:34:00.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:00.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:00.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:01.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:01.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:00 smithi153 bash[23852]: audit 2024-04-06T03:33:59.611877+0000 mgr.y (mgr.24368) 482 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:01.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:00 smithi153 bash[23852]: audit 2024-04-06T03:33:59.832939+0000 mon.c (mon.2) 68 : audit 1 from='client.? 172.21.15.50:0/3750775211' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:34:01.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:00 smithi153 bash[23852]: audit 2024-04-06T03:33:59.833282+0000 mon.a (mon.0) 1312 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:34:01.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:00 smithi153 bash[23852]: cluster 2024-04-06T03:34:00.336450+0000 osd.0 (osd.0) 298 : cluster 0 3.1 scrub starts 2024-04-06T03:34:01.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:00 smithi153 bash[23852]: cluster 2024-04-06T03:34:00.341044+0000 osd.0 (osd.0) 299 : cluster 0 3.1 scrub ok 2024-04-06T03:34:01.047 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:01.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:01.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:01.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:01.317 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:01.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:01.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:01.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-06T03:34:01.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:01.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:01.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:01 smithi050 bash[29221]: cluster 2024-04-06T03:34:00.247025+0000 mgr.y (mgr.24368) 483 : cluster 0 pgmap v497: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:01 smithi050 bash[29221]: audit 2024-04-06T03:34:00.596623+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:34:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:01 smithi050 bash[29221]: cluster 2024-04-06T03:34:00.600449+0000 mon.a (mon.0) 1314 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-04-06T03:34:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:01 smithi050 bash[21467]: cluster 2024-04-06T03:34:00.247025+0000 mgr.y (mgr.24368) 483 : cluster 0 pgmap v497: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:01 smithi050 bash[21467]: audit 2024-04-06T03:34:00.596623+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:34:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:01 smithi050 bash[21467]: cluster 2024-04-06T03:34:00.600449+0000 mon.a (mon.0) 1314 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-04-06T03:34:01.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:01.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:01.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:01.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:01.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:01 smithi153 bash[23852]: cluster 2024-04-06T03:34:00.247025+0000 mgr.y (mgr.24368) 483 : cluster 0 pgmap v497: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:01 smithi153 bash[23852]: audit 2024-04-06T03:34:00.596623+0000 mon.a (mon.0) 1313 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:34:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:01 smithi153 bash[23852]: cluster 2024-04-06T03:34:00.600449+0000 mon.a (mon.0) 1314 : cluster 0 osdmap e227: 8 total, 8 up, 8 in 2024-04-06T03:34:02.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:02.091 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:02.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:02.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:02.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:02.348 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:02.451 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:02.477 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:02.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:02.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:02.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:02.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:02.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:02.870 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:02.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:02.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:03.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:03.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:03.028 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-04-06T03:33:58.245301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2423,"num_read_kb":2237,"num_write":213711,"num_write_kb":6806174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86491,"ondisk_log_size":86491,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1730592,"kb_used_data":8220,"kb_used_omap":12,"kb_used_meta":1722291,"kb_avail":748230624,"statfs":{"total":767960285184,"available":766188158976,"internally_reserved":0,"allocated":8417280,"data_stored":5740722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1763626545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003811"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":346,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569930+0000","last_change":"2024-04-06T03:33:10.115456+0000","last_active":"2024-04-06T03:33:55.569930+0000","last_peered":"2024-04-06T03:33:55.569930+0000","last_clean":"2024-04-06T03:33:55.569930+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:33:55.569930+0000","last_undegraded":"2024-04-06T03:33:55.569930+0000","last_fullsized":"2024-04-06T03:33:55.569930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:10.115343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.115343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:22.099127+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":340,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881489+0000","last_change":"2024-04-06T03:31:18.032945+0000","last_active":"2024-04-06T03:33:41.881489+0000","last_peered":"2024-04-06T03:33:41.881489+0000","last_clean":"2024-04-06T03:33:41.881489+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:33:41.881489+0000","last_undegraded":"2024-04-06T03:33:41.881489+0000","last_fullsized":"2024-04-06T03:33:41.881489+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:18.032814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:31:18.032814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":367,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207557+0000","last_change":"2024-04-06T03:33:36.100803+0000","last_active":"2024-04-06T03:33:42.207557+0000","last_peered":"2024-04-06T03:33:42.207557+0000","last_clean":"2024-04-06T03:33:42.207557+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:33:42.207557+0000","last_undegraded":"2024-04-06T03:33:42.207557+0000","last_fullsized":"2024-04-06T03:33:42.207557+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.100494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.100494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:52.847869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"199'32","reported_seq":403,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571714+0000","last_change":"2024-04-06T03:33:18.029775+0000","last_active":"2024-04-06T03:33:55.571714+0000","last_peered":"2024-04-06T03:33:55.571714+0000","last_clean":"2024-04-06T03:33:55.571714+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:33:55.571714+0000","last_undegraded":"2024-04-06T03:33:55.571714+0000","last_fullsized":"2024-04-06T03:33:55.571714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'32","last_scrub_stamp":"2024-04-06T03:33:18.029641+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:33:18.029641+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:35.199015+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":344,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881763+0000","last_change":"2024-04-06T03:31:26.012721+0000","last_active":"2024-04-06T03:33:41.881763+0000","last_peered":"2024-04-06T03:33:41.881763+0000","last_clean":"2024-04-06T03:33:41.881763+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:33:41.881763+0000","last_undegraded":"2024-04-06T03:33:41.881763+0000","last_fullsized":"2024-04-06T03:33:41.881763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:26.012621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:26.012621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":227,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.209887+0000","last_change":"2024-04-06T03:33:24.344205+0000","last_active":"2024-04-06T03:33:42.209887+0000","last_peered":"2024-04-06T03:33:42.209887+0000","last_clean":"2024-04-06T03:33:42.209887+0000","last_became_active":"2024-04-06T03:25:54.480600+0000","last_became_peered":"2024-04-06T03:25:54.480600+0000","last_unstale":"2024-04-06T03:33:42.209887+0000","last_undegraded":"2024-04-06T03:33:42.209887+0000","last_fullsized":"2024-04-06T03:33:42.209887+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:24.344088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.344088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:24.660837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":359,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571497+0000","last_change":"2024-04-06T03:33:39.516560+0000","last_active":"2024-04-06T03:33:55.571497+0000","last_peered":"2024-04-06T03:33:55.571497+0000","last_clean":"2024-04-06T03:33:55.571497+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:33:55.571497+0000","last_undegraded":"2024-04-06T03:33:55.571497+0000","last_fullsized":"2024-04-06T03:33:55.571497+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:39.516225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:33:39.516225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:56.903888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"199'31","reported_seq":400,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207423+0000","last_change":"2024-04-06T03:33:34.113495+0000","last_active":"2024-04-06T03:33:42.207423+0000","last_peered":"2024-04-06T03:33:42.207423+0000","last_clean":"2024-04-06T03:33:42.207423+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:33:42.207423+0000","last_undegraded":"2024-04-06T03:33:42.207423+0000","last_fullsized":"2024-04-06T03:33:42.207423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:33:34.113196+0000","last_deep_scrub":"125'23","last_deep_scrub_stamp":"2024-04-06T03:25:17.013823+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.113196+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:34.184916+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":231,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212119+0000","last_change":"2024-04-06T03:33:26.400774+0000","last_active":"2024-04-06T03:33:42.212119+0000","last_peered":"2024-04-06T03:33:42.212119+0000","last_clean":"2024-04-06T03:33:42.212119+0000","last_became_active":"2024-04-06T03:25:54.484120+0000","last_became_peered":"2024-04-06T03:25:54.484120+0000","last_unstale":"2024-04-06T03:33:42.212119+0000","last_undegraded":"2024-04-06T03:33:42.212119+0000","last_fullsized":"2024-04-06T03:33:42.212119+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:26.400649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:26.400649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:54.665737+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":346,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.208957+0000","last_change":"2024-04-06T03:33:29.327349+0000","last_active":"2024-04-06T03:33:42.208957+0000","last_peered":"2024-04-06T03:33:42.208957+0000","last_clean":"2024-04-06T03:33:42.208957+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:33:42.208957+0000","last_undegraded":"2024-04-06T03:33:42.208957+0000","last_fullsized":"2024-04-06T03:33:42.208957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:29.327145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.327145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:57.257491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":370,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:48.097772+0000","last_change":"2024-04-06T03:33:48.097842+0000","last_active":"2024-04-06T03:33:48.097772+0000","last_peered":"2024-04-06T03:33:48.097772+0000","last_clean":"2024-04-06T03:33:48.097772+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:33:48.097772+0000","last_undegraded":"2024-04-06T03:33:48.097772+0000","last_fullsized":"2024-04-06T03:33:48.097772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:33:48.097599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:48.097599+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:14.712480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"199'34","reported_seq":344,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882914+0000","last_change":"2024-04-06T03:31:19.996318+0000","last_active":"2024-04-06T03:33:41.882914+0000","last_peered":"2024-04-06T03:33:41.882914+0000","last_clean":"2024-04-06T03:33:41.882914+0000","last_became_active":"2024-04-06T03:25:54.489462+0000","last_became_peered":"2024-04-06T03:25:54.489462+0000","last_unstale":"2024-04-06T03:33:41.882914+0000","last_undegraded":"2024-04-06T03:33:41.882914+0000","last_fullsized":"2024-04-06T03:33:41.882914+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'34","last_scrub_stamp":"2024-04-06T03:31:19.996221+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:31:19.996221+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":342,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206118+0000","last_change":"2024-04-06T03:33:39.245506+0000","last_active":"2024-04-06T03:33:42.206118+0000","last_peered":"2024-04-06T03:33:42.206118+0000","last_clean":"2024-04-06T03:33:42.206118+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:33:42.206118+0000","last_undegraded":"2024-04-06T03:33:42.206118+0000","last_fullsized":"2024-04-06T03:33:42.206118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:39.245352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:39.245352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:03.800079+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":223,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206666+0000","last_change":"2024-04-06T03:33:13.537781+0000","last_active":"2024-04-06T03:33:42.206666+0000","last_peered":"2024-04-06T03:33:42.206666+0000","last_clean":"2024-04-06T03:33:42.206666+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:33:42.206666+0000","last_undegraded":"2024-04-06T03:33:42.206666+0000","last_fullsized":"2024-04-06T03:33:42.206666+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.537704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.537704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.198431+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.209646+0000","last_change":"2024-04-06T03:33:34.322488+0000","last_active":"2024-04-06T03:33:42.209646+0000","last_peered":"2024-04-06T03:33:42.209646+0000","last_clean":"2024-04-06T03:33:42.209646+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:33:42.209646+0000","last_undegraded":"2024-04-06T03:33:42.209646+0000","last_fullsized":"2024-04-06T03:33:42.209646+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:34.322401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:46.087127+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.322401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:37.775476+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"199'30","reported_seq":322,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.884082+0000","last_change":"2024-04-06T03:33:09.484528+0000","last_active":"2024-04-06T03:33:41.884082+0000","last_peered":"2024-04-06T03:33:41.884082+0000","last_clean":"2024-04-06T03:33:41.884082+0000","last_became_active":"2024-04-06T03:25:54.483976+0000","last_became_peered":"2024-04-06T03:25:54.483976+0000","last_unstale":"2024-04-06T03:33:41.884082+0000","last_undegraded":"2024-04-06T03:33:41.884082+0000","last_fullsized":"2024-04-06T03:33:41.884082+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'30","last_scrub_stamp":"2024-04-06T03:33:09.484404+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.484404+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:28.202176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":347,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571928+0000","last_change":"2024-04-06T03:33:37.571108+0000","last_active":"2024-04-06T03:33:55.571928+0000","last_peered":"2024-04-06T03:33:55.571928+0000","last_clean":"2024-04-06T03:33:55.571928+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:33:55.571928+0000","last_undegraded":"2024-04-06T03:33:55.571928+0000","last_fullsized":"2024-04-06T03:33:55.571928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.570981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.570981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:55.343543+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":356,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882289+0000","last_change":"2024-04-06T03:31:37.182250+0000","last_active":"2024-04-06T03:33:41.882289+0000","last_peered":"2024-04-06T03:33:41.882289+0000","last_clean":"2024-04-06T03:33:41.882289+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:33:41.882289+0000","last_undegraded":"2024-04-06T03:33:41.882289+0000","last_fullsized":"2024-04-06T03:33:41.882289+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:37.182127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:31:37.182127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":350,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570099+0000","last_change":"2024-04-06T03:31:54.579860+0000","last_active":"2024-04-06T03:33:55.570099+0000","last_peered":"2024-04-06T03:33:55.570099+0000","last_clean":"2024-04-06T03:33:55.570099+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:33:55.570099+0000","last_undegraded":"2024-04-06T03:33:55.570099+0000","last_fullsized":"2024-04-06T03:33:55.570099+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:54.579728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:54.579728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"199'33","reported_seq":415,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882078+0000","last_change":"2024-04-06T03:33:21.449448+0000","last_active":"2024-04-06T03:33:41.882078+0000","last_peered":"2024-04-06T03:33:41.882078+0000","last_clean":"2024-04-06T03:33:41.882078+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:33:41.882078+0000","last_undegraded":"2024-04-06T03:33:41.882078+0000","last_fullsized":"2024-04-06T03:33:41.882078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:33:21.449361+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:30:12.438411+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.449361+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.839536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":345,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883135+0000","last_change":"2024-04-06T03:33:15.602204+0000","last_active":"2024-04-06T03:33:41.883135+0000","last_peered":"2024-04-06T03:33:41.883135+0000","last_clean":"2024-04-06T03:33:41.883135+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:33:41.883135+0000","last_undegraded":"2024-04-06T03:33:41.883135+0000","last_fullsized":"2024-04-06T03:33:41.883135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:15.602002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.602002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.728882+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212769+0000","last_change":"2024-04-06T03:33:25.355380+0000","last_active":"2024-04-06T03:33:42.212769+0000","last_peered":"2024-04-06T03:33:42.212769+0000","last_clean":"2024-04-06T03:33:42.212769+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:33:42.212769+0000","last_undegraded":"2024-04-06T03:33:42.212769+0000","last_fullsized":"2024-04-06T03:33:42.212769+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:25.355264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:33:25.355264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.984353+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":209,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882143+0000","last_change":"2024-04-06T03:31:10.147767+0000","last_active":"2024-04-06T03:33:41.882143+0000","last_peered":"2024-04-06T03:33:41.882143+0000","last_clean":"2024-04-06T03:33:41.882143+0000","last_became_active":"2024-04-06T03:26:18.617121+0000","last_became_peered":"2024-04-06T03:26:18.617121+0000","last_unstale":"2024-04-06T03:33:41.882143+0000","last_undegraded":"2024-04-06T03:33:41.882143+0000","last_fullsized":"2024-04-06T03:33:41.882143+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:10.147685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:10.147685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"224'16","reported_seq":386,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:51.165167+0000","last_change":"2024-04-06T03:33:51.165215+0000","last_active":"2024-04-06T03:33:51.165167+0000","last_peered":"2024-04-06T03:33:51.165167+0000","last_clean":"2024-04-06T03:33:51.165167+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:33:51.165167+0000","last_undegraded":"2024-04-06T03:33:51.165167+0000","last_fullsized":"2024-04-06T03:33:51.165167+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16","last_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.165059+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.035906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":226,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882297+0000","last_change":"2024-04-06T03:33:23.526390+0000","last_active":"2024-04-06T03:33:41.882297+0000","last_peered":"2024-04-06T03:33:41.882297+0000","last_clean":"2024-04-06T03:33:41.882297+0000","last_became_active":"2024-04-06T03:25:54.483894+0000","last_became_peered":"2024-04-06T03:25:54.483894+0000","last_unstale":"2024-04-06T03:33:41.882297+0000","last_undegraded":"2024-04-06T03:33:41.882297+0000","last_fullsized":"2024-04-06T03:33:41.882297+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.526306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.526306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.600671+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":347,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207212+0000","last_change":"2024-04-06T03:31:52.591764+0000","last_active":"2024-04-06T03:33:42.207212+0000","last_peered":"2024-04-06T03:33:42.207212+0000","last_clean":"2024-04-06T03:33:42.207212+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:33:42.207212+0000","last_undegraded":"2024-04-06T03:33:42.207212+0000","last_fullsized":"2024-04-06T03:33:42.207212+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:52.591670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:31:52.591670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":351,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569291+0000","last_change":"2024-04-06T03:31:58.603076+0000","last_active":"2024-04-06T03:33:55.569291+0000","last_peered":"2024-04-06T03:33:55.569291+0000","last_clean":"2024-04-06T03:33:55.569291+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:33:55.569291+0000","last_undegraded":"2024-04-06T03:33:55.569291+0000","last_fullsized":"2024-04-06T03:33:55.569291+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:31:58.602934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:58.602934+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"199'26","reported_seq":388,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.568998+0000","last_change":"2024-04-06T03:33:30.273551+0000","last_active":"2024-04-06T03:33:55.568998+0000","last_peered":"2024-04-06T03:33:55.568998+0000","last_clean":"2024-04-06T03:33:55.568998+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:33:55.568998+0000","last_undegraded":"2024-04-06T03:33:55.568998+0000","last_fullsized":"2024-04-06T03:33:55.568998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'26","last_scrub_stamp":"2024-04-06T03:33:30.273225+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.273225+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:58.185138+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":345,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.163558+0000","last_change":"2024-04-06T03:33:46.163699+0000","last_active":"2024-04-06T03:33:46.163558+0000","last_peered":"2024-04-06T03:33:46.163558+0000","last_clean":"2024-04-06T03:33:46.163558+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:33:46.163558+0000","last_undegraded":"2024-04-06T03:33:46.163558+0000","last_fullsized":"2024-04-06T03:33:46.163558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:33:46.163357+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:31:51.905353+0000","last_clean_scrub_stamp":"2024-04-06T03:33:46.163357+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.363841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"139'13","reported_seq":383,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.813185+0000","last_change":"2024-04-06T03:31:14.162166+0000","last_active":"2024-04-06T03:33:46.813185+0000","last_peered":"2024-04-06T03:33:46.813185+0000","last_clean":"2024-04-06T03:33:46.813185+0000","last_became_active":"2024-04-06T03:26:18.615209+0000","last_became_peered":"2024-04-06T03:26:18.615209+0000","last_unstale":"2024-04-06T03:33:46.813185+0000","last_undegraded":"2024-04-06T03:33:46.813185+0000","last_fullsized":"2024-04-06T03:33:46.813185+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'13","last_scrub_stamp":"2024-04-06T03:31:14.161673+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:31:14.161673+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":223,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881396+0000","last_change":"2024-04-06T03:31:25.010990+0000","last_active":"2024-04-06T03:33:41.881396+0000","last_peered":"2024-04-06T03:33:41.881396+0000","last_clean":"2024-04-06T03:33:41.881396+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:33:41.881396+0000","last_undegraded":"2024-04-06T03:33:41.881396+0000","last_fullsized":"2024-04-06T03:33:41.881396+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:25.010770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:31:25.010770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'62","reported_seq":460,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.211729+0000","last_change":"2024-04-06T03:33:22.362330+0000","last_active":"2024-04-06T03:33:42.211729+0000","last_peered":"2024-04-06T03:33:42.211729+0000","last_clean":"2024-04-06T03:33:42.211729+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:33:42.211729+0000","last_undegraded":"2024-04-06T03:33:42.211729+0000","last_fullsized":"2024-04-06T03:33:42.211729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'62","last_scrub_stamp":"2024-04-06T03:33:22.362144+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:33:22.362144+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:34.254236+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570574+0000","last_change":"2024-04-06T03:32:07.755529+0000","last_active":"2024-04-06T03:33:55.570574+0000","last_peered":"2024-04-06T03:33:55.570574+0000","last_clean":"2024-04-06T03:33:55.570574+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:33:55.570574+0000","last_undegraded":"2024-04-06T03:33:55.570574+0000","last_fullsized":"2024-04-06T03:33:55.570574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:07.755302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:32:07.755302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":346,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.210236+0000","last_change":"2024-04-06T03:33:13.378754+0000","last_active":"2024-04-06T03:33:42.210236+0000","last_peered":"2024-04-06T03:33:42.210236+0000","last_clean":"2024-04-06T03:33:42.210236+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:33:42.210236+0000","last_undegraded":"2024-04-06T03:33:42.210236+0000","last_fullsized":"2024-04-06T03:33:42.210236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.378611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.000214+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.378611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:27.510378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":225,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.886093+0000","last_change":"2024-04-06T03:33:15.332805+0000","last_active":"2024-04-06T03:33:41.886093+0000","last_peered":"2024-04-06T03:33:41.886093+0000","last_clean":"2024-04-06T03:33:41.886093+0000","last_became_active":"2024-04-06T03:26:18.635349+0000","last_became_peered":"2024-04-06T03:26:18.635349+0000","last_unstale":"2024-04-06T03:33:41.886093+0000","last_undegraded":"2024-04-06T03:33:41.886093+0000","last_fullsized":"2024-04-06T03:33:41.886093+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:15.332597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.332597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.883739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"199'27","reported_seq":283,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212640+0000","last_change":"2024-04-06T03:31:50.901734+0000","last_active":"2024-04-06T03:33:42.212640+0000","last_peered":"2024-04-06T03:33:42.212640+0000","last_clean":"2024-04-06T03:33:42.212640+0000","last_became_active":"2024-04-06T03:25:54.491804+0000","last_became_peered":"2024-04-06T03:25:54.491804+0000","last_unstale":"2024-04-06T03:33:42.212640+0000","last_undegraded":"2024-04-06T03:33:42.212640+0000","last_fullsized":"2024-04-06T03:33:42.212640+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:31:50.901646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:50.901646+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":342,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212393+0000","last_change":"2024-04-06T03:33:21.336526+0000","last_active":"2024-04-06T03:33:42.212393+0000","last_peered":"2024-04-06T03:33:42.212393+0000","last_clean":"2024-04-06T03:33:42.212393+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:33:42.212393+0000","last_undegraded":"2024-04-06T03:33:42.212393+0000","last_fullsized":"2024-04-06T03:33:42.212393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:21.336152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.336152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.892584+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":218,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:45.381593+0000","last_change":"2024-04-06T03:33:45.381648+0000","last_active":"2024-04-06T03:33:45.381593+0000","last_peered":"2024-04-06T03:33:45.381593+0000","last_clean":"2024-04-06T03:33:45.381593+0000","last_became_active":"2024-04-06T03:26:18.618151+0000","last_became_peered":"2024-04-06T03:26:18.618151+0000","last_unstale":"2024-04-06T03:33:45.381593+0000","last_undegraded":"2024-04-06T03:33:45.381593+0000","last_fullsized":"2024-04-06T03:33:45.381593+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:45.381432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:45.381432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:04.135911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":228,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.888443+0000","last_change":"2024-04-06T03:33:20.475825+0000","last_active":"2024-04-06T03:33:41.888443+0000","last_peered":"2024-04-06T03:33:41.888443+0000","last_clean":"2024-04-06T03:33:41.888443+0000","last_became_active":"2024-04-06T03:26:18.637825+0000","last_became_peered":"2024-04-06T03:26:18.637825+0000","last_unstale":"2024-04-06T03:33:41.888443+0000","last_undegraded":"2024-04-06T03:33:41.888443+0000","last_fullsized":"2024-04-06T03:33:41.888443+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:20.475708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.475708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:44.828060+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"199'22","reported_seq":392,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207308+0000","last_change":"2024-04-06T03:33:37.125222+0000","last_active":"2024-04-06T03:33:42.207308+0000","last_peered":"2024-04-06T03:33:42.207308+0000","last_clean":"2024-04-06T03:33:42.207308+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:33:42.207308+0000","last_undegraded":"2024-04-06T03:33:42.207308+0000","last_fullsized":"2024-04-06T03:33:42.207308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'22","last_scrub_stamp":"2024-04-06T03:33:37.125033+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.125033+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:45.455271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569181+0000","last_change":"2024-04-06T03:32:02.669696+0000","last_active":"2024-04-06T03:33:55.569181+0000","last_peered":"2024-04-06T03:33:55.569181+0000","last_clean":"2024-04-06T03:33:55.569181+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:33:55.569181+0000","last_undegraded":"2024-04-06T03:33:55.569181+0000","last_fullsized":"2024-04-06T03:33:55.569181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:02.669327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:32:02.669327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212266+0000","last_change":"2024-04-06T03:33:24.363187+0000","last_active":"2024-04-06T03:33:42.212266+0000","last_peered":"2024-04-06T03:33:42.212266+0000","last_clean":"2024-04-06T03:33:42.212266+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:33:42.212266+0000","last_undegraded":"2024-04-06T03:33:42.212266+0000","last_fullsized":"2024-04-06T03:33:42.212266+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:24.363062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.363062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:45.096055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":370,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212574+0000","last_change":"2024-04-06T03:33:30.460549+0000","last_active":"2024-04-06T03:33:42.212574+0000","last_peered":"2024-04-06T03:33:42.212574+0000","last_clean":"2024-04-06T03:33:42.212574+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:33:42.212574+0000","last_undegraded":"2024-04-06T03:33:42.212574+0000","last_fullsized":"2024-04-06T03:33:42.212574+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:30.460430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.460430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.441443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"199'33","reported_seq":415,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:53.327006+0000","last_change":"2024-04-06T03:33:53.327065+0000","last_active":"2024-04-06T03:33:53.327006+0000","last_peered":"2024-04-06T03:33:53.327006+0000","last_clean":"2024-04-06T03:33:53.327006+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:33:53.327006+0000","last_undegraded":"2024-04-06T03:33:53.327006+0000","last_fullsized":"2024-04-06T03:33:53.327006+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:33:53.326886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.326886+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:18.112004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":342,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.205983+0000","last_change":"2024-04-06T03:33:38.193397+0000","last_active":"2024-04-06T03:33:42.205983+0000","last_peered":"2024-04-06T03:33:42.205983+0000","last_clean":"2024-04-06T03:33:42.205983+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:33:42.205983+0000","last_undegraded":"2024-04-06T03:33:42.205983+0000","last_fullsized":"2024-04-06T03:33:42.205983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:38.193266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:38.193266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:04.273565+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":342,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570339+0000","last_change":"2024-04-06T03:31:35.385717+0000","last_active":"2024-04-06T03:33:55.570339+0000","last_peered":"2024-04-06T03:33:55.570339+0000","last_clean":"2024-04-06T03:33:55.570339+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:33:55.570339+0000","last_undegraded":"2024-04-06T03:33:55.570339+0000","last_fullsized":"2024-04-06T03:33:55.570339+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:35.385380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_clean_scrub_stamp":"2024-04-06T03:31:35.385380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":211,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569299+0000","last_change":"2024-04-06T03:33:20.106580+0000","last_active":"2024-04-06T03:33:55.569299+0000","last_peered":"2024-04-06T03:33:55.569299+0000","last_clean":"2024-04-06T03:33:55.569299+0000","last_became_active":"2024-04-06T03:26:18.643806+0000","last_became_peered":"2024-04-06T03:26:18.643806+0000","last_unstale":"2024-04-06T03:33:55.569299+0000","last_undegraded":"2024-04-06T03:33:55.569299+0000","last_fullsized":"2024-04-06T03:33:55.569299+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:20.106508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.106508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:34.460864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"199'21","reported_seq":387,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:44.226518+0000","last_change":"2024-04-06T03:33:44.226611+0000","last_active":"2024-04-06T03:33:44.226518+0000","last_peered":"2024-04-06T03:33:44.226518+0000","last_clean":"2024-04-06T03:33:44.226518+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:33:44.226518+0000","last_undegraded":"2024-04-06T03:33:44.226518+0000","last_fullsized":"2024-04-06T03:33:44.226518+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:33:44.226337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:44.226337+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:56.430775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":352,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883965+0000","last_change":"2024-04-06T03:33:37.480817+0000","last_active":"2024-04-06T03:33:41.883965+0000","last_peered":"2024-04-06T03:33:41.883965+0000","last_clean":"2024-04-06T03:33:41.883965+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:33:41.883965+0000","last_undegraded":"2024-04-06T03:33:41.883965+0000","last_fullsized":"2024-04-06T03:33:41.883965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.480549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.480549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:03.468270+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":510,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569400+0000","last_change":"2024-04-06T03:33:31.421434+0000","last_active":"2024-04-06T03:33:55.569400+0000","last_peered":"2024-04-06T03:33:55.569400+0000","last_clean":"2024-04-06T03:33:55.569400+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:33:55.569400+0000","last_undegraded":"2024-04-06T03:33:55.569400+0000","last_fullsized":"2024-04-06T03:33:55.569400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:33:31.421277+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:33:31.421277+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:47.402352+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206844+0000","last_change":"2024-04-06T03:33:23.775542+0000","last_active":"2024-04-06T03:33:42.206844+0000","last_peered":"2024-04-06T03:33:42.206844+0000","last_clean":"2024-04-06T03:33:42.206844+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:33:42.206844+0000","last_undegraded":"2024-04-06T03:33:42.206844+0000","last_fullsized":"2024-04-06T03:33:42.206844+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.775408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.775408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:43.932054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"199'15","reported_seq":375,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206372+0000","last_change":"2024-04-06T03:33:09.505741+0000","last_active":"2024-04-06T03:33:42.206372+0000","last_peered":"2024-04-06T03:33:42.206372+0000","last_clean":"2024-04-06T03:33:42.206372+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:33:42.206372+0000","last_undegraded":"2024-04-06T03:33:42.206372+0000","last_fullsized":"2024-04-06T03:33:42.206372+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'15","last_scrub_stamp":"2024-04-06T03:33:09.505412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.505412+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:11.838492+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":328,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207336+0000","last_change":"2024-04-06T03:31:50.701295+0000","last_active":"2024-04-06T03:33:42.207336+0000","last_peered":"2024-04-06T03:33:42.207336+0000","last_clean":"2024-04-06T03:33:42.207336+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:33:42.207336+0000","last_undegraded":"2024-04-06T03:33:42.207336+0000","last_fullsized":"2024-04-06T03:33:42.207336+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:50.701170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:31:50.701170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"154'13","reported_seq":286,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.817226+0000","last_change":"2024-04-06T03:33:19.694033+0000","last_active":"2024-04-06T03:33:46.817226+0000","last_peered":"2024-04-06T03:33:46.817226+0000","last_clean":"2024-04-06T03:33:46.817226+0000","last_became_active":"2024-04-06T03:27:10.498254+0000","last_became_peered":"2024-04-06T03:27:10.498254+0000","last_unstale":"2024-04-06T03:33:46.817226+0000","last_undegraded":"2024-04-06T03:33:46.817226+0000","last_fullsized":"2024-04-06T03:33:46.817226+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'13","last_scrub_stamp":"2024-04-06T03:33:19.693903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:19.693903+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:44.105035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":223,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570692+0000","last_change":"2024-04-06T03:33:42.516624+0000","last_active":"2024-04-06T03:33:55.570692+0000","last_peered":"2024-04-06T03:33:55.570692+0000","last_clean":"2024-04-06T03:33:55.570692+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:33:55.570692+0000","last_undegraded":"2024-04-06T03:33:55.570692+0000","last_fullsized":"2024-04-06T03:33:55.570692+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_clean_scrub_stamp":"2024-04-06T03:33:42.516373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:02.496309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"199'24","reported_seq":386,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.213284+0000","last_change":"2024-04-06T03:33:15.375572+0000","last_active":"2024-04-06T03:33:42.213284+0000","last_peered":"2024-04-06T03:33:42.213284+0000","last_clean":"2024-04-06T03:33:42.213284+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:33:42.213284+0000","last_undegraded":"2024-04-06T03:33:42.213284+0000","last_fullsized":"2024-04-06T03:33:42.213284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:33:15.375394+0000","last_deep_scrub":"168'16","last_deep_scrub_stamp":"2024-04-06T03:28:39.973804+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.375394+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:36.915506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":342,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212572+0000","last_change":"2024-04-06T03:31:57.853509+0000","last_active":"2024-04-06T03:33:42.212572+0000","last_peered":"2024-04-06T03:33:42.212572+0000","last_clean":"2024-04-06T03:33:42.212572+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:33:42.212572+0000","last_undegraded":"2024-04-06T03:33:42.212572+0000","last_fullsized":"2024-04-06T03:33:42.212572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:57.853321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:31:57.853321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":217,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571492+0000","last_change":"2024-04-06T03:33:46.541535+0000","last_active":"2024-04-06T03:33:55.571492+0000","last_peered":"2024-04-06T03:33:55.571492+0000","last_clean":"2024-04-06T03:33:55.571492+0000","last_became_active":"2024-04-06T03:25:54.485733+0000","last_became_peered":"2024-04-06T03:25:54.485733+0000","last_unstale":"2024-04-06T03:33:55.571492+0000","last_undegraded":"2024-04-06T03:33:55.571492+0000","last_fullsized":"2024-04-06T03:33:55.571492+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:46.541240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:46.541240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.998515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":209,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881754+0000","last_change":"2024-04-06T03:31:23.008787+0000","last_active":"2024-04-06T03:33:41.881754+0000","last_peered":"2024-04-06T03:33:41.881754+0000","last_clean":"2024-04-06T03:33:41.881754+0000","last_became_active":"2024-04-06T03:26:18.615468+0000","last_became_peered":"2024-04-06T03:26:18.615468+0000","last_unstale":"2024-04-06T03:33:41.881754+0000","last_undegraded":"2024-04-06T03:33:41.881754+0000","last_fullsized":"2024-04-06T03:33:41.881754+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:23.008712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:23.008712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"199'10","reported_seq":304,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206367+0000","last_change":"2024-04-06T03:33:15.548008+0000","last_active":"2024-04-06T03:33:42.206367+0000","last_peered":"2024-04-06T03:33:42.206367+0000","last_clean":"2024-04-06T03:33:42.206367+0000","last_became_active":"2024-04-06T03:26:18.639720+0000","last_became_peered":"2024-04-06T03:26:18.639720+0000","last_unstale":"2024-04-06T03:33:42.206367+0000","last_undegraded":"2024-04-06T03:33:42.206367+0000","last_fullsized":"2024-04-06T03:33:42.206367+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10","last_scrub_stamp":"2024-04-06T03:33:15.547658+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.547658+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:30.362669+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.885594+0000","last_change":"2024-04-06T03:33:32.787376+0000","last_active":"2024-04-06T03:33:41.885594+0000","last_peered":"2024-04-06T03:33:41.885594+0000","last_clean":"2024-04-06T03:33:41.885594+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:33:41.885594+0000","last_undegraded":"2024-04-06T03:33:41.885594+0000","last_fullsized":"2024-04-06T03:33:41.885594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:33:32.787297+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:33:32.787297+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:59.143038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":342,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571987+0000","last_change":"2024-04-06T03:31:27.375923+0000","last_active":"2024-04-06T03:33:55.571987+0000","last_peered":"2024-04-06T03:33:55.571987+0000","last_clean":"2024-04-06T03:33:55.571987+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:33:55.571987+0000","last_undegraded":"2024-04-06T03:33:55.571987+0000","last_fullsized":"2024-04-06T03:33:55.571987+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:27.375797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:31:27.375797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":372,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883596+0000","last_change":"2024-04-06T03:33:13.288997+0000","last_active":"2024-04-06T03:33:41.883596+0000","last_peered":"2024-04-06T03:33:41.883596+0000","last_clean":"2024-04-06T03:33:41.883596+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:33:41.883596+0000","last_undegraded":"2024-04-06T03:33:41.883596+0000","last_fullsized":"2024-04-06T03:33:41.883596+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.288879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.288879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:20.772672+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"199'31","reported_seq":399,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.209616+0000","last_change":"2024-04-06T03:32:05.754069+0000","last_active":"2024-04-06T03:33:42.209616+0000","last_peered":"2024-04-06T03:33:42.209616+0000","last_clean":"2024-04-06T03:33:42.209616+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:33:42.209616+0000","last_undegraded":"2024-04-06T03:33:42.209616+0000","last_fullsized":"2024-04-06T03:33:42.209616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:32:05.753981+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:32:05.753981+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"221'13408","reported_seq":13568,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.884036+0000","last_change":"2024-04-06T03:31:49.271108+0000","last_active":"2024-04-06T03:33:41.884036+0000","last_peered":"2024-04-06T03:33:41.884036+0000","last_clean":"2024-04-06T03:33:41.884036+0000","last_became_active":"2024-04-06T03:29:40.995877+0000","last_became_peered":"2024-04-06T03:29:40.995877+0000","last_unstale":"2024-04-06T03:33:41.884036+0000","last_undegraded":"2024-04-06T03:33:41.884036+0000","last_fullsized":"2024-04-06T03:33:41.884036+0000","mapping_epoch":182,"log_start":"207'8400","ondisk_log_start":"207'8400","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'7740","last_scrub_stamp":"2024-04-06T03:31:49.270771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:49.270771+0000","objects_scrubbed":5698,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1171,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13408,"num_write_kb":429056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570450+0000","last_change":"2024-04-06T03:31:07.223520+0000","last_active":"2024-04-06T03:33:55.570450+0000","last_peered":"2024-04-06T03:33:55.570450+0000","last_clean":"2024-04-06T03:33:55.570450+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:33:55.570450+0000","last_undegraded":"2024-04-06T03:33:55.570450+0000","last_fullsized":"2024-04-06T03:33:55.570450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:07.223320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:07.223320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":382,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882330+0000","last_change":"2024-04-06T03:33:14.584324+0000","last_active":"2024-04-06T03:33:41.882330+0000","last_peered":"2024-04-06T03:33:41.882330+0000","last_clean":"2024-04-06T03:33:41.882330+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:33:41.882330+0000","last_undegraded":"2024-04-06T03:33:41.882330+0000","last_fullsized":"2024-04-06T03:33:41.882330+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:33:14.584201+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:33:14.584201+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:43.220224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'31","reported_seq":407,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.211753+0000","last_change":"2024-04-06T03:33:35.363415+0000","last_active":"2024-04-06T03:33:42.211753+0000","last_peered":"2024-04-06T03:33:42.211753+0000","last_clean":"2024-04-06T03:33:42.211753+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:33:42.211753+0000","last_undegraded":"2024-04-06T03:33:42.211753+0000","last_fullsized":"2024-04-06T03:33:42.211753+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'31","last_scrub_stamp":"2024-04-06T03:33:35.363029+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.363029+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:58.052022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":364,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883510+0000","last_change":"2024-04-06T03:33:35.767909+0000","last_active":"2024-04-06T03:33:41.883510+0000","last_peered":"2024-04-06T03:33:41.883510+0000","last_clean":"2024-04-06T03:33:41.883510+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:33:41.883510+0000","last_undegraded":"2024-04-06T03:33:41.883510+0000","last_fullsized":"2024-04-06T03:33:41.883510+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:35.767611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.767611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:39.607144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"221'13550","reported_seq":13721,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207119+0000","last_change":"2024-04-06T03:31:43.415047+0000","last_active":"2024-04-06T03:33:42.207119+0000","last_peered":"2024-04-06T03:33:42.207119+0000","last_clean":"2024-04-06T03:33:42.207119+0000","last_became_active":"2024-04-06T03:29:41.008290+0000","last_became_peered":"2024-04-06T03:29:41.008290+0000","last_unstale":"2024-04-06T03:33:42.207119+0000","last_undegraded":"2024-04-06T03:33:42.207119+0000","last_fullsized":"2024-04-06T03:33:42.207119+0000","mapping_epoch":182,"log_start":"207'8100","ondisk_log_start":"207'8100","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"205'7562","last_scrub_stamp":"2024-04-06T03:31:43.414611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:43.414611+0000","objects_scrubbed":6194,"log_size":5450,"log_dups_size":0,"ondisk_log_size":5450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"queued for scrub","scrub_duration":6027,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":433600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":343,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206055+0000","last_change":"2024-04-06T03:33:31.120147+0000","last_active":"2024-04-06T03:33:42.206055+0000","last_peered":"2024-04-06T03:33:42.206055+0000","last_clean":"2024-04-06T03:33:42.206055+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:33:42.206055+0000","last_undegraded":"2024-04-06T03:33:42.206055+0000","last_fullsized":"2024-04-06T03:33:42.206055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:31.120067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:33:31.120067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.540592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":303,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206922+0000","last_change":"2024-04-06T03:31:54.539188+0000","last_active":"2024-04-06T03:33:42.206922+0000","last_peered":"2024-04-06T03:33:42.206922+0000","last_clean":"2024-04-06T03:33:42.206922+0000","last_became_active":"2024-04-06T03:27:10.500422+0000","last_became_peered":"2024-04-06T03:27:10.500422+0000","last_unstale":"2024-04-06T03:33:42.206922+0000","last_undegraded":"2024-04-06T03:33:42.206922+0000","last_fullsized":"2024-04-06T03:33:42.206922+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:31:54.539063+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:31:54.539063+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"199'37","reported_seq":382,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571412+0000","last_change":"2024-04-06T03:31:39.459165+0000","last_active":"2024-04-06T03:33:55.571412+0000","last_peered":"2024-04-06T03:33:55.571412+0000","last_clean":"2024-04-06T03:33:55.571412+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:33:55.571412+0000","last_undegraded":"2024-04-06T03:33:55.571412+0000","last_fullsized":"2024-04-06T03:33:55.571412+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:31:39.459040+0000","last_deep_scrub":"168'25","last_deep_scrub_stamp":"2024-04-06T03:28:40.744518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:39.459040+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":367,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212341+0000","last_change":"2024-04-06T03:33:14.416136+0000","last_active":"2024-04-06T03:33:42.212341+0000","last_peered":"2024-04-06T03:33:42.212341+0000","last_clean":"2024-04-06T03:33:42.212341+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:33:42.212341+0000","last_undegraded":"2024-04-06T03:33:42.212341+0000","last_fullsized":"2024-04-06T03:33:42.212341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:33:14.416001+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:33:14.416001+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:24.765524+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":514,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.817231+0000","last_change":"2024-04-06T03:33:41.242743+0000","last_active":"2024-04-06T03:33:46.817231+0000","last_peered":"2024-04-06T03:33:46.817231+0000","last_clean":"2024-04-06T03:33:46.817231+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:33:46.817231+0000","last_undegraded":"2024-04-06T03:33:46.817231+0000","last_fullsized":"2024-04-06T03:33:46.817231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:33:41.242500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.242500+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:51.641654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"221'13106","reported_seq":13217,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.568860+0000","last_change":"2024-04-06T03:33:21.219044+0000","last_active":"2024-04-06T03:33:55.568860+0000","last_peered":"2024-04-06T03:33:55.568860+0000","last_clean":"2024-04-06T03:33:55.568860+0000","last_became_active":"2024-04-06T03:29:41.007379+0000","last_became_peered":"2024-04-06T03:29:41.007379+0000","last_unstale":"2024-04-06T03:33:55.568860+0000","last_undegraded":"2024-04-06T03:33:55.568860+0000","last_fullsized":"2024-04-06T03:33:55.568860+0000","mapping_epoch":182,"log_start":"207'7700","ondisk_log_start":"207'7700","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'12462","last_scrub_stamp":"2024-04-06T03:33:21.218855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.218855+0000","objects_scrubbed":644,"log_size":5406,"log_dups_size":0,"ondisk_log_size":5406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:22.247624+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13106,"num_write_kb":419392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571635+0000","last_change":"2024-04-06T03:31:32.350130+0000","last_active":"2024-04-06T03:33:55.571635+0000","last_peered":"2024-04-06T03:33:55.571635+0000","last_clean":"2024-04-06T03:33:55.571635+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:33:55.571635+0000","last_undegraded":"2024-04-06T03:33:55.571635+0000","last_fullsized":"2024-04-06T03:33:55.571635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:32.350015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:32.350015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":230,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.211951+0000","last_change":"2024-04-06T03:33:23.364994+0000","last_active":"2024-04-06T03:33:42.211951+0000","last_peered":"2024-04-06T03:33:42.211951+0000","last_clean":"2024-04-06T03:33:42.211951+0000","last_became_active":"2024-04-06T03:25:54.484235+0000","last_became_peered":"2024-04-06T03:25:54.484235+0000","last_unstale":"2024-04-06T03:33:42.211951+0000","last_undegraded":"2024-04-06T03:33:42.211951+0000","last_fullsized":"2024-04-06T03:33:42.211951+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.364862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.364862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.849695+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"199'44","reported_seq":425,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883030+0000","last_change":"2024-04-06T03:33:13.560964+0000","last_active":"2024-04-06T03:33:41.883030+0000","last_peered":"2024-04-06T03:33:41.883030+0000","last_clean":"2024-04-06T03:33:41.883030+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:33:41.883030+0000","last_undegraded":"2024-04-06T03:33:41.883030+0000","last_fullsized":"2024-04-06T03:33:41.883030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'44","last_scrub_stamp":"2024-04-06T03:33:13.560818+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.560818+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.496227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":347,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206901+0000","last_change":"2024-04-06T03:33:35.119000+0000","last_active":"2024-04-06T03:33:42.206901+0000","last_peered":"2024-04-06T03:33:42.206901+0000","last_clean":"2024-04-06T03:33:42.206901+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:33:42.206901+0000","last_undegraded":"2024-04-06T03:33:42.206901+0000","last_fullsized":"2024-04-06T03:33:42.206901+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:35.118863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.118863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.048886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"221'13256","reported_seq":13364,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569816+0000","last_change":"2024-04-06T03:33:29.385513+0000","last_active":"2024-04-06T03:33:55.569816+0000","last_peered":"2024-04-06T03:33:55.569816+0000","last_clean":"2024-04-06T03:33:55.569816+0000","last_became_active":"2024-04-06T03:29:41.012653+0000","last_became_peered":"2024-04-06T03:29:41.012653+0000","last_unstale":"2024-04-06T03:33:55.569816+0000","last_undegraded":"2024-04-06T03:33:55.569816+0000","last_fullsized":"2024-04-06T03:33:55.569816+0000","mapping_epoch":182,"log_start":"207'7800","ondisk_log_start":"207'7800","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'13001","last_scrub_stamp":"2024-04-06T03:33:29.385375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.385375+0000","objects_scrubbed":258,"log_size":5456,"log_dups_size":0,"ondisk_log_size":5456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.318580+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13256,"num_write_kb":424192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":343,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212756+0000","last_change":"2024-04-06T03:33:09.452964+0000","last_active":"2024-04-06T03:33:42.212756+0000","last_peered":"2024-04-06T03:33:42.212756+0000","last_clean":"2024-04-06T03:33:42.212756+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:33:42.212756+0000","last_undegraded":"2024-04-06T03:33:42.212756+0000","last_fullsized":"2024-04-06T03:33:42.212756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:09.452871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.452871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.657429+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"199'42","reported_seq":280,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883854+0000","last_change":"2024-04-06T03:33:24.513812+0000","last_active":"2024-04-06T03:33:41.883854+0000","last_peered":"2024-04-06T03:33:41.883854+0000","last_clean":"2024-04-06T03:33:41.883854+0000","last_became_active":"2024-04-06T03:27:10.495663+0000","last_became_peered":"2024-04-06T03:27:10.495663+0000","last_unstale":"2024-04-06T03:33:41.883854+0000","last_undegraded":"2024-04-06T03:33:41.883854+0000","last_fullsized":"2024-04-06T03:33:41.883854+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'42","last_scrub_stamp":"2024-04-06T03:33:24.513666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.513666+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:49.284041+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":373,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.883980+0000","last_change":"2024-04-06T03:33:37.772754+0000","last_active":"2024-04-06T03:33:41.883980+0000","last_peered":"2024-04-06T03:33:41.883980+0000","last_clean":"2024-04-06T03:33:41.883980+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:33:41.883980+0000","last_undegraded":"2024-04-06T03:33:41.883980+0000","last_fullsized":"2024-04-06T03:33:41.883980+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.772672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.772672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:44.857148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":364,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.888668+0000","last_change":"2024-04-06T03:33:36.737236+0000","last_active":"2024-04-06T03:33:41.888668+0000","last_peered":"2024-04-06T03:33:41.888668+0000","last_clean":"2024-04-06T03:33:41.888668+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:33:41.888668+0000","last_undegraded":"2024-04-06T03:33:41.888668+0000","last_fullsized":"2024-04-06T03:33:41.888668+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.737080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.737080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:50.451973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"221'13192","reported_seq":13372,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206623+0000","last_change":"2024-04-06T03:33:25.085230+0000","last_active":"2024-04-06T03:33:42.206623+0000","last_peered":"2024-04-06T03:33:42.206623+0000","last_clean":"2024-04-06T03:33:42.206623+0000","last_became_active":"2024-04-06T03:29:41.004725+0000","last_became_peered":"2024-04-06T03:29:41.004725+0000","last_unstale":"2024-04-06T03:33:42.206623+0000","last_undegraded":"2024-04-06T03:33:42.206623+0000","last_fullsized":"2024-04-06T03:33:42.206623+0000","mapping_epoch":182,"log_start":"207'7800","ondisk_log_start":"207'7800","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'12720","last_scrub_stamp":"2024-04-06T03:33:25.084870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:25.084870+0000","objects_scrubbed":484,"log_size":5392,"log_dups_size":0,"ondisk_log_size":5392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:38.142424+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13192,"num_write_kb":422144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":210,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.572041+0000","last_change":"2024-04-06T03:31:43.475492+0000","last_active":"2024-04-06T03:33:55.572041+0000","last_peered":"2024-04-06T03:33:55.572041+0000","last_clean":"2024-04-06T03:33:55.572041+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:33:55.572041+0000","last_undegraded":"2024-04-06T03:33:55.572041+0000","last_fullsized":"2024-04-06T03:33:55.572041+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:43.475384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:43.475384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"224'32","reported_seq":403,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570523+0000","last_change":"2024-04-06T03:33:50.491583+0000","last_active":"2024-04-06T03:33:55.570523+0000","last_peered":"2024-04-06T03:33:55.570523+0000","last_clean":"2024-04-06T03:33:55.570523+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:33:55.570523+0000","last_undegraded":"2024-04-06T03:33:55.570523+0000","last_fullsized":"2024-04-06T03:33:55.570523+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'32","last_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_deep_scrub":"224'32","last_deep_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_clean_scrub_stamp":"2024-04-06T03:33:50.491205+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:01.976482+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":231,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.884340+0000","last_change":"2024-04-06T03:33:27.712256+0000","last_active":"2024-04-06T03:33:41.884340+0000","last_peered":"2024-04-06T03:33:41.884340+0000","last_clean":"2024-04-06T03:33:41.884340+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:33:41.884340+0000","last_undegraded":"2024-04-06T03:33:41.884340+0000","last_fullsized":"2024-04-06T03:33:41.884340+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:27.712145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:27.712145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:54.454326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":614,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.209550+0000","last_change":"2024-04-06T03:33:33.336069+0000","last_active":"2024-04-06T03:33:42.209550+0000","last_peered":"2024-04-06T03:33:42.209550+0000","last_clean":"2024-04-06T03:33:42.209550+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:33:42.209550+0000","last_undegraded":"2024-04-06T03:33:42.209550+0000","last_fullsized":"2024-04-06T03:33:42.209550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:33:33.335954+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:33:33.335954+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:38.640750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":347,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.210480+0000","last_change":"2024-04-06T03:33:16.345677+0000","last_active":"2024-04-06T03:33:42.210480+0000","last_peered":"2024-04-06T03:33:42.210480+0000","last_clean":"2024-04-06T03:33:42.210480+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:33:42.210480+0000","last_undegraded":"2024-04-06T03:33:42.210480+0000","last_fullsized":"2024-04-06T03:33:42.210480+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:16.345560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:33:16.345560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:19.936111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"221'13262","reported_seq":13447,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212581+0000","last_change":"2024-04-06T03:33:13.873628+0000","last_active":"2024-04-06T03:33:42.212581+0000","last_peered":"2024-04-06T03:33:42.212581+0000","last_clean":"2024-04-06T03:33:42.212581+0000","last_became_active":"2024-04-06T03:29:41.004565+0000","last_became_peered":"2024-04-06T03:29:41.004565+0000","last_unstale":"2024-04-06T03:33:42.212581+0000","last_undegraded":"2024-04-06T03:33:42.212581+0000","last_fullsized":"2024-04-06T03:33:42.212581+0000","mapping_epoch":182,"log_start":"206'7700","ondisk_log_start":"206'7700","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'12210","last_scrub_stamp":"2024-04-06T03:33:13.873384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.873384+0000","objects_scrubbed":1066,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:22.098457+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13262,"num_write_kb":424384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":351,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212179+0000","last_change":"2024-04-06T03:33:37.385401+0000","last_active":"2024-04-06T03:33:42.212179+0000","last_peered":"2024-04-06T03:33:42.212179+0000","last_clean":"2024-04-06T03:33:42.212179+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:33:42.212179+0000","last_undegraded":"2024-04-06T03:33:42.212179+0000","last_fullsized":"2024-04-06T03:33:42.212179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.385288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:31:49.387394+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.385288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:51.597710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"199'35","reported_seq":406,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206256+0000","last_change":"2024-04-06T03:31:56.607540+0000","last_active":"2024-04-06T03:33:42.206256+0000","last_peered":"2024-04-06T03:33:42.206256+0000","last_clean":"2024-04-06T03:33:42.206256+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:33:42.206256+0000","last_undegraded":"2024-04-06T03:33:42.206256+0000","last_fullsized":"2024-04-06T03:33:42.206256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'35","last_scrub_stamp":"2024-04-06T03:31:56.607450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:56.607450+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":355,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.209302+0000","last_change":"2024-04-06T03:33:41.282140+0000","last_active":"2024-04-06T03:33:42.209302+0000","last_peered":"2024-04-06T03:33:42.209302+0000","last_clean":"2024-04-06T03:33:42.209302+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:33:42.209302+0000","last_undegraded":"2024-04-06T03:33:42.209302+0000","last_fullsized":"2024-04-06T03:33:42.209302+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.281919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.281919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:11.014377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":230,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212897+0000","last_change":"2024-04-06T03:33:41.412972+0000","last_active":"2024-04-06T03:33:42.212897+0000","last_peered":"2024-04-06T03:33:42.212897+0000","last_clean":"2024-04-06T03:33:42.212897+0000","last_became_active":"2024-04-06T03:25:54.492757+0000","last_became_peered":"2024-04-06T03:25:54.492757+0000","last_unstale":"2024-04-06T03:33:42.212897+0000","last_undegraded":"2024-04-06T03:33:42.212897+0000","last_fullsized":"2024-04-06T03:33:42.212897+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.412744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.371401+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"221'13412","reported_seq":13524,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571570+0000","last_change":"2024-04-06T03:33:23.512998+0000","last_active":"2024-04-06T03:33:55.571570+0000","last_peered":"2024-04-06T03:33:55.571570+0000","last_clean":"2024-04-06T03:33:55.571570+0000","last_became_active":"2024-04-06T03:29:41.012594+0000","last_became_peered":"2024-04-06T03:29:41.012594+0000","last_unstale":"2024-04-06T03:33:55.571570+0000","last_undegraded":"2024-04-06T03:33:55.571570+0000","last_fullsized":"2024-04-06T03:33:55.571570+0000","mapping_epoch":182,"log_start":"207'8000","ondisk_log_start":"207'8000","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'12848","last_scrub_stamp":"2024-04-06T03:33:23.512657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.512657+0000","objects_scrubbed":570,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:29.306409+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13412,"num_write_kb":429184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":351,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212061+0000","last_change":"2024-04-06T03:33:09.337484+0000","last_active":"2024-04-06T03:33:42.212061+0000","last_peered":"2024-04-06T03:33:42.212061+0000","last_clean":"2024-04-06T03:33:42.212061+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:33:42.212061+0000","last_undegraded":"2024-04-06T03:33:42.212061+0000","last_fullsized":"2024-04-06T03:33:42.212061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:09.336838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.336838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:26.093869+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":355,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212929+0000","last_change":"2024-04-06T03:33:07.517308+0000","last_active":"2024-04-06T03:33:42.212929+0000","last_peered":"2024-04-06T03:33:42.212929+0000","last_clean":"2024-04-06T03:33:42.212929+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:33:42.212929+0000","last_undegraded":"2024-04-06T03:33:42.212929+0000","last_fullsized":"2024-04-06T03:33:42.212929+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:07.517188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:33:07.517188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.903244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"199'50","reported_seq":455,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.817353+0000","last_change":"2024-04-06T03:33:17.349616+0000","last_active":"2024-04-06T03:33:46.817353+0000","last_peered":"2024-04-06T03:33:46.817353+0000","last_clean":"2024-04-06T03:33:46.817353+0000","last_became_active":"2024-04-06T03:25:19.551712+0000","last_became_peered":"2024-04-06T03:25:19.551712+0000","last_unstale":"2024-04-06T03:33:46.817353+0000","last_undegraded":"2024-04-06T03:33:46.817353+0000","last_fullsized":"2024-04-06T03:33:46.817353+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'50","last_scrub_stamp":"2024-04-06T03:33:17.349177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:17.349177+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.372881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":218,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:53.214572+0000","last_change":"2024-04-06T03:33:53.214635+0000","last_active":"2024-04-06T03:33:53.214572+0000","last_peered":"2024-04-06T03:33:53.214572+0000","last_clean":"2024-04-06T03:33:53.214572+0000","last_became_active":"2024-04-06T03:26:18.643845+0000","last_became_peered":"2024-04-06T03:26:18.643845+0000","last_unstale":"2024-04-06T03:33:53.214572+0000","last_undegraded":"2024-04-06T03:33:53.214572+0000","last_fullsized":"2024-04-06T03:33:53.214572+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:53.214388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.214388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:04.878344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"221'13120","reported_seq":13281,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206765+0000","last_change":"2024-04-06T03:31:44.824995+0000","last_active":"2024-04-06T03:33:42.206765+0000","last_peered":"2024-04-06T03:33:42.206765+0000","last_clean":"2024-04-06T03:33:42.206765+0000","last_became_active":"2024-04-06T03:29:41.008933+0000","last_became_peered":"2024-04-06T03:29:41.008933+0000","last_unstale":"2024-04-06T03:33:42.206765+0000","last_undegraded":"2024-04-06T03:33:42.206765+0000","last_fullsized":"2024-04-06T03:33:42.206765+0000","mapping_epoch":182,"log_start":"207'7700","ondisk_log_start":"207'7700","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'7349","last_scrub_stamp":"2024-04-06T03:31:44.824628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:44.824628+0000","objects_scrubbed":5797,"log_size":5420,"log_dups_size":0,"ondisk_log_size":5420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13120,"num_write_kb":419840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":345,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881893+0000","last_change":"2024-04-06T03:31:28.291293+0000","last_active":"2024-04-06T03:33:41.881893+0000","last_peered":"2024-04-06T03:33:41.881893+0000","last_clean":"2024-04-06T03:33:41.881893+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:33:41.881893+0000","last_undegraded":"2024-04-06T03:33:41.881893+0000","last_fullsized":"2024-04-06T03:33:41.881893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:28.291003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:31:28.291003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":364,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212755+0000","last_change":"2024-04-06T03:33:15.352460+0000","last_active":"2024-04-06T03:33:42.212755+0000","last_peered":"2024-04-06T03:33:42.212755+0000","last_clean":"2024-04-06T03:33:42.212755+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:33:42.212755+0000","last_undegraded":"2024-04-06T03:33:42.212755+0000","last_fullsized":"2024-04-06T03:33:42.212755+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:33:15.352215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.352215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:26.305203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"199'23","reported_seq":400,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571894+0000","last_change":"2024-04-06T03:33:49.512729+0000","last_active":"2024-04-06T03:33:55.571894+0000","last_peered":"2024-04-06T03:33:55.571894+0000","last_clean":"2024-04-06T03:33:55.571894+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:33:55.571894+0000","last_undegraded":"2024-04-06T03:33:55.571894+0000","last_fullsized":"2024-04-06T03:33:55.571894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'23","last_scrub_stamp":"2024-04-06T03:33:49.512394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:49.512394+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:09.757626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1122,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569343+0000","last_change":"2024-04-06T03:33:35.485693+0000","last_active":"2024-04-06T03:33:55.569343+0000","last_peered":"2024-04-06T03:33:55.569343+0000","last_clean":"2024-04-06T03:33:55.569343+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:33:55.569343+0000","last_undegraded":"2024-04-06T03:33:55.569343+0000","last_fullsized":"2024-04-06T03:33:55.569343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:33:35.485602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.485602+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.043247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":759,"num_read_kb":754,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":349,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881356+0000","last_change":"2024-04-06T03:32:00.109040+0000","last_active":"2024-04-06T03:33:41.881356+0000","last_peered":"2024-04-06T03:33:41.881356+0000","last_clean":"2024-04-06T03:33:41.881356+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:33:41.881356+0000","last_undegraded":"2024-04-06T03:33:41.881356+0000","last_fullsized":"2024-04-06T03:33:41.881356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:00.108953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:32:00.108953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"221'13430","reported_seq":13593,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.213330+0000","last_change":"2024-04-06T03:31:45.597755+0000","last_active":"2024-04-06T03:33:42.213330+0000","last_peered":"2024-04-06T03:33:42.213330+0000","last_clean":"2024-04-06T03:33:42.213330+0000","last_became_active":"2024-04-06T03:29:40.996502+0000","last_became_peered":"2024-04-06T03:29:40.996502+0000","last_unstale":"2024-04-06T03:33:42.213330+0000","last_undegraded":"2024-04-06T03:33:42.213330+0000","last_fullsized":"2024-04-06T03:33:42.213330+0000","mapping_epoch":182,"log_start":"210'8700","ondisk_log_start":"210'8700","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'7586","last_scrub_stamp":"2024-04-06T03:31:45.597453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:45.597453+0000","objects_scrubbed":5879,"log_size":4730,"log_dups_size":0,"ondisk_log_size":4730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13430,"num_write_kb":429697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":343,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212819+0000","last_change":"2024-04-06T03:33:28.341482+0000","last_active":"2024-04-06T03:33:42.212819+0000","last_peered":"2024-04-06T03:33:42.212819+0000","last_clean":"2024-04-06T03:33:42.212819+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:33:42.212819+0000","last_undegraded":"2024-04-06T03:33:42.212819+0000","last_fullsized":"2024-04-06T03:33:42.212819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:28.341397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:28.341397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:39.429372+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":217,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.213388+0000","last_change":"2024-04-06T03:33:32.334478+0000","last_active":"2024-04-06T03:33:42.213388+0000","last_peered":"2024-04-06T03:33:42.213388+0000","last_clean":"2024-04-06T03:33:42.213388+0000","last_became_active":"2024-04-06T03:26:18.621727+0000","last_became_peered":"2024-04-06T03:26:18.621727+0000","last_unstale":"2024-04-06T03:33:42.213388+0000","last_undegraded":"2024-04-06T03:33:42.213388+0000","last_fullsized":"2024-04-06T03:33:42.213388+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:32.334329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:33:32.334329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.543074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"199'24","reported_seq":387,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.205937+0000","last_change":"2024-04-06T03:31:48.505213+0000","last_active":"2024-04-06T03:33:42.205937+0000","last_peered":"2024-04-06T03:33:42.205937+0000","last_clean":"2024-04-06T03:33:42.205937+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:33:42.205937+0000","last_undegraded":"2024-04-06T03:33:42.205937+0000","last_fullsized":"2024-04-06T03:33:42.205937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:31:48.505061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:48.505061+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":358,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881506+0000","last_change":"2024-04-06T03:33:26.451723+0000","last_active":"2024-04-06T03:33:41.881506+0000","last_peered":"2024-04-06T03:33:41.881506+0000","last_clean":"2024-04-06T03:33:41.881506+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:33:41.881506+0000","last_undegraded":"2024-04-06T03:33:41.881506+0000","last_fullsized":"2024-04-06T03:33:41.881506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:26.451631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:10.453422+0000","last_clean_scrub_stamp":"2024-04-06T03:33:26.451631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.536367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":519,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569583+0000","last_change":"2024-04-06T03:33:09.022830+0000","last_active":"2024-04-06T03:33:55.569583+0000","last_peered":"2024-04-06T03:33:55.569583+0000","last_clean":"2024-04-06T03:33:55.569583+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:33:55.569583+0000","last_undegraded":"2024-04-06T03:33:55.569583+0000","last_fullsized":"2024-04-06T03:33:55.569583+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:33:09.022701+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.022701+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:10.225545+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"221'13416","reported_seq":13588,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881396+0000","last_change":"2024-04-06T03:31:30.545278+0000","last_active":"2024-04-06T03:33:41.881396+0000","last_peered":"2024-04-06T03:33:41.881396+0000","last_clean":"2024-04-06T03:33:41.881396+0000","last_became_active":"2024-04-06T03:29:40.997223+0000","last_became_peered":"2024-04-06T03:29:40.997223+0000","last_unstale":"2024-04-06T03:33:41.881396+0000","last_undegraded":"2024-04-06T03:33:41.881396+0000","last_fullsized":"2024-04-06T03:33:41.881396+0000","mapping_epoch":182,"log_start":"207'8400","ondisk_log_start":"207'8400","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'6772","last_scrub_stamp":"2024-04-06T03:31:30.545090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:30.545090+0000","objects_scrubbed":6682,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13416,"num_write_kb":429312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":338,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571143+0000","last_change":"2024-04-06T03:31:33.297229+0000","last_active":"2024-04-06T03:33:55.571143+0000","last_peered":"2024-04-06T03:33:55.571143+0000","last_clean":"2024-04-06T03:33:55.571143+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:33:55.571143+0000","last_undegraded":"2024-04-06T03:33:55.571143+0000","last_fullsized":"2024-04-06T03:33:55.571143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:33.297046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:31:33.297046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":215,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882921+0000","last_change":"2024-04-06T03:31:12.165375+0000","last_active":"2024-04-06T03:33:41.882921+0000","last_peered":"2024-04-06T03:33:41.882921+0000","last_clean":"2024-04-06T03:33:41.882921+0000","last_became_active":"2024-04-06T03:25:54.491253+0000","last_became_peered":"2024-04-06T03:25:54.491253+0000","last_unstale":"2024-04-06T03:33:41.882921+0000","last_undegraded":"2024-04-06T03:33:41.882921+0000","last_fullsized":"2024-04-06T03:33:41.882921+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:12.165261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:41.912384+0000","last_clean_scrub_stamp":"2024-04-06T03:31:12.165261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":355,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:53.048866+0000","last_change":"2024-04-06T03:33:53.048866+0000","last_active":"2024-04-06T03:33:53.048866+0000","last_peered":"2024-04-06T03:33:53.048866+0000","last_clean":"2024-04-06T03:33:53.048866+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:33:53.048866+0000","last_undegraded":"2024-04-06T03:33:53.048866+0000","last_fullsized":"2024-04-06T03:33:53.048866+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:53.048558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.048558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:11.441117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"199'37","reported_seq":415,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.881482+0000","last_change":"2024-04-06T03:33:11.524425+0000","last_active":"2024-04-06T03:33:41.881482+0000","last_peered":"2024-04-06T03:33:41.881482+0000","last_clean":"2024-04-06T03:33:41.881482+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:33:41.881482+0000","last_undegraded":"2024-04-06T03:33:41.881482+0000","last_fullsized":"2024-04-06T03:33:41.881482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:33:11.524289+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:33:11.524289+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.856696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"221'13200","reported_seq":13378,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212211+0000","last_change":"2024-04-06T03:33:20.764345+0000","last_active":"2024-04-06T03:33:42.212211+0000","last_peered":"2024-04-06T03:33:42.212211+0000","last_clean":"2024-04-06T03:33:42.212211+0000","last_became_active":"2024-04-06T03:29:41.000964+0000","last_became_peered":"2024-04-06T03:29:41.000964+0000","last_unstale":"2024-04-06T03:33:42.212211+0000","last_undegraded":"2024-04-06T03:33:42.212211+0000","last_fullsized":"2024-04-06T03:33:42.212211+0000","mapping_epoch":182,"log_start":"206'7600","ondisk_log_start":"206'7600","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'12498","last_scrub_stamp":"2024-04-06T03:33:20.764059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.764059+0000","objects_scrubbed":716,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.299524+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13200,"num_write_kb":422400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":174,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206840+0000","last_change":"2024-04-06T03:31:51.619985+0000","last_active":"2024-04-06T03:33:42.206840+0000","last_peered":"2024-04-06T03:33:42.206840+0000","last_clean":"2024-04-06T03:33:42.206840+0000","last_became_active":"2024-04-06T03:27:10.497525+0000","last_became_peered":"2024-04-06T03:27:10.497525+0000","last_unstale":"2024-04-06T03:33:42.206840+0000","last_undegraded":"2024-04-06T03:33:42.206840+0000","last_fullsized":"2024-04-06T03:33:42.206840+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:51.619863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:51.619863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":524,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.813045+0000","last_change":"2024-04-06T03:33:10.496163+0000","last_active":"2024-04-06T03:33:46.813045+0000","last_peered":"2024-04-06T03:33:46.813045+0000","last_clean":"2024-04-06T03:33:46.813045+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:33:46.813045+0000","last_undegraded":"2024-04-06T03:33:46.813045+0000","last_fullsized":"2024-04-06T03:33:46.813045+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:33:10.496050+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:30:05.475373+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.496050+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:30.697349+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":362,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.211876+0000","last_change":"2024-04-06T03:33:21.373666+0000","last_active":"2024-04-06T03:33:42.211876+0000","last_peered":"2024-04-06T03:33:42.211876+0000","last_clean":"2024-04-06T03:33:42.211876+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:33:42.211876+0000","last_undegraded":"2024-04-06T03:33:42.211876+0000","last_fullsized":"2024-04-06T03:33:42.211876+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:21.373545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.373545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:47.171233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"199'39","reported_seq":421,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569162+0000","last_change":"2024-04-06T03:33:53.514744+0000","last_active":"2024-04-06T03:33:55.569162+0000","last_peered":"2024-04-06T03:33:55.569162+0000","last_clean":"2024-04-06T03:33:55.569162+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:33:55.569162+0000","last_undegraded":"2024-04-06T03:33:55.569162+0000","last_fullsized":"2024-04-06T03:33:55.569162+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:33:53.514484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.514484+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:15.902937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"221'13126","reported_seq":13242,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.568933+0000","last_change":"2024-04-06T03:33:17.539535+0000","last_active":"2024-04-06T03:33:55.568933+0000","last_peered":"2024-04-06T03:33:55.568933+0000","last_clean":"2024-04-06T03:33:55.568933+0000","last_became_active":"2024-04-06T03:29:41.012196+0000","last_became_peered":"2024-04-06T03:29:41.012196+0000","last_unstale":"2024-04-06T03:33:55.568933+0000","last_undegraded":"2024-04-06T03:33:55.568933+0000","last_fullsized":"2024-04-06T03:33:55.568933+0000","mapping_epoch":182,"log_start":"207'7700","ondisk_log_start":"207'7700","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'12223","last_scrub_stamp":"2024-04-06T03:33:17.539167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:17.539167+0000","objects_scrubbed":918,"log_size":5426,"log_dups_size":0,"ondisk_log_size":5426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.691607+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13126,"num_write_kb":420032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":344,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882828+0000","last_change":"2024-04-06T03:32:06.991866+0000","last_active":"2024-04-06T03:33:41.882828+0000","last_peered":"2024-04-06T03:33:41.882828+0000","last_clean":"2024-04-06T03:33:41.882828+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:33:41.882828+0000","last_undegraded":"2024-04-06T03:33:41.882828+0000","last_fullsized":"2024-04-06T03:33:41.882828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:06.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:32:06.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":532,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.817231+0000","last_change":"2024-04-06T03:33:30.761558+0000","last_active":"2024-04-06T03:33:46.817231+0000","last_peered":"2024-04-06T03:33:46.817231+0000","last_clean":"2024-04-06T03:33:46.817231+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:33:46.817231+0000","last_undegraded":"2024-04-06T03:33:46.817231+0000","last_fullsized":"2024-04-06T03:33:46.817231+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:33:30.761309+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.761309+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:37.611140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":235,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570409+0000","last_change":"2024-04-06T03:33:12.048860+0000","last_active":"2024-04-06T03:33:55.570409+0000","last_peered":"2024-04-06T03:33:55.570409+0000","last_clean":"2024-04-06T03:33:55.570409+0000","last_became_active":"2024-04-06T03:25:19.549690+0000","last_became_peered":"2024-04-06T03:25:19.549690+0000","last_unstale":"2024-04-06T03:33:55.570409+0000","last_undegraded":"2024-04-06T03:33:55.570409+0000","last_fullsized":"2024-04-06T03:33:55.570409+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:12.048735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:12.048735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:15.224464+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"199'21","reported_seq":391,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.570211+0000","last_change":"2024-04-06T03:33:51.487161+0000","last_active":"2024-04-06T03:33:55.570211+0000","last_peered":"2024-04-06T03:33:55.570211+0000","last_clean":"2024-04-06T03:33:55.570211+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:33:55.570211+0000","last_undegraded":"2024-04-06T03:33:55.570211+0000","last_fullsized":"2024-04-06T03:33:55.570211+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:33:51.487048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.487048+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:58.854777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1","version":"221'13252","reported_seq":13439,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212019+0000","last_change":"2024-04-06T03:33:18.765202+0000","last_active":"2024-04-06T03:33:42.212019+0000","last_peered":"2024-04-06T03:33:42.212019+0000","last_clean":"2024-04-06T03:33:42.212019+0000","last_became_active":"2024-04-06T03:29:41.005383+0000","last_became_peered":"2024-04-06T03:29:41.005383+0000","last_unstale":"2024-04-06T03:33:42.212019+0000","last_undegraded":"2024-04-06T03:33:42.212019+0000","last_fullsized":"2024-04-06T03:33:42.212019+0000","mapping_epoch":182,"log_start":"206'7600","ondisk_log_start":"206'7600","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'12431","last_scrub_stamp":"2024-04-06T03:33:18.764943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:18.764943+0000","objects_scrubbed":828,"log_size":5652,"log_dups_size":0,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:27.398665+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13252,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":360,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.886778+0000","last_change":"2024-04-06T03:33:30.117825+0000","last_active":"2024-04-06T03:33:41.886778+0000","last_peered":"2024-04-06T03:33:41.886778+0000","last_clean":"2024-04-06T03:33:41.886778+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:33:41.886778+0000","last_undegraded":"2024-04-06T03:33:41.886778+0000","last_fullsized":"2024-04-06T03:33:41.886778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:30.117646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.117646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:38.425925+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":179,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:51.362406+0000","last_change":"2024-04-06T03:33:51.362465+0000","last_active":"2024-04-06T03:33:51.362406+0000","last_peered":"2024-04-06T03:33:51.362406+0000","last_clean":"2024-04-06T03:33:51.362406+0000","last_became_active":"2024-04-06T03:27:10.501454+0000","last_became_peered":"2024-04-06T03:27:10.501454+0000","last_unstale":"2024-04-06T03:33:51.362406+0000","last_undegraded":"2024-04-06T03:33:51.362406+0000","last_fullsized":"2024-04-06T03:33:51.362406+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:51.362253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.362253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:01.543224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":354,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.213324+0000","last_change":"2024-04-06T03:33:06.482662+0000","last_active":"2024-04-06T03:33:42.213324+0000","last_peered":"2024-04-06T03:33:42.213324+0000","last_clean":"2024-04-06T03:33:42.213324+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:33:42.213324+0000","last_undegraded":"2024-04-06T03:33:42.213324+0000","last_fullsized":"2024-04-06T03:33:42.213324+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:06.482503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:06.482503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:20.723359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"199'41","reported_seq":303,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882366+0000","last_change":"2024-04-06T03:31:51.059197+0000","last_active":"2024-04-06T03:33:41.882366+0000","last_peered":"2024-04-06T03:33:41.882366+0000","last_clean":"2024-04-06T03:33:41.882366+0000","last_became_active":"2024-04-06T03:26:18.616944+0000","last_became_peered":"2024-04-06T03:26:18.616944+0000","last_unstale":"2024-04-06T03:33:41.882366+0000","last_undegraded":"2024-04-06T03:33:41.882366+0000","last_fullsized":"2024-04-06T03:33:41.882366+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'41","last_scrub_stamp":"2024-04-06T03:31:51.059107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:51.059107+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"221'18065","reported_seq":18382,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.207134+0000","last_change":"2024-04-06T03:32:12.953532+0000","last_active":"2024-04-06T03:33:42.207134+0000","last_peered":"2024-04-06T03:33:42.207134+0000","last_clean":"2024-04-06T03:33:42.207134+0000","last_became_active":"2024-04-06T03:29:41.003109+0000","last_became_peered":"2024-04-06T03:29:41.003109+0000","last_unstale":"2024-04-06T03:33:42.207134+0000","last_undegraded":"2024-04-06T03:33:42.207134+0000","last_fullsized":"2024-04-06T03:33:42.207134+0000","mapping_epoch":182,"log_start":"210'12700","ondisk_log_start":"210'12700","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'13603","last_scrub_stamp":"2024-04-06T03:32:12.953411+0000","last_deep_scrub":"211'13603","last_deep_scrub_stamp":"2024-04-06T03:32:12.953411+0000","last_clean_scrub_stamp":"2024-04-06T03:32:12.953411+0000","objects_scrubbed":5255,"log_size":5365,"log_dups_size":0,"ondisk_log_size":5365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":35,"scrub_schedule":"queued for scrub","scrub_duration":34499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13134,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":334,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.209005+0000","last_change":"2024-04-06T03:31:59.850617+0000","last_active":"2024-04-06T03:33:42.209005+0000","last_peered":"2024-04-06T03:33:42.209005+0000","last_clean":"2024-04-06T03:33:42.209005+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:33:42.209005+0000","last_undegraded":"2024-04-06T03:33:42.209005+0000","last_fullsized":"2024-04-06T03:33:42.209005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:59.850525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:59.850525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":364,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.884073+0000","last_change":"2024-04-06T03:33:34.760395+0000","last_active":"2024-04-06T03:33:41.884073+0000","last_peered":"2024-04-06T03:33:41.884073+0000","last_clean":"2024-04-06T03:33:41.884073+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:33:41.884073+0000","last_undegraded":"2024-04-06T03:33:41.884073+0000","last_fullsized":"2024-04-06T03:33:41.884073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:34.760314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.760314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.747440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":364,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.885040+0000","last_change":"2024-04-06T03:33:10.359026+0000","last_active":"2024-04-06T03:33:41.885040+0000","last_peered":"2024-04-06T03:33:41.885040+0000","last_clean":"2024-04-06T03:33:41.885040+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:33:41.885040+0000","last_undegraded":"2024-04-06T03:33:41.885040+0000","last_fullsized":"2024-04-06T03:33:41.885040+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:10.358787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.358787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:33.659388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"199'39","reported_seq":407,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:51.057588+0000","last_change":"2024-04-06T03:33:51.057651+0000","last_active":"2024-04-06T03:33:51.057588+0000","last_peered":"2024-04-06T03:33:51.057588+0000","last_clean":"2024-04-06T03:33:51.057588+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:33:51.057588+0000","last_undegraded":"2024-04-06T03:33:51.057588+0000","last_fullsized":"2024-04-06T03:33:51.057588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:33:51.057496+0000","last_deep_scrub":"188'25","last_deep_scrub_stamp":"2024-04-06T03:30:07.204766+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.057496+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:09.212488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"221'13178","reported_seq":13293,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.571696+0000","last_change":"2024-04-06T03:33:14.187139+0000","last_active":"2024-04-06T03:33:55.571696+0000","last_peered":"2024-04-06T03:33:55.571696+0000","last_clean":"2024-04-06T03:33:55.571696+0000","last_became_active":"2024-04-06T03:29:41.012171+0000","last_became_peered":"2024-04-06T03:29:41.012171+0000","last_unstale":"2024-04-06T03:33:55.571696+0000","last_undegraded":"2024-04-06T03:33:55.571696+0000","last_fullsized":"2024-04-06T03:33:55.571696+0000","mapping_epoch":182,"log_start":"207'7800","ondisk_log_start":"207'7800","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'12111","last_scrub_stamp":"2024-04-06T03:33:14.186999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:33:14.186999+0000","objects_scrubbed":1067,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:33.954078+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13178,"num_write_kb":421696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":342,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206492+0000","last_change":"2024-04-06T03:33:11.574828+0000","last_active":"2024-04-06T03:33:42.206492+0000","last_peered":"2024-04-06T03:33:42.206492+0000","last_clean":"2024-04-06T03:33:42.206492+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:33:42.206492+0000","last_undegraded":"2024-04-06T03:33:42.206492+0000","last_fullsized":"2024-04-06T03:33:42.206492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:11.574716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:11.574716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:30.231399+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":364,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.889370+0000","last_change":"2024-04-06T03:33:41.788390+0000","last_active":"2024-04-06T03:33:41.889370+0000","last_peered":"2024-04-06T03:33:41.889370+0000","last_clean":"2024-04-06T03:33:41.889370+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:33:41.889370+0000","last_undegraded":"2024-04-06T03:33:41.889370+0000","last_fullsized":"2024-04-06T03:33:41.889370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.788284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.906520+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.788284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.869472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":370,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.212771+0000","last_change":"2024-04-06T03:33:36.374904+0000","last_active":"2024-04-06T03:33:42.212771+0000","last_peered":"2024-04-06T03:33:42.212771+0000","last_clean":"2024-04-06T03:33:42.212771+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:33:42.212771+0000","last_undegraded":"2024-04-06T03:33:42.212771+0000","last_fullsized":"2024-04-06T03:33:42.212771+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.374779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:31:40.521795+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.374779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:50.400465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"199'25","reported_seq":359,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-04-06T03:33:55.569173+0000","last_change":"2024-04-06T03:31:57.560644+0000","last_active":"2024-04-06T03:33:55.569173+0000","last_peered":"2024-04-06T03:33:55.569173+0000","last_clean":"2024-04-06T03:33:55.569173+0000","last_became_active":"2024-04-06T03:25:54.495696+0000","last_became_peered":"2024-04-06T03:25:54.495696+0000","last_unstale":"2024-04-06T03:33:55.569173+0000","last_undegraded":"2024-04-06T03:33:55.569173+0000","last_fullsized":"2024-04-06T03:33:55.569173+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'25","last_scrub_stamp":"2024-04-06T03:31:57.560377+0000","last_deep_scrub":"125'15","last_deep_scrub_stamp":"2024-04-06T03:25:15.970079+0000","last_clean_scrub_stamp":"2024-04-06T03:31:57.560377+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"221'13576","reported_seq":13734,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.882187+0000","last_change":"2024-04-06T03:31:54.372277+0000","last_active":"2024-04-06T03:33:41.882187+0000","last_peered":"2024-04-06T03:33:41.882187+0000","last_clean":"2024-04-06T03:33:41.882187+0000","last_became_active":"2024-04-06T03:29:40.994257+0000","last_became_peered":"2024-04-06T03:29:40.994257+0000","last_unstale":"2024-04-06T03:33:41.882187+0000","last_undegraded":"2024-04-06T03:33:41.882187+0000","last_fullsized":"2024-04-06T03:33:41.882187+0000","mapping_epoch":182,"log_start":"207'8600","ondisk_log_start":"207'8600","created":182,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8142","last_scrub_stamp":"2024-04-06T03:31:54.372103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:29:39.967518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:54.372103+0000","objects_scrubbed":5472,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13576,"num_write_kb":434432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":213,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.206120+0000","last_change":"2024-04-06T03:33:29.059858+0000","last_active":"2024-04-06T03:33:42.206120+0000","last_peered":"2024-04-06T03:33:42.206120+0000","last_clean":"2024-04-06T03:33:42.206120+0000","last_became_active":"2024-04-06T03:26:18.641975+0000","last_became_peered":"2024-04-06T03:26:18.641975+0000","last_unstale":"2024-04-06T03:33:42.206120+0000","last_undegraded":"2024-04-06T03:33:42.206120+0000","last_fullsized":"2024-04-06T03:33:42.206120+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:29.059614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.059614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.973063+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":372,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:41.887797+0000","last_change":"2024-04-06T03:33:18.399220+0000","last_active":"2024-04-06T03:33:41.887797+0000","last_peered":"2024-04-06T03:33:41.887797+0000","last_clean":"2024-04-06T03:33:41.887797+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:33:41.887797+0000","last_undegraded":"2024-04-06T03:33:41.887797+0000","last_fullsized":"2024-04-06T03:33:41.887797+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:18.398992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:33:18.398992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.757970+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"199'27","reported_seq":393,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:42.213678+0000","last_change":"2024-04-06T03:33:10.424862+0000","last_active":"2024-04-06T03:33:42.213678+0000","last_peered":"2024-04-06T03:33:42.213678+0000","last_clean":"2024-04-06T03:33:42.213678+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:33:42.213678+0000","last_undegraded":"2024-04-06T03:33:42.213678+0000","last_fullsized":"2024-04-06T03:33:42.213678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:33:10.424675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.424675+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:22.839071+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'12","reported_seq":514,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-04-06T03:33:46.813188+0000","last_change":"2024-04-06T03:32:06.805270+0000","last_active":"2024-04-06T03:33:46.813188+0000","last_peered":"2024-04-06T03:33:46.813188+0000","last_clean":"2024-04-06T03:33:46.813188+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:33:46.813188+0000","last_undegraded":"2024-04-06T03:33:46.813188+0000","last_fullsized":"2024-04-06T03:33:46.813188+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'12","last_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:32:06.805186+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":212618,"num_write_kb":6803713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85249,"ondisk_log_size":85249,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1414,"num_read_kb":1139,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":994,"ondisk_log_size":994,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":759,"num_read_kb":754,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743591,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189452,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":93555700,"statfs":{"total":95995035648,"available":95801036800,"internally_reserved":0,"allocated":1445888,"data_stored":1088126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":192543178},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136049,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185256,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93559896,"statfs":{"total":95995035648,"available":95805333504,"internally_reserved":0,"allocated":1409024,"data_stored":1086406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575078,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274932,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93470220,"statfs":{"total":95995035648,"available":95713505280,"internally_reserved":0,"allocated":831488,"data_stored":495892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":263036,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482116,"statfs":{"total":95995035648,"available":95725686784,"internally_reserved":0,"allocated":1429504,"data_stored":1087459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378844,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190252,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554900,"statfs":{"total":95995035648,"available":95800217600,"internally_reserved":0,"allocated":819200,"data_stored":495636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084496,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262432,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482720,"statfs":{"total":95995035648,"available":95726305280,"internally_reserved":0,"allocated":811008,"data_stored":495550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":267909575},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855559,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162508,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582644,"statfs":{"total":95995035648,"available":95828627456,"internally_reserved":0,"allocated":856064,"data_stored":496121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495805,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202724,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93542428,"statfs":{"total":95995035648,"available":95787446272,"internally_reserved":0,"allocated":815104,"data_stored":495532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:34:03.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:34:03.055 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-06T03:34:03.055 INFO:tasks.radosbench:Beginning radosbench... 2024-04-06T03:34:03.056 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-04-06T03:34:03.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool create unique_pool_2 16 2024-04-06T03:34:03.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:03.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:03.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:03.302 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:03.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:03.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:03.493 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:34:03.513 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:34:03.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:03.535 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:34:03.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:03.556 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:34:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:03.581 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:34:03.606 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:34:03.631 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:34:03.655 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:34:03.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:03.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:03 smithi050 bash[29221]: cluster 2024-04-06T03:34:02.247638+0000 mgr.y (mgr.24368) 484 : cluster 0 pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:03.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:03 smithi050 bash[29221]: audit 2024-04-06T03:34:03.476321+0000 mon.a (mon.0) 1315 : audit 0 from='client.? 172.21.15.50:0/269073111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:03.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:03 smithi050 bash[21467]: cluster 2024-04-06T03:34:02.247638+0000 mgr.y (mgr.24368) 484 : cluster 0 pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:03.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:03 smithi050 bash[21467]: audit 2024-04-06T03:34:03.476321+0000 mon.a (mon.0) 1315 : audit 0 from='client.? 172.21.15.50:0/269073111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:03.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:03.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:03.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:03 smithi153 bash[23852]: cluster 2024-04-06T03:34:02.247638+0000 mgr.y (mgr.24368) 484 : cluster 0 pgmap v499: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:03.795 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:03 smithi153 bash[23852]: audit 2024-04-06T03:34:03.476321+0000 mon.a (mon.0) 1315 : audit 0 from='client.? 172.21.15.50:0/269073111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:03.811 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:03.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:03.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:03.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:04.041 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:04.069 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:04.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:04.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:04.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:04.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:04.331 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:04.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:04.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:04.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:04.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:04.627 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:34:04.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:04.727 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: audit 2024-04-06T03:34:03.955294+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.50:0/611493260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: audit 2024-04-06T03:34:03.955652+0000 mon.a (mon.0) 1316 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.029572+0000 osd.7 (osd.7) 333 : cluster 0 5.9 scrub starts 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.032379+0000 osd.7 (osd.7) 334 : cluster 0 5.9 scrub ok 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: audit 2024-04-06T03:34:04.296023+0000 mon.a (mon.0) 1317 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.401608+0000 osd.0 (osd.0) 300 : cluster 0 3.6 scrub starts 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.406015+0000 osd.0 (osd.0) 301 : cluster 0 3.6 scrub ok 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: audit 2024-04-06T03:34:03.955294+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.50:0/611493260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:34:04.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: audit 2024-04-06T03:34:03.955652+0000 mon.a (mon.0) 1316 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:34:04.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.029572+0000 osd.7 (osd.7) 333 : cluster 0 5.9 scrub starts 2024-04-06T03:34:04.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.032379+0000 osd.7 (osd.7) 334 : cluster 0 5.9 scrub ok 2024-04-06T03:34:04.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: audit 2024-04-06T03:34:04.296023+0000 mon.a (mon.0) 1317 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:04.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.401608+0000 osd.0 (osd.0) 300 : cluster 0 3.6 scrub starts 2024-04-06T03:34:04.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:04 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.406015+0000 osd.0 (osd.0) 301 : cluster 0 3.6 scrub ok 2024-04-06T03:34:04.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:04.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:04.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:04.974 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: audit 2024-04-06T03:34:03.955294+0000 mon.c (mon.2) 69 : audit 1 from='client.? 172.21.15.50:0/611493260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:34:04.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: audit 2024-04-06T03:34:03.955652+0000 mon.a (mon.0) 1316 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:34:04.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.029572+0000 osd.7 (osd.7) 333 : cluster 0 5.9 scrub starts 2024-04-06T03:34:04.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.032379+0000 osd.7 (osd.7) 334 : cluster 0 5.9 scrub ok 2024-04-06T03:34:04.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: audit 2024-04-06T03:34:04.296023+0000 mon.a (mon.0) 1317 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:04.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.401608+0000 osd.0 (osd.0) 300 : cluster 0 3.6 scrub starts 2024-04-06T03:34:04.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:04 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.406015+0000 osd.0 (osd.0) 301 : cluster 0 3.6 scrub ok 2024-04-06T03:34:04.992 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:04.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:05.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:05.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:05.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:05.250 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:05.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:05.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:05.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:05.511 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:05.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:05.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:05.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:05.774 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:05.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:05 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.248357+0000 mgr.y (mgr.24368) 485 : cluster 0 pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:34:05.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:05 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.616847+0000 mon.a (mon.0) 1318 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:05.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:05 smithi153 bash[23852]: audit 2024-04-06T03:34:04.625318+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:34:05.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:05 smithi153 bash[23852]: cluster 2024-04-06T03:34:04.629539+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-04-06T03:34:05.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:05.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:05.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.248357+0000 mgr.y (mgr.24368) 485 : cluster 0 pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.616847+0000 mon.a (mon.0) 1318 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[29221]: audit 2024-04-06T03:34:04.625318+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[29221]: cluster 2024-04-06T03:34:04.629539+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.248357+0000 mgr.y (mgr.24368) 485 : cluster 0 pgmap v500: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.616847+0000 mon.a (mon.0) 1318 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[21467]: audit 2024-04-06T03:34:04.625318+0000 mon.a (mon.0) 1319 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:34:05.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:05 smithi050 bash[21467]: cluster 2024-04-06T03:34:04.629539+0000 mon.a (mon.0) 1320 : cluster 0 osdmap e228: 8 total, 8 up, 8 in 2024-04-06T03:34:06.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:06.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:06.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:06.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:06.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:06.293 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:06.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:06.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:06.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:06.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:06.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:06.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:34:06.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:06.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:06.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:06.945 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:06.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:07.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:07.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:07.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:07.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:07.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:07.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:07.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:07.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:07.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:07.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:07.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:34:07.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:07.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:07.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:07 smithi153 bash[23852]: cluster 2024-04-06T03:34:06.249415+0000 mgr.y (mgr.24368) 486 : cluster 0 pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:34:07.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:07.847 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:07.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:07.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:07 smithi050 bash[21467]: cluster 2024-04-06T03:34:06.249415+0000 mgr.y (mgr.24368) 486 : cluster 0 pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:34:07.962 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:07 smithi050 bash[29221]: cluster 2024-04-06T03:34:06.249415+0000 mgr.y (mgr.24368) 486 : cluster 0 pgmap v502: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:34:07.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:07.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:08.082 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:08.109 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:08.112 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:08.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:08.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:08.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:08.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:08.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:08.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:08.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:08.632 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:08.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:08.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:08 smithi050 bash[21467]: cluster 2024-04-06T03:34:07.527344+0000 osd.7 (osd.7) 335 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:08.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:08 smithi050 bash[21467]: cluster 2024-04-06T03:34:07.527425+0000 osd.7 (osd.7) 336 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:08.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:08 smithi050 bash[21467]: cluster 2024-04-06T03:34:07.634841+0000 mon.a (mon.0) 1321 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-04-06T03:34:08.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:08 smithi050 bash[29221]: cluster 2024-04-06T03:34:07.527344+0000 osd.7 (osd.7) 335 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:08.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:08 smithi050 bash[29221]: cluster 2024-04-06T03:34:07.527425+0000 osd.7 (osd.7) 336 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:08.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:08 smithi050 bash[29221]: cluster 2024-04-06T03:34:07.634841+0000 mon.a (mon.0) 1321 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-04-06T03:34:08.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:08.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:08.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:08.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:08.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:09.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:09.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:08 smithi153 bash[23852]: cluster 2024-04-06T03:34:07.527344+0000 osd.7 (osd.7) 335 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:09.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:08 smithi153 bash[23852]: cluster 2024-04-06T03:34:07.527425+0000 osd.7 (osd.7) 336 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:09.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:08 smithi153 bash[23852]: cluster 2024-04-06T03:34:07.634841+0000 mon.a (mon.0) 1321 : cluster 0 osdmap e229: 8 total, 8 up, 8 in 2024-04-06T03:34:09.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:09.130 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:09.154 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:09.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:09.283 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:09.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:09.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:09.538 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:34:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:34:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:34:09.538 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:09.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:09.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:09.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:34:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:09.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:09 smithi153 bash[23852]: cluster 2024-04-06T03:34:08.250080+0000 mgr.y (mgr.24368) 487 : cluster 0 pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:09.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:09.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:09 smithi050 bash[21467]: cluster 2024-04-06T03:34:08.250080+0000 mgr.y (mgr.24368) 487 : cluster 0 pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:09.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:09 smithi050 bash[29221]: cluster 2024-04-06T03:34:08.250080+0000 mgr.y (mgr.24368) 487 : cluster 0 pgmap v504: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:09.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:09.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:09.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:10.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:10.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:10.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:10.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:10.190 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:10.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:10.320 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:10.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:10.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:10.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:10.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:10.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:10.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:10 smithi153 bash[23852]: audit 2024-04-06T03:34:09.623021+0000 mgr.y (mgr.24368) 488 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:10.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:10.843 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:10.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:10.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:10 smithi050 bash[21467]: audit 2024-04-06T03:34:09.623021+0000 mgr.y (mgr.24368) 488 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:10.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:10 smithi050 bash[29221]: audit 2024-04-06T03:34:09.623021+0000 mgr.y (mgr.24368) 488 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:10.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:10.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:11.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:11.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:11.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:11.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:11.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:11.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:11.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:11.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:11.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:11.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:11 smithi050 bash[21467]: cluster 2024-04-06T03:34:10.250804+0000 mgr.y (mgr.24368) 489 : cluster 0 pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:11 smithi050 bash[21467]: cluster 2024-04-06T03:34:11.316043+0000 osd.1 (osd.1) 265 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:11 smithi050 bash[21467]: cluster 2024-04-06T03:34:11.316170+0000 osd.1 (osd.1) 266 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:11.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:11 smithi050 bash[29221]: cluster 2024-04-06T03:34:10.250804+0000 mgr.y (mgr.24368) 489 : cluster 0 pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:11.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:11 smithi050 bash[29221]: cluster 2024-04-06T03:34:11.316043+0000 osd.1 (osd.1) 265 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:11.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:11 smithi050 bash[29221]: cluster 2024-04-06T03:34:11.316170+0000 osd.1 (osd.1) 266 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:11.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:11.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:11.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:11 smithi153 bash[23852]: cluster 2024-04-06T03:34:10.250804+0000 mgr.y (mgr.24368) 489 : cluster 0 pgmap v505: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:11.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:11 smithi153 bash[23852]: cluster 2024-04-06T03:34:11.316043+0000 osd.1 (osd.1) 265 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:11.872 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:11 smithi153 bash[23852]: cluster 2024-04-06T03:34:11.316170+0000 osd.1 (osd.1) 266 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:11.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:11.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:11.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:12.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:12.123 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:12.150 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:12.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:12.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:12.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:12.410 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:12.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:12.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:12.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:12.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:12.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:12.803 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:12.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:12.917 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:12 smithi153 bash[23852]: cluster 2024-04-06T03:34:11.660258+0000 mon.a (mon.0) 1322 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-04-06T03:34:12.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:12.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:12.960 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:12.961 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:12.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:12 smithi050 bash[21467]: cluster 2024-04-06T03:34:11.660258+0000 mon.a (mon.0) 1322 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-04-06T03:34:12.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:12 smithi050 bash[29221]: cluster 2024-04-06T03:34:11.660258+0000 mon.a (mon.0) 1322 : cluster 0 osdmap e230: 8 total, 8 up, 8 in 2024-04-06T03:34:13.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:13.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:13.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:13.195 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:13.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:13.326 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:13.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:13.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:13.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:13.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:13.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:13.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:13.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:13.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:13 smithi050 bash[21467]: cluster 2024-04-06T03:34:12.251410+0000 mgr.y (mgr.24368) 490 : cluster 0 pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:13.775 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:13 smithi050 bash[29221]: cluster 2024-04-06T03:34:12.251410+0000 mgr.y (mgr.24368) 490 : cluster 0 pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:13.822 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:13.833 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:13 smithi153 bash[23852]: cluster 2024-04-06T03:34:12.251410+0000 mgr.y (mgr.24368) 490 : cluster 0 pgmap v507: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:34:13.849 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:13.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:13.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:13.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:14.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:14.108 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:14.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:14.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:14.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:14.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:14.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:14.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:14.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:14.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:14.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:14.895 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:14.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:14.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:15.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:15.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:15.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:15.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:15.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:15.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:15.428 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:15.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:15.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:15.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:15.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:15.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:15.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:15.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:15.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:15 smithi050 bash[21467]: cluster 2024-04-06T03:34:14.252272+0000 mgr.y (mgr.24368) 491 : cluster 0 pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:15.936 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:15 smithi050 bash[29221]: cluster 2024-04-06T03:34:14.252272+0000 mgr.y (mgr.24368) 491 : cluster 0 pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:15.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:15.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:16.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:15 smithi153 bash[23852]: cluster 2024-04-06T03:34:14.252272+0000 mgr.y (mgr.24368) 491 : cluster 0 pgmap v508: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:16.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:16.083 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:16.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:16.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:16.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:16.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:16.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:16.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:16.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:16.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:16.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:16.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:16.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:16.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:16.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:16.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:17.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:17.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:17.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:17.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:17.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:17.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:17.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:17.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:17.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:17.645 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:17.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:17.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:17.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:17.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:17 smithi153 bash[23852]: cluster 2024-04-06T03:34:16.253450+0000 mgr.y (mgr.24368) 492 : cluster 0 pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:17.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:17.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:17 smithi050 bash[21467]: cluster 2024-04-06T03:34:16.253450+0000 mgr.y (mgr.24368) 492 : cluster 0 pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:17.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:17 smithi050 bash[29221]: cluster 2024-04-06T03:34:16.253450+0000 mgr.y (mgr.24368) 492 : cluster 0 pgmap v509: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:17.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:18.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:18.040 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:18.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:18.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:18.300 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:18.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:18.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:18.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:18.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:18.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:18.694 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:18.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:18.821 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:18.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:18.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:18.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:19.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:19.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:19.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:19.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:19.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:19.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:19.451 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:34:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:34:19.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:19.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:19.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:19.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:19.626 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:34:19.626 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:34:19.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:19.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:19.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:34:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:19.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:19 smithi153 bash[23852]: cluster 2024-04-06T03:34:18.254113+0000 mgr.y (mgr.24368) 493 : cluster 0 pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:19.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:19 smithi153 bash[23852]: audit 2024-04-06T03:34:19.332875+0000 mon.a (mon.0) 1323 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:19.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:19 smithi153 bash[23852]: audit 2024-04-06T03:34:19.334320+0000 mon.a (mon.0) 1324 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:19.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:19.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[21467]: cluster 2024-04-06T03:34:18.254113+0000 mgr.y (mgr.24368) 493 : cluster 0 pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:19.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[21467]: audit 2024-04-06T03:34:19.332875+0000 mon.a (mon.0) 1323 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:19.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[21467]: audit 2024-04-06T03:34:19.334320+0000 mon.a (mon.0) 1324 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:19.847 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[29221]: cluster 2024-04-06T03:34:18.254113+0000 mgr.y (mgr.24368) 493 : cluster 0 pgmap v510: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:19.847 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[29221]: audit 2024-04-06T03:34:19.332875+0000 mon.a (mon.0) 1323 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:19.847 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:19 smithi050 bash[29221]: audit 2024-04-06T03:34:19.334320+0000 mon.a (mon.0) 1324 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:19.859 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:19.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:19.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:19.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:20.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:20.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:20.252 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:20.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:20.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:20.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:20.498 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:34:20.503 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:34:20.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:20.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:20.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:20.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:20.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:20.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:20.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:20.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:20 smithi050 bash[21467]: audit 2024-04-06T03:34:19.634278+0000 mgr.y (mgr.24368) 494 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:20.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:20 smithi050 bash[29221]: audit 2024-04-06T03:34:19.634278+0000 mgr.y (mgr.24368) 494 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:20.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:20.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:20.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:21.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:21.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:21.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:20 smithi153 bash[23852]: audit 2024-04-06T03:34:19.634278+0000 mgr.y (mgr.24368) 494 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:21.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:21.156 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:21.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:21.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:21.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:21.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:21.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:21.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:21.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:21.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:21.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:21 smithi050 bash[29221]: cluster 2024-04-06T03:34:20.254743+0000 mgr.y (mgr.24368) 495 : cluster 0 pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:21.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:21 smithi050 bash[29221]: audit 2024-04-06T03:34:20.467098+0000 mgr.y (mgr.24368) 496 : audit 0 from='client.24986 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:21.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:21 smithi050 bash[21467]: cluster 2024-04-06T03:34:20.254743+0000 mgr.y (mgr.24368) 495 : cluster 0 pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:21.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:21 smithi050 bash[21467]: audit 2024-04-06T03:34:20.467098+0000 mgr.y (mgr.24368) 496 : audit 0 from='client.24986 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:21.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:21.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:21.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:22.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:21 smithi153 bash[23852]: cluster 2024-04-06T03:34:20.254743+0000 mgr.y (mgr.24368) 495 : cluster 0 pgmap v511: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:21 smithi153 bash[23852]: audit 2024-04-06T03:34:20.467098+0000 mgr.y (mgr.24368) 496 : audit 0 from='client.24986 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:34:22.075 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:22.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-06T03:34:22.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:22.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:22.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:22.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:22.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:22.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:22.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:22.591 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:22.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:22.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:22.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:22.854 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:22.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:22.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:22.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:22.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:23.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:23.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:23.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:23.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:23.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:23.457 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:23.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:23.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:23.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:23.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:23.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:23.857 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_2' created 2024-04-06T03:34:23.862 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:23.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:23 smithi050 bash[21467]: cluster 2024-04-06T03:34:22.255690+0000 mgr.y (mgr.24368) 497 : cluster 0 pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-04-06T03:34:23.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:23 smithi050 bash[29221]: cluster 2024-04-06T03:34:22.255690+0000 mgr.y (mgr.24368) 497 : cluster 0 pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-04-06T03:34:23.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:23.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:23.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:24.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:24.099 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:23 smithi153 bash[23852]: cluster 2024-04-06T03:34:22.255690+0000 mgr.y (mgr.24368) 497 : cluster 0 pgmap v512: 132 pgs: 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-04-06T03:34:24.115 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:24.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:24.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:24.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:24.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:24.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:24.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:24.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:24.626 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-04-06T03:34:20.254324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2442,"num_read_kb":2256,"num_write":1093,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1242,"ondisk_log_size":1242,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1731676,"kb_used_data":7868,"kb_used_omap":12,"kb_used_meta":1723699,"kb_avail":748229540,"statfs":{"total":767960285184,"available":766187048960,"internally_reserved":0,"allocated":8056832,"data_stored":5471018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1765068337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004021"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":356,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228219+0000","last_change":"2024-04-06T03:33:10.115456+0000","last_active":"2024-04-06T03:34:13.228219+0000","last_peered":"2024-04-06T03:34:13.228219+0000","last_clean":"2024-04-06T03:34:13.228219+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:34:13.228219+0000","last_undegraded":"2024-04-06T03:34:13.228219+0000","last_fullsized":"2024-04-06T03:34:13.228219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:10.115343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.115343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:22.099127+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":360,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652309+0000","last_change":"2024-04-06T03:33:54.305360+0000","last_active":"2024-04-06T03:34:11.652309+0000","last_peered":"2024-04-06T03:34:11.652309+0000","last_clean":"2024-04-06T03:34:11.652309+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:34:11.652309+0000","last_undegraded":"2024-04-06T03:34:11.652309+0000","last_fullsized":"2024-04-06T03:34:11.652309+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:54.305146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:33:54.305146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:17.726287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":379,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.661290+0000","last_change":"2024-04-06T03:33:36.100803+0000","last_active":"2024-04-06T03:34:11.661290+0000","last_peered":"2024-04-06T03:34:11.661290+0000","last_clean":"2024-04-06T03:34:11.661290+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:34:11.661290+0000","last_undegraded":"2024-04-06T03:34:11.661290+0000","last_fullsized":"2024-04-06T03:34:11.661290+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.100494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.100494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:52.847869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"199'32","reported_seq":413,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229880+0000","last_change":"2024-04-06T03:33:18.029775+0000","last_active":"2024-04-06T03:34:13.229880+0000","last_peered":"2024-04-06T03:34:13.229880+0000","last_clean":"2024-04-06T03:34:13.229880+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:34:13.229880+0000","last_undegraded":"2024-04-06T03:34:13.229880+0000","last_fullsized":"2024-04-06T03:34:13.229880+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'32","last_scrub_stamp":"2024-04-06T03:33:18.029641+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:33:18.029641+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:35.199015+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":356,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652474+0000","last_change":"2024-04-06T03:31:26.012721+0000","last_active":"2024-04-06T03:34:11.652474+0000","last_peered":"2024-04-06T03:34:11.652474+0000","last_clean":"2024-04-06T03:34:11.652474+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:34:11.652474+0000","last_undegraded":"2024-04-06T03:34:11.652474+0000","last_fullsized":"2024-04-06T03:34:11.652474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:26.012621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:26.012621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282228+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":239,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223105+0000","last_change":"2024-04-06T03:33:24.344205+0000","last_active":"2024-04-06T03:34:13.223105+0000","last_peered":"2024-04-06T03:34:13.223105+0000","last_clean":"2024-04-06T03:34:13.223105+0000","last_became_active":"2024-04-06T03:25:54.480600+0000","last_became_peered":"2024-04-06T03:25:54.480600+0000","last_unstale":"2024-04-06T03:34:13.223105+0000","last_undegraded":"2024-04-06T03:34:13.223105+0000","last_fullsized":"2024-04-06T03:34:13.223105+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:24.344088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.344088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:24.660837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":369,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229682+0000","last_change":"2024-04-06T03:33:39.516560+0000","last_active":"2024-04-06T03:34:13.229682+0000","last_peered":"2024-04-06T03:34:13.229682+0000","last_clean":"2024-04-06T03:34:13.229682+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:34:13.229682+0000","last_undegraded":"2024-04-06T03:34:13.229682+0000","last_fullsized":"2024-04-06T03:34:13.229682+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:39.516225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:33:39.516225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:56.903888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"199'31","reported_seq":412,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.661154+0000","last_change":"2024-04-06T03:33:34.113495+0000","last_active":"2024-04-06T03:34:11.661154+0000","last_peered":"2024-04-06T03:34:11.661154+0000","last_clean":"2024-04-06T03:34:11.661154+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:34:11.661154+0000","last_undegraded":"2024-04-06T03:34:11.661154+0000","last_fullsized":"2024-04-06T03:34:11.661154+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:33:34.113196+0000","last_deep_scrub":"125'23","last_deep_scrub_stamp":"2024-04-06T03:25:17.013823+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.113196+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:34.184916+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":243,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598257+0000","last_change":"2024-04-06T03:33:26.400774+0000","last_active":"2024-04-06T03:34:12.598257+0000","last_peered":"2024-04-06T03:34:12.598257+0000","last_clean":"2024-04-06T03:34:12.598257+0000","last_became_active":"2024-04-06T03:25:54.484120+0000","last_became_peered":"2024-04-06T03:25:54.484120+0000","last_unstale":"2024-04-06T03:34:12.598257+0000","last_undegraded":"2024-04-06T03:34:12.598257+0000","last_fullsized":"2024-04-06T03:34:12.598257+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:26.400649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:26.400649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:54.665737+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":358,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222653+0000","last_change":"2024-04-06T03:33:29.327349+0000","last_active":"2024-04-06T03:34:13.222653+0000","last_peered":"2024-04-06T03:34:13.222653+0000","last_clean":"2024-04-06T03:34:13.222653+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:34:13.222653+0000","last_undegraded":"2024-04-06T03:34:13.222653+0000","last_fullsized":"2024-04-06T03:34:13.222653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:29.327145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.327145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:57.257491+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":382,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224178+0000","last_change":"2024-04-06T03:33:48.097842+0000","last_active":"2024-04-06T03:34:13.224178+0000","last_peered":"2024-04-06T03:34:13.224178+0000","last_clean":"2024-04-06T03:34:13.224178+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:34:13.224178+0000","last_undegraded":"2024-04-06T03:34:13.224178+0000","last_fullsized":"2024-04-06T03:34:13.224178+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:33:48.097599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:48.097599+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:14.712480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"199'34","reported_seq":364,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653331+0000","last_change":"2024-04-06T03:33:55.304122+0000","last_active":"2024-04-06T03:34:11.653331+0000","last_peered":"2024-04-06T03:34:11.653331+0000","last_clean":"2024-04-06T03:34:11.653331+0000","last_became_active":"2024-04-06T03:25:54.489462+0000","last_became_peered":"2024-04-06T03:25:54.489462+0000","last_unstale":"2024-04-06T03:34:11.653331+0000","last_undegraded":"2024-04-06T03:34:11.653331+0000","last_fullsized":"2024-04-06T03:34:11.653331+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'34","last_scrub_stamp":"2024-04-06T03:33:55.303979+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:33:55.303979+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:19.792669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":354,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658661+0000","last_change":"2024-04-06T03:33:39.245506+0000","last_active":"2024-04-06T03:34:11.658661+0000","last_peered":"2024-04-06T03:34:11.658661+0000","last_clean":"2024-04-06T03:34:11.658661+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:34:11.658661+0000","last_undegraded":"2024-04-06T03:34:11.658661+0000","last_fullsized":"2024-04-06T03:34:11.658661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:39.245352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:39.245352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:03.800079+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":235,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.659926+0000","last_change":"2024-04-06T03:33:13.537781+0000","last_active":"2024-04-06T03:34:11.659926+0000","last_peered":"2024-04-06T03:34:11.659926+0000","last_clean":"2024-04-06T03:34:11.659926+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:34:11.659926+0000","last_undegraded":"2024-04-06T03:34:11.659926+0000","last_fullsized":"2024-04-06T03:34:11.659926+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.537704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.537704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.198431+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":366,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222429+0000","last_change":"2024-04-06T03:33:34.322488+0000","last_active":"2024-04-06T03:34:13.222429+0000","last_peered":"2024-04-06T03:34:13.222429+0000","last_clean":"2024-04-06T03:34:13.222429+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:34:13.222429+0000","last_undegraded":"2024-04-06T03:34:13.222429+0000","last_fullsized":"2024-04-06T03:34:13.222429+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:34.322401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:46.087127+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.322401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:37.775476+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"199'30","reported_seq":334,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653442+0000","last_change":"2024-04-06T03:33:09.484528+0000","last_active":"2024-04-06T03:34:11.653442+0000","last_peered":"2024-04-06T03:34:11.653442+0000","last_clean":"2024-04-06T03:34:11.653442+0000","last_became_active":"2024-04-06T03:25:54.483976+0000","last_became_peered":"2024-04-06T03:25:54.483976+0000","last_unstale":"2024-04-06T03:34:11.653442+0000","last_undegraded":"2024-04-06T03:34:11.653442+0000","last_fullsized":"2024-04-06T03:34:11.653442+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'30","last_scrub_stamp":"2024-04-06T03:33:09.484404+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.484404+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:28.202176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":357,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229102+0000","last_change":"2024-04-06T03:33:37.571108+0000","last_active":"2024-04-06T03:34:13.229102+0000","last_peered":"2024-04-06T03:34:13.229102+0000","last_clean":"2024-04-06T03:34:13.229102+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:34:13.229102+0000","last_undegraded":"2024-04-06T03:34:13.229102+0000","last_fullsized":"2024-04-06T03:34:13.229102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.570981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.570981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:55.343543+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":368,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652997+0000","last_change":"2024-04-06T03:31:37.182250+0000","last_active":"2024-04-06T03:34:11.652997+0000","last_peered":"2024-04-06T03:34:11.652997+0000","last_clean":"2024-04-06T03:34:11.652997+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:34:11.652997+0000","last_undegraded":"2024-04-06T03:34:11.652997+0000","last_fullsized":"2024-04-06T03:34:11.652997+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:37.182127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:31:37.182127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":360,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228504+0000","last_change":"2024-04-06T03:31:54.579860+0000","last_active":"2024-04-06T03:34:13.228504+0000","last_peered":"2024-04-06T03:34:13.228504+0000","last_clean":"2024-04-06T03:34:13.228504+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:34:13.228504+0000","last_undegraded":"2024-04-06T03:34:13.228504+0000","last_fullsized":"2024-04-06T03:34:13.228504+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:54.579728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:54.579728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"199'33","reported_seq":427,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652564+0000","last_change":"2024-04-06T03:33:21.449448+0000","last_active":"2024-04-06T03:34:11.652564+0000","last_peered":"2024-04-06T03:34:11.652564+0000","last_clean":"2024-04-06T03:34:11.652564+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:34:11.652564+0000","last_undegraded":"2024-04-06T03:34:11.652564+0000","last_fullsized":"2024-04-06T03:34:11.652564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:33:21.449361+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:30:12.438411+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.449361+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.839536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":357,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653718+0000","last_change":"2024-04-06T03:33:15.602204+0000","last_active":"2024-04-06T03:34:11.653718+0000","last_peered":"2024-04-06T03:34:11.653718+0000","last_clean":"2024-04-06T03:34:11.653718+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:34:11.653718+0000","last_undegraded":"2024-04-06T03:34:11.653718+0000","last_fullsized":"2024-04-06T03:34:11.653718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:15.602002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.602002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.728882+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":366,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598679+0000","last_change":"2024-04-06T03:33:25.355380+0000","last_active":"2024-04-06T03:34:12.598679+0000","last_peered":"2024-04-06T03:34:12.598679+0000","last_clean":"2024-04-06T03:34:12.598679+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:34:12.598679+0000","last_undegraded":"2024-04-06T03:34:12.598679+0000","last_fullsized":"2024-04-06T03:34:12.598679+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:25.355264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:33:25.355264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.984353+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":221,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652654+0000","last_change":"2024-04-06T03:31:10.147767+0000","last_active":"2024-04-06T03:34:11.652654+0000","last_peered":"2024-04-06T03:34:11.652654+0000","last_clean":"2024-04-06T03:34:11.652654+0000","last_became_active":"2024-04-06T03:26:18.617121+0000","last_became_peered":"2024-04-06T03:26:18.617121+0000","last_unstale":"2024-04-06T03:34:11.652654+0000","last_undegraded":"2024-04-06T03:34:11.652654+0000","last_fullsized":"2024-04-06T03:34:11.652654+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:10.147685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:10.147685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"224'16","reported_seq":398,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.661032+0000","last_change":"2024-04-06T03:33:51.165215+0000","last_active":"2024-04-06T03:34:11.661032+0000","last_peered":"2024-04-06T03:34:11.661032+0000","last_clean":"2024-04-06T03:34:11.661032+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:34:11.661032+0000","last_undegraded":"2024-04-06T03:34:11.661032+0000","last_fullsized":"2024-04-06T03:34:11.661032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16","last_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.165059+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.035906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":238,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653000+0000","last_change":"2024-04-06T03:33:23.526390+0000","last_active":"2024-04-06T03:34:11.653000+0000","last_peered":"2024-04-06T03:34:11.653000+0000","last_clean":"2024-04-06T03:34:11.653000+0000","last_became_active":"2024-04-06T03:25:54.483894+0000","last_became_peered":"2024-04-06T03:25:54.483894+0000","last_unstale":"2024-04-06T03:34:11.653000+0000","last_undegraded":"2024-04-06T03:34:11.653000+0000","last_fullsized":"2024-04-06T03:34:11.653000+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.526306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.526306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.600671+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":359,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660204+0000","last_change":"2024-04-06T03:31:52.591764+0000","last_active":"2024-04-06T03:34:11.660204+0000","last_peered":"2024-04-06T03:34:11.660204+0000","last_clean":"2024-04-06T03:34:11.660204+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:34:11.660204+0000","last_undegraded":"2024-04-06T03:34:11.660204+0000","last_fullsized":"2024-04-06T03:34:11.660204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:52.591670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:31:52.591670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.281940+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":361,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227675+0000","last_change":"2024-04-06T03:31:58.603076+0000","last_active":"2024-04-06T03:34:13.227675+0000","last_peered":"2024-04-06T03:34:13.227675+0000","last_clean":"2024-04-06T03:34:13.227675+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:34:13.227675+0000","last_undegraded":"2024-04-06T03:34:13.227675+0000","last_fullsized":"2024-04-06T03:34:13.227675+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:31:58.602934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:58.602934+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419713+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"199'26","reported_seq":398,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227358+0000","last_change":"2024-04-06T03:33:30.273551+0000","last_active":"2024-04-06T03:34:13.227358+0000","last_peered":"2024-04-06T03:34:13.227358+0000","last_clean":"2024-04-06T03:34:13.227358+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:34:13.227358+0000","last_undegraded":"2024-04-06T03:34:13.227358+0000","last_fullsized":"2024-04-06T03:34:13.227358+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'26","last_scrub_stamp":"2024-04-06T03:33:30.273225+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.273225+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:58.185138+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":357,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224095+0000","last_change":"2024-04-06T03:33:46.163699+0000","last_active":"2024-04-06T03:34:13.224095+0000","last_peered":"2024-04-06T03:34:13.224095+0000","last_clean":"2024-04-06T03:34:13.224095+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:34:13.224095+0000","last_undegraded":"2024-04-06T03:34:13.224095+0000","last_fullsized":"2024-04-06T03:34:13.224095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:33:46.163357+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:31:51.905353+0000","last_clean_scrub_stamp":"2024-04-06T03:33:46.163357+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.363841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"139'13","reported_seq":408,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.832829+0000","last_change":"2024-04-06T03:33:58.297492+0000","last_active":"2024-04-06T03:34:11.832829+0000","last_peered":"2024-04-06T03:34:11.832829+0000","last_clean":"2024-04-06T03:34:11.832829+0000","last_became_active":"2024-04-06T03:26:18.615209+0000","last_became_peered":"2024-04-06T03:26:18.615209+0000","last_unstale":"2024-04-06T03:34:11.832829+0000","last_undegraded":"2024-04-06T03:34:11.832829+0000","last_fullsized":"2024-04-06T03:34:11.832829+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'13","last_scrub_stamp":"2024-04-06T03:33:58.297262+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:33:58.297262+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:23.424504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":235,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652266+0000","last_change":"2024-04-06T03:31:25.010990+0000","last_active":"2024-04-06T03:34:11.652266+0000","last_peered":"2024-04-06T03:34:11.652266+0000","last_clean":"2024-04-06T03:34:11.652266+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:34:11.652266+0000","last_undegraded":"2024-04-06T03:34:11.652266+0000","last_fullsized":"2024-04-06T03:34:11.652266+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:25.010770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:31:25.010770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'62","reported_seq":472,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.597849+0000","last_change":"2024-04-06T03:33:22.362330+0000","last_active":"2024-04-06T03:34:12.597849+0000","last_peered":"2024-04-06T03:34:12.597849+0000","last_clean":"2024-04-06T03:34:12.597849+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:34:12.597849+0000","last_undegraded":"2024-04-06T03:34:12.597849+0000","last_fullsized":"2024-04-06T03:34:12.597849+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'62","last_scrub_stamp":"2024-04-06T03:33:22.362144+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:33:22.362144+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:34.254236+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":348,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229092+0000","last_change":"2024-04-06T03:32:07.755529+0000","last_active":"2024-04-06T03:34:13.229092+0000","last_peered":"2024-04-06T03:34:13.229092+0000","last_clean":"2024-04-06T03:34:13.229092+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:34:13.229092+0000","last_undegraded":"2024-04-06T03:34:13.229092+0000","last_fullsized":"2024-04-06T03:34:13.229092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:07.755302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:32:07.755302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419733+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":358,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224369+0000","last_change":"2024-04-06T03:33:13.378754+0000","last_active":"2024-04-06T03:34:13.224369+0000","last_peered":"2024-04-06T03:34:13.224369+0000","last_clean":"2024-04-06T03:34:13.224369+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:34:13.224369+0000","last_undegraded":"2024-04-06T03:34:13.224369+0000","last_fullsized":"2024-04-06T03:34:13.224369+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.378611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.000214+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.378611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:27.510378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":237,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658712+0000","last_change":"2024-04-06T03:33:15.332805+0000","last_active":"2024-04-06T03:34:11.658712+0000","last_peered":"2024-04-06T03:34:11.658712+0000","last_clean":"2024-04-06T03:34:11.658712+0000","last_became_active":"2024-04-06T03:26:18.635349+0000","last_became_peered":"2024-04-06T03:26:18.635349+0000","last_unstale":"2024-04-06T03:34:11.658712+0000","last_undegraded":"2024-04-06T03:34:11.658712+0000","last_fullsized":"2024-04-06T03:34:11.658712+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:15.332597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.332597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.883739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"199'27","reported_seq":295,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224437+0000","last_change":"2024-04-06T03:31:50.901734+0000","last_active":"2024-04-06T03:34:13.224437+0000","last_peered":"2024-04-06T03:34:13.224437+0000","last_clean":"2024-04-06T03:34:13.224437+0000","last_became_active":"2024-04-06T03:25:54.491804+0000","last_became_peered":"2024-04-06T03:25:54.491804+0000","last_unstale":"2024-04-06T03:34:13.224437+0000","last_undegraded":"2024-04-06T03:34:13.224437+0000","last_fullsized":"2024-04-06T03:34:13.224437+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:31:50.901646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:50.901646+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":354,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222468+0000","last_change":"2024-04-06T03:33:21.336526+0000","last_active":"2024-04-06T03:34:13.222468+0000","last_peered":"2024-04-06T03:34:13.222468+0000","last_clean":"2024-04-06T03:34:13.222468+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:34:13.222468+0000","last_undegraded":"2024-04-06T03:34:13.222468+0000","last_fullsized":"2024-04-06T03:34:13.222468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:21.336152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.336152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.892584+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":230,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652308+0000","last_change":"2024-04-06T03:33:45.381648+0000","last_active":"2024-04-06T03:34:11.652308+0000","last_peered":"2024-04-06T03:34:11.652308+0000","last_clean":"2024-04-06T03:34:11.652308+0000","last_became_active":"2024-04-06T03:26:18.618151+0000","last_became_peered":"2024-04-06T03:26:18.618151+0000","last_unstale":"2024-04-06T03:34:11.652308+0000","last_undegraded":"2024-04-06T03:34:11.652308+0000","last_fullsized":"2024-04-06T03:34:11.652308+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:45.381432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:45.381432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:04.135911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":240,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658504+0000","last_change":"2024-04-06T03:33:20.475825+0000","last_active":"2024-04-06T03:34:11.658504+0000","last_peered":"2024-04-06T03:34:11.658504+0000","last_clean":"2024-04-06T03:34:11.658504+0000","last_became_active":"2024-04-06T03:26:18.637825+0000","last_became_peered":"2024-04-06T03:26:18.637825+0000","last_unstale":"2024-04-06T03:34:11.658504+0000","last_undegraded":"2024-04-06T03:34:11.658504+0000","last_fullsized":"2024-04-06T03:34:11.658504+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:20.475708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.475708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:44.828060+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"199'22","reported_seq":404,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660991+0000","last_change":"2024-04-06T03:33:37.125222+0000","last_active":"2024-04-06T03:34:11.660991+0000","last_peered":"2024-04-06T03:34:11.660991+0000","last_clean":"2024-04-06T03:34:11.660991+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:34:11.660991+0000","last_undegraded":"2024-04-06T03:34:11.660991+0000","last_fullsized":"2024-04-06T03:34:11.660991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'22","last_scrub_stamp":"2024-04-06T03:33:37.125033+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.125033+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:45.455271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":348,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227507+0000","last_change":"2024-04-06T03:32:02.669696+0000","last_active":"2024-04-06T03:34:13.227507+0000","last_peered":"2024-04-06T03:34:13.227507+0000","last_clean":"2024-04-06T03:34:13.227507+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:34:13.227507+0000","last_undegraded":"2024-04-06T03:34:13.227507+0000","last_fullsized":"2024-04-06T03:34:13.227507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:02.669327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:32:02.669327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419753+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":366,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598407+0000","last_change":"2024-04-06T03:33:24.363187+0000","last_active":"2024-04-06T03:34:12.598407+0000","last_peered":"2024-04-06T03:34:12.598407+0000","last_clean":"2024-04-06T03:34:12.598407+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:34:12.598407+0000","last_undegraded":"2024-04-06T03:34:12.598407+0000","last_fullsized":"2024-04-06T03:34:12.598407+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:24.363062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.363062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:45.096055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":382,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598388+0000","last_change":"2024-04-06T03:33:30.460549+0000","last_active":"2024-04-06T03:34:12.598388+0000","last_peered":"2024-04-06T03:34:12.598388+0000","last_clean":"2024-04-06T03:34:12.598388+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:34:12.598388+0000","last_undegraded":"2024-04-06T03:34:12.598388+0000","last_fullsized":"2024-04-06T03:34:12.598388+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:30.460430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.460430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.441443+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"199'33","reported_seq":427,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653257+0000","last_change":"2024-04-06T03:33:53.327065+0000","last_active":"2024-04-06T03:34:11.653257+0000","last_peered":"2024-04-06T03:34:11.653257+0000","last_clean":"2024-04-06T03:34:11.653257+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:34:11.653257+0000","last_undegraded":"2024-04-06T03:34:11.653257+0000","last_fullsized":"2024-04-06T03:34:11.653257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:33:53.326886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.326886+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:18.112004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":354,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658096+0000","last_change":"2024-04-06T03:33:38.193397+0000","last_active":"2024-04-06T03:34:11.658096+0000","last_peered":"2024-04-06T03:34:11.658096+0000","last_clean":"2024-04-06T03:34:11.658096+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:34:11.658096+0000","last_undegraded":"2024-04-06T03:34:11.658096+0000","last_fullsized":"2024-04-06T03:34:11.658096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:38.193266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:38.193266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:04.273565+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":352,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228806+0000","last_change":"2024-04-06T03:31:35.385717+0000","last_active":"2024-04-06T03:34:13.228806+0000","last_peered":"2024-04-06T03:34:13.228806+0000","last_clean":"2024-04-06T03:34:13.228806+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:34:13.228806+0000","last_undegraded":"2024-04-06T03:34:13.228806+0000","last_fullsized":"2024-04-06T03:34:13.228806+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:35.385380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_clean_scrub_stamp":"2024-04-06T03:31:35.385380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419643+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":221,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229130+0000","last_change":"2024-04-06T03:33:20.106580+0000","last_active":"2024-04-06T03:34:13.229130+0000","last_peered":"2024-04-06T03:34:13.229130+0000","last_clean":"2024-04-06T03:34:13.229130+0000","last_became_active":"2024-04-06T03:26:18.643806+0000","last_became_peered":"2024-04-06T03:26:18.643806+0000","last_unstale":"2024-04-06T03:34:13.229130+0000","last_undegraded":"2024-04-06T03:34:13.229130+0000","last_fullsized":"2024-04-06T03:34:13.229130+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:20.106508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.106508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:34.460864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"199'21","reported_seq":399,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224348+0000","last_change":"2024-04-06T03:33:44.226611+0000","last_active":"2024-04-06T03:34:13.224348+0000","last_peered":"2024-04-06T03:34:13.224348+0000","last_clean":"2024-04-06T03:34:13.224348+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:34:13.224348+0000","last_undegraded":"2024-04-06T03:34:13.224348+0000","last_fullsized":"2024-04-06T03:34:13.224348+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:33:44.226337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:44.226337+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:56.430775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":364,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653741+0000","last_change":"2024-04-06T03:33:37.480817+0000","last_active":"2024-04-06T03:34:11.653741+0000","last_peered":"2024-04-06T03:34:11.653741+0000","last_clean":"2024-04-06T03:34:11.653741+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:34:11.653741+0000","last_undegraded":"2024-04-06T03:34:11.653741+0000","last_fullsized":"2024-04-06T03:34:11.653741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.480549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.480549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:03.468270+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":524,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229300+0000","last_change":"2024-04-06T03:33:31.421434+0000","last_active":"2024-04-06T03:34:13.229300+0000","last_peered":"2024-04-06T03:34:13.229300+0000","last_clean":"2024-04-06T03:34:13.229300+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:34:13.229300+0000","last_undegraded":"2024-04-06T03:34:13.229300+0000","last_fullsized":"2024-04-06T03:34:13.229300+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:33:31.421277+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:33:31.421277+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:47.402352+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":366,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.661040+0000","last_change":"2024-04-06T03:33:23.775542+0000","last_active":"2024-04-06T03:34:11.661040+0000","last_peered":"2024-04-06T03:34:11.661040+0000","last_clean":"2024-04-06T03:34:11.661040+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:34:11.661040+0000","last_undegraded":"2024-04-06T03:34:11.661040+0000","last_fullsized":"2024-04-06T03:34:11.661040+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.775408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.775408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:43.932054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"199'15","reported_seq":387,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660038+0000","last_change":"2024-04-06T03:33:09.505741+0000","last_active":"2024-04-06T03:34:11.660038+0000","last_peered":"2024-04-06T03:34:11.660038+0000","last_clean":"2024-04-06T03:34:11.660038+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:34:11.660038+0000","last_undegraded":"2024-04-06T03:34:11.660038+0000","last_fullsized":"2024-04-06T03:34:11.660038+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'15","last_scrub_stamp":"2024-04-06T03:33:09.505412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.505412+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:11.838492+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":340,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.661069+0000","last_change":"2024-04-06T03:31:50.701295+0000","last_active":"2024-04-06T03:34:11.661069+0000","last_peered":"2024-04-06T03:34:11.661069+0000","last_clean":"2024-04-06T03:34:11.661069+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:34:11.661069+0000","last_undegraded":"2024-04-06T03:34:11.661069+0000","last_fullsized":"2024-04-06T03:34:11.661069+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:50.701170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:31:50.701170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.281875+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"154'13","reported_seq":302,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.661003+0000","last_change":"2024-04-06T03:33:19.694033+0000","last_active":"2024-04-06T03:34:11.661003+0000","last_peered":"2024-04-06T03:34:11.661003+0000","last_clean":"2024-04-06T03:34:11.661003+0000","last_became_active":"2024-04-06T03:27:10.498254+0000","last_became_peered":"2024-04-06T03:27:10.498254+0000","last_unstale":"2024-04-06T03:34:11.661003+0000","last_undegraded":"2024-04-06T03:34:11.661003+0000","last_fullsized":"2024-04-06T03:34:11.661003+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'13","last_scrub_stamp":"2024-04-06T03:33:19.693903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:19.693903+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:44.105035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":233,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228318+0000","last_change":"2024-04-06T03:33:42.516624+0000","last_active":"2024-04-06T03:34:13.228318+0000","last_peered":"2024-04-06T03:34:13.228318+0000","last_clean":"2024-04-06T03:34:13.228318+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:34:13.228318+0000","last_undegraded":"2024-04-06T03:34:13.228318+0000","last_fullsized":"2024-04-06T03:34:13.228318+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_clean_scrub_stamp":"2024-04-06T03:33:42.516373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:02.496309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"199'24","reported_seq":398,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222986+0000","last_change":"2024-04-06T03:33:15.375572+0000","last_active":"2024-04-06T03:34:13.222986+0000","last_peered":"2024-04-06T03:34:13.222986+0000","last_clean":"2024-04-06T03:34:13.222986+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:34:13.222986+0000","last_undegraded":"2024-04-06T03:34:13.222986+0000","last_fullsized":"2024-04-06T03:34:13.222986+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:33:15.375394+0000","last_deep_scrub":"168'16","last_deep_scrub_stamp":"2024-04-06T03:28:39.973804+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.375394+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:36.915506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":362,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223924+0000","last_change":"2024-04-06T03:33:55.107549+0000","last_active":"2024-04-06T03:34:13.223924+0000","last_peered":"2024-04-06T03:34:13.223924+0000","last_clean":"2024-04-06T03:34:13.223924+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:34:13.223924+0000","last_undegraded":"2024-04-06T03:34:13.223924+0000","last_fullsized":"2024-04-06T03:34:13.223924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:55.107432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:33:55.107432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:12.588476+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":227,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229436+0000","last_change":"2024-04-06T03:33:46.541535+0000","last_active":"2024-04-06T03:34:13.229436+0000","last_peered":"2024-04-06T03:34:13.229436+0000","last_clean":"2024-04-06T03:34:13.229436+0000","last_became_active":"2024-04-06T03:25:54.485733+0000","last_became_peered":"2024-04-06T03:25:54.485733+0000","last_unstale":"2024-04-06T03:34:13.229436+0000","last_undegraded":"2024-04-06T03:34:13.229436+0000","last_fullsized":"2024-04-06T03:34:13.229436+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:46.541240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:46.541240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.998515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":221,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652814+0000","last_change":"2024-04-06T03:31:23.008787+0000","last_active":"2024-04-06T03:34:11.652814+0000","last_peered":"2024-04-06T03:34:11.652814+0000","last_clean":"2024-04-06T03:34:11.652814+0000","last_became_active":"2024-04-06T03:26:18.615468+0000","last_became_peered":"2024-04-06T03:26:18.615468+0000","last_unstale":"2024-04-06T03:34:11.652814+0000","last_undegraded":"2024-04-06T03:34:11.652814+0000","last_fullsized":"2024-04-06T03:34:11.652814+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:23.008712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:31:23.008712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"199'10","reported_seq":316,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.659149+0000","last_change":"2024-04-06T03:33:15.548008+0000","last_active":"2024-04-06T03:34:11.659149+0000","last_peered":"2024-04-06T03:34:11.659149+0000","last_clean":"2024-04-06T03:34:11.659149+0000","last_became_active":"2024-04-06T03:26:18.639720+0000","last_became_peered":"2024-04-06T03:26:18.639720+0000","last_unstale":"2024-04-06T03:34:11.659149+0000","last_undegraded":"2024-04-06T03:34:11.659149+0000","last_fullsized":"2024-04-06T03:34:11.659149+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10","last_scrub_stamp":"2024-04-06T03:33:15.547658+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.547658+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:30.362669+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":374,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658252+0000","last_change":"2024-04-06T03:33:32.787376+0000","last_active":"2024-04-06T03:34:11.658252+0000","last_peered":"2024-04-06T03:34:11.658252+0000","last_clean":"2024-04-06T03:34:11.658252+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:34:11.658252+0000","last_undegraded":"2024-04-06T03:34:11.658252+0000","last_fullsized":"2024-04-06T03:34:11.658252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:33:32.787297+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:33:32.787297+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:59.143038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":352,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229343+0000","last_change":"2024-04-06T03:31:27.375923+0000","last_active":"2024-04-06T03:34:13.229343+0000","last_peered":"2024-04-06T03:34:13.229343+0000","last_clean":"2024-04-06T03:34:13.229343+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:34:13.229343+0000","last_undegraded":"2024-04-06T03:34:13.229343+0000","last_fullsized":"2024-04-06T03:34:13.229343+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:27.375797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:31:27.375797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419557+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":384,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.657398+0000","last_change":"2024-04-06T03:33:13.288997+0000","last_active":"2024-04-06T03:34:11.657398+0000","last_peered":"2024-04-06T03:34:11.657398+0000","last_clean":"2024-04-06T03:34:11.657398+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:34:11.657398+0000","last_undegraded":"2024-04-06T03:34:11.657398+0000","last_fullsized":"2024-04-06T03:34:11.657398+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.288879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.288879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:20.772672+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"199'31","reported_seq":419,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222937+0000","last_change":"2024-04-06T03:33:58.072257+0000","last_active":"2024-04-06T03:34:13.222937+0000","last_peered":"2024-04-06T03:34:13.222937+0000","last_clean":"2024-04-06T03:34:13.222937+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:34:13.222937+0000","last_undegraded":"2024-04-06T03:34:13.222937+0000","last_fullsized":"2024-04-06T03:34:13.222937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:33:58.072001+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:33:58.072001+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:13.483390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":348,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228880+0000","last_change":"2024-04-06T03:31:07.223520+0000","last_active":"2024-04-06T03:34:13.228880+0000","last_peered":"2024-04-06T03:34:13.228880+0000","last_clean":"2024-04-06T03:34:13.228880+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:34:13.228880+0000","last_undegraded":"2024-04-06T03:34:13.228880+0000","last_fullsized":"2024-04-06T03:34:13.228880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:07.223320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:07.223320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419508+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":394,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653474+0000","last_change":"2024-04-06T03:33:14.584324+0000","last_active":"2024-04-06T03:34:11.653474+0000","last_peered":"2024-04-06T03:34:11.653474+0000","last_clean":"2024-04-06T03:34:11.653474+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:34:11.653474+0000","last_undegraded":"2024-04-06T03:34:11.653474+0000","last_fullsized":"2024-04-06T03:34:11.653474+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:33:14.584201+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:33:14.584201+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:43.220224+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'31","reported_seq":419,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224164+0000","last_change":"2024-04-06T03:33:35.363415+0000","last_active":"2024-04-06T03:34:13.224164+0000","last_peered":"2024-04-06T03:34:13.224164+0000","last_clean":"2024-04-06T03:34:13.224164+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:34:13.224164+0000","last_undegraded":"2024-04-06T03:34:13.224164+0000","last_fullsized":"2024-04-06T03:34:13.224164+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'31","last_scrub_stamp":"2024-04-06T03:33:35.363029+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.363029+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:58.052022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":377,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.657229+0000","last_change":"2024-04-06T03:33:35.767909+0000","last_active":"2024-04-06T03:34:11.657229+0000","last_peered":"2024-04-06T03:34:11.657229+0000","last_clean":"2024-04-06T03:34:11.657229+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:34:11.657229+0000","last_undegraded":"2024-04-06T03:34:11.657229+0000","last_fullsized":"2024-04-06T03:34:11.657229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:35.767611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.767611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:39.607144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":355,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658400+0000","last_change":"2024-04-06T03:33:31.120147+0000","last_active":"2024-04-06T03:34:11.658400+0000","last_peered":"2024-04-06T03:34:11.658400+0000","last_clean":"2024-04-06T03:34:11.658400+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:34:11.658400+0000","last_undegraded":"2024-04-06T03:34:11.658400+0000","last_fullsized":"2024-04-06T03:34:11.658400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:31.120067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:33:31.120067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.540592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":315,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.659987+0000","last_change":"2024-04-06T03:31:54.539188+0000","last_active":"2024-04-06T03:34:11.659987+0000","last_peered":"2024-04-06T03:34:11.659987+0000","last_clean":"2024-04-06T03:34:11.659987+0000","last_became_active":"2024-04-06T03:27:10.500422+0000","last_became_peered":"2024-04-06T03:27:10.500422+0000","last_unstale":"2024-04-06T03:34:11.659987+0000","last_undegraded":"2024-04-06T03:34:11.659987+0000","last_fullsized":"2024-04-06T03:34:11.659987+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:31:54.539063+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:31:54.539063+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.281900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"199'37","reported_seq":397,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229088+0000","last_change":"2024-04-06T03:31:39.459165+0000","last_active":"2024-04-06T03:34:13.229088+0000","last_peered":"2024-04-06T03:34:13.229088+0000","last_clean":"2024-04-06T03:34:13.229088+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:34:13.229088+0000","last_undegraded":"2024-04-06T03:34:13.229088+0000","last_fullsized":"2024-04-06T03:34:13.229088+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:31:39.459040+0000","last_deep_scrub":"168'25","last_deep_scrub_stamp":"2024-04-06T03:28:40.744518+0000","last_clean_scrub_stamp":"2024-04-06T03:31:39.459040+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":379,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598555+0000","last_change":"2024-04-06T03:33:14.416136+0000","last_active":"2024-04-06T03:34:12.598555+0000","last_peered":"2024-04-06T03:34:12.598555+0000","last_clean":"2024-04-06T03:34:12.598555+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:34:12.598555+0000","last_undegraded":"2024-04-06T03:34:12.598555+0000","last_fullsized":"2024-04-06T03:34:12.598555+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:33:14.416001+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:33:14.416001+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:24.765524+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":530,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660956+0000","last_change":"2024-04-06T03:33:41.242743+0000","last_active":"2024-04-06T03:34:11.660956+0000","last_peered":"2024-04-06T03:34:11.660956+0000","last_clean":"2024-04-06T03:34:11.660956+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:34:11.660956+0000","last_undegraded":"2024-04-06T03:34:11.660956+0000","last_fullsized":"2024-04-06T03:34:11.660956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:33:41.242500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.242500+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:51.641654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":349,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229701+0000","last_change":"2024-04-06T03:31:32.350130+0000","last_active":"2024-04-06T03:34:13.229701+0000","last_peered":"2024-04-06T03:34:13.229701+0000","last_clean":"2024-04-06T03:34:13.229701+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:34:13.229701+0000","last_undegraded":"2024-04-06T03:34:13.229701+0000","last_fullsized":"2024-04-06T03:34:13.229701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:32.350015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:32.350015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419537+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":243,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598233+0000","last_change":"2024-04-06T03:33:23.364994+0000","last_active":"2024-04-06T03:34:12.598233+0000","last_peered":"2024-04-06T03:34:12.598233+0000","last_clean":"2024-04-06T03:34:12.598233+0000","last_became_active":"2024-04-06T03:25:54.484235+0000","last_became_peered":"2024-04-06T03:25:54.484235+0000","last_unstale":"2024-04-06T03:34:12.598233+0000","last_undegraded":"2024-04-06T03:34:12.598233+0000","last_fullsized":"2024-04-06T03:34:12.598233+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.364862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.364862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.849695+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"199'44","reported_seq":437,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653225+0000","last_change":"2024-04-06T03:33:13.560964+0000","last_active":"2024-04-06T03:34:11.653225+0000","last_peered":"2024-04-06T03:34:11.653225+0000","last_clean":"2024-04-06T03:34:11.653225+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:34:11.653225+0000","last_undegraded":"2024-04-06T03:34:11.653225+0000","last_fullsized":"2024-04-06T03:34:11.653225+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'44","last_scrub_stamp":"2024-04-06T03:33:13.560818+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.560818+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.496227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":359,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660215+0000","last_change":"2024-04-06T03:33:35.119000+0000","last_active":"2024-04-06T03:34:11.660215+0000","last_peered":"2024-04-06T03:34:11.660215+0000","last_clean":"2024-04-06T03:34:11.660215+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:34:11.660215+0000","last_undegraded":"2024-04-06T03:34:11.660215+0000","last_fullsized":"2024-04-06T03:34:11.660215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:35.118863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.118863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:41.048886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":355,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223594+0000","last_change":"2024-04-06T03:33:09.452964+0000","last_active":"2024-04-06T03:34:13.223594+0000","last_peered":"2024-04-06T03:34:13.223594+0000","last_clean":"2024-04-06T03:34:13.223594+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:34:13.223594+0000","last_undegraded":"2024-04-06T03:34:13.223594+0000","last_fullsized":"2024-04-06T03:34:13.223594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:09.452871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.452871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.657429+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"199'42","reported_seq":292,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.657370+0000","last_change":"2024-04-06T03:33:24.513812+0000","last_active":"2024-04-06T03:34:11.657370+0000","last_peered":"2024-04-06T03:34:11.657370+0000","last_clean":"2024-04-06T03:34:11.657370+0000","last_became_active":"2024-04-06T03:27:10.495663+0000","last_became_peered":"2024-04-06T03:27:10.495663+0000","last_unstale":"2024-04-06T03:34:11.657370+0000","last_undegraded":"2024-04-06T03:34:11.657370+0000","last_fullsized":"2024-04-06T03:34:11.657370+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'42","last_scrub_stamp":"2024-04-06T03:33:24.513666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.513666+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:49.284041+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":385,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.657348+0000","last_change":"2024-04-06T03:33:37.772754+0000","last_active":"2024-04-06T03:34:11.657348+0000","last_peered":"2024-04-06T03:34:11.657348+0000","last_clean":"2024-04-06T03:34:11.657348+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:34:11.657348+0000","last_undegraded":"2024-04-06T03:34:11.657348+0000","last_fullsized":"2024-04-06T03:34:11.657348+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.772672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.772672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:44.857148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":377,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658631+0000","last_change":"2024-04-06T03:33:36.737236+0000","last_active":"2024-04-06T03:34:11.658631+0000","last_peered":"2024-04-06T03:34:11.658631+0000","last_clean":"2024-04-06T03:34:11.658631+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:34:11.658631+0000","last_undegraded":"2024-04-06T03:34:11.658631+0000","last_fullsized":"2024-04-06T03:34:11.658631+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.737080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.737080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:50.451973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":220,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229340+0000","last_change":"2024-04-06T03:31:43.475492+0000","last_active":"2024-04-06T03:34:13.229340+0000","last_peered":"2024-04-06T03:34:13.229340+0000","last_clean":"2024-04-06T03:34:13.229340+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:34:13.229340+0000","last_undegraded":"2024-04-06T03:34:13.229340+0000","last_fullsized":"2024-04-06T03:34:13.229340+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:43.475384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:43.475384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419601+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"224'32","reported_seq":413,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227941+0000","last_change":"2024-04-06T03:33:50.491583+0000","last_active":"2024-04-06T03:34:13.227941+0000","last_peered":"2024-04-06T03:34:13.227941+0000","last_clean":"2024-04-06T03:34:13.227941+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:34:13.227941+0000","last_undegraded":"2024-04-06T03:34:13.227941+0000","last_fullsized":"2024-04-06T03:34:13.227941+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'32","last_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_deep_scrub":"224'32","last_deep_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_clean_scrub_stamp":"2024-04-06T03:33:50.491205+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:01.976482+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":244,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658030+0000","last_change":"2024-04-06T03:33:27.712256+0000","last_active":"2024-04-06T03:34:11.658030+0000","last_peered":"2024-04-06T03:34:11.658030+0000","last_clean":"2024-04-06T03:34:11.658030+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:34:11.658030+0000","last_undegraded":"2024-04-06T03:34:11.658030+0000","last_fullsized":"2024-04-06T03:34:11.658030+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:27.712145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:27.712145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:54.454326+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":627,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224566+0000","last_change":"2024-04-06T03:33:33.336069+0000","last_active":"2024-04-06T03:34:13.224566+0000","last_peered":"2024-04-06T03:34:13.224566+0000","last_clean":"2024-04-06T03:34:13.224566+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:34:13.224566+0000","last_undegraded":"2024-04-06T03:34:13.224566+0000","last_fullsized":"2024-04-06T03:34:13.224566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:33:33.335954+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:33:33.335954+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:38.640750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":359,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223896+0000","last_change":"2024-04-06T03:33:16.345677+0000","last_active":"2024-04-06T03:34:13.223896+0000","last_peered":"2024-04-06T03:34:13.223896+0000","last_clean":"2024-04-06T03:34:13.223896+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:34:13.223896+0000","last_undegraded":"2024-04-06T03:34:13.223896+0000","last_fullsized":"2024-04-06T03:34:13.223896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:16.345560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:33:16.345560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:19.936111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":363,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598162+0000","last_change":"2024-04-06T03:33:37.385401+0000","last_active":"2024-04-06T03:34:12.598162+0000","last_peered":"2024-04-06T03:34:12.598162+0000","last_clean":"2024-04-06T03:34:12.598162+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:34:12.598162+0000","last_undegraded":"2024-04-06T03:34:12.598162+0000","last_fullsized":"2024-04-06T03:34:12.598162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.385288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:31:49.387394+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.385288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:51.597710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"199'35","reported_seq":426,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.659620+0000","last_change":"2024-04-06T03:34:00.341333+0000","last_active":"2024-04-06T03:34:11.659620+0000","last_peered":"2024-04-06T03:34:11.659620+0000","last_clean":"2024-04-06T03:34:11.659620+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:34:11.659620+0000","last_undegraded":"2024-04-06T03:34:11.659620+0000","last_fullsized":"2024-04-06T03:34:11.659620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'35","last_scrub_stamp":"2024-04-06T03:34:00.341058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:34:00.341058+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:20.757960+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":367,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223822+0000","last_change":"2024-04-06T03:33:41.282140+0000","last_active":"2024-04-06T03:34:13.223822+0000","last_peered":"2024-04-06T03:34:13.223822+0000","last_clean":"2024-04-06T03:34:13.223822+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:34:13.223822+0000","last_undegraded":"2024-04-06T03:34:13.223822+0000","last_fullsized":"2024-04-06T03:34:13.223822+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.281919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.281919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:11.014377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":242,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598790+0000","last_change":"2024-04-06T03:33:41.412972+0000","last_active":"2024-04-06T03:34:12.598790+0000","last_peered":"2024-04-06T03:34:12.598790+0000","last_clean":"2024-04-06T03:34:12.598790+0000","last_became_active":"2024-04-06T03:25:54.492757+0000","last_became_peered":"2024-04-06T03:25:54.492757+0000","last_unstale":"2024-04-06T03:34:12.598790+0000","last_undegraded":"2024-04-06T03:34:12.598790+0000","last_fullsized":"2024-04-06T03:34:12.598790+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.412744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.371401+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":363,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.598070+0000","last_change":"2024-04-06T03:33:09.337484+0000","last_active":"2024-04-06T03:34:12.598070+0000","last_peered":"2024-04-06T03:34:12.598070+0000","last_clean":"2024-04-06T03:34:12.598070+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:34:12.598070+0000","last_undegraded":"2024-04-06T03:34:12.598070+0000","last_fullsized":"2024-04-06T03:34:12.598070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:09.336838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.336838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:26.093869+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":367,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223742+0000","last_change":"2024-04-06T03:33:07.517308+0000","last_active":"2024-04-06T03:34:13.223742+0000","last_peered":"2024-04-06T03:34:13.223742+0000","last_clean":"2024-04-06T03:34:13.223742+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:34:13.223742+0000","last_undegraded":"2024-04-06T03:34:13.223742+0000","last_fullsized":"2024-04-06T03:34:13.223742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:07.517188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:33:07.517188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:31.903244+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"199'50","reported_seq":472,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.838545+0000","last_change":"2024-04-06T03:33:17.349616+0000","last_active":"2024-04-06T03:34:11.838545+0000","last_peered":"2024-04-06T03:34:11.838545+0000","last_clean":"2024-04-06T03:34:11.838545+0000","last_became_active":"2024-04-06T03:25:19.551712+0000","last_became_peered":"2024-04-06T03:25:19.551712+0000","last_unstale":"2024-04-06T03:34:11.838545+0000","last_undegraded":"2024-04-06T03:34:11.838545+0000","last_fullsized":"2024-04-06T03:34:11.838545+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'50","last_scrub_stamp":"2024-04-06T03:33:17.349177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:17.349177+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.372881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":230,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660021+0000","last_change":"2024-04-06T03:33:53.214635+0000","last_active":"2024-04-06T03:34:11.660021+0000","last_peered":"2024-04-06T03:34:11.660021+0000","last_clean":"2024-04-06T03:34:11.660021+0000","last_became_active":"2024-04-06T03:26:18.643845+0000","last_became_peered":"2024-04-06T03:26:18.643845+0000","last_unstale":"2024-04-06T03:34:11.660021+0000","last_undegraded":"2024-04-06T03:34:11.660021+0000","last_fullsized":"2024-04-06T03:34:11.660021+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:53.214388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.214388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:04.878344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":357,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652532+0000","last_change":"2024-04-06T03:31:28.291293+0000","last_active":"2024-04-06T03:34:11.652532+0000","last_peered":"2024-04-06T03:34:11.652532+0000","last_clean":"2024-04-06T03:34:11.652532+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:34:11.652532+0000","last_undegraded":"2024-04-06T03:34:11.652532+0000","last_fullsized":"2024-04-06T03:34:11.652532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:28.291003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:31:28.291003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282299+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":376,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.597719+0000","last_change":"2024-04-06T03:33:15.352460+0000","last_active":"2024-04-06T03:34:12.597719+0000","last_peered":"2024-04-06T03:34:12.597719+0000","last_clean":"2024-04-06T03:34:12.597719+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:34:12.597719+0000","last_undegraded":"2024-04-06T03:34:12.597719+0000","last_fullsized":"2024-04-06T03:34:12.597719+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:33:15.352215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.352215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:26.305203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"199'23","reported_seq":410,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229013+0000","last_change":"2024-04-06T03:33:49.512729+0000","last_active":"2024-04-06T03:34:13.229013+0000","last_peered":"2024-04-06T03:34:13.229013+0000","last_clean":"2024-04-06T03:34:13.229013+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:34:13.229013+0000","last_undegraded":"2024-04-06T03:34:13.229013+0000","last_fullsized":"2024-04-06T03:34:13.229013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'23","last_scrub_stamp":"2024-04-06T03:33:49.512394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:49.512394+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:09.757626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1152,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:14.598212+0000","last_change":"2024-04-06T03:33:35.485693+0000","last_active":"2024-04-06T03:34:14.598212+0000","last_peered":"2024-04-06T03:34:14.598212+0000","last_clean":"2024-04-06T03:34:14.598212+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:34:14.598212+0000","last_undegraded":"2024-04-06T03:34:14.598212+0000","last_fullsized":"2024-04-06T03:34:14.598212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:33:35.485602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.485602+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.043247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":779,"num_read_kb":774,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":361,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652465+0000","last_change":"2024-04-06T03:32:00.109040+0000","last_active":"2024-04-06T03:34:11.652465+0000","last_peered":"2024-04-06T03:34:11.652465+0000","last_clean":"2024-04-06T03:34:11.652465+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:34:11.652465+0000","last_undegraded":"2024-04-06T03:34:11.652465+0000","last_fullsized":"2024-04-06T03:34:11.652465+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:00.108953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:32:00.108953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282427+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":355,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222586+0000","last_change":"2024-04-06T03:33:28.341482+0000","last_active":"2024-04-06T03:34:13.222586+0000","last_peered":"2024-04-06T03:34:13.222586+0000","last_clean":"2024-04-06T03:34:13.222586+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:34:13.222586+0000","last_undegraded":"2024-04-06T03:34:13.222586+0000","last_fullsized":"2024-04-06T03:34:13.222586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:28.341397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:28.341397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:39.429372+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":229,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224583+0000","last_change":"2024-04-06T03:33:32.334478+0000","last_active":"2024-04-06T03:34:13.224583+0000","last_peered":"2024-04-06T03:34:13.224583+0000","last_clean":"2024-04-06T03:34:13.224583+0000","last_became_active":"2024-04-06T03:26:18.621727+0000","last_became_peered":"2024-04-06T03:26:18.621727+0000","last_unstale":"2024-04-06T03:34:13.224583+0000","last_undegraded":"2024-04-06T03:34:13.224583+0000","last_fullsized":"2024-04-06T03:34:13.224583+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:32.334329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:33:32.334329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.543074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"199'24","reported_seq":407,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660251+0000","last_change":"2024-04-06T03:34:04.406266+0000","last_active":"2024-04-06T03:34:11.660251+0000","last_peered":"2024-04-06T03:34:11.660251+0000","last_clean":"2024-04-06T03:34:11.660251+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:34:11.660251+0000","last_undegraded":"2024-04-06T03:34:11.660251+0000","last_fullsized":"2024-04-06T03:34:11.660251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:34:04.406029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:34:04.406029+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.069272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":370,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652597+0000","last_change":"2024-04-06T03:33:26.451723+0000","last_active":"2024-04-06T03:34:11.652597+0000","last_peered":"2024-04-06T03:34:11.652597+0000","last_clean":"2024-04-06T03:34:11.652597+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:34:11.652597+0000","last_undegraded":"2024-04-06T03:34:11.652597+0000","last_fullsized":"2024-04-06T03:34:11.652597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:26.451631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:10.453422+0000","last_clean_scrub_stamp":"2024-04-06T03:33:26.451631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:32.536367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":533,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227821+0000","last_change":"2024-04-06T03:33:09.022830+0000","last_active":"2024-04-06T03:34:13.227821+0000","last_peered":"2024-04-06T03:34:13.227821+0000","last_clean":"2024-04-06T03:34:13.227821+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:34:13.227821+0000","last_undegraded":"2024-04-06T03:34:13.227821+0000","last_fullsized":"2024-04-06T03:34:13.227821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:33:09.022701+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.022701+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419774+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":348,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.229469+0000","last_change":"2024-04-06T03:31:33.297229+0000","last_active":"2024-04-06T03:34:13.229469+0000","last_peered":"2024-04-06T03:34:13.229469+0000","last_clean":"2024-04-06T03:34:13.229469+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:34:13.229469+0000","last_undegraded":"2024-04-06T03:34:13.229469+0000","last_fullsized":"2024-04-06T03:34:13.229469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:33.297046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:31:33.297046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":227,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653698+0000","last_change":"2024-04-06T03:31:12.165375+0000","last_active":"2024-04-06T03:34:11.653698+0000","last_peered":"2024-04-06T03:34:11.653698+0000","last_clean":"2024-04-06T03:34:11.653698+0000","last_became_active":"2024-04-06T03:25:54.491253+0000","last_became_peered":"2024-04-06T03:25:54.491253+0000","last_unstale":"2024-04-06T03:34:11.653698+0000","last_undegraded":"2024-04-06T03:34:11.653698+0000","last_fullsized":"2024-04-06T03:34:11.653698+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:12.165261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:41.912384+0000","last_clean_scrub_stamp":"2024-04-06T03:31:12.165261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":367,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224059+0000","last_change":"2024-04-06T03:33:53.048866+0000","last_active":"2024-04-06T03:34:13.224059+0000","last_peered":"2024-04-06T03:34:13.224059+0000","last_clean":"2024-04-06T03:34:13.224059+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:34:13.224059+0000","last_undegraded":"2024-04-06T03:34:13.224059+0000","last_fullsized":"2024-04-06T03:34:13.224059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:53.048558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.048558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:11.441117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"199'37","reported_seq":428,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652599+0000","last_change":"2024-04-06T03:33:11.524425+0000","last_active":"2024-04-06T03:34:11.652599+0000","last_peered":"2024-04-06T03:34:11.652599+0000","last_clean":"2024-04-06T03:34:11.652599+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:34:11.652599+0000","last_undegraded":"2024-04-06T03:34:11.652599+0000","last_fullsized":"2024-04-06T03:34:11.652599+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:33:11.524289+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:33:11.524289+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:25.856696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":187,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.660079+0000","last_change":"2024-04-06T03:31:51.619985+0000","last_active":"2024-04-06T03:34:11.660079+0000","last_peered":"2024-04-06T03:34:11.660079+0000","last_clean":"2024-04-06T03:34:11.660079+0000","last_became_active":"2024-04-06T03:27:10.497525+0000","last_became_peered":"2024-04-06T03:27:10.497525+0000","last_unstale":"2024-04-06T03:34:11.660079+0000","last_undegraded":"2024-04-06T03:34:11.660079+0000","last_fullsized":"2024-04-06T03:34:11.660079+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:51.619863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:51.619863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.281921+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":541,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.832829+0000","last_change":"2024-04-06T03:33:10.496163+0000","last_active":"2024-04-06T03:34:11.832829+0000","last_peered":"2024-04-06T03:34:11.832829+0000","last_clean":"2024-04-06T03:34:11.832829+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:34:11.832829+0000","last_undegraded":"2024-04-06T03:34:11.832829+0000","last_fullsized":"2024-04-06T03:34:11.832829+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:33:10.496050+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:30:05.475373+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.496050+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:30.697349+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":375,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.597932+0000","last_change":"2024-04-06T03:33:21.373666+0000","last_active":"2024-04-06T03:34:12.597932+0000","last_peered":"2024-04-06T03:34:12.597932+0000","last_clean":"2024-04-06T03:34:12.597932+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:34:12.597932+0000","last_undegraded":"2024-04-06T03:34:12.597932+0000","last_fullsized":"2024-04-06T03:34:12.597932+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:21.373545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.373545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:47.171233+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"199'39","reported_seq":432,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227504+0000","last_change":"2024-04-06T03:33:53.514744+0000","last_active":"2024-04-06T03:34:13.227504+0000","last_peered":"2024-04-06T03:34:13.227504+0000","last_clean":"2024-04-06T03:34:13.227504+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:34:13.227504+0000","last_undegraded":"2024-04-06T03:34:13.227504+0000","last_fullsized":"2024-04-06T03:34:13.227504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:33:53.514484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.514484+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:15.902937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":356,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653626+0000","last_change":"2024-04-06T03:32:06.991866+0000","last_active":"2024-04-06T03:34:11.653626+0000","last_peered":"2024-04-06T03:34:11.653626+0000","last_clean":"2024-04-06T03:34:11.653626+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:34:11.653626+0000","last_undegraded":"2024-04-06T03:34:11.653626+0000","last_fullsized":"2024-04-06T03:34:11.653626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:06.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:32:06.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282404+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":549,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.837719+0000","last_change":"2024-04-06T03:33:30.761558+0000","last_active":"2024-04-06T03:34:11.837719+0000","last_peered":"2024-04-06T03:34:11.837719+0000","last_clean":"2024-04-06T03:34:11.837719+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:34:11.837719+0000","last_undegraded":"2024-04-06T03:34:11.837719+0000","last_fullsized":"2024-04-06T03:34:11.837719+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:33:30.761309+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.761309+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:37.611140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":246,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.227740+0000","last_change":"2024-04-06T03:33:12.048860+0000","last_active":"2024-04-06T03:34:13.227740+0000","last_peered":"2024-04-06T03:34:13.227740+0000","last_clean":"2024-04-06T03:34:13.227740+0000","last_became_active":"2024-04-06T03:25:19.549690+0000","last_became_peered":"2024-04-06T03:25:19.549690+0000","last_unstale":"2024-04-06T03:34:13.227740+0000","last_undegraded":"2024-04-06T03:34:13.227740+0000","last_fullsized":"2024-04-06T03:34:13.227740+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:12.048735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:12.048735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:15.224464+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"199'21","reported_seq":401,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228582+0000","last_change":"2024-04-06T03:33:51.487161+0000","last_active":"2024-04-06T03:34:13.228582+0000","last_peered":"2024-04-06T03:34:13.228582+0000","last_clean":"2024-04-06T03:34:13.228582+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:34:13.228582+0000","last_undegraded":"2024-04-06T03:34:13.228582+0000","last_fullsized":"2024-04-06T03:34:13.228582+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:33:51.487048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.487048+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:58.854777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":372,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658395+0000","last_change":"2024-04-06T03:33:30.117825+0000","last_active":"2024-04-06T03:34:11.658395+0000","last_peered":"2024-04-06T03:34:11.658395+0000","last_clean":"2024-04-06T03:34:11.658395+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:34:11.658395+0000","last_undegraded":"2024-04-06T03:34:11.658395+0000","last_fullsized":"2024-04-06T03:34:11.658395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:30.117646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.117646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:38.425925+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":192,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.653132+0000","last_change":"2024-04-06T03:33:51.362465+0000","last_active":"2024-04-06T03:34:11.653132+0000","last_peered":"2024-04-06T03:34:11.653132+0000","last_clean":"2024-04-06T03:34:11.653132+0000","last_became_active":"2024-04-06T03:27:10.501454+0000","last_became_peered":"2024-04-06T03:27:10.501454+0000","last_unstale":"2024-04-06T03:34:11.653132+0000","last_undegraded":"2024-04-06T03:34:11.653132+0000","last_fullsized":"2024-04-06T03:34:11.653132+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:51.362253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.362253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:01.543224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":366,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.222840+0000","last_change":"2024-04-06T03:33:06.482662+0000","last_active":"2024-04-06T03:34:13.222840+0000","last_peered":"2024-04-06T03:34:13.222840+0000","last_clean":"2024-04-06T03:34:13.222840+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:34:13.222840+0000","last_undegraded":"2024-04-06T03:34:13.222840+0000","last_fullsized":"2024-04-06T03:34:13.222840+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:06.482503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:06.482503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:20.723359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"199'41","reported_seq":315,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.652802+0000","last_change":"2024-04-06T03:31:51.059197+0000","last_active":"2024-04-06T03:34:11.652802+0000","last_peered":"2024-04-06T03:34:11.652802+0000","last_clean":"2024-04-06T03:34:11.652802+0000","last_became_active":"2024-04-06T03:26:18.616944+0000","last_became_peered":"2024-04-06T03:26:18.616944+0000","last_unstale":"2024-04-06T03:34:11.652802+0000","last_undegraded":"2024-04-06T03:34:11.652802+0000","last_fullsized":"2024-04-06T03:34:11.652802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'41","last_scrub_stamp":"2024-04-06T03:31:51.059107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:51.059107+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:14.282353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":347,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223683+0000","last_change":"2024-04-06T03:31:59.850617+0000","last_active":"2024-04-06T03:34:13.223683+0000","last_peered":"2024-04-06T03:34:13.223683+0000","last_clean":"2024-04-06T03:34:13.223683+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:34:13.223683+0000","last_undegraded":"2024-04-06T03:34:13.223683+0000","last_fullsized":"2024-04-06T03:34:13.223683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:59.850525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:31:59.850525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":376,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658402+0000","last_change":"2024-04-06T03:33:34.760395+0000","last_active":"2024-04-06T03:34:11.658402+0000","last_peered":"2024-04-06T03:34:11.658402+0000","last_clean":"2024-04-06T03:34:11.658402+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:34:11.658402+0000","last_undegraded":"2024-04-06T03:34:11.658402+0000","last_fullsized":"2024-04-06T03:34:11.658402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:34.760314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.760314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:00.747440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":376,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658006+0000","last_change":"2024-04-06T03:33:10.359026+0000","last_active":"2024-04-06T03:34:11.658006+0000","last_peered":"2024-04-06T03:34:11.658006+0000","last_clean":"2024-04-06T03:34:11.658006+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:34:11.658006+0000","last_undegraded":"2024-04-06T03:34:11.658006+0000","last_fullsized":"2024-04-06T03:34:11.658006+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:10.358787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.358787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:33.659388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"199'39","reported_seq":419,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224156+0000","last_change":"2024-04-06T03:33:51.057651+0000","last_active":"2024-04-06T03:34:13.224156+0000","last_peered":"2024-04-06T03:34:13.224156+0000","last_clean":"2024-04-06T03:34:13.224156+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:34:13.224156+0000","last_undegraded":"2024-04-06T03:34:13.224156+0000","last_fullsized":"2024-04-06T03:34:13.224156+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:33:51.057496+0000","last_deep_scrub":"188'25","last_deep_scrub_stamp":"2024-04-06T03:30:07.204766+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.057496+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:09.212488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":355,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.659748+0000","last_change":"2024-04-06T03:33:11.574828+0000","last_active":"2024-04-06T03:34:11.659748+0000","last_peered":"2024-04-06T03:34:11.659748+0000","last_clean":"2024-04-06T03:34:11.659748+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:34:11.659748+0000","last_undegraded":"2024-04-06T03:34:11.659748+0000","last_fullsized":"2024-04-06T03:34:11.659748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:11.574716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:11.574716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:30.231399+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":377,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.657744+0000","last_change":"2024-04-06T03:33:41.788390+0000","last_active":"2024-04-06T03:34:11.657744+0000","last_peered":"2024-04-06T03:34:11.657744+0000","last_clean":"2024-04-06T03:34:11.657744+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:34:11.657744+0000","last_undegraded":"2024-04-06T03:34:11.657744+0000","last_fullsized":"2024-04-06T03:34:11.657744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.788284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.906520+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.788284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:05.869472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":382,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:12.597713+0000","last_change":"2024-04-06T03:33:36.374904+0000","last_active":"2024-04-06T03:34:12.597713+0000","last_peered":"2024-04-06T03:34:12.597713+0000","last_clean":"2024-04-06T03:34:12.597713+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:34:12.597713+0000","last_undegraded":"2024-04-06T03:34:12.597713+0000","last_fullsized":"2024-04-06T03:34:12.597713+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.374779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:31:40.521795+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.374779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:50.400465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"199'25","reported_seq":369,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.228790+0000","last_change":"2024-04-06T03:31:57.560644+0000","last_active":"2024-04-06T03:34:13.228790+0000","last_peered":"2024-04-06T03:34:13.228790+0000","last_clean":"2024-04-06T03:34:13.228790+0000","last_became_active":"2024-04-06T03:25:54.495696+0000","last_became_peered":"2024-04-06T03:25:54.495696+0000","last_unstale":"2024-04-06T03:34:13.228790+0000","last_undegraded":"2024-04-06T03:34:13.228790+0000","last_fullsized":"2024-04-06T03:34:13.228790+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'25","last_scrub_stamp":"2024-04-06T03:31:57.560377+0000","last_deep_scrub":"125'15","last_deep_scrub_stamp":"2024-04-06T03:25:15.970079+0000","last_clean_scrub_stamp":"2024-04-06T03:31:57.560377+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:16.419687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":225,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.659749+0000","last_change":"2024-04-06T03:33:29.059858+0000","last_active":"2024-04-06T03:34:11.659749+0000","last_peered":"2024-04-06T03:34:11.659749+0000","last_clean":"2024-04-06T03:34:11.659749+0000","last_became_active":"2024-04-06T03:26:18.641975+0000","last_became_peered":"2024-04-06T03:26:18.641975+0000","last_unstale":"2024-04-06T03:34:11.659749+0000","last_undegraded":"2024-04-06T03:34:11.659749+0000","last_fullsized":"2024-04-06T03:34:11.659749+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:29.059614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.059614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.973063+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":384,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:11.658194+0000","last_change":"2024-04-06T03:33:18.399220+0000","last_active":"2024-04-06T03:34:11.658194+0000","last_peered":"2024-04-06T03:34:11.658194+0000","last_clean":"2024-04-06T03:34:11.658194+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:34:11.658194+0000","last_undegraded":"2024-04-06T03:34:11.658194+0000","last_fullsized":"2024-04-06T03:34:11.658194+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:18.398992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:33:18.398992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:42.757970+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"199'27","reported_seq":405,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.224678+0000","last_change":"2024-04-06T03:33:10.424862+0000","last_active":"2024-04-06T03:34:13.224678+0000","last_peered":"2024-04-06T03:34:13.224678+0000","last_clean":"2024-04-06T03:34:13.224678+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:34:13.224678+0000","last_undegraded":"2024-04-06T03:34:13.224678+0000","last_fullsized":"2024-04-06T03:34:13.224678+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:33:10.424675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.424675+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:34:22.839071+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'12","reported_seq":540,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-04-06T03:34:13.223422+0000","last_change":"2024-04-06T03:34:04.032595+0000","last_active":"2024-04-06T03:34:13.223422+0000","last_peered":"2024-04-06T03:34:13.223422+0000","last_clean":"2024-04-06T03:34:13.223422+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:34:13.223422+0000","last_undegraded":"2024-04-06T03:34:13.223422+0000","last_fullsized":"2024-04-06T03:34:13.223422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'12","last_scrub_stamp":"2024-04-06T03:34:04.032388+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:34:04.032388+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:25.833036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1414,"num_read_kb":1139,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":994,"ondisk_log_size":994,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":779,"num_read_kb":774,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743596,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172772,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93572380,"statfs":{"total":95995035648,"available":95818117120,"internally_reserved":0,"allocated":1400832,"data_stored":1054413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":175503818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136054,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185216,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93559936,"statfs":{"total":95995035648,"available":95805374464,"internally_reserved":0,"allocated":1363968,"data_stored":1052693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292940,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93452212,"statfs":{"total":95995035648,"available":95695065088,"internally_reserved":0,"allocated":786432,"data_stored":462179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738612,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262996,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482156,"statfs":{"total":95995035648,"available":95725727744,"internally_reserved":0,"allocated":1384448,"data_stored":1053746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378849,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190208,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554944,"statfs":{"total":95995035648,"available":95800262656,"internally_reserved":0,"allocated":774144,"data_stored":461923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084500,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262392,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93482760,"statfs":{"total":95995035648,"available":95726346240,"internally_reserved":0,"allocated":765952,"data_stored":461837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":267909575},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855564,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162468,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582684,"statfs":{"total":95995035648,"available":95828668416,"internally_reserved":0,"allocated":811008,"data_stored":462408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495810,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202684,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93542468,"statfs":{"total":95995035648,"available":95787487232,"internally_reserved":0,"allocated":770048,"data_stored":461819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:34:24.631 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-06T03:34:24.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:34:24.632 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-06T03:34:24.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-06T03:34:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:24.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:24.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:24.761 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:24.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:24.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:24.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[29221]: audit 2024-04-06T03:34:23.334479+0000 mon.c (mon.2) 70 : audit 1 from='client.? 172.21.15.50:0/3781172614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-06T03:34:24.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[29221]: audit 2024-04-06T03:34:23.334872+0000 mon.a (mon.0) 1325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-06T03:34:24.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[29221]: audit 2024-04-06T03:34:23.849324+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-06T03:34:24.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[29221]: cluster 2024-04-06T03:34:23.858746+0000 mon.a (mon.0) 1327 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-04-06T03:34:24.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[29221]: cluster 2024-04-06T03:34:24.256588+0000 mgr.y (mgr.24368) 498 : cluster 0 pgmap v514: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[21467]: audit 2024-04-06T03:34:23.334479+0000 mon.c (mon.2) 70 : audit 1 from='client.? 172.21.15.50:0/3781172614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-06T03:34:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[21467]: audit 2024-04-06T03:34:23.334872+0000 mon.a (mon.0) 1325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-06T03:34:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[21467]: audit 2024-04-06T03:34:23.849324+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-06T03:34:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[21467]: cluster 2024-04-06T03:34:23.858746+0000 mon.a (mon.0) 1327 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-04-06T03:34:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:24 smithi050 bash[21467]: cluster 2024-04-06T03:34:24.256588+0000 mgr.y (mgr.24368) 498 : cluster 0 pgmap v514: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:24.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:24.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:24.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:25.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:24 smithi153 bash[23852]: audit 2024-04-06T03:34:23.334479+0000 mon.c (mon.2) 70 : audit 1 from='client.? 172.21.15.50:0/3781172614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-06T03:34:25.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:24 smithi153 bash[23852]: audit 2024-04-06T03:34:23.334872+0000 mon.a (mon.0) 1325 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-04-06T03:34:25.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:24 smithi153 bash[23852]: audit 2024-04-06T03:34:23.849324+0000 mon.a (mon.0) 1326 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-04-06T03:34:25.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:24 smithi153 bash[23852]: cluster 2024-04-06T03:34:23.858746+0000 mon.a (mon.0) 1327 : cluster 0 osdmap e231: 8 total, 8 up, 8 in 2024-04-06T03:34:25.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:24 smithi153 bash[23852]: cluster 2024-04-06T03:34:24.256588+0000 mgr.y (mgr.24368) 498 : cluster 0 pgmap v514: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:25.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:25.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:25.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:25.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:25.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:25.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:25.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:25.544 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:25.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:25.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:25.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:25.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:25.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:25.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:25.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:25.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[21467]: cluster 2024-04-06T03:34:24.849244+0000 mon.a (mon.0) 1328 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:34:25.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[21467]: cluster 2024-04-06T03:34:24.854722+0000 osd.4 (osd.4) 307 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:25.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[21467]: cluster 2024-04-06T03:34:24.854852+0000 osd.4 (osd.4) 308 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:25.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[21467]: cluster 2024-04-06T03:34:24.863638+0000 mon.a (mon.0) 1329 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-04-06T03:34:25.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[29221]: cluster 2024-04-06T03:34:24.849244+0000 mon.a (mon.0) 1328 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:34:25.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[29221]: cluster 2024-04-06T03:34:24.854722+0000 osd.4 (osd.4) 307 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:25.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[29221]: cluster 2024-04-06T03:34:24.854852+0000 osd.4 (osd.4) 308 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:25.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:25 smithi050 bash[29221]: cluster 2024-04-06T03:34:24.863638+0000 mon.a (mon.0) 1329 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-04-06T03:34:26.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:26.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:25 smithi153 bash[23852]: cluster 2024-04-06T03:34:24.849244+0000 mon.a (mon.0) 1328 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:34:26.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:25 smithi153 bash[23852]: cluster 2024-04-06T03:34:24.854722+0000 osd.4 (osd.4) 307 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:26.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:25 smithi153 bash[23852]: cluster 2024-04-06T03:34:24.854852+0000 osd.4 (osd.4) 308 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:26.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:25 smithi153 bash[23852]: cluster 2024-04-06T03:34:24.863638+0000 mon.a (mon.0) 1329 : cluster 0 osdmap e232: 8 total, 8 up, 8 in 2024-04-06T03:34:26.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:26.172 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:26.199 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:26.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:26.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:26.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:26.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:26.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:26.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:26.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:26.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-04-06T03:34:26.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:26.724 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:26.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:26.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:26.856 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:26.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:26.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:26.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:27.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:26 smithi153 bash[23852]: cluster 2024-04-06T03:34:25.868959+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-04-06T03:34:27.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:26 smithi153 bash[23852]: cluster 2024-04-06T03:34:26.257529+0000 mgr.y (mgr.24368) 499 : cluster 0 pgmap v517: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:34:27.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:27.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:27.226 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:34:27.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:26 smithi050 bash[21467]: cluster 2024-04-06T03:34:25.868959+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-04-06T03:34:27.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:26 smithi050 bash[21467]: cluster 2024-04-06T03:34:26.257529+0000 mgr.y (mgr.24368) 499 : cluster 0 pgmap v517: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:34:27.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:26 smithi050 bash[29221]: cluster 2024-04-06T03:34:25.868959+0000 mon.a (mon.0) 1330 : cluster 0 osdmap e233: 8 total, 8 up, 8 in 2024-04-06T03:34:27.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:26 smithi050 bash[29221]: cluster 2024-04-06T03:34:26.257529+0000 mgr.y (mgr.24368) 499 : cluster 0 pgmap v517: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:34:27.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:27.248 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:34:27.270 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:34:27.295 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:34:27.320 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:34:27.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:27.346 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:34:27.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:27.371 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:34:27.396 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:34:27.398 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:34:27.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:27.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:27.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:27.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:27.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:27.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:27.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:27.877 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:34:27.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:27 smithi050 bash[21467]: audit 2024-04-06T03:34:27.211022+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.50:0/2495957364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:27 smithi050 bash[21467]: audit 2024-04-06T03:34:27.653393+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.50:0/3391049457' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:34:27.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:27 smithi050 bash[21467]: audit 2024-04-06T03:34:27.653830+0000 mon.a (mon.0) 1331 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:34:27.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:27 smithi050 bash[29221]: audit 2024-04-06T03:34:27.211022+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.50:0/2495957364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:27.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:27 smithi050 bash[29221]: audit 2024-04-06T03:34:27.653393+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.50:0/3391049457' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:34:27.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:27 smithi050 bash[29221]: audit 2024-04-06T03:34:27.653830+0000 mon.a (mon.0) 1331 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:34:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:27.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:27 smithi153 bash[23852]: audit 2024-04-06T03:34:27.211022+0000 mon.c (mon.2) 71 : audit 0 from='client.? 172.21.15.50:0/2495957364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:27 smithi153 bash[23852]: audit 2024-04-06T03:34:27.653393+0000 mon.c (mon.2) 72 : audit 1 from='client.? 172.21.15.50:0/3391049457' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:34:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:27 smithi153 bash[23852]: audit 2024-04-06T03:34:27.653830+0000 mon.a (mon.0) 1331 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:34:28.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:28.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:28.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:28.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:28.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:28.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:28.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:28.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:28.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:28.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:28.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:28.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:28.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:28.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:29.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:29.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[21467]: cluster 2024-04-06T03:34:27.867219+0000 mon.a (mon.0) 1332 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:29.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[21467]: audit 2024-04-06T03:34:27.875069+0000 mon.a (mon.0) 1333 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:34:29.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[21467]: cluster 2024-04-06T03:34:27.881015+0000 mon.a (mon.0) 1334 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-04-06T03:34:29.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[21467]: cluster 2024-04-06T03:34:28.258100+0000 mgr.y (mgr.24368) 500 : cluster 0 pgmap v519: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:34:29.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[29221]: cluster 2024-04-06T03:34:27.867219+0000 mon.a (mon.0) 1332 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:29.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[29221]: audit 2024-04-06T03:34:27.875069+0000 mon.a (mon.0) 1333 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:34:29.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[29221]: cluster 2024-04-06T03:34:27.881015+0000 mon.a (mon.0) 1334 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-04-06T03:34:29.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:28 smithi050 bash[29221]: cluster 2024-04-06T03:34:28.258100+0000 mgr.y (mgr.24368) 500 : cluster 0 pgmap v519: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:34:29.040 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:29.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:28 smithi153 bash[23852]: cluster 2024-04-06T03:34:27.867219+0000 mon.a (mon.0) 1332 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:28 smithi153 bash[23852]: audit 2024-04-06T03:34:27.875069+0000 mon.a (mon.0) 1333 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:34:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:28 smithi153 bash[23852]: cluster 2024-04-06T03:34:27.881015+0000 mon.a (mon.0) 1334 : cluster 0 osdmap e234: 8 total, 8 up, 8 in 2024-04-06T03:34:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:28 smithi153 bash[23852]: cluster 2024-04-06T03:34:28.258100+0000 mgr.y (mgr.24368) 500 : cluster 0 pgmap v519: 148 pgs: 9 creating+peering, 139 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:34:29.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:29.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:29.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:29.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:29.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:29.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:29.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:29.560 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:29.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:29.674 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:34:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:29.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:29.736 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:34:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:34:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:34:29.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:29.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:29.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:29.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:29.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:30.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:29 smithi153 bash[23852]: audit 2024-04-06T03:34:29.645575+0000 mgr.y (mgr.24368) 501 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:30.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:30.087 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:30.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:30.201 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:29 smithi050 bash[29221]: audit 2024-04-06T03:34:29.645575+0000 mgr.y (mgr.24368) 501 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:30.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:29 smithi050 bash[21467]: audit 2024-04-06T03:34:29.645575+0000 mgr.y (mgr.24368) 501 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:30.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:30.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:30.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:30.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:30.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:30.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:30.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:30.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:30.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:30.766 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:30.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:34:30.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:30.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:30.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:30.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:30.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:30 smithi050 bash[29221]: cluster 2024-04-06T03:34:30.258882+0000 mgr.y (mgr.24368) 502 : cluster 0 pgmap v520: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:34:30.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:30 smithi050 bash[21467]: cluster 2024-04-06T03:34:30.258882+0000 mgr.y (mgr.24368) 502 : cluster 0 pgmap v520: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:34:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:31.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:31.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:31.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:31.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:31.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:34:31.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:30 smithi153 bash[23852]: cluster 2024-04-06T03:34:30.258882+0000 mgr.y (mgr.24368) 502 : cluster 0 pgmap v520: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:34:31.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:31.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:31.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:31.643 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:31.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:31.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:31.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:31.904 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:31.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:31.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:32.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:32.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:32.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:32.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:32.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:32.295 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:32.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:32.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:32.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:32.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:32.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:32.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:32.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:32.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:32.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:32.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:33.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:33.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:33.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:33.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:33.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:33.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[29221]: cluster 2024-04-06T03:34:32.105081+0000 osd.2 (osd.2) 261 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:33.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[29221]: cluster 2024-04-06T03:34:32.105235+0000 osd.2 (osd.2) 262 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:33.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[29221]: cluster 2024-04-06T03:34:32.161593+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-04-06T03:34:33.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[29221]: cluster 2024-04-06T03:34:32.259368+0000 mgr.y (mgr.24368) 503 : cluster 0 pgmap v522: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-04-06T03:34:33.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[21467]: cluster 2024-04-06T03:34:32.105081+0000 osd.2 (osd.2) 261 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:33.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[21467]: cluster 2024-04-06T03:34:32.105235+0000 osd.2 (osd.2) 262 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:33.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[21467]: cluster 2024-04-06T03:34:32.161593+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-04-06T03:34:33.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:33 smithi050 bash[21467]: cluster 2024-04-06T03:34:32.259368+0000 mgr.y (mgr.24368) 503 : cluster 0 pgmap v522: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-04-06T03:34:33.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:33.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:33.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:33 smithi153 bash[23852]: cluster 2024-04-06T03:34:32.105081+0000 osd.2 (osd.2) 261 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:33.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:33 smithi153 bash[23852]: cluster 2024-04-06T03:34:32.105235+0000 osd.2 (osd.2) 262 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:33.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:33 smithi153 bash[23852]: cluster 2024-04-06T03:34:32.161593+0000 mon.a (mon.0) 1335 : cluster 0 osdmap e235: 8 total, 8 up, 8 in 2024-04-06T03:34:33.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:33 smithi153 bash[23852]: cluster 2024-04-06T03:34:32.259368+0000 mgr.y (mgr.24368) 503 : cluster 0 pgmap v522: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 800 B/s rd, 0 op/s 2024-04-06T03:34:33.470 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:33.505 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:33.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:33.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:33.727 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:33.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:33.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:33.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:33.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:33.982 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:33.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:33.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:34.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:34.115 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:34.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:34.244 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:34.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:34.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:34.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:34.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:34.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:34.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:34.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:34.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:34.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:34.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:34.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:35.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:35.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:35.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:35.162 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:35.265 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:35.294 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:35.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:35.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:35 smithi153 bash[23852]: cluster 2024-04-06T03:34:34.260429+0000 mgr.y (mgr.24368) 504 : cluster 0 pgmap v523: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:35.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:35 smithi153 bash[23852]: audit 2024-04-06T03:34:34.366625+0000 mon.a (mon.0) 1336 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:35.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:35 smithi153 bash[23852]: audit 2024-04-06T03:34:34.369100+0000 mon.a (mon.0) 1337 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:35.408 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:35 smithi153 bash[23852]: audit 2024-04-06T03:34:35.313318+0000 mon.a (mon.0) 1338 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:34:35.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:35.430 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:34:35.430 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:34:35.430 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:34:35.431 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:34:35.431 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[29221]: cluster 2024-04-06T03:34:34.260429+0000 mgr.y (mgr.24368) 504 : cluster 0 pgmap v523: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[29221]: audit 2024-04-06T03:34:34.366625+0000 mon.a (mon.0) 1336 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[29221]: audit 2024-04-06T03:34:34.369100+0000 mon.a (mon.0) 1337 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[29221]: audit 2024-04-06T03:34:35.313318+0000 mon.a (mon.0) 1338 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[21467]: cluster 2024-04-06T03:34:34.260429+0000 mgr.y (mgr.24368) 504 : cluster 0 pgmap v523: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[21467]: audit 2024-04-06T03:34:34.366625+0000 mon.a (mon.0) 1336 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[21467]: audit 2024-04-06T03:34:34.369100+0000 mon.a (mon.0) 1337 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:35 smithi050 bash[21467]: audit 2024-04-06T03:34:35.313318+0000 mon.a (mon.0) 1338 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:34:35.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:35.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:35.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:35.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:35.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:35.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:35.854 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:35.854 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:35.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:35.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:35.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:36.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:36.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:36.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:36.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:36.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:36.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:36.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:36.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:36.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[29221]: cluster 2024-04-06T03:34:34.879526+0000 osd.6 (osd.6) 91 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:36.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[29221]: cluster 2024-04-06T03:34:34.879652+0000 osd.6 (osd.6) 92 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:36.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[29221]: cluster 2024-04-06T03:34:35.809617+0000 mon.a (mon.0) 1339 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-04-06T03:34:36.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[29221]: audit 2024-04-06T03:34:35.987742+0000 mon.a (mon.0) 1340 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:34:36.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[29221]: audit 2024-04-06T03:34:35.988683+0000 mon.a (mon.0) 1341 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:34:36.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[29221]: audit 2024-04-06T03:34:35.993747+0000 mon.a (mon.0) 1342 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:36.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[21467]: cluster 2024-04-06T03:34:34.879526+0000 osd.6 (osd.6) 91 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:36.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[21467]: cluster 2024-04-06T03:34:34.879652+0000 osd.6 (osd.6) 92 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:36.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[21467]: cluster 2024-04-06T03:34:35.809617+0000 mon.a (mon.0) 1339 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-04-06T03:34:36.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[21467]: audit 2024-04-06T03:34:35.987742+0000 mon.a (mon.0) 1340 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:34:36.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[21467]: audit 2024-04-06T03:34:35.988683+0000 mon.a (mon.0) 1341 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:34:36.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:36 smithi050 bash[21467]: audit 2024-04-06T03:34:35.993747+0000 mon.a (mon.0) 1342 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:36.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:36 smithi153 bash[23852]: cluster 2024-04-06T03:34:34.879526+0000 osd.6 (osd.6) 91 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:36 smithi153 bash[23852]: cluster 2024-04-06T03:34:34.879652+0000 osd.6 (osd.6) 92 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:36 smithi153 bash[23852]: cluster 2024-04-06T03:34:35.809617+0000 mon.a (mon.0) 1339 : cluster 0 osdmap e236: 8 total, 8 up, 8 in 2024-04-06T03:34:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:36 smithi153 bash[23852]: audit 2024-04-06T03:34:35.987742+0000 mon.a (mon.0) 1340 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:34:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:36 smithi153 bash[23852]: audit 2024-04-06T03:34:35.988683+0000 mon.a (mon.0) 1341 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:34:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:36 smithi153 bash[23852]: audit 2024-04-06T03:34:35.993747+0000 mon.a (mon.0) 1342 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:36.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:36.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:36.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:36.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:36.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:36.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:36.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:36.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:37.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:37.091 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:37.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:37.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.358 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:37.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:37.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:37.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:37 smithi050 bash[21467]: cluster 2024-04-06T03:34:36.260886+0000 mgr.y (mgr.24368) 505 : cluster 0 pgmap v525: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:37.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:37 smithi050 bash[29221]: cluster 2024-04-06T03:34:36.260886+0000 mgr.y (mgr.24368) 505 : cluster 0 pgmap v525: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:37.487 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:37.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:37.618 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:37.620 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:37 smithi153 bash[23852]: cluster 2024-04-06T03:34:36.260886+0000 mgr.y (mgr.24368) 505 : cluster 0 pgmap v525: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:37.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:37.750 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:37.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:37.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:37.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:37.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:38.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:38.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:38.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:38.145 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:38.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-06T03:34:38.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:38.275 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:38.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:38.403 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:38.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:38.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:38.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:38.664 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:38.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:38.793 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:38.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:38.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:39.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:39.055 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:39.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:39.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:39.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:39.316 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:39.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:39.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:39 smithi050 bash[21467]: cluster 2024-04-06T03:34:38.261376+0000 mgr.y (mgr.24368) 506 : cluster 0 pgmap v526: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:39.431 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:34:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:34:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:34:39.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:39 smithi050 bash[29221]: cluster 2024-04-06T03:34:38.261376+0000 mgr.y (mgr.24368) 506 : cluster 0 pgmap v526: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:39.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:39.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:39.561 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:39 smithi153 bash[23852]: cluster 2024-04-06T03:34:38.261376+0000 mgr.y (mgr.24368) 506 : cluster 0 pgmap v526: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:39.578 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:39.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:39.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:39.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:39.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:39.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:39.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:39.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:40.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:34:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:40.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:40.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:40.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:40.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:40.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:40.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:40.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:40.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:40.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:40 smithi050 bash[21467]: audit 2024-04-06T03:34:39.656393+0000 mgr.y (mgr.24368) 507 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:40.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:40 smithi050 bash[29221]: audit 2024-04-06T03:34:39.656393+0000 mgr.y (mgr.24368) 507 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:40.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:40.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:40.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:40.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:40.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:40 smithi153 bash[23852]: audit 2024-04-06T03:34:39.656393+0000 mgr.y (mgr.24368) 507 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:40.757 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:40.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:40.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:40.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:41.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:41.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:41.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:41.267 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:41.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:41.397 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:41.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:41.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:41 smithi153 bash[23852]: cluster 2024-04-06T03:34:40.262162+0000 mgr.y (mgr.24368) 508 : cluster 0 pgmap v527: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-04-06T03:34:41.525 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:41.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:41 smithi050 bash[21467]: cluster 2024-04-06T03:34:40.262162+0000 mgr.y (mgr.24368) 508 : cluster 0 pgmap v527: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-04-06T03:34:41.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:41 smithi050 bash[29221]: cluster 2024-04-06T03:34:40.262162+0000 mgr.y (mgr.24368) 508 : cluster 0 pgmap v527: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-04-06T03:34:41.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:41.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:41.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:41.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:41.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:41.917 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:41.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:42.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:42.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:42.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:42.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:42.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:42.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:42.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:42.437 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:42.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:42.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:42.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:42.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:42.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:42.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:42.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:42.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:43.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:43.090 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:43.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:43.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:43.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:43.339 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 236, 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-06T03:10:26.969128+0000", 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-06T03:34:35.805093+0000", 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-06T03:20:09.072730+0000", 2024-04-06T03:34:43.360 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-06T03:19:42.545394+0000", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5806080, 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "nodeep-scrub", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "noscrub", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 18, 2024-04-06T03:34:43.361 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 9, 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-06T03:34:43.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:15:48.096900+0000", 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.363 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.364 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:34:43.365 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.366 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.367 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.368 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 6, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 6, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.5, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:50.011930+0000", 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.370 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "79", 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.372 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:34:43.374 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.3200000524520874, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.3200000524520874, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-06T03:34:43.375 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:51.687715+0000", 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:34:43.376 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "81", 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.377 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.378 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.379 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:34:43.380 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.301921+0000", 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.381 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.382 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "88", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 88, 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.383 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.385 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.1699999570846558, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.810316+0000", 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:34:43.386 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:34:43.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "83", 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.388 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.389 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.390 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.391 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:56.321502+0000", 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "85", 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 9, 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "unique_pool_2", 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:34:23.797117+0000", 2024-04-06T03:34:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 16, 2024-04-06T03:34:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 16, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 16, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 16, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 16, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "233", 2024-04-06T03:34:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:34:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:34:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": {}, 2024-04-06T03:34:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.8799999952316284, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.8799999952316284, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.5, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.5, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "e607f527-4592-4acd-a47f-76575615ae1b", 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 231, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.405 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/3531754646", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/3531754646", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/3531754646", 2024-04-06T03:34:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/3531754646", 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10", 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0.5613250732421875, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 231, 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/4128410069", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/4128410069", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/4128410069", 2024-04-06T03:34:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/4128410069", 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b", 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 19, 2024-04-06T03:34:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 231, 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/1307247904", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/1307247904", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/1307247904", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/1307247904", 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240", 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 231, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/1397888508", 2024-04-06T03:34:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/1397888508", 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/1397888508", 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/1397888508", 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48", 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 182, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6800", 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6801", 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6802", 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6803", 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6806", 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6807", 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6804", 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6805", 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6801/3377683473", 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6803/3377683473", 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6807/3377683473", 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6805/3377683473", 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241", 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 46, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 127, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6808", 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6809", 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6810", 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6811", 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.433 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6814", 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6815", 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6812", 2024-04-06T03:34:43.434 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6813", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6809/357179117", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6811/357179117", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6815/357179117", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6813/357179117", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.435 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "98027d53-4324-44d1-89ce-8da85be58efb", 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:34:43.436 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6816", 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6817", 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6818", 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6819", 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.438 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6822", 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6823", 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6820", 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6821", 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.440 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6817/1552538208", 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6819/1552538208", 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6823/1552538208", 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6821/1552538208", 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd", 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:34:43.441 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 231, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6824", 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.442 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6825", 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6826", 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6827", 2024-04-06T03:34:43.443 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6830", 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6831", 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6828", 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6829", 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6825/2479193924", 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6827/2479193924", 2024-04-06T03:34:43.445 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6831/2479193924", 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6829/2479193924", 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.446 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:33:41.622690+0000", 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:34:11.316176+0000", 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.447 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:34:32.105250+0000", 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:33:57.470136+0000", 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:34:24.855509+0000", 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.449 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:33:54.589744+0000", 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:34:34.879661+0000", 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.450 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:34:07.527428+0000", 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.12", 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-06T03:34:43.451 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/283275195": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2738019433": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1268273764": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/567103210": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:34:43.452 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2891059051": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2170219983": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/4082841962": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/570623774": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1379780741": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2880368026": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2044358308": "2024-04-07T03:21:47.915758+0000" 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-06T03:34:43.453 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-06T03:34:43.454 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-06T03:34:43.455 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:34:43.455 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:34:43.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:43.456 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[21467]: cluster 2024-04-06T03:34:42.263431+0000 mgr.y (mgr.24368) 509 : cluster 0 pgmap v528: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[21467]: audit 2024-04-06T03:34:42.925283+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.50:0/587781934' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[21467]: audit 2024-04-06T03:34:42.925846+0000 mon.a (mon.0) 1343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[21467]: audit 2024-04-06T03:34:43.363448+0000 mon.a (mon.0) 1344 : audit 0 from='client.? 172.21.15.50:0/577761020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[29221]: cluster 2024-04-06T03:34:42.263431+0000 mgr.y (mgr.24368) 509 : cluster 0 pgmap v528: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[29221]: audit 2024-04-06T03:34:42.925283+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.50:0/587781934' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[29221]: audit 2024-04-06T03:34:42.925846+0000 mon.a (mon.0) 1343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:34:43.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:43 smithi050 bash[29221]: audit 2024-04-06T03:34:43.363448+0000 mon.a (mon.0) 1344 : audit 0 from='client.? 172.21.15.50:0/577761020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:34:43.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:43.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:43.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:43.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:43.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:43 smithi153 bash[23852]: cluster 2024-04-06T03:34:42.263431+0000 mgr.y (mgr.24368) 509 : cluster 0 pgmap v528: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:43.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:43 smithi153 bash[23852]: audit 2024-04-06T03:34:42.925283+0000 mon.c (mon.2) 73 : audit 1 from='client.? 172.21.15.50:0/587781934' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:34:43.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:43 smithi153 bash[23852]: audit 2024-04-06T03:34:42.925846+0000 mon.a (mon.0) 1343 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:34:43.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:43 smithi153 bash[23852]: audit 2024-04-06T03:34:43.363448+0000 mon.a (mon.0) 1344 : audit 0 from='client.? 172.21.15.50:0/577761020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:34:43.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:43.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:43.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:43.995 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:43.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:44.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:44.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:44.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:44.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:44.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:44.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:44.501 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:44 smithi153 bash[23852]: audit 2024-04-06T03:34:43.382194+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:34:44.501 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:44 smithi153 bash[23852]: cluster 2024-04-06T03:34:43.385230+0000 mon.a (mon.0) 1346 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-04-06T03:34:44.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:44.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:44.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:44 smithi050 bash[29221]: audit 2024-04-06T03:34:43.382194+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:34:44.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:44 smithi050 bash[29221]: cluster 2024-04-06T03:34:43.385230+0000 mon.a (mon.0) 1346 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-04-06T03:34:44.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:44 smithi050 bash[21467]: audit 2024-04-06T03:34:43.382194+0000 mon.a (mon.0) 1345 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:34:44.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:44 smithi050 bash[21467]: cluster 2024-04-06T03:34:43.385230+0000 mon.a (mon.0) 1346 : cluster 0 osdmap e237: 8 total, 8 up, 8 in 2024-04-06T03:34:44.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:44.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:44.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:45.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:45.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:45.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:45.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:45.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:45.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:45.436 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:45.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:45 smithi050 bash[21467]: cluster 2024-04-06T03:34:44.264543+0000 mgr.y (mgr.24368) 510 : cluster 0 pgmap v530: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-04-06T03:34:45.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:45 smithi050 bash[21467]: cluster 2024-04-06T03:34:44.390486+0000 mon.a (mon.0) 1347 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:34:45.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:45 smithi050 bash[29221]: cluster 2024-04-06T03:34:44.264543+0000 mgr.y (mgr.24368) 510 : cluster 0 pgmap v530: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-04-06T03:34:45.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:45 smithi050 bash[29221]: cluster 2024-04-06T03:34:44.390486+0000 mon.a (mon.0) 1347 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:34:45.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:45.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:45 smithi153 bash[23852]: cluster 2024-04-06T03:34:44.264543+0000 mgr.y (mgr.24368) 510 : cluster 0 pgmap v530: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-04-06T03:34:45.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:45 smithi153 bash[23852]: cluster 2024-04-06T03:34:44.390486+0000 mon.a (mon.0) 1347 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:34:45.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:45.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:45.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:45.694 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:45.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:45.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:45.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:45.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:45.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:46.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:46.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:46.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:46.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:46.306 INFO:tasks.thrashosds.thrasher:j is {'epoch': 236, 'fsid': '0313e520-f3c3-11ee-b649-cb9ed24678a4', 'created': '2024-04-06T03:10:26.969128+0000', 'modified': '2024-04-06T03:34:35.805093+0000', 'last_up_change': '2024-04-06T03:20:09.072730+0000', 'last_in_change': '2024-04-06T03:19:42.545394+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:15:48.096900+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-06T03:20:50.011930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-06T03:20:51.687715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-06T03:20:53.301921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-06T03:20:53.810316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-06T03:20:56.321502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-04-06T03:34:23.797117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'e607f527-4592-4acd-a47f-76575615ae1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 3531754646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 3531754646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3531754646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 3531754646}]}, 'public_addr': '172.21.15.50:6803/3531754646', 'cluster_addr': '172.21.15.50:6805/3531754646', 'heartbeat_back_addr': '172.21.15.50:6809/3531754646', 'heartbeat_front_addr': '172.21.15.50:6807/3531754646', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2ae8f784-d36a-4633-8630-2aed04e4ff10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5613250732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 4128410069}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 4128410069}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 4128410069}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 4128410069}]}, 'public_addr': '172.21.15.50:6811/4128410069', 'cluster_addr': '172.21.15.50:6813/4128410069', 'heartbeat_back_addr': '172.21.15.50:6817/4128410069', 'heartbeat_front_addr': '172.21.15.50:6815/4128410069', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '43f31d09-5c16-4aea-8822-e0d877cc931b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1307247904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1307247904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 1307247904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1307247904}]}, 'public_addr': '172.21.15.50:6819/1307247904', 'cluster_addr': '172.21.15.50:6821/1307247904', 'heartbeat_back_addr': '172.21.15.50:6825/1307247904', 'heartbeat_front_addr': '172.21.15.50:6823/1307247904', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4c913325-442e-4ee5-94f1-8b5bca0c3240', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 1397888508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 1397888508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 1397888508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 1397888508}]}, 'public_addr': '172.21.15.50:6827/1397888508', 'cluster_addr': '172.21.15.50:6829/1397888508', 'heartbeat_back_addr': '172.21.15.50:6833/1397888508', 'heartbeat_front_addr': '172.21.15.50:6831/1397888508', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50c02bfc-c2f2-4aae-b1d5-c38a86726b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 182, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6800', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6801', 'nonce': 3377683473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6802', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6803', 'nonce': 3377683473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6806', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6807', 'nonce': 3377683473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6804', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6805', 'nonce': 3377683473}]}, 'public_addr': '172.21.15.153:6801/3377683473', 'cluster_addr': '172.21.15.153:6803/3377683473', 'heartbeat_back_addr': '172.21.15.153:6807/3377683473', 'heartbeat_front_addr': '172.21.15.153:6805/3377683473', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'c3e8194b-2c20-4ffb-859d-00c33902b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6808', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6809', 'nonce': 357179117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6810', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6811', 'nonce': 357179117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6814', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6815', 'nonce': 357179117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6812', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6813', 'nonce': 357179117}]}, 'public_addr': '172.21.15.153:6809/357179117', 'cluster_addr': '172.21.15.153:6811/357179117', 'heartbeat_back_addr': '172.21.15.153:6815/357179117', 'heartbeat_front_addr': '172.21.15.153:6813/357179117', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98027d53-4324-44d1-89ce-8da85be58efb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6816', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6817', 'nonce': 1552538208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6818', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6819', 'nonce': 1552538208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6822', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6823', 'nonce': 1552538208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6820', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6821', 'nonce': 1552538208}]}, 'public_addr': '172.21.15.153:6817/1552538208', 'cluster_addr': '172.21.15.153:6819/1552538208', 'heartbeat_back_addr': '172.21.15.153:6823/1552538208', 'heartbeat_front_addr': '172.21.15.153:6821/1552538208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6b47d95a-1228-4fb1-a315-5371d9b42bbd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6824', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6825', 'nonce': 2479193924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6826', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6827', 'nonce': 2479193924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6830', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6831', 'nonce': 2479193924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6828', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6829', 'nonce': 2479193924}]}, 'public_addr': '172.21.15.153:6825/2479193924', 'cluster_addr': '172.21.15.153:6827/2479193924', 'heartbeat_back_addr': '172.21.15.153:6831/2479193924', 'heartbeat_front_addr': '172.21.15.153:6829/2479193924', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:33:41.622690+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:34:11.316176+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:34:32.105250+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:33:57.470136+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:34:24.855509+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:33:54.589744+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:34:34.879661+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:34:07.527428+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.12', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/283275195': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2738019433': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1268273764': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/567103210': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/2891059051': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2170219983': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/4082841962': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6800/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/570623774': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6801/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1379780741': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2880368026': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2044358308': '2024-04-07T03:21:47.915758+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-06T03:34:46.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:34:46.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:46.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:46.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:46.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:46.641 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:46.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:46.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:46.772 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:46.798 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:46.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:46.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:46.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:46.960 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:34:46.983 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:34:47.004 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:34:47.026 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:34:47.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:47.051 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:34:47.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:47.073 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:34:47.095 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:34:47.117 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:34:47.119 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:34:47.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:47.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:47.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:47.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:47.341 DEBUG:teuthology.orchestra.run.smithi050:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-04-06T03:34:47.344 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-06T03:34:47.344 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-06T03:34:47.390 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:hints = 1 2024-04-06T03:34:47.390 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-06T03:34:47.390 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object prefix: benchmark_data_smithi050_105098 2024-04-06T03:34:47.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:34:47.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 0 0 0 0 0 0 - 0 2024-04-06T03:34:47.400 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:34:47.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:47.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:47 smithi050 bash[21467]: cluster 2024-04-06T03:34:46.265297+0000 mgr.y (mgr.24368) 511 : cluster 0 pgmap v531: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:47.424 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:47 smithi050 bash[21467]: audit 2024-04-06T03:34:46.945323+0000 mon.a (mon.0) 1348 : audit 0 from='client.? 172.21.15.50:0/1119861474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:47.424 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:47 smithi050 bash[21467]: audit 2024-04-06T03:34:47.161665+0000 mon.a (mon.0) 1349 : audit 1 from='client.? 172.21.15.50:0/3855915281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:34:47.424 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:47 smithi050 bash[29221]: cluster 2024-04-06T03:34:46.265297+0000 mgr.y (mgr.24368) 511 : cluster 0 pgmap v531: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:47.424 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:47 smithi050 bash[29221]: audit 2024-04-06T03:34:46.945323+0000 mon.a (mon.0) 1348 : audit 0 from='client.? 172.21.15.50:0/1119861474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:47.424 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:47 smithi050 bash[29221]: audit 2024-04-06T03:34:47.161665+0000 mon.a (mon.0) 1349 : audit 1 from='client.? 172.21.15.50:0/3855915281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:34:47.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:47 smithi153 bash[23852]: cluster 2024-04-06T03:34:46.265297+0000 mgr.y (mgr.24368) 511 : cluster 0 pgmap v531: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:34:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:47 smithi153 bash[23852]: audit 2024-04-06T03:34:46.945323+0000 mon.a (mon.0) 1348 : audit 0 from='client.? 172.21.15.50:0/1119861474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:34:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:47 smithi153 bash[23852]: audit 2024-04-06T03:34:47.161665+0000 mon.a (mon.0) 1349 : audit 1 from='client.? 172.21.15.50:0/3855915281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:34:47.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:47.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:47.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:47.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:47.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:47.847 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:47.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:47.976 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:47.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:48.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:48.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:48.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:48.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:48.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:48.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:48.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:48.365 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:48.390 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 1 16 2238 2222 138.868 138.875 0.0038985 0.00716141 2024-04-06T03:34:48.468 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:48.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:48.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:48 smithi153 bash[23852]: cluster 2024-04-06T03:34:46.922924+0000 osd.0 (osd.0) 302 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:48.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:48 smithi153 bash[23852]: cluster 2024-04-06T03:34:46.922998+0000 osd.0 (osd.0) 303 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:48.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:48 smithi153 bash[23852]: cluster 2024-04-06T03:34:47.395900+0000 mon.a (mon.0) 1350 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:48.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:48 smithi153 bash[23852]: audit 2024-04-06T03:34:47.397338+0000 mon.a (mon.0) 1351 : audit 1 from='client.? 172.21.15.50:0/3855915281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:34:48.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:48 smithi153 bash[23852]: cluster 2024-04-06T03:34:47.402454+0000 mon.a (mon.0) 1352 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-04-06T03:34:48.623 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:48.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:48.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[21467]: cluster 2024-04-06T03:34:46.922924+0000 osd.0 (osd.0) 302 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[21467]: cluster 2024-04-06T03:34:46.922998+0000 osd.0 (osd.0) 303 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[21467]: cluster 2024-04-06T03:34:47.395900+0000 mon.a (mon.0) 1350 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[21467]: audit 2024-04-06T03:34:47.397338+0000 mon.a (mon.0) 1351 : audit 1 from='client.? 172.21.15.50:0/3855915281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[21467]: cluster 2024-04-06T03:34:47.402454+0000 mon.a (mon.0) 1352 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[29221]: cluster 2024-04-06T03:34:46.922924+0000 osd.0 (osd.0) 302 : cluster 0 purged_snaps scrub starts 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[29221]: cluster 2024-04-06T03:34:46.922998+0000 osd.0 (osd.0) 303 : cluster 0 purged_snaps scrub ok 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[29221]: cluster 2024-04-06T03:34:47.395900+0000 mon.a (mon.0) 1350 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[29221]: audit 2024-04-06T03:34:47.397338+0000 mon.a (mon.0) 1351 : audit 1 from='client.? 172.21.15.50:0/3855915281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:34:48.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:48 smithi050 bash[29221]: cluster 2024-04-06T03:34:47.402454+0000 mon.a (mon.0) 1352 : cluster 0 osdmap e238: 8 total, 8 up, 8 in 2024-04-06T03:34:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:48.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:48.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:48.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:48.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:49.012 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:49.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:49.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:49.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:49.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:49.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:49.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 2 16 4390 4374 136.678 134.5 0.221893 0.00689688 2024-04-06T03:34:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:49.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:49.503 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:49 smithi153 bash[23852]: cluster 2024-04-06T03:34:48.265647+0000 mgr.y (mgr.24368) 512 : cluster 0 pgmap v533: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:49.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:49 smithi153 bash[23852]: cluster 2024-04-06T03:34:48.351964+0000 osd.0 (osd.0) 304 : cluster 0 3.17 scrub starts 2024-04-06T03:34:49.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:49 smithi153 bash[23852]: cluster 2024-04-06T03:34:48.358045+0000 osd.0 (osd.0) 305 : cluster 0 3.17 scrub ok 2024-04-06T03:34:49.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:49 smithi153 bash[23852]: audit 2024-04-06T03:34:49.408497+0000 mon.a (mon.0) 1353 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:49.504 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:49 smithi153 bash[23852]: audit 2024-04-06T03:34:49.412071+0000 mon.a (mon.0) 1354 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:49.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:49.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:49.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:49.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:34:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:49.767 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[21467]: cluster 2024-04-06T03:34:48.265647+0000 mgr.y (mgr.24368) 512 : cluster 0 pgmap v533: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[21467]: cluster 2024-04-06T03:34:48.351964+0000 osd.0 (osd.0) 304 : cluster 0 3.17 scrub starts 2024-04-06T03:34:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[21467]: cluster 2024-04-06T03:34:48.358045+0000 osd.0 (osd.0) 305 : cluster 0 3.17 scrub ok 2024-04-06T03:34:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[21467]: audit 2024-04-06T03:34:49.408497+0000 mon.a (mon.0) 1353 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:49.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[21467]: audit 2024-04-06T03:34:49.412071+0000 mon.a (mon.0) 1354 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:49.769 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:34:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:34:49.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[29221]: cluster 2024-04-06T03:34:48.265647+0000 mgr.y (mgr.24368) 512 : cluster 0 pgmap v533: 148 pgs: 148 active+clean; 582 KiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:34:49.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[29221]: cluster 2024-04-06T03:34:48.351964+0000 osd.0 (osd.0) 304 : cluster 0 3.17 scrub starts 2024-04-06T03:34:49.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[29221]: cluster 2024-04-06T03:34:48.358045+0000 osd.0 (osd.0) 305 : cluster 0 3.17 scrub ok 2024-04-06T03:34:49.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[29221]: audit 2024-04-06T03:34:49.408497+0000 mon.a (mon.0) 1353 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:34:49.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:49 smithi050 bash[29221]: audit 2024-04-06T03:34:49.412071+0000 mon.a (mon.0) 1354 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:34:49.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:49.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:49.898 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:49.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:50.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:50.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:50.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:50.149 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:50.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:50.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:50.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:50.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 3 15 6530 6515 135.718 133.812 0.00438469 0.00735624 2024-04-06T03:34:50.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:50.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:50.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:50 smithi153 bash[23852]: cluster 2024-04-06T03:34:48.450297+0000 osd.1 (osd.1) 267 : cluster 0 3.19 scrub starts 2024-04-06T03:34:50.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:50 smithi153 bash[23852]: cluster 2024-04-06T03:34:48.461759+0000 osd.1 (osd.1) 268 : cluster 0 3.19 scrub ok 2024-04-06T03:34:50.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:50 smithi153 bash[23852]: cluster 2024-04-06T03:34:49.335801+0000 osd.4 (osd.4) 309 : cluster 0 3.1d scrub starts 2024-04-06T03:34:50.521 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:50 smithi153 bash[23852]: cluster 2024-04-06T03:34:49.339785+0000 osd.4 (osd.4) 310 : cluster 0 3.1d scrub ok 2024-04-06T03:34:50.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:50 smithi153 bash[23852]: audit 2024-04-06T03:34:49.664443+0000 mgr.y (mgr.24368) 513 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:50.535 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:50.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:50.663 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:50.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[21467]: cluster 2024-04-06T03:34:48.450297+0000 osd.1 (osd.1) 267 : cluster 0 3.19 scrub starts 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[21467]: cluster 2024-04-06T03:34:48.461759+0000 osd.1 (osd.1) 268 : cluster 0 3.19 scrub ok 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[21467]: cluster 2024-04-06T03:34:49.335801+0000 osd.4 (osd.4) 309 : cluster 0 3.1d scrub starts 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[21467]: cluster 2024-04-06T03:34:49.339785+0000 osd.4 (osd.4) 310 : cluster 0 3.1d scrub ok 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[21467]: audit 2024-04-06T03:34:49.664443+0000 mgr.y (mgr.24368) 513 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[29221]: cluster 2024-04-06T03:34:48.450297+0000 osd.1 (osd.1) 267 : cluster 0 3.19 scrub starts 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[29221]: cluster 2024-04-06T03:34:48.461759+0000 osd.1 (osd.1) 268 : cluster 0 3.19 scrub ok 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[29221]: cluster 2024-04-06T03:34:49.335801+0000 osd.4 (osd.4) 309 : cluster 0 3.1d scrub starts 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[29221]: cluster 2024-04-06T03:34:49.339785+0000 osd.4 (osd.4) 310 : cluster 0 3.1d scrub ok 2024-04-06T03:34:50.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:50 smithi050 bash[29221]: audit 2024-04-06T03:34:49.664443+0000 mgr.y (mgr.24368) 513 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:34:50.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:50.788 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:50.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:50.916 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:50.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:51.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:51.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:51.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:51.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:51.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:51.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:51.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:51.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 4 15 9165 9150 142.957 164.688 0.00336423 0.00698647 2024-04-06T03:34:51.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:51.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:51.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:51 smithi153 bash[23852]: cluster 2024-04-06T03:34:49.438681+0000 osd.1 (osd.1) 269 : cluster 0 2.9 scrub starts 2024-04-06T03:34:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:51 smithi153 bash[23852]: cluster 2024-04-06T03:34:49.441879+0000 osd.1 (osd.1) 270 : cluster 0 2.9 scrub ok 2024-04-06T03:34:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:51 smithi153 bash[23852]: cluster 2024-04-06T03:34:50.172374+0000 osd.2 (osd.2) 263 : cluster 0 2.10 scrub starts 2024-04-06T03:34:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:51 smithi153 bash[23852]: cluster 2024-04-06T03:34:50.174206+0000 osd.2 (osd.2) 264 : cluster 0 2.10 scrub ok 2024-04-06T03:34:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:51 smithi153 bash[23852]: cluster 2024-04-06T03:34:50.266415+0000 mgr.y (mgr.24368) 514 : cluster 0 pgmap v534: 148 pgs: 148 active+clean; 162 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 20 MiB/s wr, 323 op/s 2024-04-06T03:34:51.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:51.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:51.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:51.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:51.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:51.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:51.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[21467]: cluster 2024-04-06T03:34:49.438681+0000 osd.1 (osd.1) 269 : cluster 0 2.9 scrub starts 2024-04-06T03:34:51.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[21467]: cluster 2024-04-06T03:34:49.441879+0000 osd.1 (osd.1) 270 : cluster 0 2.9 scrub ok 2024-04-06T03:34:51.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[21467]: cluster 2024-04-06T03:34:50.172374+0000 osd.2 (osd.2) 263 : cluster 0 2.10 scrub starts 2024-04-06T03:34:51.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[21467]: cluster 2024-04-06T03:34:50.174206+0000 osd.2 (osd.2) 264 : cluster 0 2.10 scrub ok 2024-04-06T03:34:51.947 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[21467]: cluster 2024-04-06T03:34:50.266415+0000 mgr.y (mgr.24368) 514 : cluster 0 pgmap v534: 148 pgs: 148 active+clean; 162 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 20 MiB/s wr, 323 op/s 2024-04-06T03:34:51.947 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[29221]: cluster 2024-04-06T03:34:49.438681+0000 osd.1 (osd.1) 269 : cluster 0 2.9 scrub starts 2024-04-06T03:34:51.948 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[29221]: cluster 2024-04-06T03:34:49.441879+0000 osd.1 (osd.1) 270 : cluster 0 2.9 scrub ok 2024-04-06T03:34:51.948 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[29221]: cluster 2024-04-06T03:34:50.172374+0000 osd.2 (osd.2) 263 : cluster 0 2.10 scrub starts 2024-04-06T03:34:51.948 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[29221]: cluster 2024-04-06T03:34:50.174206+0000 osd.2 (osd.2) 264 : cluster 0 2.10 scrub ok 2024-04-06T03:34:51.948 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:51 smithi050 bash[29221]: cluster 2024-04-06T03:34:50.266415+0000 mgr.y (mgr.24368) 514 : cluster 0 pgmap v534: 148 pgs: 148 active+clean; 162 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 20 MiB/s wr, 323 op/s 2024-04-06T03:34:51.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:51.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:52.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:52.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:52.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:52.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:52.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:52.335 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:52.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 5 16 10322 10306 128.814 72.25 0.00250794 0.00760611 2024-04-06T03:34:52.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:52.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:52.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:52 smithi050 bash[21467]: cluster 2024-04-06T03:34:51.471662+0000 osd.0 (osd.0) 306 : cluster 0 6.16 scrub starts 2024-04-06T03:34:52.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:52 smithi050 bash[21467]: cluster 2024-04-06T03:34:51.514937+0000 osd.0 (osd.0) 307 : cluster 0 6.16 scrub ok 2024-04-06T03:34:52.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:52 smithi050 bash[29221]: cluster 2024-04-06T03:34:51.471662+0000 osd.0 (osd.0) 306 : cluster 0 6.16 scrub starts 2024-04-06T03:34:52.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:52 smithi050 bash[29221]: cluster 2024-04-06T03:34:51.514937+0000 osd.0 (osd.0) 307 : cluster 0 6.16 scrub ok 2024-04-06T03:34:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:52.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:52.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:52 smithi153 bash[23852]: cluster 2024-04-06T03:34:51.471662+0000 osd.0 (osd.0) 306 : cluster 0 6.16 scrub starts 2024-04-06T03:34:52.599 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:52 smithi153 bash[23852]: cluster 2024-04-06T03:34:51.514937+0000 osd.0 (osd.0) 307 : cluster 0 6.16 scrub ok 2024-04-06T03:34:52.615 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:52.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:52.742 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:52.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:52.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:52.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:52.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:53.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:53.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:53.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:53.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:53.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:53.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:53.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 6 16 12598 12582 131.051 142.25 0.00552869 0.00762303 2024-04-06T03:34:53.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:53.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:53.524 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:53.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[21467]: cluster 2024-04-06T03:34:52.267161+0000 mgr.y (mgr.24368) 515 : cluster 0 pgmap v535: 148 pgs: 148 active+clean; 360 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 648 op/s 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[21467]: cluster 2024-04-06T03:34:52.368027+0000 osd.1 (osd.1) 271 : cluster 0 4.2 scrub starts 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[21467]: cluster 2024-04-06T03:34:52.369911+0000 osd.1 (osd.1) 272 : cluster 0 4.2 scrub ok 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[21467]: cluster 2024-04-06T03:34:52.466171+0000 osd.3 (osd.3) 319 : cluster 0 6.b scrub starts 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[21467]: cluster 2024-04-06T03:34:53.077357+0000 osd.7 (osd.7) 337 : cluster 0 3.16 scrub starts 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[21467]: cluster 2024-04-06T03:34:53.085932+0000 osd.7 (osd.7) 338 : cluster 0 3.16 scrub ok 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[29221]: cluster 2024-04-06T03:34:52.267161+0000 mgr.y (mgr.24368) 515 : cluster 0 pgmap v535: 148 pgs: 148 active+clean; 360 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 648 op/s 2024-04-06T03:34:53.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[29221]: cluster 2024-04-06T03:34:52.368027+0000 osd.1 (osd.1) 271 : cluster 0 4.2 scrub starts 2024-04-06T03:34:53.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[29221]: cluster 2024-04-06T03:34:52.369911+0000 osd.1 (osd.1) 272 : cluster 0 4.2 scrub ok 2024-04-06T03:34:53.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[29221]: cluster 2024-04-06T03:34:52.466171+0000 osd.3 (osd.3) 319 : cluster 0 6.b scrub starts 2024-04-06T03:34:53.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[29221]: cluster 2024-04-06T03:34:53.077357+0000 osd.7 (osd.7) 337 : cluster 0 3.16 scrub starts 2024-04-06T03:34:53.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:53 smithi050 bash[29221]: cluster 2024-04-06T03:34:53.085932+0000 osd.7 (osd.7) 338 : cluster 0 3.16 scrub ok 2024-04-06T03:34:53.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:53.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:53.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:53 smithi153 bash[23852]: cluster 2024-04-06T03:34:52.267161+0000 mgr.y (mgr.24368) 515 : cluster 0 pgmap v535: 148 pgs: 148 active+clean; 360 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 648 op/s 2024-04-06T03:34:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:53 smithi153 bash[23852]: cluster 2024-04-06T03:34:52.368027+0000 osd.1 (osd.1) 271 : cluster 0 4.2 scrub starts 2024-04-06T03:34:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:53 smithi153 bash[23852]: cluster 2024-04-06T03:34:52.369911+0000 osd.1 (osd.1) 272 : cluster 0 4.2 scrub ok 2024-04-06T03:34:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:53 smithi153 bash[23852]: cluster 2024-04-06T03:34:52.466171+0000 osd.3 (osd.3) 319 : cluster 0 6.b scrub starts 2024-04-06T03:34:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:53 smithi153 bash[23852]: cluster 2024-04-06T03:34:53.077357+0000 osd.7 (osd.7) 337 : cluster 0 3.16 scrub starts 2024-04-06T03:34:53.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:53 smithi153 bash[23852]: cluster 2024-04-06T03:34:53.085932+0000 osd.7 (osd.7) 338 : cluster 0 3.16 scrub ok 2024-04-06T03:34:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:53.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:53.924 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:53.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:54.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:54.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:54.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:54.182 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:54.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:54.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:54.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 7 16 14464 14448 128.988 116.625 0.00141998 0.00754943 2024-04-06T03:34:54.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:54.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:54.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:54.563 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:54.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:52.570524+0000 osd.3 (osd.3) 320 : cluster 0 6.b scrub ok 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:53.308345+0000 osd.4 (osd.4) 311 : cluster 0 2.1a scrub starts 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:53.309992+0000 osd.4 (osd.4) 312 : cluster 0 2.1a scrub ok 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:53.414856+0000 osd.3 (osd.3) 321 : cluster 0 6.6 scrub starts 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:53.424430+0000 osd.3 (osd.3) 322 : cluster 0 6.6 scrub ok 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:54.349065+0000 osd.1 (osd.1) 273 : cluster 0 2.2 scrub starts 2024-04-06T03:34:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[21467]: cluster 2024-04-06T03:34:54.350575+0000 osd.1 (osd.1) 274 : cluster 0 2.2 scrub ok 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:52.570524+0000 osd.3 (osd.3) 320 : cluster 0 6.b scrub ok 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:53.308345+0000 osd.4 (osd.4) 311 : cluster 0 2.1a scrub starts 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:53.309992+0000 osd.4 (osd.4) 312 : cluster 0 2.1a scrub ok 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:53.414856+0000 osd.3 (osd.3) 321 : cluster 0 6.6 scrub starts 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:53.424430+0000 osd.3 (osd.3) 322 : cluster 0 6.6 scrub ok 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:54.349065+0000 osd.1 (osd.1) 273 : cluster 0 2.2 scrub starts 2024-04-06T03:34:54.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:54 smithi050 bash[29221]: cluster 2024-04-06T03:34:54.350575+0000 osd.1 (osd.1) 274 : cluster 0 2.2 scrub ok 2024-04-06T03:34:54.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:52.570524+0000 osd.3 (osd.3) 320 : cluster 0 6.b scrub ok 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:53.308345+0000 osd.4 (osd.4) 311 : cluster 0 2.1a scrub starts 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:53.309992+0000 osd.4 (osd.4) 312 : cluster 0 2.1a scrub ok 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:53.414856+0000 osd.3 (osd.3) 321 : cluster 0 6.6 scrub starts 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:53.424430+0000 osd.3 (osd.3) 322 : cluster 0 6.6 scrub ok 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:54.349065+0000 osd.1 (osd.1) 273 : cluster 0 2.2 scrub starts 2024-04-06T03:34:54.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:54 smithi153 bash[23852]: cluster 2024-04-06T03:34:54.350575+0000 osd.1 (osd.1) 274 : cluster 0 2.2 scrub ok 2024-04-06T03:34:54.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:54.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:34:54.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:54.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:34:54.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:54.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:55.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:55.097 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:55.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:55.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:55.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:55.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 8 16 16012 15996 124.957 96.75 0.00467766 0.00799505 2024-04-06T03:34:55.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:55.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:55.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:55.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:55.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:55.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:55 smithi050 bash[21467]: cluster 2024-04-06T03:34:54.267815+0000 mgr.y (mgr.24368) 516 : cluster 0 pgmap v536: 148 pgs: 148 active+clean; 557 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 891 op/s 2024-04-06T03:34:55.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:55 smithi050 bash[21467]: cluster 2024-04-06T03:34:55.361767+0000 osd.1 (osd.1) 275 : cluster 0 5.1d scrub starts 2024-04-06T03:34:55.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:55 smithi050 bash[21467]: cluster 2024-04-06T03:34:55.364637+0000 osd.1 (osd.1) 276 : cluster 0 5.1d scrub ok 2024-04-06T03:34:55.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:55 smithi050 bash[29221]: cluster 2024-04-06T03:34:54.267815+0000 mgr.y (mgr.24368) 516 : cluster 0 pgmap v536: 148 pgs: 148 active+clean; 557 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 891 op/s 2024-04-06T03:34:55.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:55 smithi050 bash[29221]: cluster 2024-04-06T03:34:55.361767+0000 osd.1 (osd.1) 275 : cluster 0 5.1d scrub starts 2024-04-06T03:34:55.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:55 smithi050 bash[29221]: cluster 2024-04-06T03:34:55.364637+0000 osd.1 (osd.1) 276 : cluster 0 5.1d scrub ok 2024-04-06T03:34:55.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:55 smithi153 bash[23852]: cluster 2024-04-06T03:34:54.267815+0000 mgr.y (mgr.24368) 516 : cluster 0 pgmap v536: 148 pgs: 148 active+clean; 557 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 891 op/s 2024-04-06T03:34:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:55 smithi153 bash[23852]: cluster 2024-04-06T03:34:55.361767+0000 osd.1 (osd.1) 275 : cluster 0 5.1d scrub starts 2024-04-06T03:34:55.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:55 smithi153 bash[23852]: cluster 2024-04-06T03:34:55.364637+0000 osd.1 (osd.1) 276 : cluster 0 5.1d scrub ok 2024-04-06T03:34:55.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:55.881 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:55.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:55.963 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:55.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:56.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:56.139 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:56.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:56.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:56.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:56.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 9 16 18159 18143 125.981 134.188 0.012221 0.00793048 2024-04-06T03:34:56.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:56.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:56.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:56.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:56.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:56.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:56.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:56 smithi153 bash[23852]: cluster 2024-04-06T03:34:55.140875+0000 osd.2 (osd.2) 265 : cluster 0 2.17 scrub starts 2024-04-06T03:34:56.780 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:56 smithi153 bash[23852]: cluster 2024-04-06T03:34:55.142194+0000 osd.2 (osd.2) 266 : cluster 0 2.17 scrub ok 2024-04-06T03:34:56.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:56 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.031735+0000 osd.7 (osd.7) 339 : cluster 0 6.8 scrub starts 2024-04-06T03:34:56.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:56 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.033852+0000 osd.7 (osd.7) 340 : cluster 0 6.8 scrub ok 2024-04-06T03:34:56.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:56 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.369553+0000 osd.1 (osd.1) 277 : cluster 0 6.1 scrub starts 2024-04-06T03:34:56.781 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:56 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.405145+0000 osd.1 (osd.1) 278 : cluster 0 6.1 scrub ok 2024-04-06T03:34:56.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:56.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:56.920 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:56.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:57.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:57.050 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:57.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:56 smithi050 bash[21467]: cluster 2024-04-06T03:34:55.140875+0000 osd.2 (osd.2) 265 : cluster 0 2.17 scrub starts 2024-04-06T03:34:57.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:56 smithi050 bash[21467]: cluster 2024-04-06T03:34:55.142194+0000 osd.2 (osd.2) 266 : cluster 0 2.17 scrub ok 2024-04-06T03:34:57.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:56 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.031735+0000 osd.7 (osd.7) 339 : cluster 0 6.8 scrub starts 2024-04-06T03:34:57.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:56 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.033852+0000 osd.7 (osd.7) 340 : cluster 0 6.8 scrub ok 2024-04-06T03:34:57.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:56 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.369553+0000 osd.1 (osd.1) 277 : cluster 0 6.1 scrub starts 2024-04-06T03:34:57.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:56 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.405145+0000 osd.1 (osd.1) 278 : cluster 0 6.1 scrub ok 2024-04-06T03:34:57.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:57.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:57.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:57.320 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:34:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:34:57.391 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 10 16 20282 20266 126.651 132.688 0.00254684 0.0078887 2024-04-06T03:34:57.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:57.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:55.140875+0000 osd.2 (osd.2) 265 : cluster 0 2.17 scrub starts 2024-04-06T03:34:57.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:55.142194+0000 osd.2 (osd.2) 266 : cluster 0 2.17 scrub ok 2024-04-06T03:34:57.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.031735+0000 osd.7 (osd.7) 339 : cluster 0 6.8 scrub starts 2024-04-06T03:34:57.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.033852+0000 osd.7 (osd.7) 340 : cluster 0 6.8 scrub ok 2024-04-06T03:34:57.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.369553+0000 osd.1 (osd.1) 277 : cluster 0 6.1 scrub starts 2024-04-06T03:34:57.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.405145+0000 osd.1 (osd.1) 278 : cluster 0 6.1 scrub ok 2024-04-06T03:34:57.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:57.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:57.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:57.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.268987+0000 mgr.y (mgr.24368) 517 : cluster 0 pgmap v537: 148 pgs: 148 active+clean; 966 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 97 MiB/s wr, 1.54k op/s 2024-04-06T03:34:57.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.327862+0000 osd.4 (osd.4) 313 : cluster 0 6.1a scrub starts 2024-04-06T03:34:57.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.377338+0000 osd.4 (osd.4) 314 : cluster 0 6.1a scrub ok 2024-04-06T03:34:57.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.432965+0000 osd.3 (osd.3) 323 : cluster 0 5.17 scrub starts 2024-04-06T03:34:57.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[21467]: cluster 2024-04-06T03:34:56.435922+0000 osd.3 (osd.3) 324 : cluster 0 5.17 scrub ok 2024-04-06T03:34:57.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.268987+0000 mgr.y (mgr.24368) 517 : cluster 0 pgmap v537: 148 pgs: 148 active+clean; 966 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 97 MiB/s wr, 1.54k op/s 2024-04-06T03:34:57.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.327862+0000 osd.4 (osd.4) 313 : cluster 0 6.1a scrub starts 2024-04-06T03:34:57.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.377338+0000 osd.4 (osd.4) 314 : cluster 0 6.1a scrub ok 2024-04-06T03:34:57.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.432965+0000 osd.3 (osd.3) 323 : cluster 0 5.17 scrub starts 2024-04-06T03:34:57.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:57 smithi050 bash[29221]: cluster 2024-04-06T03:34:56.435922+0000 osd.3 (osd.3) 324 : cluster 0 5.17 scrub ok 2024-04-06T03:34:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:57 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.268987+0000 mgr.y (mgr.24368) 517 : cluster 0 pgmap v537: 148 pgs: 148 active+clean; 966 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 97 MiB/s wr, 1.54k op/s 2024-04-06T03:34:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:57 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.327862+0000 osd.4 (osd.4) 313 : cluster 0 6.1a scrub starts 2024-04-06T03:34:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:57 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.377338+0000 osd.4 (osd.4) 314 : cluster 0 6.1a scrub ok 2024-04-06T03:34:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:57 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.432965+0000 osd.3 (osd.3) 323 : cluster 0 5.17 scrub starts 2024-04-06T03:34:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:57 smithi153 bash[23852]: cluster 2024-04-06T03:34:56.435922+0000 osd.3 (osd.3) 324 : cluster 0 5.17 scrub ok 2024-04-06T03:34:57.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:57.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:57.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:57.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:57.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:58.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:58.096 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:58.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:58.222 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:58.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:58.349 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:58.392 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 11 16 22054 22038 125.204 110.75 0.00645301 0.00797944 2024-04-06T03:34:58.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:58.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:58.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:58.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:57.185108+0000 osd.2 (osd.2) 267 : cluster 0 2.c scrub starts 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:57.188691+0000 osd.2 (osd.2) 268 : cluster 0 2.c scrub ok 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:57.357588+0000 osd.4 (osd.4) 315 : cluster 0 5.f scrub starts 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:57.359012+0000 osd.4 (osd.4) 316 : cluster 0 5.f scrub ok 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:57.492988+0000 osd.0 (osd.0) 308 : cluster 0 2.4 scrub starts 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:57.496840+0000 osd.0 (osd.0) 309 : cluster 0 2.4 scrub ok 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:58.377700+0000 osd.1 (osd.1) 279 : cluster 0 2.6 scrub starts 2024-04-06T03:34:58.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[21467]: cluster 2024-04-06T03:34:58.382071+0000 osd.1 (osd.1) 280 : cluster 0 2.6 scrub ok 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:57.185108+0000 osd.2 (osd.2) 267 : cluster 0 2.c scrub starts 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:57.188691+0000 osd.2 (osd.2) 268 : cluster 0 2.c scrub ok 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:57.357588+0000 osd.4 (osd.4) 315 : cluster 0 5.f scrub starts 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:57.359012+0000 osd.4 (osd.4) 316 : cluster 0 5.f scrub ok 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:57.492988+0000 osd.0 (osd.0) 308 : cluster 0 2.4 scrub starts 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:57.496840+0000 osd.0 (osd.0) 309 : cluster 0 2.4 scrub ok 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:58.377700+0000 osd.1 (osd.1) 279 : cluster 0 2.6 scrub starts 2024-04-06T03:34:58.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:58 smithi050 bash[29221]: cluster 2024-04-06T03:34:58.382071+0000 osd.1 (osd.1) 280 : cluster 0 2.6 scrub ok 2024-04-06T03:34:58.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:58.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:57.185108+0000 osd.2 (osd.2) 267 : cluster 0 2.c scrub starts 2024-04-06T03:34:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:57.188691+0000 osd.2 (osd.2) 268 : cluster 0 2.c scrub ok 2024-04-06T03:34:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:57.357588+0000 osd.4 (osd.4) 315 : cluster 0 5.f scrub starts 2024-04-06T03:34:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:57.359012+0000 osd.4 (osd.4) 316 : cluster 0 5.f scrub ok 2024-04-06T03:34:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:57.492988+0000 osd.0 (osd.0) 308 : cluster 0 2.4 scrub starts 2024-04-06T03:34:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:57.496840+0000 osd.0 (osd.0) 309 : cluster 0 2.4 scrub ok 2024-04-06T03:34:58.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:58.377700+0000 osd.1 (osd.1) 279 : cluster 0 2.6 scrub starts 2024-04-06T03:34:58.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:58 smithi153 bash[23852]: cluster 2024-04-06T03:34:58.382071+0000 osd.1 (osd.1) 280 : cluster 0 2.6 scrub ok 2024-04-06T03:34:58.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:58.873 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:34:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:58.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:34:58.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:59.003 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:34:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:34:59.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:59.131 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:34:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:34:59.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:59.262 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:59.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:59.388 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:34:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:34:59.392 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 12 16 24114 24098 125.499 128.75 0.00923375 0.00796128 2024-04-06T03:34:59.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:59.495 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[21467]: cluster 2024-04-06T03:34:58.269338+0000 mgr.y (mgr.24368) 518 : cluster 0 pgmap v538: 148 pgs: 148 active+clean; 966 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 847 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[21467]: cluster 2024-04-06T03:34:58.360040+0000 osd.4 (osd.4) 317 : cluster 0 2.18 scrub starts 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[21467]: cluster 2024-04-06T03:34:58.361224+0000 osd.4 (osd.4) 318 : cluster 0 2.18 scrub ok 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[21467]: cluster 2024-04-06T03:34:59.378831+0000 osd.1 (osd.1) 281 : cluster 0 2.15 scrub starts 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[21467]: cluster 2024-04-06T03:34:59.380357+0000 osd.1 (osd.1) 282 : cluster 0 2.15 scrub ok 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[29221]: cluster 2024-04-06T03:34:58.269338+0000 mgr.y (mgr.24368) 518 : cluster 0 pgmap v538: 148 pgs: 148 active+clean; 966 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 847 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[29221]: cluster 2024-04-06T03:34:58.360040+0000 osd.4 (osd.4) 317 : cluster 0 2.18 scrub starts 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[29221]: cluster 2024-04-06T03:34:58.361224+0000 osd.4 (osd.4) 318 : cluster 0 2.18 scrub ok 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[29221]: cluster 2024-04-06T03:34:59.378831+0000 osd.1 (osd.1) 281 : cluster 0 2.15 scrub starts 2024-04-06T03:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[29221]: cluster 2024-04-06T03:34:59.380357+0000 osd.1 (osd.1) 282 : cluster 0 2.15 scrub ok 2024-04-06T03:34:59.510 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:34:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:34:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:34:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:34:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:34:59.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:59.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:34:59.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:59 smithi153 bash[23852]: cluster 2024-04-06T03:34:58.269338+0000 mgr.y (mgr.24368) 518 : cluster 0 pgmap v538: 148 pgs: 148 active+clean; 966 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 847 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-04-06T03:34:59.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:59 smithi153 bash[23852]: cluster 2024-04-06T03:34:58.360040+0000 osd.4 (osd.4) 317 : cluster 0 2.18 scrub starts 2024-04-06T03:34:59.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:59 smithi153 bash[23852]: cluster 2024-04-06T03:34:58.361224+0000 osd.4 (osd.4) 318 : cluster 0 2.18 scrub ok 2024-04-06T03:34:59.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:59 smithi153 bash[23852]: cluster 2024-04-06T03:34:59.378831+0000 osd.1 (osd.1) 281 : cluster 0 2.15 scrub starts 2024-04-06T03:34:59.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:34:59 smithi153 bash[23852]: cluster 2024-04-06T03:34:59.380357+0000 osd.1 (osd.1) 282 : cluster 0 2.15 scrub ok 2024-04-06T03:34:59.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:59.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:34:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:34:59.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:34:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:34:59.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:59.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:34:59.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:34:59.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:00.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:00.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:00.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:00.177 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:00.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:00.305 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:00.392 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 13 16 25788 25772 123.892 104.625 0.00335384 0.00806568 2024-04-06T03:35:00.408 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:00.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:00.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:00 smithi153 bash[23852]: cluster 2024-04-06T03:34:59.514181+0000 osd.0 (osd.0) 310 : cluster 0 6.d scrub starts 2024-04-06T03:35:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:00 smithi153 bash[23852]: audit 2024-04-06T03:34:59.672279+0000 mgr.y (mgr.24368) 519 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:00.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:00.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:00.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:00.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:00 smithi050 bash[21467]: cluster 2024-04-06T03:34:59.514181+0000 osd.0 (osd.0) 310 : cluster 0 6.d scrub starts 2024-04-06T03:35:00.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:00 smithi050 bash[21467]: audit 2024-04-06T03:34:59.672279+0000 mgr.y (mgr.24368) 519 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:00.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:00 smithi050 bash[29221]: cluster 2024-04-06T03:34:59.514181+0000 osd.0 (osd.0) 310 : cluster 0 6.d scrub starts 2024-04-06T03:35:00.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:00 smithi050 bash[29221]: audit 2024-04-06T03:34:59.672279+0000 mgr.y (mgr.24368) 519 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:00.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:00.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:00.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:00.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:01.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:01.096 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:01.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:01.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:01.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:01.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:01.392 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 14 16 27041 27025 120.636 78.3125 0.00200777 0.0081128 2024-04-06T03:35:01.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:01.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:01.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:01.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:01.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[21467]: cluster 2024-04-06T03:34:59.619574+0000 osd.0 (osd.0) 311 : cluster 0 6.d scrub ok 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[21467]: cluster 2024-04-06T03:35:00.270214+0000 mgr.y (mgr.24368) 520 : cluster 0 pgmap v539: 148 pgs: 148 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[21467]: cluster 2024-04-06T03:35:00.330868+0000 osd.2 (osd.2) 269 : cluster 0 5.d scrub starts 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[21467]: cluster 2024-04-06T03:35:00.333684+0000 osd.2 (osd.2) 270 : cluster 0 5.d scrub ok 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[21467]: cluster 2024-04-06T03:35:00.394366+0000 osd.3 (osd.3) 325 : cluster 0 3.5 scrub starts 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[21467]: cluster 2024-04-06T03:35:00.398297+0000 osd.3 (osd.3) 326 : cluster 0 3.5 scrub ok 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[29221]: cluster 2024-04-06T03:34:59.619574+0000 osd.0 (osd.0) 311 : cluster 0 6.d scrub ok 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[29221]: cluster 2024-04-06T03:35:00.270214+0000 mgr.y (mgr.24368) 520 : cluster 0 pgmap v539: 148 pgs: 148 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[29221]: cluster 2024-04-06T03:35:00.330868+0000 osd.2 (osd.2) 269 : cluster 0 5.d scrub starts 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[29221]: cluster 2024-04-06T03:35:00.333684+0000 osd.2 (osd.2) 270 : cluster 0 5.d scrub ok 2024-04-06T03:35:01.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[29221]: cluster 2024-04-06T03:35:00.394366+0000 osd.3 (osd.3) 325 : cluster 0 3.5 scrub starts 2024-04-06T03:35:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:01 smithi050 bash[29221]: cluster 2024-04-06T03:35:00.398297+0000 osd.3 (osd.3) 326 : cluster 0 3.5 scrub ok 2024-04-06T03:35:01.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:01.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:01 smithi153 bash[23852]: cluster 2024-04-06T03:34:59.619574+0000 osd.0 (osd.0) 311 : cluster 0 6.d scrub ok 2024-04-06T03:35:01.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:01 smithi153 bash[23852]: cluster 2024-04-06T03:35:00.270214+0000 mgr.y (mgr.24368) 520 : cluster 0 pgmap v539: 148 pgs: 148 active+clean; 1.3 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-04-06T03:35:01.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:01 smithi153 bash[23852]: cluster 2024-04-06T03:35:00.330868+0000 osd.2 (osd.2) 269 : cluster 0 5.d scrub starts 2024-04-06T03:35:01.752 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:01 smithi153 bash[23852]: cluster 2024-04-06T03:35:00.333684+0000 osd.2 (osd.2) 270 : cluster 0 5.d scrub ok 2024-04-06T03:35:01.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:01 smithi153 bash[23852]: cluster 2024-04-06T03:35:00.394366+0000 osd.3 (osd.3) 325 : cluster 0 3.5 scrub starts 2024-04-06T03:35:01.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:01 smithi153 bash[23852]: cluster 2024-04-06T03:35:00.398297+0000 osd.3 (osd.3) 326 : cluster 0 3.5 scrub ok 2024-04-06T03:35:01.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:01.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:01.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:01.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:01.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:02.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:02.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:02.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:02.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:02.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:02.392 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 15 16 28044 28028 116.768 62.6875 0.0164041 0.00855731 2024-04-06T03:35:02.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:02.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:02.533 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:02.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:02.646 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:02 smithi050 bash[21467]: cluster 2024-04-06T03:35:01.534197+0000 osd.0 (osd.0) 312 : cluster 0 5.1e scrub starts 2024-04-06T03:35:02.646 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:02 smithi050 bash[21467]: cluster 2024-04-06T03:35:01.539694+0000 osd.0 (osd.0) 313 : cluster 0 5.1e scrub ok 2024-04-06T03:35:02.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:02 smithi050 bash[29221]: cluster 2024-04-06T03:35:01.534197+0000 osd.0 (osd.0) 312 : cluster 0 5.1e scrub starts 2024-04-06T03:35:02.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:02 smithi050 bash[29221]: cluster 2024-04-06T03:35:01.539694+0000 osd.0 (osd.0) 313 : cluster 0 5.1e scrub ok 2024-04-06T03:35:02.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:02.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:02.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:02 smithi153 bash[23852]: cluster 2024-04-06T03:35:01.534197+0000 osd.0 (osd.0) 312 : cluster 0 5.1e scrub starts 2024-04-06T03:35:02.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:02 smithi153 bash[23852]: cluster 2024-04-06T03:35:01.539694+0000 osd.0 (osd.0) 313 : cluster 0 5.1e scrub ok 2024-04-06T03:35:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:02.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:02.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:02.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:03.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:03.055 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:03.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:03.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:03.287 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:03.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:03.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 16 15 29793 29778 116.306 109.375 0.00694105 0.00859256 2024-04-06T03:35:03.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:03.441 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:03.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:03.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:03 smithi153 bash[23852]: cluster 2024-04-06T03:35:02.270902+0000 mgr.y (mgr.24368) 521 : cluster 0 pgmap v540: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-04-06T03:35:03.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:03 smithi153 bash[23852]: cluster 2024-04-06T03:35:02.447967+0000 osd.2 (osd.2) 271 : cluster 0 6.f scrub starts 2024-04-06T03:35:03.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:03 smithi153 bash[23852]: cluster 2024-04-06T03:35:02.582184+0000 osd.2 (osd.2) 272 : cluster 0 6.f scrub ok 2024-04-06T03:35:03.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:03.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:03.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:03.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:03.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:03.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:03.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:03 smithi050 bash[29221]: cluster 2024-04-06T03:35:02.270902+0000 mgr.y (mgr.24368) 521 : cluster 0 pgmap v540: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-04-06T03:35:03.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:03 smithi050 bash[29221]: cluster 2024-04-06T03:35:02.447967+0000 osd.2 (osd.2) 271 : cluster 0 6.f scrub starts 2024-04-06T03:35:03.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:03 smithi050 bash[29221]: cluster 2024-04-06T03:35:02.582184+0000 osd.2 (osd.2) 272 : cluster 0 6.f scrub ok 2024-04-06T03:35:03.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:03 smithi050 bash[21467]: cluster 2024-04-06T03:35:02.270902+0000 mgr.y (mgr.24368) 521 : cluster 0 pgmap v540: 148 pgs: 148 active+clean; 1.5 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-04-06T03:35:03.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:03 smithi050 bash[21467]: cluster 2024-04-06T03:35:02.447967+0000 osd.2 (osd.2) 271 : cluster 0 6.f scrub starts 2024-04-06T03:35:03.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:03 smithi050 bash[21467]: cluster 2024-04-06T03:35:02.582184+0000 osd.2 (osd.2) 272 : cluster 0 6.f scrub ok 2024-04-06T03:35:03.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:03.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:03.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:04.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:04.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:04.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:04.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:04.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:04.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:04.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 17 16 30726 30710 112.89 58.25 0.00821245 0.00883626 2024-04-06T03:35:04.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:04.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:04.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:04.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:03.378637+0000 osd.4 (osd.4) 319 : cluster 0 6.2 scrub starts 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:03.380667+0000 osd.4 (osd.4) 320 : cluster 0 6.2 scrub ok 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:03.474592+0000 osd.0 (osd.0) 314 : cluster 0 3.12 scrub starts 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:03.477222+0000 osd.0 (osd.0) 315 : cluster 0 3.12 scrub ok 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.146446+0000 osd.7 (osd.7) 341 : cluster 0 5.4 scrub starts 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.150552+0000 osd.7 (osd.7) 342 : cluster 0 5.4 scrub ok 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.394904+0000 osd.1 (osd.1) 283 : cluster 0 5.12 scrub starts 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.398399+0000 osd.1 (osd.1) 284 : cluster 0 5.12 scrub ok 2024-04-06T03:35:04.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[21467]: audit 2024-04-06T03:35:04.448877+0000 mon.a (mon.0) 1355 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:03.378637+0000 osd.4 (osd.4) 319 : cluster 0 6.2 scrub starts 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:03.380667+0000 osd.4 (osd.4) 320 : cluster 0 6.2 scrub ok 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:03.474592+0000 osd.0 (osd.0) 314 : cluster 0 3.12 scrub starts 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:03.477222+0000 osd.0 (osd.0) 315 : cluster 0 3.12 scrub ok 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.146446+0000 osd.7 (osd.7) 341 : cluster 0 5.4 scrub starts 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.150552+0000 osd.7 (osd.7) 342 : cluster 0 5.4 scrub ok 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.394904+0000 osd.1 (osd.1) 283 : cluster 0 5.12 scrub starts 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.398399+0000 osd.1 (osd.1) 284 : cluster 0 5.12 scrub ok 2024-04-06T03:35:04.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:04 smithi050 bash[29221]: audit 2024-04-06T03:35:04.448877+0000 mon.a (mon.0) 1355 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:04.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:04.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:04.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:03.378637+0000 osd.4 (osd.4) 319 : cluster 0 6.2 scrub starts 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:03.380667+0000 osd.4 (osd.4) 320 : cluster 0 6.2 scrub ok 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:03.474592+0000 osd.0 (osd.0) 314 : cluster 0 3.12 scrub starts 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:03.477222+0000 osd.0 (osd.0) 315 : cluster 0 3.12 scrub ok 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.146446+0000 osd.7 (osd.7) 341 : cluster 0 5.4 scrub starts 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.150552+0000 osd.7 (osd.7) 342 : cluster 0 5.4 scrub ok 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.394904+0000 osd.1 (osd.1) 283 : cluster 0 5.12 scrub starts 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.398399+0000 osd.1 (osd.1) 284 : cluster 0 5.12 scrub ok 2024-04-06T03:35:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:04 smithi153 bash[23852]: audit 2024-04-06T03:35:04.448877+0000 mon.a (mon.0) 1355 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:04.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:04.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:04.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:04.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:04.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:05.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:05.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:05.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:05.254 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:05.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:05.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:05.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:05.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 18 16 31732 31716 110.111 62.875 0.00378059 0.00907565 2024-04-06T03:35:05.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:05.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:05.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:05.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:03.870601+0000 osd.3 (osd.3) 327 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:05.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:03.870734+0000 osd.3 (osd.3) 328 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:05.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.271694+0000 mgr.y (mgr.24368) 522 : cluster 0 pgmap v541: 148 pgs: 148 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-04-06T03:35:05.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.469356+0000 osd.2 (osd.2) 273 : cluster 0 5.8 scrub starts 2024-04-06T03:35:05.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.470680+0000 osd.2 (osd.2) 274 : cluster 0 5.8 scrub ok 2024-04-06T03:35:05.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.482162+0000 osd.0 (osd.0) 316 : cluster 0 6.9 scrub starts 2024-04-06T03:35:05.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.562150+0000 osd.0 (osd.0) 317 : cluster 0 6.9 scrub ok 2024-04-06T03:35:05.633 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:05 smithi153 bash[23852]: cluster 2024-04-06T03:35:04.820441+0000 mon.a (mon.0) 1356 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-04-06T03:35:05.643 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:05.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:05.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:05.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:03.870601+0000 osd.3 (osd.3) 327 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:03.870734+0000 osd.3 (osd.3) 328 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.271694+0000 mgr.y (mgr.24368) 522 : cluster 0 pgmap v541: 148 pgs: 148 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.469356+0000 osd.2 (osd.2) 273 : cluster 0 5.8 scrub starts 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.470680+0000 osd.2 (osd.2) 274 : cluster 0 5.8 scrub ok 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.482162+0000 osd.0 (osd.0) 316 : cluster 0 6.9 scrub starts 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.562150+0000 osd.0 (osd.0) 317 : cluster 0 6.9 scrub ok 2024-04-06T03:35:05.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[21467]: cluster 2024-04-06T03:35:04.820441+0000 mon.a (mon.0) 1356 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:03.870601+0000 osd.3 (osd.3) 327 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:03.870734+0000 osd.3 (osd.3) 328 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.271694+0000 mgr.y (mgr.24368) 522 : cluster 0 pgmap v541: 148 pgs: 148 active+clean; 1.7 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 767 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.469356+0000 osd.2 (osd.2) 273 : cluster 0 5.8 scrub starts 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.470680+0000 osd.2 (osd.2) 274 : cluster 0 5.8 scrub ok 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.482162+0000 osd.0 (osd.0) 316 : cluster 0 6.9 scrub starts 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.562150+0000 osd.0 (osd.0) 317 : cluster 0 6.9 scrub ok 2024-04-06T03:35:05.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:05 smithi050 bash[29221]: cluster 2024-04-06T03:35:04.820441+0000 mon.a (mon.0) 1356 : cluster 0 osdmap e239: 8 total, 8 up, 8 in 2024-04-06T03:35:05.909 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:05.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:05.920 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:35:05.921 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:35:06.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:06.038 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:06.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:06.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:06.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:06.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:06.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 19 16 33367 33351 109.694 102.188 0.00872508 0.00911044 2024-04-06T03:35:06.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:06.429 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:06.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:06 smithi153 bash[23852]: cluster 2024-04-06T03:35:05.506223+0000 osd.0 (osd.0) 318 : cluster 0 3.10 scrub starts 2024-04-06T03:35:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:06 smithi153 bash[23852]: cluster 2024-04-06T03:35:05.512123+0000 osd.0 (osd.0) 319 : cluster 0 3.10 scrub ok 2024-04-06T03:35:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:06 smithi153 bash[23852]: cluster 2024-04-06T03:35:06.325917+0000 osd.1 (osd.1) 285 : cluster 0 2.d scrub starts 2024-04-06T03:35:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:06 smithi153 bash[23852]: cluster 2024-04-06T03:35:06.330486+0000 osd.1 (osd.1) 286 : cluster 0 2.d scrub ok 2024-04-06T03:35:06.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:06.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:06.664 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:06.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:06.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[21467]: cluster 2024-04-06T03:35:05.506223+0000 osd.0 (osd.0) 318 : cluster 0 3.10 scrub starts 2024-04-06T03:35:06.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[21467]: cluster 2024-04-06T03:35:05.512123+0000 osd.0 (osd.0) 319 : cluster 0 3.10 scrub ok 2024-04-06T03:35:06.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[21467]: cluster 2024-04-06T03:35:06.325917+0000 osd.1 (osd.1) 285 : cluster 0 2.d scrub starts 2024-04-06T03:35:06.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[21467]: cluster 2024-04-06T03:35:06.330486+0000 osd.1 (osd.1) 286 : cluster 0 2.d scrub ok 2024-04-06T03:35:06.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[29221]: cluster 2024-04-06T03:35:05.506223+0000 osd.0 (osd.0) 318 : cluster 0 3.10 scrub starts 2024-04-06T03:35:06.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[29221]: cluster 2024-04-06T03:35:05.512123+0000 osd.0 (osd.0) 319 : cluster 0 3.10 scrub ok 2024-04-06T03:35:06.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[29221]: cluster 2024-04-06T03:35:06.325917+0000 osd.1 (osd.1) 285 : cluster 0 2.d scrub starts 2024-04-06T03:35:06.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:06 smithi050 bash[29221]: cluster 2024-04-06T03:35:06.330486+0000 osd.1 (osd.1) 286 : cluster 0 2.d scrub ok 2024-04-06T03:35:06.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:06.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:06.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:06.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:06.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:07.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:07.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:07.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:07.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:07.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:07.335 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:07.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:35:07.399443+0000 min lat: 0.00108908 max lat: 0.428594 avg lat: 0.0091821 2024-04-06T03:35:07.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:35:07.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 20 16 34847 34831 108.834 92.5 0.00329497 0.0091821 2024-04-06T03:35:07.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:07.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:07.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:07.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:07 smithi153 bash[23852]: cluster 2024-04-06T03:35:06.272330+0000 mgr.y (mgr.24368) 523 : cluster 0 pgmap v543: 148 pgs: 148 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:35:07.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:07 smithi153 bash[23852]: cluster 2024-04-06T03:35:06.889243+0000 osd.7 (osd.7) 343 : cluster 0 2.b scrub starts 2024-04-06T03:35:07.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:07 smithi153 bash[23852]: cluster 2024-04-06T03:35:06.892281+0000 osd.7 (osd.7) 344 : cluster 0 2.b scrub ok 2024-04-06T03:35:07.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:07.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:07.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:07.863 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:07.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:07.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:07.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:07 smithi050 bash[21467]: cluster 2024-04-06T03:35:06.272330+0000 mgr.y (mgr.24368) 523 : cluster 0 pgmap v543: 148 pgs: 148 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:35:07.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:07 smithi050 bash[21467]: cluster 2024-04-06T03:35:06.889243+0000 osd.7 (osd.7) 343 : cluster 0 2.b scrub starts 2024-04-06T03:35:07.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:07 smithi050 bash[21467]: cluster 2024-04-06T03:35:06.892281+0000 osd.7 (osd.7) 344 : cluster 0 2.b scrub ok 2024-04-06T03:35:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:07 smithi050 bash[29221]: cluster 2024-04-06T03:35:06.272330+0000 mgr.y (mgr.24368) 523 : cluster 0 pgmap v543: 148 pgs: 148 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:35:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:07 smithi050 bash[29221]: cluster 2024-04-06T03:35:06.889243+0000 osd.7 (osd.7) 343 : cluster 0 2.b scrub starts 2024-04-06T03:35:07.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:07 smithi050 bash[29221]: cluster 2024-04-06T03:35:06.892281+0000 osd.7 (osd.7) 344 : cluster 0 2.b scrub ok 2024-04-06T03:35:07.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:07.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:08.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:08.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:08.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:08.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:08.358 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:08.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 21 16 36609 36593 108.895 110.125 0.00654562 0.00917723 2024-04-06T03:35:08.394 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:08.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:08.523 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:08.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:08.646 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:08.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:08.777 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:08.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:08.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:08.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:09.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:09.034 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:09.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:09.167 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:09.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:09.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:09.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:09.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 22 16 37539 37523 106.587 58.125 0.00248433 0.00932096 2024-04-06T03:35:09.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:09.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:09.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:09.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.272733+0000 mgr.y (mgr.24368) 524 : cluster 0 pgmap v544: 148 pgs: 148 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:35:09.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.675469+0000 osd.2 (osd.2) 275 : cluster 0 3.4 scrub starts 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.685408+0000 osd.2 (osd.2) 276 : cluster 0 3.4 scrub ok 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.815271+0000 mon.a (mon.0) 1357 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.860831+0000 osd.7 (osd.7) 345 : cluster 0 3.f scrub starts 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.862820+0000 osd.7 (osd.7) 346 : cluster 0 3.f scrub ok 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.272733+0000 mgr.y (mgr.24368) 524 : cluster 0 pgmap v544: 148 pgs: 148 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.675469+0000 osd.2 (osd.2) 275 : cluster 0 3.4 scrub starts 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.685408+0000 osd.2 (osd.2) 276 : cluster 0 3.4 scrub ok 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.815271+0000 mon.a (mon.0) 1357 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.860831+0000 osd.7 (osd.7) 345 : cluster 0 3.f scrub starts 2024-04-06T03:35:09.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.862820+0000 osd.7 (osd.7) 346 : cluster 0 3.f scrub ok 2024-04-06T03:35:09.532 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:35:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:35:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:35:09.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:09.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:09.681 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:09.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:09.793 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:35:09.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.272733+0000 mgr.y (mgr.24368) 524 : cluster 0 pgmap v544: 148 pgs: 148 active+clean; 1.9 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.60k op/s 2024-04-06T03:35:09.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.675469+0000 osd.2 (osd.2) 275 : cluster 0 3.4 scrub starts 2024-04-06T03:35:09.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.685408+0000 osd.2 (osd.2) 276 : cluster 0 3.4 scrub ok 2024-04-06T03:35:09.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.815271+0000 mon.a (mon.0) 1357 : cluster 0 osdmap e240: 8 total, 8 up, 8 in 2024-04-06T03:35:09.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.860831+0000 osd.7 (osd.7) 345 : cluster 0 3.f scrub starts 2024-04-06T03:35:09.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:09 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.862820+0000 osd.7 (osd.7) 346 : cluster 0 3.f scrub ok 2024-04-06T03:35:09.805 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:09.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:09.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:09.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:10.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:10.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:10.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:10.196 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:10.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:10.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:10.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 23 16 38355 38339 104.17 51 0.00710328 0.00959485 2024-04-06T03:35:10.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:10.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:10.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.688132+0000 osd.3 (osd.3) 329 : cluster 0 6.7 scrub starts 2024-04-06T03:35:10.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[21467]: cluster 2024-04-06T03:35:08.795531+0000 osd.3 (osd.3) 330 : cluster 0 6.7 scrub ok 2024-04-06T03:35:10.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[21467]: cluster 2024-04-06T03:35:09.527131+0000 osd.0 (osd.0) 320 : cluster 0 5.1b scrub starts 2024-04-06T03:35:10.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[21467]: cluster 2024-04-06T03:35:09.528835+0000 osd.0 (osd.0) 321 : cluster 0 5.1b scrub ok 2024-04-06T03:35:10.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[21467]: audit 2024-04-06T03:35:09.680937+0000 mgr.y (mgr.24368) 525 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:10.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.688132+0000 osd.3 (osd.3) 329 : cluster 0 6.7 scrub starts 2024-04-06T03:35:10.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[29221]: cluster 2024-04-06T03:35:08.795531+0000 osd.3 (osd.3) 330 : cluster 0 6.7 scrub ok 2024-04-06T03:35:10.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[29221]: cluster 2024-04-06T03:35:09.527131+0000 osd.0 (osd.0) 320 : cluster 0 5.1b scrub starts 2024-04-06T03:35:10.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[29221]: cluster 2024-04-06T03:35:09.528835+0000 osd.0 (osd.0) 321 : cluster 0 5.1b scrub ok 2024-04-06T03:35:10.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:10 smithi050 bash[29221]: audit 2024-04-06T03:35:09.680937+0000 mgr.y (mgr.24368) 525 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:10.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:10.599 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:10.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:10.729 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:10 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.688132+0000 osd.3 (osd.3) 329 : cluster 0 6.7 scrub starts 2024-04-06T03:35:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:10 smithi153 bash[23852]: cluster 2024-04-06T03:35:08.795531+0000 osd.3 (osd.3) 330 : cluster 0 6.7 scrub ok 2024-04-06T03:35:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:10 smithi153 bash[23852]: cluster 2024-04-06T03:35:09.527131+0000 osd.0 (osd.0) 320 : cluster 0 5.1b scrub starts 2024-04-06T03:35:10.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:10 smithi153 bash[23852]: cluster 2024-04-06T03:35:09.528835+0000 osd.0 (osd.0) 321 : cluster 0 5.1b scrub ok 2024-04-06T03:35:10.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:10 smithi153 bash[23852]: audit 2024-04-06T03:35:09.680937+0000 mgr.y (mgr.24368) 525 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:10.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:10.859 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:10.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-06T03:35:10.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:10.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:10.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:11.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:11.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:11.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:11.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:11.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:11.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 24 16 39663 39647 103.235 81.75 0.0104441 0.00968023 2024-04-06T03:35:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:11.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:11.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:11.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:11.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:09.688651+0000 osd.3 (osd.3) 331 : cluster 0 6.c scrub starts 2024-04-06T03:35:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:09.691978+0000 osd.3 (osd.3) 332 : cluster 0 6.c scrub ok 2024-04-06T03:35:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.256729+0000 osd.1 (osd.1) 287 : cluster 0 6.4 scrub starts 2024-04-06T03:35:11.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.265418+0000 osd.1 (osd.1) 288 : cluster 0 6.4 scrub ok 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.274249+0000 mgr.y (mgr.24368) 526 : cluster 0 pgmap v546: 148 pgs: 148 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.472386+0000 osd.7 (osd.7) 347 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.472454+0000 osd.7 (osd.7) 348 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.533291+0000 mon.a (mon.0) 1358 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.577922+0000 osd.0 (osd.0) 322 : cluster 0 6.0 scrub starts 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.652029+0000 osd.3 (osd.3) 333 : cluster 0 5.13 scrub starts 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.661041+0000 osd.3 (osd.3) 334 : cluster 0 5.13 scrub ok 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.824583+0000 osd.7 (osd.7) 349 : cluster 0 5.19 scrub starts 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.831851+0000 osd.7 (osd.7) 350 : cluster 0 5.19 scrub ok 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:09.688651+0000 osd.3 (osd.3) 331 : cluster 0 6.c scrub starts 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:09.691978+0000 osd.3 (osd.3) 332 : cluster 0 6.c scrub ok 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.256729+0000 osd.1 (osd.1) 287 : cluster 0 6.4 scrub starts 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.265418+0000 osd.1 (osd.1) 288 : cluster 0 6.4 scrub ok 2024-04-06T03:35:11.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.274249+0000 mgr.y (mgr.24368) 526 : cluster 0 pgmap v546: 148 pgs: 148 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.472386+0000 osd.7 (osd.7) 347 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.472454+0000 osd.7 (osd.7) 348 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.533291+0000 mon.a (mon.0) 1358 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.577922+0000 osd.0 (osd.0) 322 : cluster 0 6.0 scrub starts 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.652029+0000 osd.3 (osd.3) 333 : cluster 0 5.13 scrub starts 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.661041+0000 osd.3 (osd.3) 334 : cluster 0 5.13 scrub ok 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.824583+0000 osd.7 (osd.7) 349 : cluster 0 5.19 scrub starts 2024-04-06T03:35:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:11 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.831851+0000 osd.7 (osd.7) 350 : cluster 0 5.19 scrub ok 2024-04-06T03:35:11.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:11.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:09.688651+0000 osd.3 (osd.3) 331 : cluster 0 6.c scrub starts 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:09.691978+0000 osd.3 (osd.3) 332 : cluster 0 6.c scrub ok 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.256729+0000 osd.1 (osd.1) 287 : cluster 0 6.4 scrub starts 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.265418+0000 osd.1 (osd.1) 288 : cluster 0 6.4 scrub ok 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.274249+0000 mgr.y (mgr.24368) 526 : cluster 0 pgmap v546: 148 pgs: 148 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.472386+0000 osd.7 (osd.7) 347 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.472454+0000 osd.7 (osd.7) 348 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.533291+0000 mon.a (mon.0) 1358 : cluster 0 osdmap e241: 8 total, 8 up, 8 in 2024-04-06T03:35:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.577922+0000 osd.0 (osd.0) 322 : cluster 0 6.0 scrub starts 2024-04-06T03:35:11.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.652029+0000 osd.3 (osd.3) 333 : cluster 0 5.13 scrub starts 2024-04-06T03:35:11.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.661041+0000 osd.3 (osd.3) 334 : cluster 0 5.13 scrub ok 2024-04-06T03:35:11.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.824583+0000 osd.7 (osd.7) 349 : cluster 0 5.19 scrub starts 2024-04-06T03:35:11.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:11 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.831851+0000 osd.7 (osd.7) 350 : cluster 0 5.19 scrub ok 2024-04-06T03:35:11.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:11.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:11.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:12.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:12.049 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:12.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:12.186 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:12.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:12.330 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:12.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 25 16 40781 40765 101.9 69.875 0.00500064 0.00980669 2024-04-06T03:35:12.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:12.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:12.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:12.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:12.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:12.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:12 smithi153 bash[23852]: cluster 2024-04-06T03:35:10.631282+0000 osd.0 (osd.0) 323 : cluster 0 6.0 scrub ok 2024-04-06T03:35:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:12 smithi153 bash[23852]: cluster 2024-04-06T03:35:11.377780+0000 osd.4 (osd.4) 321 : cluster 0 2.11 scrub starts 2024-04-06T03:35:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:12 smithi153 bash[23852]: cluster 2024-04-06T03:35:11.378871+0000 osd.4 (osd.4) 322 : cluster 0 2.11 scrub ok 2024-04-06T03:35:12.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:12.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:12 smithi050 bash[21467]: cluster 2024-04-06T03:35:10.631282+0000 osd.0 (osd.0) 323 : cluster 0 6.0 scrub ok 2024-04-06T03:35:12.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:12 smithi050 bash[21467]: cluster 2024-04-06T03:35:11.377780+0000 osd.4 (osd.4) 321 : cluster 0 2.11 scrub starts 2024-04-06T03:35:12.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:12 smithi050 bash[21467]: cluster 2024-04-06T03:35:11.378871+0000 osd.4 (osd.4) 322 : cluster 0 2.11 scrub ok 2024-04-06T03:35:12.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:12 smithi050 bash[29221]: cluster 2024-04-06T03:35:10.631282+0000 osd.0 (osd.0) 323 : cluster 0 6.0 scrub ok 2024-04-06T03:35:12.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:12 smithi050 bash[29221]: cluster 2024-04-06T03:35:11.377780+0000 osd.4 (osd.4) 321 : cluster 0 2.11 scrub starts 2024-04-06T03:35:12.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:12 smithi050 bash[29221]: cluster 2024-04-06T03:35:11.378871+0000 osd.4 (osd.4) 322 : cluster 0 2.11 scrub ok 2024-04-06T03:35:12.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:12.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:12.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:12.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:13.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:13.123 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:13.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:13.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:13.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:13.390 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:13.393 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 26 16 42063 42047 101.063 80.125 0.0102347 0.00988894 2024-04-06T03:35:13.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:13.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:13.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:13.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:13.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:13.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:13.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:13 smithi153 bash[23852]: cluster 2024-04-06T03:35:12.275823+0000 mgr.y (mgr.24368) 527 : cluster 0 pgmap v548: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-06T03:35:13.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:13 smithi153 bash[23852]: cluster 2024-04-06T03:35:12.662631+0000 osd.3 (osd.3) 335 : cluster 0 2.1b scrub starts 2024-04-06T03:35:13.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:13 smithi153 bash[23852]: cluster 2024-04-06T03:35:12.664967+0000 osd.3 (osd.3) 336 : cluster 0 2.1b scrub ok 2024-04-06T03:35:13.781 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:13.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:13 smithi050 bash[21467]: cluster 2024-04-06T03:35:12.275823+0000 mgr.y (mgr.24368) 527 : cluster 0 pgmap v548: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-06T03:35:13.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:13 smithi050 bash[21467]: cluster 2024-04-06T03:35:12.662631+0000 osd.3 (osd.3) 335 : cluster 0 2.1b scrub starts 2024-04-06T03:35:13.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:13 smithi050 bash[21467]: cluster 2024-04-06T03:35:12.664967+0000 osd.3 (osd.3) 336 : cluster 0 2.1b scrub ok 2024-04-06T03:35:13.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:13 smithi050 bash[29221]: cluster 2024-04-06T03:35:12.275823+0000 mgr.y (mgr.24368) 527 : cluster 0 pgmap v548: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-06T03:35:13.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:13 smithi050 bash[29221]: cluster 2024-04-06T03:35:12.662631+0000 osd.3 (osd.3) 335 : cluster 0 2.1b scrub starts 2024-04-06T03:35:13.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:13 smithi050 bash[29221]: cluster 2024-04-06T03:35:12.664967+0000 osd.3 (osd.3) 336 : cluster 0 2.1b scrub ok 2024-04-06T03:35:13.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:13.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:13.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:13.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:14.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:14.047 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:14.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:14.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:14.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:14.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:14.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 27 16 43228 43212 100.016 72.8125 0.01096 0.00999269 2024-04-06T03:35:14.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:14.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:14.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.481231+0000 osd.0 (osd.0) 324 : cluster 0 3.1f scrub starts 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.487054+0000 osd.0 (osd.0) 325 : cluster 0 3.1f scrub ok 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.698522+0000 osd.3 (osd.3) 337 : cluster 0 3.c scrub starts 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.701229+0000 osd.3 (osd.3) 338 : cluster 0 3.c scrub ok 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.771826+0000 osd.7 (osd.7) 351 : cluster 0 6.12 scrub starts 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.784000+0000 osd.7 (osd.7) 352 : cluster 0 6.12 scrub ok 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[21467]: cluster 2024-04-06T03:35:13.822799+0000 mon.a (mon.0) 1359 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.481231+0000 osd.0 (osd.0) 324 : cluster 0 3.1f scrub starts 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.487054+0000 osd.0 (osd.0) 325 : cluster 0 3.1f scrub ok 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.698522+0000 osd.3 (osd.3) 337 : cluster 0 3.c scrub starts 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.701229+0000 osd.3 (osd.3) 338 : cluster 0 3.c scrub ok 2024-04-06T03:35:14.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.771826+0000 osd.7 (osd.7) 351 : cluster 0 6.12 scrub starts 2024-04-06T03:35:14.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.784000+0000 osd.7 (osd.7) 352 : cluster 0 6.12 scrub ok 2024-04-06T03:35:14.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:14 smithi050 bash[29221]: cluster 2024-04-06T03:35:13.822799+0000 mon.a (mon.0) 1359 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-04-06T03:35:14.577 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:14.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:14.712 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:14.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.481231+0000 osd.0 (osd.0) 324 : cluster 0 3.1f scrub starts 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.487054+0000 osd.0 (osd.0) 325 : cluster 0 3.1f scrub ok 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.698522+0000 osd.3 (osd.3) 337 : cluster 0 3.c scrub starts 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.701229+0000 osd.3 (osd.3) 338 : cluster 0 3.c scrub ok 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.771826+0000 osd.7 (osd.7) 351 : cluster 0 6.12 scrub starts 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.784000+0000 osd.7 (osd.7) 352 : cluster 0 6.12 scrub ok 2024-04-06T03:35:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:14 smithi153 bash[23852]: cluster 2024-04-06T03:35:13.822799+0000 mon.a (mon.0) 1359 : cluster 0 osdmap e242: 8 total, 8 up, 8 in 2024-04-06T03:35:14.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:14.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:14.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:14.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:14.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:15.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:15.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:15.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:15.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:15.345 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:15.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 28 16 44008 43992 98.185 48.75 0.00254297 0.0101534 2024-04-06T03:35:15.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:15.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:15.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:15.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:15.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:15.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:15 smithi153 bash[23852]: cluster 2024-04-06T03:35:14.276554+0000 mgr.y (mgr.24368) 528 : cluster 0 pgmap v550: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-06T03:35:15.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:15 smithi153 bash[23852]: cluster 2024-04-06T03:35:14.478818+0000 osd.2 (osd.2) 277 : cluster 0 2.e scrub starts 2024-04-06T03:35:15.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:15 smithi153 bash[23852]: cluster 2024-04-06T03:35:14.481604+0000 osd.2 (osd.2) 278 : cluster 0 2.e scrub ok 2024-04-06T03:35:15.728 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:15.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:15.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:15.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:15.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:15 smithi050 bash[21467]: cluster 2024-04-06T03:35:14.276554+0000 mgr.y (mgr.24368) 528 : cluster 0 pgmap v550: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-06T03:35:15.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:15 smithi050 bash[21467]: cluster 2024-04-06T03:35:14.478818+0000 osd.2 (osd.2) 277 : cluster 0 2.e scrub starts 2024-04-06T03:35:15.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:15 smithi050 bash[21467]: cluster 2024-04-06T03:35:14.481604+0000 osd.2 (osd.2) 278 : cluster 0 2.e scrub ok 2024-04-06T03:35:15.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:15 smithi050 bash[29221]: cluster 2024-04-06T03:35:14.276554+0000 mgr.y (mgr.24368) 528 : cluster 0 pgmap v550: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.4 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-04-06T03:35:15.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:15 smithi050 bash[29221]: cluster 2024-04-06T03:35:14.478818+0000 osd.2 (osd.2) 277 : cluster 0 2.e scrub starts 2024-04-06T03:35:15.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:15 smithi050 bash[29221]: cluster 2024-04-06T03:35:14.481604+0000 osd.2 (osd.2) 278 : cluster 0 2.e scrub ok 2024-04-06T03:35:15.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:15.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:16.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:16.114 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:16.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:16.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:16.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:16.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 29 16 45042 45026 97.0275 64.625 0.00655698 0.0103016 2024-04-06T03:35:16.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:16.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:16.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:16.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:16.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:16.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[21467]: cluster 2024-04-06T03:35:15.257217+0000 osd.1 (osd.1) 289 : cluster 0 6.19 scrub starts 2024-04-06T03:35:16.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[21467]: cluster 2024-04-06T03:35:15.414020+0000 osd.1 (osd.1) 290 : cluster 0 6.19 scrub ok 2024-04-06T03:35:16.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[21467]: cluster 2024-04-06T03:35:15.437730+0000 osd.0 (osd.0) 326 : cluster 0 5.5 scrub starts 2024-04-06T03:35:16.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[21467]: cluster 2024-04-06T03:35:15.439089+0000 osd.0 (osd.0) 327 : cluster 0 5.5 scrub ok 2024-04-06T03:35:16.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[29221]: cluster 2024-04-06T03:35:15.257217+0000 osd.1 (osd.1) 289 : cluster 0 6.19 scrub starts 2024-04-06T03:35:16.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[29221]: cluster 2024-04-06T03:35:15.414020+0000 osd.1 (osd.1) 290 : cluster 0 6.19 scrub ok 2024-04-06T03:35:16.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[29221]: cluster 2024-04-06T03:35:15.437730+0000 osd.0 (osd.0) 326 : cluster 0 5.5 scrub starts 2024-04-06T03:35:16.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:16 smithi050 bash[29221]: cluster 2024-04-06T03:35:15.439089+0000 osd.0 (osd.0) 327 : cluster 0 5.5 scrub ok 2024-04-06T03:35:16.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:16.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:16.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:16 smithi153 bash[23852]: cluster 2024-04-06T03:35:15.257217+0000 osd.1 (osd.1) 289 : cluster 0 6.19 scrub starts 2024-04-06T03:35:16.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:16 smithi153 bash[23852]: cluster 2024-04-06T03:35:15.414020+0000 osd.1 (osd.1) 290 : cluster 0 6.19 scrub ok 2024-04-06T03:35:16.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:16 smithi153 bash[23852]: cluster 2024-04-06T03:35:15.437730+0000 osd.0 (osd.0) 326 : cluster 0 5.5 scrub starts 2024-04-06T03:35:16.883 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:16 smithi153 bash[23852]: cluster 2024-04-06T03:35:15.439089+0000 osd.0 (osd.0) 327 : cluster 0 5.5 scrub ok 2024-04-06T03:35:16.899 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:16.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:17.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:17.022 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:17.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:17.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:17.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:17.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:17.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:17.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 30 16 45997 45981 95.7828 59.6875 0.0157272 0.010435 2024-04-06T03:35:17.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:17.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:17.554 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:17.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:17.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:17.745 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:35:17.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:17.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:15.712902+0000 osd.3 (osd.3) 339 : cluster 0 2.19 scrub starts 2024-04-06T03:35:17.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:15.714830+0000 osd.3 (osd.3) 340 : cluster 0 2.19 scrub ok 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.144851+0000 osd.5 (osd.5) 135 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.145001+0000 osd.5 (osd.5) 136 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.277852+0000 mgr.y (mgr.24368) 529 : cluster 0 pgmap v551: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.367746+0000 osd.4 (osd.4) 323 : cluster 0 3.a scrub starts 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.369552+0000 osd.4 (osd.4) 324 : cluster 0 3.a scrub ok 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.403437+0000 osd.0 (osd.0) 328 : cluster 0 6.a scrub starts 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.444251+0000 osd.0 (osd.0) 329 : cluster 0 6.a scrub ok 2024-04-06T03:35:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:17 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.582545+0000 mon.a (mon.0) 1360 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-04-06T03:35:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:17.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:15.712902+0000 osd.3 (osd.3) 339 : cluster 0 2.19 scrub starts 2024-04-06T03:35:17.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:15.714830+0000 osd.3 (osd.3) 340 : cluster 0 2.19 scrub ok 2024-04-06T03:35:17.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.144851+0000 osd.5 (osd.5) 135 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.145001+0000 osd.5 (osd.5) 136 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.277852+0000 mgr.y (mgr.24368) 529 : cluster 0 pgmap v551: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.367746+0000 osd.4 (osd.4) 323 : cluster 0 3.a scrub starts 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.369552+0000 osd.4 (osd.4) 324 : cluster 0 3.a scrub ok 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.403437+0000 osd.0 (osd.0) 328 : cluster 0 6.a scrub starts 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.444251+0000 osd.0 (osd.0) 329 : cluster 0 6.a scrub ok 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.582545+0000 mon.a (mon.0) 1360 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:15.712902+0000 osd.3 (osd.3) 339 : cluster 0 2.19 scrub starts 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:15.714830+0000 osd.3 (osd.3) 340 : cluster 0 2.19 scrub ok 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.144851+0000 osd.5 (osd.5) 135 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.145001+0000 osd.5 (osd.5) 136 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:17.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.277852+0000 mgr.y (mgr.24368) 529 : cluster 0 pgmap v551: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-04-06T03:35:17.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.367746+0000 osd.4 (osd.4) 323 : cluster 0 3.a scrub starts 2024-04-06T03:35:17.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.369552+0000 osd.4 (osd.4) 324 : cluster 0 3.a scrub ok 2024-04-06T03:35:17.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.403437+0000 osd.0 (osd.0) 328 : cluster 0 6.a scrub starts 2024-04-06T03:35:17.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.444251+0000 osd.0 (osd.0) 329 : cluster 0 6.a scrub ok 2024-04-06T03:35:17.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:17 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.582545+0000 mon.a (mon.0) 1360 : cluster 0 osdmap e243: 8 total, 8 up, 8 in 2024-04-06T03:35:17.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:17.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:18.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:18.096 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:18.171 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:35:18.178 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:35:18.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:18.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:18.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:18.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:18.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 31 16 47254 47238 95.2271 78.5625 0.00660316 0.0104953 2024-04-06T03:35:18.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:18.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:18.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:18.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:18.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:18.749 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:18.782 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.737729+0000 osd.3 (osd.3) 341 : cluster 0 6.13 scrub starts 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[21467]: cluster 2024-04-06T03:35:16.841010+0000 osd.3 (osd.3) 342 : cluster 0 6.13 scrub ok 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[21467]: cluster 2024-04-06T03:35:17.323227+0000 osd.4 (osd.4) 325 : cluster 0 5.1c scrub starts 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[21467]: cluster 2024-04-06T03:35:17.327701+0000 osd.4 (osd.4) 326 : cluster 0 5.1c scrub ok 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.737729+0000 osd.3 (osd.3) 341 : cluster 0 6.13 scrub starts 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[29221]: cluster 2024-04-06T03:35:16.841010+0000 osd.3 (osd.3) 342 : cluster 0 6.13 scrub ok 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[29221]: cluster 2024-04-06T03:35:17.323227+0000 osd.4 (osd.4) 325 : cluster 0 5.1c scrub starts 2024-04-06T03:35:18.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:18 smithi050 bash[29221]: cluster 2024-04-06T03:35:17.327701+0000 osd.4 (osd.4) 326 : cluster 0 5.1c scrub ok 2024-04-06T03:35:18.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:18.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:18.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:18.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:19.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:19.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:18 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.737729+0000 osd.3 (osd.3) 341 : cluster 0 6.13 scrub starts 2024-04-06T03:35:19.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:18 smithi153 bash[23852]: cluster 2024-04-06T03:35:16.841010+0000 osd.3 (osd.3) 342 : cluster 0 6.13 scrub ok 2024-04-06T03:35:19.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:18 smithi153 bash[23852]: cluster 2024-04-06T03:35:17.323227+0000 osd.4 (osd.4) 325 : cluster 0 5.1c scrub starts 2024-04-06T03:35:19.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:18 smithi153 bash[23852]: cluster 2024-04-06T03:35:17.327701+0000 osd.4 (osd.4) 326 : cluster 0 5.1c scrub ok 2024-04-06T03:35:19.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:19.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:19.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:19.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:19.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:19.393 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:35:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:35:19.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 32 16 48292 48276 94.2784 64.875 0.00211038 0.010593 2024-04-06T03:35:19.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:19.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:19.528 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:19.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:17.743955+0000 osd.3 (osd.3) 343 : cluster 0 6.11 scrub starts 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:17.775546+0000 osd.3 (osd.3) 344 : cluster 0 6.11 scrub ok 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: audit 2024-04-06T03:35:18.147251+0000 mgr.y (mgr.24368) 530 : audit 0 from='client.24781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.278288+0000 mgr.y (mgr.24368) 531 : cluster 0 pgmap v553: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.392625+0000 osd.0 (osd.0) 330 : cluster 0 2.13 scrub starts 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.396830+0000 osd.0 (osd.0) 331 : cluster 0 2.13 scrub ok 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.502847+0000 osd.2 (osd.2) 279 : cluster 0 2.16 scrub starts 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.503942+0000 osd.2 (osd.2) 280 : cluster 0 2.16 scrub ok 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.694654+0000 osd.7 (osd.7) 353 : cluster 0 5.10 scrub starts 2024-04-06T03:35:19.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.696941+0000 osd.7 (osd.7) 354 : cluster 0 5.10 scrub ok 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: cluster 2024-04-06T03:35:18.826439+0000 mon.a (mon.0) 1361 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[21467]: audit 2024-04-06T03:35:19.467170+0000 mon.a (mon.0) 1362 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:17.743955+0000 osd.3 (osd.3) 343 : cluster 0 6.11 scrub starts 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:17.775546+0000 osd.3 (osd.3) 344 : cluster 0 6.11 scrub ok 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: audit 2024-04-06T03:35:18.147251+0000 mgr.y (mgr.24368) 530 : audit 0 from='client.24781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.278288+0000 mgr.y (mgr.24368) 531 : cluster 0 pgmap v553: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.392625+0000 osd.0 (osd.0) 330 : cluster 0 2.13 scrub starts 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.396830+0000 osd.0 (osd.0) 331 : cluster 0 2.13 scrub ok 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.502847+0000 osd.2 (osd.2) 279 : cluster 0 2.16 scrub starts 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.503942+0000 osd.2 (osd.2) 280 : cluster 0 2.16 scrub ok 2024-04-06T03:35:19.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.694654+0000 osd.7 (osd.7) 353 : cluster 0 5.10 scrub starts 2024-04-06T03:35:19.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.696941+0000 osd.7 (osd.7) 354 : cluster 0 5.10 scrub ok 2024-04-06T03:35:19.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: cluster 2024-04-06T03:35:18.826439+0000 mon.a (mon.0) 1361 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-04-06T03:35:19.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:19 smithi050 bash[29221]: audit 2024-04-06T03:35:19.467170+0000 mon.a (mon.0) 1362 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:19.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:19.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:35:19.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:17.743955+0000 osd.3 (osd.3) 343 : cluster 0 6.11 scrub starts 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:17.775546+0000 osd.3 (osd.3) 344 : cluster 0 6.11 scrub ok 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: audit 2024-04-06T03:35:18.147251+0000 mgr.y (mgr.24368) 530 : audit 0 from='client.24781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.278288+0000 mgr.y (mgr.24368) 531 : cluster 0 pgmap v553: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.392625+0000 osd.0 (osd.0) 330 : cluster 0 2.13 scrub starts 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.396830+0000 osd.0 (osd.0) 331 : cluster 0 2.13 scrub ok 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.502847+0000 osd.2 (osd.2) 279 : cluster 0 2.16 scrub starts 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.503942+0000 osd.2 (osd.2) 280 : cluster 0 2.16 scrub ok 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.694654+0000 osd.7 (osd.7) 353 : cluster 0 5.10 scrub starts 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.696941+0000 osd.7 (osd.7) 354 : cluster 0 5.10 scrub ok 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: cluster 2024-04-06T03:35:18.826439+0000 mon.a (mon.0) 1361 : cluster 0 osdmap e244: 8 total, 8 up, 8 in 2024-04-06T03:35:19.779 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:19 smithi153 bash[23852]: audit 2024-04-06T03:35:19.467170+0000 mon.a (mon.0) 1362 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:19.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:19.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:19.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:19.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:20.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:20.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:20.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:20.191 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:20.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:20.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:20.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 33 16 48884 48868 92.5427 37 0.00644937 0.010799 2024-04-06T03:35:20.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:20.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:20.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:20.582 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:20.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:20.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:20.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:20.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:20.819 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:20 smithi153 bash[23852]: cluster 2024-04-06T03:35:19.231768+0000 osd.1 (osd.1) 291 : cluster 0 5.7 scrub starts 2024-04-06T03:35:20.819 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:20 smithi153 bash[23852]: cluster 2024-04-06T03:35:19.260510+0000 osd.1 (osd.1) 292 : cluster 0 5.7 scrub ok 2024-04-06T03:35:20.819 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:20 smithi153 bash[23852]: audit 2024-04-06T03:35:19.691967+0000 mgr.y (mgr.24368) 532 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:20.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:20.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:20 smithi050 bash[29221]: cluster 2024-04-06T03:35:19.231768+0000 osd.1 (osd.1) 291 : cluster 0 5.7 scrub starts 2024-04-06T03:35:20.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:20 smithi050 bash[29221]: cluster 2024-04-06T03:35:19.260510+0000 osd.1 (osd.1) 292 : cluster 0 5.7 scrub ok 2024-04-06T03:35:20.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:20 smithi050 bash[29221]: audit 2024-04-06T03:35:19.691967+0000 mgr.y (mgr.24368) 532 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:20.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:20 smithi050 bash[21467]: cluster 2024-04-06T03:35:19.231768+0000 osd.1 (osd.1) 291 : cluster 0 5.7 scrub starts 2024-04-06T03:35:20.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:20 smithi050 bash[21467]: cluster 2024-04-06T03:35:19.260510+0000 osd.1 (osd.1) 292 : cluster 0 5.7 scrub ok 2024-04-06T03:35:20.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:20 smithi050 bash[21467]: audit 2024-04-06T03:35:19.691967+0000 mgr.y (mgr.24368) 532 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:20.937 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:20.964 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:20.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:21.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:21.093 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:21.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:21.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:21.345 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:21.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 34 16 49595 49579 91.1276 44.4375 0.00582272 0.0109028 2024-04-06T03:35:21.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:21.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:21.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:21.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:21.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:21.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:21 smithi050 bash[21467]: cluster 2024-04-06T03:35:20.279396+0000 mgr.y (mgr.24368) 533 : cluster 0 pgmap v555: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 793 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-06T03:35:21.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:21 smithi050 bash[21467]: cluster 2024-04-06T03:35:20.858366+0000 osd.7 (osd.7) 355 : cluster 0 2.5 scrub starts 2024-04-06T03:35:21.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:21 smithi050 bash[21467]: cluster 2024-04-06T03:35:20.860453+0000 osd.7 (osd.7) 356 : cluster 0 2.5 scrub ok 2024-04-06T03:35:21.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:21 smithi050 bash[29221]: cluster 2024-04-06T03:35:20.279396+0000 mgr.y (mgr.24368) 533 : cluster 0 pgmap v555: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 793 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-06T03:35:21.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:21 smithi050 bash[29221]: cluster 2024-04-06T03:35:20.858366+0000 osd.7 (osd.7) 355 : cluster 0 2.5 scrub starts 2024-04-06T03:35:21.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:21 smithi050 bash[29221]: cluster 2024-04-06T03:35:20.860453+0000 osd.7 (osd.7) 356 : cluster 0 2.5 scrub ok 2024-04-06T03:35:21.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:21 smithi153 bash[23852]: cluster 2024-04-06T03:35:20.279396+0000 mgr.y (mgr.24368) 533 : cluster 0 pgmap v555: 148 pgs: 148 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 793 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-04-06T03:35:21.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:21 smithi153 bash[23852]: cluster 2024-04-06T03:35:20.858366+0000 osd.7 (osd.7) 355 : cluster 0 2.5 scrub starts 2024-04-06T03:35:21.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:21 smithi153 bash[23852]: cluster 2024-04-06T03:35:20.860453+0000 osd.7 (osd.7) 356 : cluster 0 2.5 scrub ok 2024-04-06T03:35:21.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:21.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:21.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:21.957 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2024-04-06T03:35:16.559512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2873629022,"num_objects":44069,"num_object_clones":0,"num_object_copies":132207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44069,"num_whiteouts":0,"num_read":2501,"num_read_kb":2315,"num_write":44932,"num_write_kb":2808157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45081,"ondisk_log_size":45081,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10347732,"kb_used_data":8254668,"kb_used_omap":12,"kb_used_meta":2092979,"kb_avail":739613484,"statfs":{"total":767960285184,"available":757364207616,"internally_reserved":0,"allocated":8452780032,"data_stored":8449814454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2143211057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":555483136,"num_objects":8476,"num_object_clones":0,"num_object_copies":25428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8476,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8476,"num_write_kb":542464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.286551"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":380,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833055+0000","last_change":"2024-04-06T03:33:10.115456+0000","last_active":"2024-04-06T03:35:13.833055+0000","last_peered":"2024-04-06T03:35:13.833055+0000","last_clean":"2024-04-06T03:35:13.833055+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:35:13.833055+0000","last_undegraded":"2024-04-06T03:35:13.833055+0000","last_fullsized":"2024-04-06T03:35:13.833055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:10.115343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.115343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":384,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811495+0000","last_change":"2024-04-06T03:33:54.305360+0000","last_active":"2024-04-06T03:35:13.811495+0000","last_peered":"2024-04-06T03:35:13.811495+0000","last_clean":"2024-04-06T03:35:13.811495+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:35:13.811495+0000","last_undegraded":"2024-04-06T03:35:13.811495+0000","last_fullsized":"2024-04-06T03:35:13.811495+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:54.305146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:33:54.305146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:17.726287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":403,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.831681+0000","last_change":"2024-04-06T03:33:36.100803+0000","last_active":"2024-04-06T03:35:13.831681+0000","last_peered":"2024-04-06T03:35:13.831681+0000","last_clean":"2024-04-06T03:35:13.831681+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:35:13.831681+0000","last_undegraded":"2024-04-06T03:35:13.831681+0000","last_fullsized":"2024-04-06T03:35:13.831681+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.100494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.100494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"199'32","reported_seq":438,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834155+0000","last_change":"2024-04-06T03:33:18.029775+0000","last_active":"2024-04-06T03:35:13.834155+0000","last_peered":"2024-04-06T03:35:13.834155+0000","last_clean":"2024-04-06T03:35:13.834155+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:35:13.834155+0000","last_undegraded":"2024-04-06T03:35:13.834155+0000","last_fullsized":"2024-04-06T03:35:13.834155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'32","last_scrub_stamp":"2024-04-06T03:33:18.029641+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:33:18.029641+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":388,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811814+0000","last_change":"2024-04-06T03:34:56.377428+0000","last_active":"2024-04-06T03:35:13.811814+0000","last_peered":"2024-04-06T03:35:13.811814+0000","last_clean":"2024-04-06T03:35:13.811814+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:35:13.811814+0000","last_undegraded":"2024-04-06T03:35:13.811814+0000","last_fullsized":"2024-04-06T03:35:13.811814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:56.377342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:34:56.377342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:11.927536+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":271,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817678+0000","last_change":"2024-04-06T03:35:10.831946+0000","last_active":"2024-04-06T03:35:13.817678+0000","last_peered":"2024-04-06T03:35:13.817678+0000","last_clean":"2024-04-06T03:35:13.817678+0000","last_became_active":"2024-04-06T03:25:54.480600+0000","last_became_peered":"2024-04-06T03:25:54.480600+0000","last_unstale":"2024-04-06T03:35:13.817678+0000","last_undegraded":"2024-04-06T03:35:13.817678+0000","last_fullsized":"2024-04-06T03:35:13.817678+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:10.831856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:35:10.831856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:27.779089+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":394,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833828+0000","last_change":"2024-04-06T03:33:39.516560+0000","last_active":"2024-04-06T03:35:13.833828+0000","last_peered":"2024-04-06T03:35:13.833828+0000","last_clean":"2024-04-06T03:35:13.833828+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:35:13.833828+0000","last_undegraded":"2024-04-06T03:35:13.833828+0000","last_fullsized":"2024-04-06T03:35:13.833828+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:39.516225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:33:39.516225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"199'31","reported_seq":444,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.831312+0000","last_change":"2024-04-06T03:35:13.487254+0000","last_active":"2024-04-06T03:35:13.831312+0000","last_peered":"2024-04-06T03:35:13.831312+0000","last_clean":"2024-04-06T03:35:13.831312+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:35:13.831312+0000","last_undegraded":"2024-04-06T03:35:13.831312+0000","last_fullsized":"2024-04-06T03:35:13.831312+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:35:13.487061+0000","last_deep_scrub":"125'23","last_deep_scrub_stamp":"2024-04-06T03:25:17.013823+0000","last_clean_scrub_stamp":"2024-04-06T03:35:13.487061+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:38.426499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":267,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.838325+0000","last_change":"2024-04-06T03:33:26.400774+0000","last_active":"2024-04-06T03:35:13.838325+0000","last_peered":"2024-04-06T03:35:13.838325+0000","last_clean":"2024-04-06T03:35:13.838325+0000","last_became_active":"2024-04-06T03:25:54.484120+0000","last_became_peered":"2024-04-06T03:25:54.484120+0000","last_unstale":"2024-04-06T03:35:13.838325+0000","last_undegraded":"2024-04-06T03:35:13.838325+0000","last_fullsized":"2024-04-06T03:35:13.838325+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:26.400649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:26.400649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":382,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817476+0000","last_change":"2024-04-06T03:33:29.327349+0000","last_active":"2024-04-06T03:35:13.817476+0000","last_peered":"2024-04-06T03:35:13.817476+0000","last_clean":"2024-04-06T03:35:13.817476+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:35:13.817476+0000","last_undegraded":"2024-04-06T03:35:13.817476+0000","last_fullsized":"2024-04-06T03:35:13.817476+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:29.327145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.327145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":406,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818139+0000","last_change":"2024-04-06T03:33:48.097842+0000","last_active":"2024-04-06T03:35:13.818139+0000","last_peered":"2024-04-06T03:35:13.818139+0000","last_clean":"2024-04-06T03:35:13.818139+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:35:13.818139+0000","last_undegraded":"2024-04-06T03:35:13.818139+0000","last_fullsized":"2024-04-06T03:35:13.818139+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:33:48.097599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:48.097599+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:14.712480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"199'34","reported_seq":388,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812361+0000","last_change":"2024-04-06T03:33:55.304122+0000","last_active":"2024-04-06T03:35:13.812361+0000","last_peered":"2024-04-06T03:35:13.812361+0000","last_clean":"2024-04-06T03:35:13.812361+0000","last_became_active":"2024-04-06T03:25:54.489462+0000","last_became_peered":"2024-04-06T03:25:54.489462+0000","last_unstale":"2024-04-06T03:35:13.812361+0000","last_undegraded":"2024-04-06T03:35:13.812361+0000","last_fullsized":"2024-04-06T03:35:13.812361+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'34","last_scrub_stamp":"2024-04-06T03:33:55.303979+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:33:55.303979+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:19.792669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":378,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830542+0000","last_change":"2024-04-06T03:33:39.245506+0000","last_active":"2024-04-06T03:35:13.830542+0000","last_peered":"2024-04-06T03:35:13.830542+0000","last_clean":"2024-04-06T03:35:13.830542+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:35:13.830542+0000","last_undegraded":"2024-04-06T03:35:13.830542+0000","last_fullsized":"2024-04-06T03:35:13.830542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:39.245352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:39.245352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":267,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.836952+0000","last_change":"2024-04-06T03:35:09.529033+0000","last_active":"2024-04-06T03:35:13.836952+0000","last_peered":"2024-04-06T03:35:13.836952+0000","last_clean":"2024-04-06T03:35:13.836952+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:35:13.836952+0000","last_undegraded":"2024-04-06T03:35:13.836952+0000","last_fullsized":"2024-04-06T03:35:13.836952+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:09.528843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:35:09.528843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:37.420689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":390,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817570+0000","last_change":"2024-04-06T03:33:34.322488+0000","last_active":"2024-04-06T03:35:13.817570+0000","last_peered":"2024-04-06T03:35:13.817570+0000","last_clean":"2024-04-06T03:35:13.817570+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:35:13.817570+0000","last_undegraded":"2024-04-06T03:35:13.817570+0000","last_fullsized":"2024-04-06T03:35:13.817570+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:34.322401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:46.087127+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.322401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"199'30","reported_seq":366,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812526+0000","last_change":"2024-04-06T03:34:49.340142+0000","last_active":"2024-04-06T03:35:13.812526+0000","last_peered":"2024-04-06T03:35:13.812526+0000","last_clean":"2024-04-06T03:35:13.812526+0000","last_became_active":"2024-04-06T03:25:54.483976+0000","last_became_peered":"2024-04-06T03:25:54.483976+0000","last_unstale":"2024-04-06T03:35:13.812526+0000","last_undegraded":"2024-04-06T03:35:13.812526+0000","last_fullsized":"2024-04-06T03:35:13.812526+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'30","last_scrub_stamp":"2024-04-06T03:34:49.339799+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:34:49.339799+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:10.104876+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":381,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837194+0000","last_change":"2024-04-06T03:33:37.571108+0000","last_active":"2024-04-06T03:35:13.837194+0000","last_peered":"2024-04-06T03:35:13.837194+0000","last_clean":"2024-04-06T03:35:13.837194+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:35:13.837194+0000","last_undegraded":"2024-04-06T03:35:13.837194+0000","last_fullsized":"2024-04-06T03:35:13.837194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.570981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.570981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":392,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812739+0000","last_change":"2024-04-06T03:31:37.182250+0000","last_active":"2024-04-06T03:35:13.812739+0000","last_peered":"2024-04-06T03:35:13.812739+0000","last_clean":"2024-04-06T03:35:13.812739+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:35:13.812739+0000","last_undegraded":"2024-04-06T03:35:13.812739+0000","last_fullsized":"2024-04-06T03:35:13.812739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:31:37.182127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:31:37.182127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":392,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833344+0000","last_change":"2024-04-06T03:35:12.665153+0000","last_active":"2024-04-06T03:35:13.833344+0000","last_peered":"2024-04-06T03:35:13.833344+0000","last_clean":"2024-04-06T03:35:13.833344+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:35:13.833344+0000","last_undegraded":"2024-04-06T03:35:13.833344+0000","last_fullsized":"2024-04-06T03:35:13.833344+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:12.664976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:35:12.664976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:40.877902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"199'33","reported_seq":451,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812019+0000","last_change":"2024-04-06T03:33:21.449448+0000","last_active":"2024-04-06T03:35:13.812019+0000","last_peered":"2024-04-06T03:35:13.812019+0000","last_clean":"2024-04-06T03:35:13.812019+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:35:13.812019+0000","last_undegraded":"2024-04-06T03:35:13.812019+0000","last_fullsized":"2024-04-06T03:35:13.812019+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:33:21.449361+0000","last_deep_scrub":"111'27","last_deep_scrub_stamp":"2024-04-06T03:30:12.438411+0000","last_clean_scrub_stamp":"2024-04-06T03:33:21.449361+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":382,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.813117+0000","last_change":"2024-04-06T03:33:15.602204+0000","last_active":"2024-04-06T03:35:13.813117+0000","last_peered":"2024-04-06T03:35:13.813117+0000","last_clean":"2024-04-06T03:35:13.813117+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:35:13.813117+0000","last_undegraded":"2024-04-06T03:35:13.813117+0000","last_fullsized":"2024-04-06T03:35:13.813117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:15.602002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.602002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":398,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.840432+0000","last_change":"2024-04-06T03:34:55.364726+0000","last_active":"2024-04-06T03:35:13.840432+0000","last_peered":"2024-04-06T03:35:13.840432+0000","last_clean":"2024-04-06T03:35:13.840432+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:35:13.840432+0000","last_undegraded":"2024-04-06T03:35:13.840432+0000","last_fullsized":"2024-04-06T03:35:13.840432+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:55.364645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:34:55.364645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:14.306567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":253,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812094+0000","last_change":"2024-04-06T03:34:53.310116+0000","last_active":"2024-04-06T03:35:13.812094+0000","last_peered":"2024-04-06T03:35:13.812094+0000","last_clean":"2024-04-06T03:35:13.812094+0000","last_became_active":"2024-04-06T03:26:18.617121+0000","last_became_peered":"2024-04-06T03:26:18.617121+0000","last_unstale":"2024-04-06T03:35:13.812094+0000","last_undegraded":"2024-04-06T03:35:13.812094+0000","last_fullsized":"2024-04-06T03:35:13.812094+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:53.309998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:34:53.309998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:07.856297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"224'16","reported_seq":422,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837362+0000","last_change":"2024-04-06T03:33:51.165215+0000","last_active":"2024-04-06T03:35:13.837362+0000","last_peered":"2024-04-06T03:35:13.837362+0000","last_clean":"2024-04-06T03:35:13.837362+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:35:13.837362+0000","last_undegraded":"2024-04-06T03:35:13.837362+0000","last_fullsized":"2024-04-06T03:35:13.837362+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16","last_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.165059+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":262,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812278+0000","last_change":"2024-04-06T03:33:23.526390+0000","last_active":"2024-04-06T03:35:13.812278+0000","last_peered":"2024-04-06T03:35:13.812278+0000","last_clean":"2024-04-06T03:35:13.812278+0000","last_became_active":"2024-04-06T03:25:54.483894+0000","last_became_peered":"2024-04-06T03:25:54.483894+0000","last_unstale":"2024-04-06T03:35:13.812278+0000","last_undegraded":"2024-04-06T03:35:13.812278+0000","last_fullsized":"2024-04-06T03:35:13.812278+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.526306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.526306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":392,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830382+0000","last_change":"2024-04-06T03:35:01.539829+0000","last_active":"2024-04-06T03:35:13.830382+0000","last_peered":"2024-04-06T03:35:13.830382+0000","last_clean":"2024-04-06T03:35:13.830382+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:35:13.830382+0000","last_undegraded":"2024-04-06T03:35:13.830382+0000","last_fullsized":"2024-04-06T03:35:13.830382+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:01.539704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:35:01.539704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:20.891311+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":393,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.714934+0000","last_change":"2024-04-06T03:35:15.714968+0000","last_active":"2024-04-06T03:35:15.714934+0000","last_peered":"2024-04-06T03:35:15.714934+0000","last_clean":"2024-04-06T03:35:15.714934+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:35:15.714934+0000","last_undegraded":"2024-04-06T03:35:15.714934+0000","last_fullsized":"2024-04-06T03:35:15.714934+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:35:15.714839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:35:15.714839+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:31.307779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"199'26","reported_seq":422,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833740+0000","last_change":"2024-04-06T03:33:30.273551+0000","last_active":"2024-04-06T03:35:13.833740+0000","last_peered":"2024-04-06T03:35:13.833740+0000","last_clean":"2024-04-06T03:35:13.833740+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:35:13.833740+0000","last_undegraded":"2024-04-06T03:35:13.833740+0000","last_fullsized":"2024-04-06T03:35:13.833740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'26","last_scrub_stamp":"2024-04-06T03:33:30.273225+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:33:30.273225+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":381,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818015+0000","last_change":"2024-04-06T03:33:46.163699+0000","last_active":"2024-04-06T03:35:13.818015+0000","last_peered":"2024-04-06T03:35:13.818015+0000","last_clean":"2024-04-06T03:35:13.818015+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:35:13.818015+0000","last_undegraded":"2024-04-06T03:35:13.818015+0000","last_fullsized":"2024-04-06T03:35:13.818015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:33:46.163357+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:31:51.905353+0000","last_clean_scrub_stamp":"2024-04-06T03:33:46.163357+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"139'13","reported_seq":444,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812749+0000","last_change":"2024-04-06T03:33:58.297492+0000","last_active":"2024-04-06T03:35:13.812749+0000","last_peered":"2024-04-06T03:35:13.812749+0000","last_clean":"2024-04-06T03:35:13.812749+0000","last_became_active":"2024-04-06T03:26:18.615209+0000","last_became_peered":"2024-04-06T03:26:18.615209+0000","last_unstale":"2024-04-06T03:35:13.812749+0000","last_undegraded":"2024-04-06T03:35:13.812749+0000","last_fullsized":"2024-04-06T03:35:13.812749+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'13","last_scrub_stamp":"2024-04-06T03:33:58.297262+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:33:58.297262+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:23.424504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":267,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811482+0000","last_change":"2024-04-06T03:34:58.361360+0000","last_active":"2024-04-06T03:35:13.811482+0000","last_peered":"2024-04-06T03:35:13.811482+0000","last_clean":"2024-04-06T03:35:13.811482+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:35:13.811482+0000","last_undegraded":"2024-04-06T03:35:13.811482+0000","last_fullsized":"2024-04-06T03:35:13.811482+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:58.361232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:34:58.361232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:14.741376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'62","reported_seq":504,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837639+0000","last_change":"2024-04-06T03:34:48.461935+0000","last_active":"2024-04-06T03:35:13.837639+0000","last_peered":"2024-04-06T03:35:13.837639+0000","last_clean":"2024-04-06T03:35:13.837639+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:35:13.837639+0000","last_undegraded":"2024-04-06T03:35:13.837639+0000","last_fullsized":"2024-04-06T03:35:13.837639+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'62","last_scrub_stamp":"2024-04-06T03:34:48.461767+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:34:48.461767+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:09.974557+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":373,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834013+0000","last_change":"2024-04-06T03:32:07.755529+0000","last_active":"2024-04-06T03:35:13.834013+0000","last_peered":"2024-04-06T03:35:13.834013+0000","last_clean":"2024-04-06T03:35:13.834013+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:35:13.834013+0000","last_undegraded":"2024-04-06T03:35:13.834013+0000","last_fullsized":"2024-04-06T03:35:13.834013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:07.755302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:32:07.755302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":383,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.816971+0000","last_change":"2024-04-06T03:33:13.378754+0000","last_active":"2024-04-06T03:35:13.816971+0000","last_peered":"2024-04-06T03:35:13.816971+0000","last_clean":"2024-04-06T03:35:13.816971+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:35:13.816971+0000","last_undegraded":"2024-04-06T03:35:13.816971+0000","last_fullsized":"2024-04-06T03:35:13.816971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:13.378611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.000214+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.378611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":269,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828072+0000","last_change":"2024-04-06T03:34:55.142330+0000","last_active":"2024-04-06T03:35:13.828072+0000","last_peered":"2024-04-06T03:35:13.828072+0000","last_clean":"2024-04-06T03:35:13.828072+0000","last_became_active":"2024-04-06T03:26:18.635349+0000","last_became_peered":"2024-04-06T03:26:18.635349+0000","last_unstale":"2024-04-06T03:35:13.828072+0000","last_undegraded":"2024-04-06T03:35:13.828072+0000","last_fullsized":"2024-04-06T03:35:13.828072+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:55.142201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:34:55.142201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:00.570454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"199'27","reported_seq":327,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818443+0000","last_change":"2024-04-06T03:34:53.086100+0000","last_active":"2024-04-06T03:35:13.818443+0000","last_peered":"2024-04-06T03:35:13.818443+0000","last_clean":"2024-04-06T03:35:13.818443+0000","last_became_active":"2024-04-06T03:25:54.491804+0000","last_became_peered":"2024-04-06T03:25:54.491804+0000","last_unstale":"2024-04-06T03:35:13.818443+0000","last_undegraded":"2024-04-06T03:35:13.818443+0000","last_fullsized":"2024-04-06T03:35:13.818443+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:34:53.085942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:34:53.085942+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:16.468336+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817371+0000","last_change":"2024-04-06T03:35:13.784139+0000","last_active":"2024-04-06T03:35:13.817371+0000","last_peered":"2024-04-06T03:35:13.817371+0000","last_clean":"2024-04-06T03:35:13.817371+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:35:13.817371+0000","last_undegraded":"2024-04-06T03:35:13.817371+0000","last_fullsized":"2024-04-06T03:35:13.817371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:13.784006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:35:13.784006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:41.792451+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":254,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811526+0000","last_change":"2024-04-06T03:33:45.381648+0000","last_active":"2024-04-06T03:35:13.811526+0000","last_peered":"2024-04-06T03:35:13.811526+0000","last_clean":"2024-04-06T03:35:13.811526+0000","last_became_active":"2024-04-06T03:26:18.618151+0000","last_became_peered":"2024-04-06T03:26:18.618151+0000","last_unstale":"2024-04-06T03:35:13.811526+0000","last_undegraded":"2024-04-06T03:35:13.811526+0000","last_fullsized":"2024-04-06T03:35:13.811526+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:45.381432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:45.381432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":264,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.836215+0000","last_change":"2024-04-06T03:33:20.475825+0000","last_active":"2024-04-06T03:35:13.836215+0000","last_peered":"2024-04-06T03:35:13.836215+0000","last_clean":"2024-04-06T03:35:13.836215+0000","last_became_active":"2024-04-06T03:26:18.637825+0000","last_became_peered":"2024-04-06T03:26:18.637825+0000","last_unstale":"2024-04-06T03:35:13.836215+0000","last_undegraded":"2024-04-06T03:35:13.836215+0000","last_fullsized":"2024-04-06T03:35:13.836215+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:20.475708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.475708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"199'22","reported_seq":436,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.831089+0000","last_change":"2024-04-06T03:34:48.358199+0000","last_active":"2024-04-06T03:35:13.831089+0000","last_peered":"2024-04-06T03:35:13.831089+0000","last_clean":"2024-04-06T03:35:13.831089+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:35:13.831089+0000","last_undegraded":"2024-04-06T03:35:13.831089+0000","last_fullsized":"2024-04-06T03:35:13.831089+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'22","last_scrub_stamp":"2024-04-06T03:34:48.358053+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:34:48.358053+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:18.012741+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":373,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834033+0000","last_change":"2024-04-06T03:32:02.669696+0000","last_active":"2024-04-06T03:35:13.834033+0000","last_peered":"2024-04-06T03:35:13.834033+0000","last_clean":"2024-04-06T03:35:13.834033+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:35:13.834033+0000","last_undegraded":"2024-04-06T03:35:13.834033+0000","last_fullsized":"2024-04-06T03:35:13.834033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:02.669327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:32:02.669327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":399,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.853252+0000","last_change":"2024-04-06T03:35:04.398559+0000","last_active":"2024-04-06T03:35:13.853252+0000","last_peered":"2024-04-06T03:35:13.853252+0000","last_clean":"2024-04-06T03:35:13.853252+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:35:13.853252+0000","last_undegraded":"2024-04-06T03:35:13.853252+0000","last_fullsized":"2024-04-06T03:35:13.853252+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:04.398408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:35:04.398408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:23.351508+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":414,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.840052+0000","last_change":"2024-04-06T03:34:59.380496+0000","last_active":"2024-04-06T03:35:13.840052+0000","last_peered":"2024-04-06T03:35:13.840052+0000","last_clean":"2024-04-06T03:35:13.840052+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:35:13.840052+0000","last_undegraded":"2024-04-06T03:35:13.840052+0000","last_fullsized":"2024-04-06T03:35:13.840052+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:59.380362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:34:59.380362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:10.580888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"199'33","reported_seq":452,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812475+0000","last_change":"2024-04-06T03:33:53.327065+0000","last_active":"2024-04-06T03:35:13.812475+0000","last_peered":"2024-04-06T03:35:13.812475+0000","last_clean":"2024-04-06T03:35:13.812475+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:35:13.812475+0000","last_undegraded":"2024-04-06T03:35:13.812475+0000","last_fullsized":"2024-04-06T03:35:13.812475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:33:53.326886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.326886+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:18.112004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":379,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827006+0000","last_change":"2024-04-06T03:33:38.193397+0000","last_active":"2024-04-06T03:35:13.827006+0000","last_peered":"2024-04-06T03:35:13.827006+0000","last_clean":"2024-04-06T03:35:13.827006+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:35:13.827006+0000","last_undegraded":"2024-04-06T03:35:13.827006+0000","last_fullsized":"2024-04-06T03:35:13.827006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:38.193266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:38.193266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":385,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833700+0000","last_change":"2024-04-06T03:35:10.661156+0000","last_active":"2024-04-06T03:35:13.833700+0000","last_peered":"2024-04-06T03:35:13.833700+0000","last_clean":"2024-04-06T03:35:13.833700+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:35:13.833700+0000","last_undegraded":"2024-04-06T03:35:13.833700+0000","last_fullsized":"2024-04-06T03:35:13.833700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:10.661047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_clean_scrub_stamp":"2024-04-06T03:35:10.661047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:13.687883+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":245,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.835332+0000","last_change":"2024-04-06T03:33:20.106580+0000","last_active":"2024-04-06T03:35:13.835332+0000","last_peered":"2024-04-06T03:35:13.835332+0000","last_clean":"2024-04-06T03:35:13.835332+0000","last_became_active":"2024-04-06T03:26:18.643806+0000","last_became_peered":"2024-04-06T03:26:18.643806+0000","last_unstale":"2024-04-06T03:35:13.835332+0000","last_undegraded":"2024-04-06T03:35:13.835332+0000","last_fullsized":"2024-04-06T03:35:13.835332+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:20.106508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:33:20.106508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"199'21","reported_seq":423,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818218+0000","last_change":"2024-04-06T03:33:44.226611+0000","last_active":"2024-04-06T03:35:13.818218+0000","last_peered":"2024-04-06T03:35:13.818218+0000","last_clean":"2024-04-06T03:35:13.818218+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:35:13.818218+0000","last_undegraded":"2024-04-06T03:35:13.818218+0000","last_fullsized":"2024-04-06T03:35:13.818218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:33:44.226337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:44.226337+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":388,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812880+0000","last_change":"2024-04-06T03:33:37.480817+0000","last_active":"2024-04-06T03:35:13.812880+0000","last_peered":"2024-04-06T03:35:13.812880+0000","last_clean":"2024-04-06T03:35:13.812880+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:35:13.812880+0000","last_undegraded":"2024-04-06T03:35:13.812880+0000","last_fullsized":"2024-04-06T03:35:13.812880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.480549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.480549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":561,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.835485+0000","last_change":"2024-04-06T03:33:31.421434+0000","last_active":"2024-04-06T03:35:13.835485+0000","last_peered":"2024-04-06T03:35:13.835485+0000","last_clean":"2024-04-06T03:35:13.835485+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:35:13.835485+0000","last_undegraded":"2024-04-06T03:35:13.835485+0000","last_fullsized":"2024-04-06T03:35:13.835485+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:33:31.421277+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:33:31.421277+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":391,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837527+0000","last_change":"2024-04-06T03:33:23.775542+0000","last_active":"2024-04-06T03:35:13.837527+0000","last_peered":"2024-04-06T03:35:13.837527+0000","last_clean":"2024-04-06T03:35:13.837527+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:35:13.837527+0000","last_undegraded":"2024-04-06T03:35:13.837527+0000","last_fullsized":"2024-04-06T03:35:13.837527+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:23.775408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:23.775408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"199'15","reported_seq":420,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.829424+0000","last_change":"2024-04-06T03:35:03.478320+0000","last_active":"2024-04-06T03:35:13.829424+0000","last_peered":"2024-04-06T03:35:13.829424+0000","last_clean":"2024-04-06T03:35:13.829424+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:35:13.829424+0000","last_undegraded":"2024-04-06T03:35:13.829424+0000","last_fullsized":"2024-04-06T03:35:13.829424+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'15","last_scrub_stamp":"2024-04-06T03:35:03.477228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:35:03.477228+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:08.609200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":372,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830578+0000","last_change":"2024-04-06T03:34:51.515070+0000","last_active":"2024-04-06T03:35:13.830578+0000","last_peered":"2024-04-06T03:35:13.830578+0000","last_clean":"2024-04-06T03:35:13.830578+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:35:13.830578+0000","last_undegraded":"2024-04-06T03:35:13.830578+0000","last_fullsized":"2024-04-06T03:35:13.830578+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:51.514943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:34:51.514943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:08.042406+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"154'13","reported_seq":339,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.831150+0000","last_change":"2024-04-06T03:33:19.694033+0000","last_active":"2024-04-06T03:35:13.831150+0000","last_peered":"2024-04-06T03:35:13.831150+0000","last_clean":"2024-04-06T03:35:13.831150+0000","last_became_active":"2024-04-06T03:27:10.498254+0000","last_became_peered":"2024-04-06T03:27:10.498254+0000","last_unstale":"2024-04-06T03:35:13.831150+0000","last_undegraded":"2024-04-06T03:35:13.831150+0000","last_fullsized":"2024-04-06T03:35:13.831150+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'13","last_scrub_stamp":"2024-04-06T03:33:19.693903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:19.693903+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":258,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834036+0000","last_change":"2024-04-06T03:33:42.516624+0000","last_active":"2024-04-06T03:35:13.834036+0000","last_peered":"2024-04-06T03:35:13.834036+0000","last_clean":"2024-04-06T03:35:13.834036+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:35:13.834036+0000","last_undegraded":"2024-04-06T03:35:13.834036+0000","last_fullsized":"2024-04-06T03:35:13.834036+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_clean_scrub_stamp":"2024-04-06T03:33:42.516373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"199'24","reported_seq":423,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817695+0000","last_change":"2024-04-06T03:33:15.375572+0000","last_active":"2024-04-06T03:35:13.817695+0000","last_peered":"2024-04-06T03:35:13.817695+0000","last_clean":"2024-04-06T03:35:13.817695+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:35:13.817695+0000","last_undegraded":"2024-04-06T03:35:13.817695+0000","last_fullsized":"2024-04-06T03:35:13.817695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:33:15.375394+0000","last_deep_scrub":"168'16","last_deep_scrub_stamp":"2024-04-06T03:28:39.973804+0000","last_clean_scrub_stamp":"2024-04-06T03:33:15.375394+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":386,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817957+0000","last_change":"2024-04-06T03:33:55.107549+0000","last_active":"2024-04-06T03:35:13.817957+0000","last_peered":"2024-04-06T03:35:13.817957+0000","last_clean":"2024-04-06T03:35:13.817957+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:35:13.817957+0000","last_undegraded":"2024-04-06T03:35:13.817957+0000","last_fullsized":"2024-04-06T03:35:13.817957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:55.107432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:33:55.107432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":251,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834393+0000","last_change":"2024-04-06T03:33:46.541535+0000","last_active":"2024-04-06T03:35:13.834393+0000","last_peered":"2024-04-06T03:35:13.834393+0000","last_clean":"2024-04-06T03:35:13.834393+0000","last_became_active":"2024-04-06T03:25:54.485733+0000","last_became_peered":"2024-04-06T03:25:54.485733+0000","last_unstale":"2024-04-06T03:35:13.834393+0000","last_undegraded":"2024-04-06T03:35:13.834393+0000","last_fullsized":"2024-04-06T03:35:13.834393+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:46.541240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:46.541240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":253,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811825+0000","last_change":"2024-04-06T03:35:11.379061+0000","last_active":"2024-04-06T03:35:13.811825+0000","last_peered":"2024-04-06T03:35:13.811825+0000","last_clean":"2024-04-06T03:35:13.811825+0000","last_became_active":"2024-04-06T03:26:18.615468+0000","last_became_peered":"2024-04-06T03:26:18.615468+0000","last_unstale":"2024-04-06T03:35:13.811825+0000","last_undegraded":"2024-04-06T03:35:13.811825+0000","last_fullsized":"2024-04-06T03:35:13.811825+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:11.378882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:35:11.378882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:20.214525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"199'10","reported_seq":348,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.831276+0000","last_change":"2024-04-06T03:35:05.512221+0000","last_active":"2024-04-06T03:35:13.831276+0000","last_peered":"2024-04-06T03:35:13.831276+0000","last_clean":"2024-04-06T03:35:13.831276+0000","last_became_active":"2024-04-06T03:26:18.639720+0000","last_became_peered":"2024-04-06T03:26:18.639720+0000","last_unstale":"2024-04-06T03:35:13.831276+0000","last_undegraded":"2024-04-06T03:35:13.831276+0000","last_fullsized":"2024-04-06T03:35:13.831276+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10","last_scrub_stamp":"2024-04-06T03:35:05.512127+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:35:05.512127+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:29.579207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":399,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828401+0000","last_change":"2024-04-06T03:33:32.787376+0000","last_active":"2024-04-06T03:35:13.828401+0000","last_peered":"2024-04-06T03:35:13.828401+0000","last_clean":"2024-04-06T03:35:13.828401+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:35:13.828401+0000","last_undegraded":"2024-04-06T03:35:13.828401+0000","last_fullsized":"2024-04-06T03:35:13.828401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:33:32.787297+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:33:32.787297+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":384,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837391+0000","last_change":"2024-04-06T03:34:56.436008+0000","last_active":"2024-04-06T03:35:13.837391+0000","last_peered":"2024-04-06T03:35:13.837391+0000","last_clean":"2024-04-06T03:35:13.837391+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:35:13.837391+0000","last_undegraded":"2024-04-06T03:35:13.837391+0000","last_fullsized":"2024-04-06T03:35:13.837391+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:56.435927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:34:56.435927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:04.241352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":417,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827322+0000","last_change":"2024-04-06T03:34:50.174297+0000","last_active":"2024-04-06T03:35:13.827322+0000","last_peered":"2024-04-06T03:35:13.827322+0000","last_clean":"2024-04-06T03:35:13.827322+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:35:13.827322+0000","last_undegraded":"2024-04-06T03:35:13.827322+0000","last_fullsized":"2024-04-06T03:35:13.827322+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:50.174210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:34:50.174210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:13.233705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"199'31","reported_seq":444,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817686+0000","last_change":"2024-04-06T03:33:58.072257+0000","last_active":"2024-04-06T03:35:13.817686+0000","last_peered":"2024-04-06T03:35:13.817686+0000","last_clean":"2024-04-06T03:35:13.817686+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:35:13.817686+0000","last_undegraded":"2024-04-06T03:35:13.817686+0000","last_fullsized":"2024-04-06T03:35:13.817686+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:33:58.072001+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:33:58.072001+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"242'2784","reported_seq":2816,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.768245+0000","last_change":"2024-04-06T03:34:24.867428+0000","last_active":"2024-04-06T03:35:14.768245+0000","last_peered":"2024-04-06T03:35:14.768245+0000","last_clean":"2024-04-06T03:35:14.768245+0000","last_became_active":"2024-04-06T03:34:24.867100+0000","last_became_peered":"2024-04-06T03:34:24.867100+0000","last_unstale":"2024-04-06T03:35:14.768245+0000","last_undegraded":"2024-04-06T03:35:14.768245+0000","last_fullsized":"2024-04-06T03:35:14.768245+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2784,"log_dups_size":0,"ondisk_log_size":2784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:48.026372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182452224,"num_objects":2784,"num_object_clones":0,"num_object_copies":8352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2784,"num_write_kb":178176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":381,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833848+0000","last_change":"2024-04-06T03:34:52.570618+0000","last_active":"2024-04-06T03:35:13.833848+0000","last_peered":"2024-04-06T03:35:13.833848+0000","last_clean":"2024-04-06T03:35:13.833848+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:35:13.833848+0000","last_undegraded":"2024-04-06T03:35:13.833848+0000","last_fullsized":"2024-04-06T03:35:13.833848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:52.570528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:34:52.570528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:09.780862+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":419,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812534+0000","last_change":"2024-04-06T03:33:14.584324+0000","last_active":"2024-04-06T03:35:13.812534+0000","last_peered":"2024-04-06T03:35:13.812534+0000","last_clean":"2024-04-06T03:35:13.812534+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:35:13.812534+0000","last_undegraded":"2024-04-06T03:35:13.812534+0000","last_fullsized":"2024-04-06T03:35:13.812534+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:33:14.584201+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:33:14.584201+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'31","reported_seq":444,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818342+0000","last_change":"2024-04-06T03:33:35.363415+0000","last_active":"2024-04-06T03:35:13.818342+0000","last_peered":"2024-04-06T03:35:13.818342+0000","last_clean":"2024-04-06T03:35:13.818342+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:35:13.818342+0000","last_undegraded":"2024-04-06T03:35:13.818342+0000","last_fullsized":"2024-04-06T03:35:13.818342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'31","last_scrub_stamp":"2024-04-06T03:33:35.363029+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.363029+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":409,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827096+0000","last_change":"2024-04-06T03:35:04.470775+0000","last_active":"2024-04-06T03:35:13.827096+0000","last_peered":"2024-04-06T03:35:13.827096+0000","last_clean":"2024-04-06T03:35:13.827096+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:35:13.827096+0000","last_undegraded":"2024-04-06T03:35:13.827096+0000","last_fullsized":"2024-04-06T03:35:13.827096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:04.470684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:35:04.470684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:34.082551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"242'2733","reported_seq":2765,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.765352+0000","last_change":"2024-04-06T03:34:25.030982+0000","last_active":"2024-04-06T03:35:14.765352+0000","last_peered":"2024-04-06T03:35:14.765352+0000","last_clean":"2024-04-06T03:35:14.765352+0000","last_became_active":"2024-04-06T03:34:25.029320+0000","last_became_peered":"2024-04-06T03:34:25.029320+0000","last_unstale":"2024-04-06T03:35:14.765352+0000","last_undegraded":"2024-04-06T03:35:14.765352+0000","last_fullsized":"2024-04-06T03:35:14.765352+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:26.714990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179109888,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":174912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":388,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828938+0000","last_change":"2024-04-06T03:35:10.631412+0000","last_active":"2024-04-06T03:35:13.828938+0000","last_peered":"2024-04-06T03:35:13.828938+0000","last_clean":"2024-04-06T03:35:13.828938+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:35:13.828938+0000","last_undegraded":"2024-04-06T03:35:13.828938+0000","last_fullsized":"2024-04-06T03:35:13.828938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:10.631289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:35:10.631289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:15.420859+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":347,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830175+0000","last_change":"2024-04-06T03:34:57.496930+0000","last_active":"2024-04-06T03:35:13.830175+0000","last_peered":"2024-04-06T03:35:13.830175+0000","last_clean":"2024-04-06T03:35:13.830175+0000","last_became_active":"2024-04-06T03:27:10.500422+0000","last_became_peered":"2024-04-06T03:27:10.500422+0000","last_unstale":"2024-04-06T03:35:13.830175+0000","last_undegraded":"2024-04-06T03:35:13.830175+0000","last_fullsized":"2024-04-06T03:35:13.830175+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:34:57.496845+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:34:57.496845+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:26.431550+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"199'37","reported_seq":441,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833443+0000","last_change":"2024-04-06T03:35:00.398455+0000","last_active":"2024-04-06T03:35:13.833443+0000","last_peered":"2024-04-06T03:35:13.833443+0000","last_clean":"2024-04-06T03:35:13.833443+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:35:13.833443+0000","last_undegraded":"2024-04-06T03:35:13.833443+0000","last_fullsized":"2024-04-06T03:35:13.833443+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:35:00.398304+0000","last_deep_scrub":"168'25","last_deep_scrub_stamp":"2024-04-06T03:28:40.744518+0000","last_clean_scrub_stamp":"2024-04-06T03:35:00.398304+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:21.750260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":411,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.853377+0000","last_change":"2024-04-06T03:34:52.370119+0000","last_active":"2024-04-06T03:35:13.853377+0000","last_peered":"2024-04-06T03:35:13.853377+0000","last_clean":"2024-04-06T03:35:13.853377+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:35:13.853377+0000","last_undegraded":"2024-04-06T03:35:13.853377+0000","last_fullsized":"2024-04-06T03:35:13.853377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:34:52.369920+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:34:52.369920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:56.770691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":566,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837110+0000","last_change":"2024-04-06T03:33:41.242743+0000","last_active":"2024-04-06T03:35:13.837110+0000","last_peered":"2024-04-06T03:35:13.837110+0000","last_clean":"2024-04-06T03:35:13.837110+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:35:13.837110+0000","last_undegraded":"2024-04-06T03:35:13.837110+0000","last_fullsized":"2024-04-06T03:35:13.837110+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:33:41.242500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.242500+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"242'2828","reported_seq":2860,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.587395+0000","last_change":"2024-04-06T03:34:25.023097+0000","last_active":"2024-04-06T03:35:15.587395+0000","last_peered":"2024-04-06T03:35:15.587395+0000","last_clean":"2024-04-06T03:35:15.587395+0000","last_became_active":"2024-04-06T03:34:25.022147+0000","last_became_peered":"2024-04-06T03:34:25.022147+0000","last_unstale":"2024-04-06T03:35:15.587395+0000","last_undegraded":"2024-04-06T03:35:15.587395+0000","last_fullsized":"2024-04-06T03:35:15.587395+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:50.284723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185335808,"num_objects":2828,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2828,"num_write_kb":180992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":381,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833952+0000","last_change":"2024-04-06T03:34:53.424517+0000","last_active":"2024-04-06T03:35:13.833952+0000","last_peered":"2024-04-06T03:35:13.833952+0000","last_clean":"2024-04-06T03:35:13.833952+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:35:13.833952+0000","last_undegraded":"2024-04-06T03:35:13.833952+0000","last_fullsized":"2024-04-06T03:35:13.833952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:53.424434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:34:53.424434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:09.965705+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":275,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.852890+0000","last_change":"2024-04-06T03:34:54.350652+0000","last_active":"2024-04-06T03:35:13.852890+0000","last_peered":"2024-04-06T03:35:13.852890+0000","last_clean":"2024-04-06T03:35:13.852890+0000","last_became_active":"2024-04-06T03:25:54.484235+0000","last_became_peered":"2024-04-06T03:25:54.484235+0000","last_unstale":"2024-04-06T03:35:13.852890+0000","last_undegraded":"2024-04-06T03:35:13.852890+0000","last_fullsized":"2024-04-06T03:35:13.852890+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:54.350578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:34:54.350578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:20.586500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"199'44","reported_seq":461,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812416+0000","last_change":"2024-04-06T03:33:13.560964+0000","last_active":"2024-04-06T03:35:13.812416+0000","last_peered":"2024-04-06T03:35:13.812416+0000","last_clean":"2024-04-06T03:35:13.812416+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:35:13.812416+0000","last_undegraded":"2024-04-06T03:35:13.812416+0000","last_fullsized":"2024-04-06T03:35:13.812416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'44","last_scrub_stamp":"2024-04-06T03:33:13.560818+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:33:13.560818+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":391,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.439163+0000","last_change":"2024-04-06T03:35:15.439191+0000","last_active":"2024-04-06T03:35:15.439163+0000","last_peered":"2024-04-06T03:35:15.439163+0000","last_clean":"2024-04-06T03:35:15.439163+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:35:15.439163+0000","last_undegraded":"2024-04-06T03:35:15.439163+0000","last_fullsized":"2024-04-06T03:35:15.439163+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:15.439094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:35:15.439094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:39.649602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"242'2731","reported_seq":2763,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.201668+0000","last_change":"2024-04-06T03:34:25.019742+0000","last_active":"2024-04-06T03:35:14.201668+0000","last_peered":"2024-04-06T03:35:14.201668+0000","last_clean":"2024-04-06T03:35:14.201668+0000","last_became_active":"2024-04-06T03:34:25.019101+0000","last_became_peered":"2024-04-06T03:34:25.019101+0000","last_unstale":"2024-04-06T03:35:14.201668+0000","last_undegraded":"2024-04-06T03:35:14.201668+0000","last_fullsized":"2024-04-06T03:35:14.201668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:29.104041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":174784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":379,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817999+0000","last_change":"2024-04-06T03:33:09.452964+0000","last_active":"2024-04-06T03:35:13.817999+0000","last_peered":"2024-04-06T03:35:13.817999+0000","last_clean":"2024-04-06T03:35:13.817999+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:35:13.817999+0000","last_undegraded":"2024-04-06T03:35:13.817999+0000","last_fullsized":"2024-04-06T03:35:13.817999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:09.452871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.452871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"199'42","reported_seq":317,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827556+0000","last_change":"2024-04-06T03:33:24.513812+0000","last_active":"2024-04-06T03:35:13.827556+0000","last_peered":"2024-04-06T03:35:13.827556+0000","last_clean":"2024-04-06T03:35:13.827556+0000","last_became_active":"2024-04-06T03:27:10.495663+0000","last_became_peered":"2024-04-06T03:27:10.495663+0000","last_unstale":"2024-04-06T03:35:13.827556+0000","last_undegraded":"2024-04-06T03:35:13.827556+0000","last_fullsized":"2024-04-06T03:35:13.827556+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'42","last_scrub_stamp":"2024-04-06T03:33:24.513666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:24.513666+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":409,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827726+0000","last_change":"2024-04-06T03:33:37.772754+0000","last_active":"2024-04-06T03:35:13.827726+0000","last_peered":"2024-04-06T03:35:13.827726+0000","last_clean":"2024-04-06T03:35:13.827726+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:35:13.827726+0000","last_undegraded":"2024-04-06T03:35:13.827726+0000","last_fullsized":"2024-04-06T03:35:13.827726+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:37.772672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:33:37.772672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.836398+0000","last_change":"2024-04-06T03:33:36.737236+0000","last_active":"2024-04-06T03:35:13.836398+0000","last_peered":"2024-04-06T03:35:13.836398+0000","last_clean":"2024-04-06T03:35:13.836398+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:35:13.836398+0000","last_undegraded":"2024-04-06T03:35:13.836398+0000","last_fullsized":"2024-04-06T03:35:13.836398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:36.737080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:36.737080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"242'2734","reported_seq":2766,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.791295+0000","last_change":"2024-04-06T03:34:25.022987+0000","last_active":"2024-04-06T03:35:15.791295+0000","last_peered":"2024-04-06T03:35:15.791295+0000","last_clean":"2024-04-06T03:35:15.791295+0000","last_became_active":"2024-04-06T03:34:25.022028+0000","last_became_peered":"2024-04-06T03:34:25.022028+0000","last_unstale":"2024-04-06T03:35:15.791295+0000","last_undegraded":"2024-04-06T03:35:15.791295+0000","last_fullsized":"2024-04-06T03:35:15.791295+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2734,"log_dups_size":0,"ondisk_log_size":2734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:45.192539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179175424,"num_objects":2734,"num_object_clones":0,"num_object_copies":8202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":174976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":253,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837715+0000","last_change":"2024-04-06T03:35:08.795669+0000","last_active":"2024-04-06T03:35:13.837715+0000","last_peered":"2024-04-06T03:35:13.837715+0000","last_clean":"2024-04-06T03:35:13.837715+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:35:13.837715+0000","last_undegraded":"2024-04-06T03:35:13.837715+0000","last_fullsized":"2024-04-06T03:35:13.837715+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:08.795537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:35:08.795537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:11.108502+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"224'32","reported_seq":437,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833635+0000","last_change":"2024-04-06T03:33:50.491583+0000","last_active":"2024-04-06T03:35:13.833635+0000","last_peered":"2024-04-06T03:35:13.833635+0000","last_clean":"2024-04-06T03:35:13.833635+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:35:13.833635+0000","last_undegraded":"2024-04-06T03:35:13.833635+0000","last_fullsized":"2024-04-06T03:35:13.833635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'32","last_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_deep_scrub":"224'32","last_deep_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_clean_scrub_stamp":"2024-04-06T03:33:50.491205+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":268,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827915+0000","last_change":"2024-04-06T03:33:27.712256+0000","last_active":"2024-04-06T03:35:13.827915+0000","last_peered":"2024-04-06T03:35:13.827915+0000","last_clean":"2024-04-06T03:35:13.827915+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:35:13.827915+0000","last_undegraded":"2024-04-06T03:35:13.827915+0000","last_fullsized":"2024-04-06T03:35:13.827915+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:27.712145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:27.712145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":651,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.816828+0000","last_change":"2024-04-06T03:33:33.336069+0000","last_active":"2024-04-06T03:35:13.816828+0000","last_peered":"2024-04-06T03:35:13.816828+0000","last_clean":"2024-04-06T03:35:13.816828+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:35:13.816828+0000","last_undegraded":"2024-04-06T03:35:13.816828+0000","last_fullsized":"2024-04-06T03:35:13.816828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:33:33.335954+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:33:33.335954+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":391,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817842+0000","last_change":"2024-04-06T03:35:04.150949+0000","last_active":"2024-04-06T03:35:13.817842+0000","last_peered":"2024-04-06T03:35:13.817842+0000","last_clean":"2024-04-06T03:35:13.817842+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:35:13.817842+0000","last_undegraded":"2024-04-06T03:35:13.817842+0000","last_fullsized":"2024-04-06T03:35:13.817842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:04.150570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:35:04.150570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:31.900245+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"242'2736","reported_seq":2768,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.600845+0000","last_change":"2024-04-06T03:34:25.028877+0000","last_active":"2024-04-06T03:35:15.600845+0000","last_peered":"2024-04-06T03:35:15.600845+0000","last_clean":"2024-04-06T03:35:15.600845+0000","last_became_active":"2024-04-06T03:34:25.027991+0000","last_became_peered":"2024-04-06T03:34:25.027991+0000","last_unstale":"2024-04-06T03:35:15.600845+0000","last_undegraded":"2024-04-06T03:35:15.600845+0000","last_fullsized":"2024-04-06T03:35:15.600845+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2736,"log_dups_size":0,"ondisk_log_size":2736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:39.631164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179306496,"num_objects":2736,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":175104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837082+0000","last_change":"2024-04-06T03:35:10.265756+0000","last_active":"2024-04-06T03:35:13.837082+0000","last_peered":"2024-04-06T03:35:13.837082+0000","last_clean":"2024-04-06T03:35:13.837082+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:35:13.837082+0000","last_undegraded":"2024-04-06T03:35:13.837082+0000","last_fullsized":"2024-04-06T03:35:13.837082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:10.265436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:31:49.387394+0000","last_clean_scrub_stamp":"2024-04-06T03:35:10.265436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:26.678475+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"199'35","reported_seq":450,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.829760+0000","last_change":"2024-04-06T03:34:00.341333+0000","last_active":"2024-04-06T03:35:13.829760+0000","last_peered":"2024-04-06T03:35:13.829760+0000","last_clean":"2024-04-06T03:35:13.829760+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:35:13.829760+0000","last_undegraded":"2024-04-06T03:35:13.829760+0000","last_fullsized":"2024-04-06T03:35:13.829760+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'35","last_scrub_stamp":"2024-04-06T03:34:00.341058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:34:00.341058+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:20.757960+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":392,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.816793+0000","last_change":"2024-04-06T03:33:41.282140+0000","last_active":"2024-04-06T03:35:13.816793+0000","last_peered":"2024-04-06T03:35:13.816793+0000","last_clean":"2024-04-06T03:35:13.816793+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:35:13.816793+0000","last_undegraded":"2024-04-06T03:35:13.816793+0000","last_fullsized":"2024-04-06T03:35:13.816793+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.281919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.281919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":267,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.841992+0000","last_change":"2024-04-06T03:33:41.412972+0000","last_active":"2024-04-06T03:35:13.841992+0000","last_peered":"2024-04-06T03:35:13.841992+0000","last_clean":"2024-04-06T03:35:13.841992+0000","last_became_active":"2024-04-06T03:25:54.492757+0000","last_became_peered":"2024-04-06T03:25:54.492757+0000","last_unstale":"2024-04-06T03:35:13.841992+0000","last_undegraded":"2024-04-06T03:35:13.841992+0000","last_fullsized":"2024-04-06T03:35:13.841992+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.412744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"242'2722","reported_seq":2751,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.229513+0000","last_change":"2024-04-06T03:34:24.874930+0000","last_active":"2024-04-06T03:35:14.229513+0000","last_peered":"2024-04-06T03:35:14.229513+0000","last_clean":"2024-04-06T03:35:14.229513+0000","last_became_active":"2024-04-06T03:34:24.874001+0000","last_became_peered":"2024-04-06T03:34:24.874001+0000","last_unstale":"2024-04-06T03:35:14.229513+0000","last_undegraded":"2024-04-06T03:35:14.229513+0000","last_fullsized":"2024-04-06T03:35:14.229513+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2722,"log_dups_size":0,"ondisk_log_size":2722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:49.820226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178388992,"num_objects":2722,"num_object_clones":0,"num_object_copies":8166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":174208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.838179+0000","last_change":"2024-04-06T03:34:56.405276+0000","last_active":"2024-04-06T03:35:13.838179+0000","last_peered":"2024-04-06T03:35:13.838179+0000","last_clean":"2024-04-06T03:35:13.838179+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:35:13.838179+0000","last_undegraded":"2024-04-06T03:35:13.838179+0000","last_fullsized":"2024-04-06T03:35:13.838179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:56.405152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:34:56.405152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:15.065582+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":391,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818085+0000","last_change":"2024-04-06T03:33:07.517308+0000","last_active":"2024-04-06T03:35:13.818085+0000","last_peered":"2024-04-06T03:35:13.818085+0000","last_clean":"2024-04-06T03:35:13.818085+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:35:13.818085+0000","last_undegraded":"2024-04-06T03:35:13.818085+0000","last_fullsized":"2024-04-06T03:35:13.818085+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:07.517188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:33:07.517188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"199'50","reported_seq":515,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828338+0000","last_change":"2024-04-06T03:35:08.685744+0000","last_active":"2024-04-06T03:35:13.828338+0000","last_peered":"2024-04-06T03:35:13.828338+0000","last_clean":"2024-04-06T03:35:13.828338+0000","last_became_active":"2024-04-06T03:25:19.551712+0000","last_became_peered":"2024-04-06T03:25:19.551712+0000","last_unstale":"2024-04-06T03:35:13.828338+0000","last_undegraded":"2024-04-06T03:35:13.828338+0000","last_fullsized":"2024-04-06T03:35:13.828338+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'50","last_scrub_stamp":"2024-04-06T03:35:08.685421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:35:08.685421+0000","objects_scrubbed":11,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:21.369715+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":255,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.829472+0000","last_change":"2024-04-06T03:33:53.214635+0000","last_active":"2024-04-06T03:35:13.829472+0000","last_peered":"2024-04-06T03:35:13.829472+0000","last_clean":"2024-04-06T03:35:13.829472+0000","last_became_active":"2024-04-06T03:26:18.643845+0000","last_became_peered":"2024-04-06T03:26:18.643845+0000","last_unstale":"2024-04-06T03:35:13.829472+0000","last_undegraded":"2024-04-06T03:35:13.829472+0000","last_fullsized":"2024-04-06T03:35:13.829472+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:53.214388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.214388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"242'2690","reported_seq":2722,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.218363+0000","last_change":"2024-04-06T03:34:25.016970+0000","last_active":"2024-04-06T03:35:14.218363+0000","last_peered":"2024-04-06T03:35:14.218363+0000","last_clean":"2024-04-06T03:35:14.218363+0000","last_became_active":"2024-04-06T03:34:25.016601+0000","last_became_peered":"2024-04-06T03:34:25.016601+0000","last_unstale":"2024-04-06T03:35:14.218363+0000","last_undegraded":"2024-04-06T03:35:14.218363+0000","last_fullsized":"2024-04-06T03:35:14.218363+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2690,"log_dups_size":0,"ondisk_log_size":2690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:38.657022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176291840,"num_objects":2690,"num_object_clones":0,"num_object_copies":8070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2690,"num_write_kb":172160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":389,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812013+0000","last_change":"2024-04-06T03:35:03.380786+0000","last_active":"2024-04-06T03:35:13.812013+0000","last_peered":"2024-04-06T03:35:13.812013+0000","last_clean":"2024-04-06T03:35:13.812013+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:35:13.812013+0000","last_undegraded":"2024-04-06T03:35:13.812013+0000","last_fullsized":"2024-04-06T03:35:13.812013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:03.380671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:35:03.380671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:17.089073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":408,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.850560+0000","last_change":"2024-04-06T03:34:58.382217+0000","last_active":"2024-04-06T03:35:13.850560+0000","last_peered":"2024-04-06T03:35:13.850560+0000","last_clean":"2024-04-06T03:35:13.850560+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:35:13.850560+0000","last_undegraded":"2024-04-06T03:35:13.850560+0000","last_fullsized":"2024-04-06T03:35:13.850560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:34:58.382079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:34:58.382079+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:03.283505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"199'23","reported_seq":434,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.836964+0000","last_change":"2024-04-06T03:33:49.512729+0000","last_active":"2024-04-06T03:35:13.836964+0000","last_peered":"2024-04-06T03:35:13.836964+0000","last_clean":"2024-04-06T03:35:13.836964+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:35:13.836964+0000","last_undegraded":"2024-04-06T03:35:13.836964+0000","last_fullsized":"2024-04-06T03:35:13.836964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'23","last_scrub_stamp":"2024-04-06T03:33:49.512394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:49.512394+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1236,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.117744+0000","last_change":"2024-04-06T03:33:35.485693+0000","last_active":"2024-04-06T03:35:14.117744+0000","last_peered":"2024-04-06T03:35:14.117744+0000","last_clean":"2024-04-06T03:35:14.117744+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:35:14.117744+0000","last_undegraded":"2024-04-06T03:35:14.117744+0000","last_fullsized":"2024-04-06T03:35:14.117744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:33:35.485602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:33:35.485602+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":838,"num_read_kb":833,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":385,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811573+0000","last_change":"2024-04-06T03:32:00.109040+0000","last_active":"2024-04-06T03:35:13.811573+0000","last_peered":"2024-04-06T03:35:13.811573+0000","last_clean":"2024-04-06T03:35:13.811573+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:35:13.811573+0000","last_undegraded":"2024-04-06T03:35:13.811573+0000","last_fullsized":"2024-04-06T03:35:13.811573+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:00.108953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:32:00.108953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"242'2749","reported_seq":2781,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.772516+0000","last_change":"2024-04-06T03:34:24.871668+0000","last_active":"2024-04-06T03:35:15.772516+0000","last_peered":"2024-04-06T03:35:15.772516+0000","last_clean":"2024-04-06T03:35:15.772516+0000","last_became_active":"2024-04-06T03:34:24.871397+0000","last_became_peered":"2024-04-06T03:34:24.871397+0000","last_unstale":"2024-04-06T03:35:15.772516+0000","last_undegraded":"2024-04-06T03:35:15.772516+0000","last_fullsized":"2024-04-06T03:35:15.772516+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:37.794765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180158464,"num_objects":2749,"num_object_clones":0,"num_object_copies":8247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2749,"num_write_kb":175936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":379,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817538+0000","last_change":"2024-04-06T03:33:28.341482+0000","last_active":"2024-04-06T03:35:13.817538+0000","last_peered":"2024-04-06T03:35:13.817538+0000","last_clean":"2024-04-06T03:35:13.817538+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:35:13.817538+0000","last_undegraded":"2024-04-06T03:35:13.817538+0000","last_fullsized":"2024-04-06T03:35:13.817538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:28.341397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:33:28.341397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":254,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818267+0000","last_change":"2024-04-06T03:33:32.334478+0000","last_active":"2024-04-06T03:35:13.818267+0000","last_peered":"2024-04-06T03:35:13.818267+0000","last_clean":"2024-04-06T03:35:13.818267+0000","last_became_active":"2024-04-06T03:26:18.621727+0000","last_became_peered":"2024-04-06T03:26:18.621727+0000","last_unstale":"2024-04-06T03:35:13.818267+0000","last_undegraded":"2024-04-06T03:35:13.818267+0000","last_fullsized":"2024-04-06T03:35:13.818267+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:32.334329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:33:32.334329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"199'24","reported_seq":431,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830195+0000","last_change":"2024-04-06T03:34:04.406266+0000","last_active":"2024-04-06T03:35:13.830195+0000","last_peered":"2024-04-06T03:35:13.830195+0000","last_clean":"2024-04-06T03:35:13.830195+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:35:13.830195+0000","last_undegraded":"2024-04-06T03:35:13.830195+0000","last_fullsized":"2024-04-06T03:35:13.830195+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:34:04.406029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:34:04.406029+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":394,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811672+0000","last_change":"2024-04-06T03:33:26.451723+0000","last_active":"2024-04-06T03:35:13.811672+0000","last_peered":"2024-04-06T03:35:13.811672+0000","last_clean":"2024-04-06T03:35:13.811672+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:35:13.811672+0000","last_undegraded":"2024-04-06T03:35:13.811672+0000","last_fullsized":"2024-04-06T03:35:13.811672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:26.451631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:10.453422+0000","last_clean_scrub_stamp":"2024-04-06T03:33:26.451631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":570,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834512+0000","last_change":"2024-04-06T03:33:09.022830+0000","last_active":"2024-04-06T03:35:13.834512+0000","last_peered":"2024-04-06T03:35:13.834512+0000","last_clean":"2024-04-06T03:35:13.834512+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:35:13.834512+0000","last_undegraded":"2024-04-06T03:35:13.834512+0000","last_fullsized":"2024-04-06T03:35:13.834512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:33:09.022701+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:33:09.022701+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"242'2637","reported_seq":2669,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.232183+0000","last_change":"2024-04-06T03:34:24.870758+0000","last_active":"2024-04-06T03:35:14.232183+0000","last_peered":"2024-04-06T03:35:14.232183+0000","last_clean":"2024-04-06T03:35:14.232183+0000","last_became_active":"2024-04-06T03:34:24.870010+0000","last_became_peered":"2024-04-06T03:34:24.870010+0000","last_unstale":"2024-04-06T03:35:14.232183+0000","last_undegraded":"2024-04-06T03:35:14.232183+0000","last_fullsized":"2024-04-06T03:35:14.232183+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2637,"log_dups_size":0,"ondisk_log_size":2637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:48.678070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172818432,"num_objects":2637,"num_object_clones":0,"num_object_copies":7911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2637,"num_write_kb":168768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":381,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837754+0000","last_change":"2024-04-06T03:35:09.692055+0000","last_active":"2024-04-06T03:35:13.837754+0000","last_peered":"2024-04-06T03:35:13.837754+0000","last_clean":"2024-04-06T03:35:13.837754+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:35:13.837754+0000","last_undegraded":"2024-04-06T03:35:13.837754+0000","last_fullsized":"2024-04-06T03:35:13.837754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:09.691982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:35:09.691982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:13.617426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":260,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812845+0000","last_change":"2024-04-06T03:34:57.359204+0000","last_active":"2024-04-06T03:35:13.812845+0000","last_peered":"2024-04-06T03:35:13.812845+0000","last_clean":"2024-04-06T03:35:13.812845+0000","last_became_active":"2024-04-06T03:25:54.491253+0000","last_became_peered":"2024-04-06T03:25:54.491253+0000","last_unstale":"2024-04-06T03:35:13.812845+0000","last_undegraded":"2024-04-06T03:35:13.812845+0000","last_fullsized":"2024-04-06T03:35:13.812845+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:57.359019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:41.912384+0000","last_clean_scrub_stamp":"2024-04-06T03:34:57.359019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:06.330354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":391,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.816892+0000","last_change":"2024-04-06T03:33:53.048866+0000","last_active":"2024-04-06T03:35:13.816892+0000","last_peered":"2024-04-06T03:35:13.816892+0000","last_clean":"2024-04-06T03:35:13.816892+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:35:13.816892+0000","last_undegraded":"2024-04-06T03:35:13.816892+0000","last_fullsized":"2024-04-06T03:35:13.816892+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:53.048558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.048558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"199'37","reported_seq":452,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.811628+0000","last_change":"2024-04-06T03:33:11.524425+0000","last_active":"2024-04-06T03:35:13.811628+0000","last_peered":"2024-04-06T03:35:13.811628+0000","last_clean":"2024-04-06T03:35:13.811628+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:35:13.811628+0000","last_undegraded":"2024-04-06T03:35:13.811628+0000","last_fullsized":"2024-04-06T03:35:13.811628+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:33:11.524289+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:33:11.524289+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"242'2723","reported_seq":2755,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.787163+0000","last_change":"2024-04-06T03:34:25.000340+0000","last_active":"2024-04-06T03:35:15.787163+0000","last_peered":"2024-04-06T03:35:15.787163+0000","last_clean":"2024-04-06T03:35:15.787163+0000","last_became_active":"2024-04-06T03:34:24.999464+0000","last_became_peered":"2024-04-06T03:34:24.999464+0000","last_unstale":"2024-04-06T03:35:15.787163+0000","last_undegraded":"2024-04-06T03:35:15.787163+0000","last_fullsized":"2024-04-06T03:35:15.787163+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2723,"log_dups_size":0,"ondisk_log_size":2723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:32.260012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178454528,"num_objects":2723,"num_object_clones":0,"num_object_copies":8169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2723,"num_write_kb":174272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":219,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830407+0000","last_change":"2024-04-06T03:34:59.619751+0000","last_active":"2024-04-06T03:35:13.830407+0000","last_peered":"2024-04-06T03:35:13.830407+0000","last_clean":"2024-04-06T03:35:13.830407+0000","last_became_active":"2024-04-06T03:27:10.497525+0000","last_became_peered":"2024-04-06T03:27:10.497525+0000","last_unstale":"2024-04-06T03:35:13.830407+0000","last_undegraded":"2024-04-06T03:35:13.830407+0000","last_fullsized":"2024-04-06T03:35:13.830407+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:59.619581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:34:59.619581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:15.888164+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":577,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812429+0000","last_change":"2024-04-06T03:33:10.496163+0000","last_active":"2024-04-06T03:35:13.812429+0000","last_peered":"2024-04-06T03:35:13.812429+0000","last_clean":"2024-04-06T03:35:13.812429+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:35:13.812429+0000","last_undegraded":"2024-04-06T03:35:13.812429+0000","last_fullsized":"2024-04-06T03:35:13.812429+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:33:10.496050+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:30:05.475373+0000","last_clean_scrub_stamp":"2024-04-06T03:33:10.496050+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":407,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.837869+0000","last_change":"2024-04-06T03:34:49.442048+0000","last_active":"2024-04-06T03:35:13.837869+0000","last_peered":"2024-04-06T03:35:13.837869+0000","last_clean":"2024-04-06T03:35:13.837869+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:35:13.837869+0000","last_undegraded":"2024-04-06T03:35:13.837869+0000","last_fullsized":"2024-04-06T03:35:13.837869+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:49.441887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:34:49.441887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:15.557318+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"199'39","reported_seq":456,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.834092+0000","last_change":"2024-04-06T03:33:53.514744+0000","last_active":"2024-04-06T03:35:13.834092+0000","last_peered":"2024-04-06T03:35:13.834092+0000","last_clean":"2024-04-06T03:35:13.834092+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:35:13.834092+0000","last_undegraded":"2024-04-06T03:35:13.834092+0000","last_fullsized":"2024-04-06T03:35:13.834092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:33:53.514484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:53.514484+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:15.902937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"242'2796","reported_seq":2826,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.804018+0000","last_change":"2024-04-06T03:34:25.026070+0000","last_active":"2024-04-06T03:35:15.804018+0000","last_peered":"2024-04-06T03:35:15.804018+0000","last_clean":"2024-04-06T03:35:15.804018+0000","last_became_active":"2024-04-06T03:34:25.025168+0000","last_became_peered":"2024-04-06T03:34:25.025168+0000","last_unstale":"2024-04-06T03:35:15.804018+0000","last_undegraded":"2024-04-06T03:35:15.804018+0000","last_fullsized":"2024-04-06T03:35:15.804018+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2796,"log_dups_size":0,"ondisk_log_size":2796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:36.920246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183238656,"num_objects":2796,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2796,"num_write_kb":178944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":381,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812713+0000","last_change":"2024-04-06T03:32:06.991866+0000","last_active":"2024-04-06T03:35:13.812713+0000","last_peered":"2024-04-06T03:35:13.812713+0000","last_clean":"2024-04-06T03:35:13.812713+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:35:13.812713+0000","last_undegraded":"2024-04-06T03:35:13.812713+0000","last_fullsized":"2024-04-06T03:35:13.812713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:32:06.991780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:32:06.991780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":593,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828743+0000","last_change":"2024-04-06T03:35:00.333842+0000","last_active":"2024-04-06T03:35:13.828743+0000","last_peered":"2024-04-06T03:35:13.828743+0000","last_clean":"2024-04-06T03:35:13.828743+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:35:13.828743+0000","last_undegraded":"2024-04-06T03:35:13.828743+0000","last_fullsized":"2024-04-06T03:35:13.828743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:35:00.333692+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:35:00.333692+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:10.817577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":270,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833438+0000","last_change":"2024-04-06T03:33:12.048860+0000","last_active":"2024-04-06T03:35:13.833438+0000","last_peered":"2024-04-06T03:35:13.833438+0000","last_clean":"2024-04-06T03:35:13.833438+0000","last_became_active":"2024-04-06T03:25:19.549690+0000","last_became_peered":"2024-04-06T03:25:19.549690+0000","last_unstale":"2024-04-06T03:35:13.833438+0000","last_undegraded":"2024-04-06T03:35:13.833438+0000","last_fullsized":"2024-04-06T03:35:13.833438+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:12.048735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:33:12.048735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"199'21","reported_seq":426,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.833521+0000","last_change":"2024-04-06T03:33:51.487161+0000","last_active":"2024-04-06T03:35:13.833521+0000","last_peered":"2024-04-06T03:35:13.833521+0000","last_clean":"2024-04-06T03:35:13.833521+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:35:13.833521+0000","last_undegraded":"2024-04-06T03:35:13.833521+0000","last_fullsized":"2024-04-06T03:35:13.833521+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:33:51.487048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.487048+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"242'2695","reported_seq":2727,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.776488+0000","last_change":"2024-04-06T03:34:25.026854+0000","last_active":"2024-04-06T03:35:14.776488+0000","last_peered":"2024-04-06T03:35:14.776488+0000","last_clean":"2024-04-06T03:35:14.776488+0000","last_became_active":"2024-04-06T03:34:25.025785+0000","last_became_peered":"2024-04-06T03:34:25.025785+0000","last_unstale":"2024-04-06T03:35:14.776488+0000","last_undegraded":"2024-04-06T03:35:14.776488+0000","last_fullsized":"2024-04-06T03:35:14.776488+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2695,"log_dups_size":0,"ondisk_log_size":2695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:45.771167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176619520,"num_objects":2695,"num_object_clones":0,"num_object_copies":8085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":172480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":405,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828516+0000","last_change":"2024-04-06T03:35:02.582338+0000","last_active":"2024-04-06T03:35:13.828516+0000","last_peered":"2024-04-06T03:35:13.828516+0000","last_clean":"2024-04-06T03:35:13.828516+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:35:13.828516+0000","last_undegraded":"2024-04-06T03:35:13.828516+0000","last_fullsized":"2024-04-06T03:35:13.828516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:02.582192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:35:02.582192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:19.339404+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":216,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812633+0000","last_change":"2024-04-06T03:33:51.362465+0000","last_active":"2024-04-06T03:35:13.812633+0000","last_peered":"2024-04-06T03:35:13.812633+0000","last_clean":"2024-04-06T03:35:13.812633+0000","last_became_active":"2024-04-06T03:27:10.501454+0000","last_became_peered":"2024-04-06T03:27:10.501454+0000","last_unstale":"2024-04-06T03:35:13.812633+0000","last_undegraded":"2024-04-06T03:35:13.812633+0000","last_fullsized":"2024-04-06T03:35:13.812633+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:51.362253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.362253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":399,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817668+0000","last_change":"2024-04-06T03:35:06.892362+0000","last_active":"2024-04-06T03:35:13.817668+0000","last_peered":"2024-04-06T03:35:13.817668+0000","last_clean":"2024-04-06T03:35:13.817668+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:35:13.817668+0000","last_undegraded":"2024-04-06T03:35:13.817668+0000","last_fullsized":"2024-04-06T03:35:13.817668+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:06.892285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:35:06.892285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:30.933299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"199'41","reported_seq":340,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.812193+0000","last_change":"2024-04-06T03:31:51.059197+0000","last_active":"2024-04-06T03:35:13.812193+0000","last_peered":"2024-04-06T03:35:13.812193+0000","last_clean":"2024-04-06T03:35:13.812193+0000","last_became_active":"2024-04-06T03:26:18.616944+0000","last_became_peered":"2024-04-06T03:26:18.616944+0000","last_unstale":"2024-04-06T03:35:13.812193+0000","last_undegraded":"2024-04-06T03:35:13.812193+0000","last_fullsized":"2024-04-06T03:35:13.812193+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'41","last_scrub_stamp":"2024-04-06T03:31:51.059107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:31:51.059107+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"242'2738","reported_seq":2770,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.235944+0000","last_change":"2024-04-06T03:34:25.019481+0000","last_active":"2024-04-06T03:35:14.235944+0000","last_peered":"2024-04-06T03:35:14.235944+0000","last_clean":"2024-04-06T03:35:14.235944+0000","last_became_active":"2024-04-06T03:34:25.019121+0000","last_became_peered":"2024-04-06T03:34:25.019121+0000","last_unstale":"2024-04-06T03:35:14.235944+0000","last_undegraded":"2024-04-06T03:35:14.235944+0000","last_fullsized":"2024-04-06T03:35:14.235944+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2738,"log_dups_size":0,"ondisk_log_size":2738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:38.303392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179437568,"num_objects":2738,"num_object_clones":0,"num_object_copies":8214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":175232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":379,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.816731+0000","last_change":"2024-04-06T03:34:56.034203+0000","last_active":"2024-04-06T03:35:13.816731+0000","last_peered":"2024-04-06T03:35:13.816731+0000","last_clean":"2024-04-06T03:35:13.816731+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:35:13.816731+0000","last_undegraded":"2024-04-06T03:35:13.816731+0000","last_fullsized":"2024-04-06T03:35:13.816731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:56.033858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:34:56.033858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:11.692678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.827827+0000","last_change":"2024-04-06T03:33:34.760395+0000","last_active":"2024-04-06T03:35:13.827827+0000","last_peered":"2024-04-06T03:35:13.827827+0000","last_clean":"2024-04-06T03:35:13.827827+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:35:13.827827+0000","last_undegraded":"2024-04-06T03:35:13.827827+0000","last_fullsized":"2024-04-06T03:35:13.827827+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:34.760314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:33:34.760314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":409,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.828213+0000","last_change":"2024-04-06T03:34:57.188807+0000","last_active":"2024-04-06T03:35:13.828213+0000","last_peered":"2024-04-06T03:35:13.828213+0000","last_clean":"2024-04-06T03:35:13.828213+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:35:13.828213+0000","last_undegraded":"2024-04-06T03:35:13.828213+0000","last_fullsized":"2024-04-06T03:35:13.828213+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:57.188699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:34:57.188699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:18.746917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"199'39","reported_seq":444,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818167+0000","last_change":"2024-04-06T03:33:51.057651+0000","last_active":"2024-04-06T03:35:13.818167+0000","last_peered":"2024-04-06T03:35:13.818167+0000","last_clean":"2024-04-06T03:35:13.818167+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:35:13.818167+0000","last_undegraded":"2024-04-06T03:35:13.818167+0000","last_fullsized":"2024-04-06T03:35:13.818167+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:33:51.057496+0000","last_deep_scrub":"188'25","last_deep_scrub_stamp":"2024-04-06T03:30:07.204766+0000","last_clean_scrub_stamp":"2024-04-06T03:33:51.057496+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"242'2775","reported_seq":2807,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.616064+0000","last_change":"2024-04-06T03:34:25.025578+0000","last_active":"2024-04-06T03:35:14.616064+0000","last_peered":"2024-04-06T03:35:14.616064+0000","last_clean":"2024-04-06T03:35:14.616064+0000","last_became_active":"2024-04-06T03:34:25.024611+0000","last_became_peered":"2024-04-06T03:34:25.024611+0000","last_unstale":"2024-04-06T03:35:14.616064+0000","last_undegraded":"2024-04-06T03:35:14.616064+0000","last_fullsized":"2024-04-06T03:35:14.616064+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2775,"log_dups_size":0,"ondisk_log_size":2775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:38.832947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181862400,"num_objects":2775,"num_object_clones":0,"num_object_copies":8325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2775,"num_write_kb":177600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.830017+0000","last_change":"2024-04-06T03:35:04.562286+0000","last_active":"2024-04-06T03:35:13.830017+0000","last_peered":"2024-04-06T03:35:13.830017+0000","last_clean":"2024-04-06T03:35:13.830017+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:35:13.830017+0000","last_undegraded":"2024-04-06T03:35:13.830017+0000","last_fullsized":"2024-04-06T03:35:13.830017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:04.562157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:35:04.562157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:18.954247+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.831273+0000","last_change":"2024-04-06T03:33:41.788390+0000","last_active":"2024-04-06T03:35:13.831273+0000","last_peered":"2024-04-06T03:35:13.831273+0000","last_clean":"2024-04-06T03:35:13.831273+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:35:13.831273+0000","last_undegraded":"2024-04-06T03:35:13.831273+0000","last_fullsized":"2024-04-06T03:35:13.831273+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:41.788284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.906520+0000","last_clean_scrub_stamp":"2024-04-06T03:33:41.788284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":415,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.840395+0000","last_change":"2024-04-06T03:35:06.330609+0000","last_active":"2024-04-06T03:35:13.840395+0000","last_peered":"2024-04-06T03:35:13.840395+0000","last_clean":"2024-04-06T03:35:13.840395+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:35:13.840395+0000","last_undegraded":"2024-04-06T03:35:13.840395+0000","last_fullsized":"2024-04-06T03:35:13.840395+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:06.330493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:31:40.521795+0000","last_clean_scrub_stamp":"2024-04-06T03:35:06.330493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:29.330483+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"199'25","reported_seq":402,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.841562+0000","last_change":"2024-04-06T03:35:13.701317+0000","last_active":"2024-04-06T03:35:13.841562+0000","last_peered":"2024-04-06T03:35:13.841562+0000","last_clean":"2024-04-06T03:35:13.841562+0000","last_became_active":"2024-04-06T03:25:54.495696+0000","last_became_peered":"2024-04-06T03:25:54.495696+0000","last_unstale":"2024-04-06T03:35:13.841562+0000","last_undegraded":"2024-04-06T03:35:13.841562+0000","last_fullsized":"2024-04-06T03:35:13.841562+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'25","last_scrub_stamp":"2024-04-06T03:35:13.701233+0000","last_deep_scrub":"125'15","last_deep_scrub_stamp":"2024-04-06T03:25:15.970079+0000","last_clean_scrub_stamp":"2024-04-06T03:35:13.701233+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:29.124214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"242'2768","reported_seq":2800,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:15.612419+0000","last_change":"2024-04-06T03:34:25.026586+0000","last_active":"2024-04-06T03:35:15.612419+0000","last_peered":"2024-04-06T03:35:15.612419+0000","last_clean":"2024-04-06T03:35:15.612419+0000","last_became_active":"2024-04-06T03:34:25.025681+0000","last_became_peered":"2024-04-06T03:34:25.025681+0000","last_unstale":"2024-04-06T03:35:15.612419+0000","last_undegraded":"2024-04-06T03:35:15.612419+0000","last_fullsized":"2024-04-06T03:35:15.612419+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:34:23.847323+0000","last_clean_scrub_stamp":"2024-04-06T03:34:23.847323+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:42.545288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":177152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":250,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.829262+0000","last_change":"2024-04-06T03:33:29.059858+0000","last_active":"2024-04-06T03:35:13.829262+0000","last_peered":"2024-04-06T03:35:13.829262+0000","last_clean":"2024-04-06T03:35:13.829262+0000","last_became_active":"2024-04-06T03:26:18.641975+0000","last_became_peered":"2024-04-06T03:26:18.641975+0000","last_unstale":"2024-04-06T03:35:13.829262+0000","last_undegraded":"2024-04-06T03:35:13.829262+0000","last_fullsized":"2024-04-06T03:35:13.829262+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:33:29.059614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:33:29.059614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":417,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:14.481696+0000","last_change":"2024-04-06T03:35:14.481733+0000","last_active":"2024-04-06T03:35:14.481696+0000","last_peered":"2024-04-06T03:35:14.481696+0000","last_clean":"2024-04-06T03:35:14.481696+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:35:14.481696+0000","last_undegraded":"2024-04-06T03:35:14.481696+0000","last_fullsized":"2024-04-06T03:35:14.481696+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:35:14.481611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:35:14.481611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:22.854883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"199'27","reported_seq":438,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.818324+0000","last_change":"2024-04-06T03:35:08.862920+0000","last_active":"2024-04-06T03:35:13.818324+0000","last_peered":"2024-04-06T03:35:13.818324+0000","last_clean":"2024-04-06T03:35:13.818324+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:35:13.818324+0000","last_undegraded":"2024-04-06T03:35:13.818324+0000","last_fullsized":"2024-04-06T03:35:13.818324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:35:08.862826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:35:08.862826+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:36:12.376069+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'12","reported_seq":575,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-04-06T03:35:13.817952+0000","last_change":"2024-04-06T03:34:04.032595+0000","last_active":"2024-04-06T03:35:13.817952+0000","last_peered":"2024-04-06T03:35:13.817952+0000","last_clean":"2024-04-06T03:35:13.817952+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:35:13.817952+0000","last_undegraded":"2024-04-06T03:35:13.817952+0000","last_fullsized":"2024-04-06T03:35:13.817952+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'12","last_scrub_stamp":"2024-04-06T03:34:04.032388+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:34:04.032388+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:35:25.833036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2873032704,"num_objects":43839,"num_object_clones":0,"num_object_copies":131517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43839,"num_write_kb":2805696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8553103360,"data_stored":8553103360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43839,"ondisk_log_size":43839,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1414,"num_read_kb":1139,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":994,"ondisk_log_size":994,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":838,"num_read_kb":833,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743607,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1775588,"kb_used_data":1531996,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91969564,"statfs":{"total":95995035648,"available":94176833536,"internally_reserved":0,"allocated":1568763904,"data_stored":1568372135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":249428426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136065,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714944,"kb_used_data":499128,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93030208,"statfs":{"total":95995035648,"available":95262932992,"internally_reserved":0,"allocated":511107072,"data_stored":510750447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575094,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850588,"kb_used_data":1548688,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91894564,"statfs":{"total":95995035648,"available":94100033536,"internally_reserved":0,"allocated":1585856512,"data_stored":1585482668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":309131718},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738624,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1010400,"kb_used_data":704344,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92734752,"statfs":{"total":95995035648,"available":94960386048,"internally_reserved":0,"allocated":721248256,"data_stored":720867963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773008,"kb_used_data":516932,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92972144,"statfs":{"total":95995035648,"available":95203475456,"internally_reserved":0,"allocated":529338368,"data_stored":528976556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1717764,"kb_used_data":1415868,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92027388,"statfs":{"total":95995035648,"available":94236045312,"internally_reserved":0,"allocated":1449848832,"data_stored":1449495126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855575,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703076,"kb_used_data":509468,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93042076,"statfs":{"total":95995035648,"available":95275085824,"internally_reserved":0,"allocated":521695232,"data_stored":521301282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495821,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1802364,"kb_used_data":1528244,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91942788,"statfs":{"total":95995035648,"available":94149414912,"internally_reserved":0,"allocated":1564921856,"data_stored":1564568277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":724172800,"data_stored":724172800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1613627392,"data_stored":1613627392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":531365888,"data_stored":531365888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1453981696,"data_stored":1453981696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":530776064,"data_stored":530776064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1583874048,"data_stored":1583874048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":522387456,"data_stored":522387456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1592918016,"data_stored":1592918016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:35:21.964 INFO:tasks.thrashosds.thrasher:Setting 3.4 to [6, 3, 5, 7, 0, 4] 2024-04-06T03:35:21.964 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.4', '6', '3', '5', '7', '0', '4'] 2024-04-06T03:35:21.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pg-upmap-items 3.4 6 3 5 7 0 4 2024-04-06T03:35:21.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:22.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:22.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:22.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:22.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:22.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:22.394 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 35 16 50812 50796 90.697 76.0625 0.0145663 0.0110204 2024-04-06T03:35:22.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:22.429 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:22.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:22.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:22.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:22.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[21467]: cluster 2024-04-06T03:35:21.613009+0000 osd.2 (osd.2) 281 : cluster 0 2.1 scrub starts 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[21467]: cluster 2024-04-06T03:35:21.616650+0000 osd.2 (osd.2) 282 : cluster 0 2.1 scrub ok 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[21467]: cluster 2024-04-06T03:35:21.694727+0000 osd.7 (osd.7) 357 : cluster 0 2.7 scrub starts 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[21467]: cluster 2024-04-06T03:35:21.696052+0000 osd.7 (osd.7) 358 : cluster 0 2.7 scrub ok 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[29221]: cluster 2024-04-06T03:35:21.613009+0000 osd.2 (osd.2) 281 : cluster 0 2.1 scrub starts 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[29221]: cluster 2024-04-06T03:35:21.616650+0000 osd.2 (osd.2) 282 : cluster 0 2.1 scrub ok 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[29221]: cluster 2024-04-06T03:35:21.694727+0000 osd.7 (osd.7) 357 : cluster 0 2.7 scrub starts 2024-04-06T03:35:22.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:22 smithi050 bash[29221]: cluster 2024-04-06T03:35:21.696052+0000 osd.7 (osd.7) 358 : cluster 0 2.7 scrub ok 2024-04-06T03:35:22.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:22.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:22 smithi153 bash[23852]: cluster 2024-04-06T03:35:21.613009+0000 osd.2 (osd.2) 281 : cluster 0 2.1 scrub starts 2024-04-06T03:35:22.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:22 smithi153 bash[23852]: cluster 2024-04-06T03:35:21.616650+0000 osd.2 (osd.2) 282 : cluster 0 2.1 scrub ok 2024-04-06T03:35:22.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:22 smithi153 bash[23852]: cluster 2024-04-06T03:35:21.694727+0000 osd.7 (osd.7) 357 : cluster 0 2.7 scrub starts 2024-04-06T03:35:22.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:22 smithi153 bash[23852]: cluster 2024-04-06T03:35:21.696052+0000 osd.7 (osd.7) 358 : cluster 0 2.7 scrub ok 2024-04-06T03:35:22.809 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:35:22.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:22.855 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:35:22.902 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:35:22.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:22.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:22.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:22.964 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:35:23.018 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:35:23.058 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:35:23.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:23.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:23.112 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:35:23.158 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:35:23.166 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:35:23.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:23.212 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:23.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:23.344 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:23.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 36 16 51970 51954 90.1879 72.375 0.00738248 0.0110831 2024-04-06T03:35:23.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:23.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:23.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:23.613 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:35:23.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:23.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: cluster 2024-04-06T03:35:22.280008+0000 mgr.y (mgr.24368) 534 : cluster 0 pgmap v556: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:35:23.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: cluster 2024-04-06T03:35:22.661059+0000 osd.2 (osd.2) 283 : cluster 0 3.0 scrub starts 2024-04-06T03:35:23.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: cluster 2024-04-06T03:35:22.674027+0000 osd.2 (osd.2) 284 : cluster 0 3.0 scrub ok 2024-04-06T03:35:23.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: audit 2024-04-06T03:35:22.764955+0000 mon.b (mon.1) 40 : audit 0 from='client.? 172.21.15.50:0/997783786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:35:23.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: cluster 2024-04-06T03:35:22.856790+0000 osd.3 (osd.3) 345 : cluster 0 5.0 scrub starts 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: audit 2024-04-06T03:35:23.037330+0000 mon.c (mon.2) 74 : audit 1 from='client.? 172.21.15.50:0/2262323828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[21467]: audit 2024-04-06T03:35:23.038266+0000 mon.a (mon.0) 1363 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: cluster 2024-04-06T03:35:22.280008+0000 mgr.y (mgr.24368) 534 : cluster 0 pgmap v556: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: cluster 2024-04-06T03:35:22.661059+0000 osd.2 (osd.2) 283 : cluster 0 3.0 scrub starts 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: cluster 2024-04-06T03:35:22.674027+0000 osd.2 (osd.2) 284 : cluster 0 3.0 scrub ok 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: audit 2024-04-06T03:35:22.764955+0000 mon.b (mon.1) 40 : audit 0 from='client.? 172.21.15.50:0/997783786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: cluster 2024-04-06T03:35:22.856790+0000 osd.3 (osd.3) 345 : cluster 0 5.0 scrub starts 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: audit 2024-04-06T03:35:23.037330+0000 mon.c (mon.2) 74 : audit 1 from='client.? 172.21.15.50:0/2262323828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:35:23.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:23 smithi050 bash[29221]: audit 2024-04-06T03:35:23.038266+0000 mon.a (mon.0) 1363 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:35:23.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:23.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:23.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: cluster 2024-04-06T03:35:22.280008+0000 mgr.y (mgr.24368) 534 : cluster 0 pgmap v556: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: cluster 2024-04-06T03:35:22.661059+0000 osd.2 (osd.2) 283 : cluster 0 3.0 scrub starts 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: cluster 2024-04-06T03:35:22.674027+0000 osd.2 (osd.2) 284 : cluster 0 3.0 scrub ok 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: audit 2024-04-06T03:35:22.764955+0000 mon.b (mon.1) 40 : audit 0 from='client.? 172.21.15.50:0/997783786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: cluster 2024-04-06T03:35:22.856790+0000 osd.3 (osd.3) 345 : cluster 0 5.0 scrub starts 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: audit 2024-04-06T03:35:23.037330+0000 mon.c (mon.2) 74 : audit 1 from='client.? 172.21.15.50:0/2262323828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:35:23.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:23 smithi153 bash[23852]: audit 2024-04-06T03:35:23.038266+0000 mon.a (mon.0) 1363 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:35:23.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:23.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:23.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:23.994 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:23.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:24.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:24.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:24.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:24.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:24.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:24.388 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:24.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 37 15 53331 53316 90.0508 85.125 0.0121048 0.0110988 2024-04-06T03:35:24.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:24.526 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:24.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[21467]: cluster 2024-04-06T03:35:22.876671+0000 osd.3 (osd.3) 346 : cluster 0 5.0 scrub ok 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[21467]: cluster 2024-04-06T03:35:23.601639+0000 mon.a (mon.0) 1364 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[21467]: cluster 2024-04-06T03:35:23.601676+0000 mon.a (mon.0) 1365 : cluster 1 Cluster is now healthy 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[21467]: audit 2024-04-06T03:35:23.604097+0000 mon.a (mon.0) 1366 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[21467]: cluster 2024-04-06T03:35:23.608681+0000 mon.a (mon.0) 1367 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[29221]: cluster 2024-04-06T03:35:22.876671+0000 osd.3 (osd.3) 346 : cluster 0 5.0 scrub ok 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[29221]: cluster 2024-04-06T03:35:23.601639+0000 mon.a (mon.0) 1364 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[29221]: cluster 2024-04-06T03:35:23.601676+0000 mon.a (mon.0) 1365 : cluster 1 Cluster is now healthy 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[29221]: audit 2024-04-06T03:35:23.604097+0000 mon.a (mon.0) 1366 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:35:24.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:24 smithi050 bash[29221]: cluster 2024-04-06T03:35:23.608681+0000 mon.a (mon.0) 1367 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-04-06T03:35:24.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:24.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:24.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:24 smithi153 bash[23852]: cluster 2024-04-06T03:35:22.876671+0000 osd.3 (osd.3) 346 : cluster 0 5.0 scrub ok 2024-04-06T03:35:24.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:24 smithi153 bash[23852]: cluster 2024-04-06T03:35:23.601639+0000 mon.a (mon.0) 1364 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:35:24.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:24 smithi153 bash[23852]: cluster 2024-04-06T03:35:23.601676+0000 mon.a (mon.0) 1365 : cluster 1 Cluster is now healthy 2024-04-06T03:35:24.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:24 smithi153 bash[23852]: audit 2024-04-06T03:35:23.604097+0000 mon.a (mon.0) 1366 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:35:24.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:24 smithi153 bash[23852]: cluster 2024-04-06T03:35:23.608681+0000 mon.a (mon.0) 1367 : cluster 0 osdmap e245: 8 total, 8 up, 8 in 2024-04-06T03:35:24.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:24.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:24.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:24.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:25.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:25.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:25.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:25.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:25.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:25.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:25.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:25.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 38 16 54644 54628 89.8388 82 0.0104756 0.0111255 2024-04-06T03:35:25.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:25.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:25.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:25.559 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:25.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:25.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:25 smithi153 bash[23852]: cluster 2024-04-06T03:35:23.903853+0000 osd.3 (osd.3) 347 : cluster 0 2.a deep-scrub starts 2024-04-06T03:35:25.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:25 smithi153 bash[23852]: cluster 2024-04-06T03:35:23.908282+0000 osd.3 (osd.3) 348 : cluster 0 2.a deep-scrub ok 2024-04-06T03:35:25.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:25 smithi153 bash[23852]: cluster 2024-04-06T03:35:24.280682+0000 mgr.y (mgr.24368) 535 : cluster 0 pgmap v558: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 663 B/s rd, 52 MiB/s wr, 837 op/s 2024-04-06T03:35:25.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:25 smithi153 bash[23852]: cluster 2024-04-06T03:35:24.697129+0000 osd.7 (osd.7) 359 : cluster 0 6.5 scrub starts 2024-04-06T03:35:25.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:25 smithi153 bash[23852]: cluster 2024-04-06T03:35:24.731179+0000 osd.7 (osd.7) 360 : cluster 0 6.5 scrub ok 2024-04-06T03:35:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:25.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:25.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:25.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[29221]: cluster 2024-04-06T03:35:23.903853+0000 osd.3 (osd.3) 347 : cluster 0 2.a deep-scrub starts 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[29221]: cluster 2024-04-06T03:35:23.908282+0000 osd.3 (osd.3) 348 : cluster 0 2.a deep-scrub ok 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[29221]: cluster 2024-04-06T03:35:24.280682+0000 mgr.y (mgr.24368) 535 : cluster 0 pgmap v558: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 663 B/s rd, 52 MiB/s wr, 837 op/s 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[29221]: cluster 2024-04-06T03:35:24.697129+0000 osd.7 (osd.7) 359 : cluster 0 6.5 scrub starts 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[29221]: cluster 2024-04-06T03:35:24.731179+0000 osd.7 (osd.7) 360 : cluster 0 6.5 scrub ok 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[21467]: cluster 2024-04-06T03:35:23.903853+0000 osd.3 (osd.3) 347 : cluster 0 2.a deep-scrub starts 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[21467]: cluster 2024-04-06T03:35:23.908282+0000 osd.3 (osd.3) 348 : cluster 0 2.a deep-scrub ok 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[21467]: cluster 2024-04-06T03:35:24.280682+0000 mgr.y (mgr.24368) 535 : cluster 0 pgmap v558: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 663 B/s rd, 52 MiB/s wr, 837 op/s 2024-04-06T03:35:25.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[21467]: cluster 2024-04-06T03:35:24.697129+0000 osd.7 (osd.7) 359 : cluster 0 6.5 scrub starts 2024-04-06T03:35:25.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:25 smithi050 bash[21467]: cluster 2024-04-06T03:35:24.731179+0000 osd.7 (osd.7) 360 : cluster 0 6.5 scrub ok 2024-04-06T03:35:25.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:25.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:26.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:26.078 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:26.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:26.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:26.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:26.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:26.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:26.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 39 16 55520 55504 88.9389 54.75 0.00541705 0.0112381 2024-04-06T03:35:26.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:26.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:26.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:26.593 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:26.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:26.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:26.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[21467]: cluster 2024-04-06T03:35:24.933617+0000 osd.3 (osd.3) 349 : cluster 0 6.1b deep-scrub starts 2024-04-06T03:35:26.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.064676+0000 osd.3 (osd.3) 350 : cluster 0 6.1b deep-scrub ok 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.215441+0000 osd.4 (osd.4) 327 : cluster 0 6.e scrub starts 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.253106+0000 osd.4 (osd.4) 328 : cluster 0 6.e scrub ok 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.725988+0000 osd.7 (osd.7) 361 : cluster 0 3.13 scrub starts 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.727682+0000 osd.7 (osd.7) 362 : cluster 0 3.13 scrub ok 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[29221]: cluster 2024-04-06T03:35:24.933617+0000 osd.3 (osd.3) 349 : cluster 0 6.1b deep-scrub starts 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.064676+0000 osd.3 (osd.3) 350 : cluster 0 6.1b deep-scrub ok 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.215441+0000 osd.4 (osd.4) 327 : cluster 0 6.e scrub starts 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.253106+0000 osd.4 (osd.4) 328 : cluster 0 6.e scrub ok 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.725988+0000 osd.7 (osd.7) 361 : cluster 0 3.13 scrub starts 2024-04-06T03:35:26.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:26 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.727682+0000 osd.7 (osd.7) 362 : cluster 0 3.13 scrub ok 2024-04-06T03:35:26.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:26.843 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:26.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:26.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:26 smithi153 bash[23852]: cluster 2024-04-06T03:35:24.933617+0000 osd.3 (osd.3) 349 : cluster 0 6.1b deep-scrub starts 2024-04-06T03:35:26.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:26 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.064676+0000 osd.3 (osd.3) 350 : cluster 0 6.1b deep-scrub ok 2024-04-06T03:35:26.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:26 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.215441+0000 osd.4 (osd.4) 327 : cluster 0 6.e scrub starts 2024-04-06T03:35:26.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:26 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.253106+0000 osd.4 (osd.4) 328 : cluster 0 6.e scrub ok 2024-04-06T03:35:26.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:26 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.725988+0000 osd.7 (osd.7) 361 : cluster 0 3.13 scrub starts 2024-04-06T03:35:26.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:26 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.727682+0000 osd.7 (osd.7) 362 : cluster 0 3.13 scrub ok 2024-04-06T03:35:26.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:26.983 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:26.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:27.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:27.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:27.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:27.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:27.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:27.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:35:27.401438+0000 min lat: 0.00108908 max lat: 0.428594 avg lat: 0.011268 2024-04-06T03:35:27.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:35:27.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 40 16 56796 56780 88.7091 79.75 0.0104845 0.011268 2024-04-06T03:35:27.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:27.514 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:27.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:27.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.903274+0000 osd.3 (osd.3) 351 : cluster 0 2.14 scrub starts 2024-04-06T03:35:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:25.911642+0000 osd.3 (osd.3) 352 : cluster 0 2.14 scrub ok 2024-04-06T03:35:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:26.281886+0000 mgr.y (mgr.24368) 536 : cluster 0 pgmap v559: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-06T03:35:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:26.661978+0000 osd.2 (osd.2) 285 : cluster 0 5.6 scrub starts 2024-04-06T03:35:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:26.663170+0000 osd.2 (osd.2) 286 : cluster 0 5.6 scrub ok 2024-04-06T03:35:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:26.721921+0000 osd.7 (osd.7) 363 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:35:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:27 smithi153 bash[23852]: cluster 2024-04-06T03:35:26.722874+0000 osd.7 (osd.7) 364 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:35:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:27.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:27.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:27.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:27.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.903274+0000 osd.3 (osd.3) 351 : cluster 0 2.14 scrub starts 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:25.911642+0000 osd.3 (osd.3) 352 : cluster 0 2.14 scrub ok 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:26.281886+0000 mgr.y (mgr.24368) 536 : cluster 0 pgmap v559: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:26.661978+0000 osd.2 (osd.2) 285 : cluster 0 5.6 scrub starts 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:26.663170+0000 osd.2 (osd.2) 286 : cluster 0 5.6 scrub ok 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:26.721921+0000 osd.7 (osd.7) 363 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:35:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[21467]: cluster 2024-04-06T03:35:26.722874+0000 osd.7 (osd.7) 364 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.903274+0000 osd.3 (osd.3) 351 : cluster 0 2.14 scrub starts 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:25.911642+0000 osd.3 (osd.3) 352 : cluster 0 2.14 scrub ok 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:26.281886+0000 mgr.y (mgr.24368) 536 : cluster 0 pgmap v559: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:26.661978+0000 osd.2 (osd.2) 285 : cluster 0 5.6 scrub starts 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:26.663170+0000 osd.2 (osd.2) 286 : cluster 0 5.6 scrub ok 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:26.721921+0000 osd.7 (osd.7) 363 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:35:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:27 smithi050 bash[29221]: cluster 2024-04-06T03:35:26.722874+0000 osd.7 (osd.7) 364 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:35:28.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:28.166 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:28.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:28.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:28.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:28.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 41 16 57656 57640 87.8563 53.75 0.0119989 0.0113771 2024-04-06T03:35:28.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:28.449 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:28.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:28.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:28.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:28 smithi153 bash[23852]: cluster 2024-04-06T03:35:27.852398+0000 osd.1 (osd.1) 293 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:28 smithi153 bash[23852]: cluster 2024-04-06T03:35:27.852494+0000 osd.1 (osd.1) 294 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:28 smithi153 bash[23852]: cluster 2024-04-06T03:35:27.874054+0000 osd.4 (osd.4) 329 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:28 smithi153 bash[23852]: cluster 2024-04-06T03:35:27.874109+0000 osd.4 (osd.4) 330 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:28.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:28.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:28.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:28.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:28.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[21467]: cluster 2024-04-06T03:35:27.852398+0000 osd.1 (osd.1) 293 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:28.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[21467]: cluster 2024-04-06T03:35:27.852494+0000 osd.1 (osd.1) 294 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:28.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[21467]: cluster 2024-04-06T03:35:27.874054+0000 osd.4 (osd.4) 329 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:28.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[21467]: cluster 2024-04-06T03:35:27.874109+0000 osd.4 (osd.4) 330 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:28.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[29221]: cluster 2024-04-06T03:35:27.852398+0000 osd.1 (osd.1) 293 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:28.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[29221]: cluster 2024-04-06T03:35:27.852494+0000 osd.1 (osd.1) 294 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:28.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[29221]: cluster 2024-04-06T03:35:27.874054+0000 osd.4 (osd.4) 329 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:28.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:28 smithi050 bash[29221]: cluster 2024-04-06T03:35:27.874109+0000 osd.4 (osd.4) 330 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:28.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:29.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:29.103 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:29.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:29.229 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:29.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:29.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:29.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:29.395 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 42 16 58730 58714 87.3624 67.125 0.0026319 0.011401 2024-04-06T03:35:29.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:29.477 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:29.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:29.614 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:29.644 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:35:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:35:29.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:29.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:35:29.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:29.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:29.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:29.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:29.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:27.887483+0000 osd.3 (osd.3) 353 : cluster 0 3.1e scrub starts 2024-04-06T03:35:29.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:27.889594+0000 osd.3 (osd.3) 354 : cluster 0 3.1e scrub ok 2024-04-06T03:35:29.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.282327+0000 mgr.y (mgr.24368) 537 : cluster 0 pgmap v560: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.297726+0000 osd.4 (osd.4) 331 : cluster 0 5.1 scrub starts 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.301942+0000 osd.4 (osd.4) 332 : cluster 0 5.1 scrub ok 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.575923+0000 osd.0 (osd.0) 332 : cluster 0 5.15 scrub starts 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.578689+0000 osd.0 (osd.0) 333 : cluster 0 5.15 scrub ok 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.653681+0000 mon.a (mon.0) 1368 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[21467]: cluster 2024-04-06T03:35:28.824935+0000 mon.a (mon.0) 1369 : cluster 0 osdmap e247: 8 total, 8 up, 8 in 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:27.887483+0000 osd.3 (osd.3) 353 : cluster 0 3.1e scrub starts 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:27.889594+0000 osd.3 (osd.3) 354 : cluster 0 3.1e scrub ok 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.282327+0000 mgr.y (mgr.24368) 537 : cluster 0 pgmap v560: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:35:29.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.297726+0000 osd.4 (osd.4) 331 : cluster 0 5.1 scrub starts 2024-04-06T03:35:29.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.301942+0000 osd.4 (osd.4) 332 : cluster 0 5.1 scrub ok 2024-04-06T03:35:29.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.575923+0000 osd.0 (osd.0) 332 : cluster 0 5.15 scrub starts 2024-04-06T03:35:29.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.578689+0000 osd.0 (osd.0) 333 : cluster 0 5.15 scrub ok 2024-04-06T03:35:29.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.653681+0000 mon.a (mon.0) 1368 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-04-06T03:35:29.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:29 smithi050 bash[29221]: cluster 2024-04-06T03:35:28.824935+0000 mon.a (mon.0) 1369 : cluster 0 osdmap e247: 8 total, 8 up, 8 in 2024-04-06T03:35:30.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:30.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:30.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:35:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:27.887483+0000 osd.3 (osd.3) 353 : cluster 0 3.1e scrub starts 2024-04-06T03:35:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:27.889594+0000 osd.3 (osd.3) 354 : cluster 0 3.1e scrub ok 2024-04-06T03:35:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.282327+0000 mgr.y (mgr.24368) 537 : cluster 0 pgmap v560: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:35:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.297726+0000 osd.4 (osd.4) 331 : cluster 0 5.1 scrub starts 2024-04-06T03:35:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.301942+0000 osd.4 (osd.4) 332 : cluster 0 5.1 scrub ok 2024-04-06T03:35:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.575923+0000 osd.0 (osd.0) 332 : cluster 0 5.15 scrub starts 2024-04-06T03:35:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.578689+0000 osd.0 (osd.0) 333 : cluster 0 5.15 scrub ok 2024-04-06T03:35:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.653681+0000 mon.a (mon.0) 1368 : cluster 0 osdmap e246: 8 total, 8 up, 8 in 2024-04-06T03:35:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:29 smithi153 bash[23852]: cluster 2024-04-06T03:35:28.824935+0000 mon.a (mon.0) 1369 : cluster 0 osdmap e247: 8 total, 8 up, 8 in 2024-04-06T03:35:30.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:30.150 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:30.171 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:30.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:30.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:30.396 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 43 16 59662 59646 86.6837 58.25 0.0375743 0.0115301 2024-04-06T03:35:30.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:30.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:30.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:30.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:30.610 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:35:30.610 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:59884 9.2 9:469c4118:::benchmark_data_smithi050_105098_object59883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:30.605658+0000", 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010256615, 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010296665, 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 59884 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605658+0000", 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605658+0000", 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605659+0000", 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0200000000000001e-07 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605684+0000", 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5627999999999999e-05 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605693+0000", 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2110000000000004e-06 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605701+0000", 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.153e-06 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.611532+0000", 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058314630000000003 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.611712+0000", 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018002099999999999 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.611867+0000", 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154611 2024-04-06T03:35:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:59885 9.1 9:812f88f0:::benchmark_data_smithi050_105098_object59884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:30.605711+0000", 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010203067, 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010339941, 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 59885 2024-04-06T03:35:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605711+0000", 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605711+0000", 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:35:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605712+0000", 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3099999999999998e-07 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605771+0000", 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8717e-05 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605780+0000", 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6339999999999996e-06 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.605788+0000", 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2960000000000001e-06 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.606719+0000", 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093133500000000004 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.606847+0000", 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012766899999999999 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.606991+0000", 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000144358 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.609909+0000", 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029184660000000002 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.613893+0000", 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039833259999999997 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15135.0:59884 9.2 e247/231)", 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:30.615943+0000", 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9999719, 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016193300000000001, 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.615943+0000", 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.615943+0000", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.615944+0000", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9399999999999993e-07 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.615952+0000", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0420000000000003e-06 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:35:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.615965+0000", 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3628999999999999e-05 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.615973+0000", 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5499999999999997e-06 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:59896 9.8 9.24fb4ac8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:30.616050+0000", 2024-04-06T03:35:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9998646, 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3999999999999995e-05, 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 59896 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.616050+0000", 2024-04-06T03:35:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.616050+0000", 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.616051+0000", 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3289999999999999e-06 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.616074+0000", 2024-04-06T03:35:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2379999999999999e-05 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.616085+0000", 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1114999999999999e-05 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.616092+0000", 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1849999999999998e-06 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:59877 9.2 9:41c4c333:::benchmark_data_smithi050_105098_object59876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:30.597712+0000", 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.018202592, 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018464589, 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 59877 2024-04-06T03:35:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.597712+0000", 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.597712+0000", 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.597713+0000", 2024-04-06T03:35:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6499999999999998e-07 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.597772+0000", 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9388999999999997e-05 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.597794+0000", 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1460000000000001e-05 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.597804+0000", 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.058e-05 2024-04-06T03:35:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.599947+0000", 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021426050000000001 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.600088+0000", 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000141722 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.607455+0000", 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073665029999999999 2024-04-06T03:35:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.614276+0000", 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068207780000000004 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:30.614999+0000", 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072292700000000005 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:35:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:35:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 5 2024-04-06T03:35:30.631 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:35:30.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:30.675 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:30 smithi153 bash[23852]: cluster 2024-04-06T03:35:29.536550+0000 osd.1 (osd.1) 295 : cluster 0 9.f scrub starts 2024-04-06T03:35:30.675 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:30 smithi153 bash[23852]: cluster 2024-04-06T03:35:29.650236+0000 osd.7 (osd.7) 365 : cluster 0 1.0 scrub starts 2024-04-06T03:35:30.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:30 smithi153 bash[23852]: cluster 2024-04-06T03:35:29.654810+0000 osd.7 (osd.7) 366 : cluster 0 1.0 scrub ok 2024-04-06T03:35:30.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:30 smithi153 bash[23852]: audit 2024-04-06T03:35:29.704607+0000 mgr.y (mgr.24368) 538 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:30.688 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:30.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:30.802 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[21467]: cluster 2024-04-06T03:35:29.536550+0000 osd.1 (osd.1) 295 : cluster 0 9.f scrub starts 2024-04-06T03:35:30.802 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[21467]: cluster 2024-04-06T03:35:29.650236+0000 osd.7 (osd.7) 365 : cluster 0 1.0 scrub starts 2024-04-06T03:35:30.802 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[21467]: cluster 2024-04-06T03:35:29.654810+0000 osd.7 (osd.7) 366 : cluster 0 1.0 scrub ok 2024-04-06T03:35:30.802 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[21467]: audit 2024-04-06T03:35:29.704607+0000 mgr.y (mgr.24368) 538 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:30.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[29221]: cluster 2024-04-06T03:35:29.536550+0000 osd.1 (osd.1) 295 : cluster 0 9.f scrub starts 2024-04-06T03:35:30.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[29221]: cluster 2024-04-06T03:35:29.650236+0000 osd.7 (osd.7) 365 : cluster 0 1.0 scrub starts 2024-04-06T03:35:30.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[29221]: cluster 2024-04-06T03:35:29.654810+0000 osd.7 (osd.7) 366 : cluster 0 1.0 scrub ok 2024-04-06T03:35:30.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:30 smithi050 bash[29221]: audit 2024-04-06T03:35:29.704607+0000 mgr.y (mgr.24368) 538 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:30.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:30.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:30.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:30.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:31.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:31.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:31.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:31.218 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:31.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:31.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.350 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:31.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:31.396 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 44 16 60859 60843 86.4138 74.8125 0.00539699 0.0115659 2024-04-06T03:35:31.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:31.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:31.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:31.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:31.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:31.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[21467]: cluster 2024-04-06T03:35:29.880977+0000 osd.3 (osd.3) 355 : cluster 0 5.14 scrub starts 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[21467]: cluster 2024-04-06T03:35:29.907019+0000 osd.3 (osd.3) 356 : cluster 0 5.14 scrub ok 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[21467]: cluster 2024-04-06T03:35:30.282977+0000 mgr.y (mgr.24368) 539 : cluster 0 pgmap v563: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 766 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[21467]: cluster 2024-04-06T03:35:30.724938+0000 osd.2 (osd.2) 287 : cluster 0 2.3 scrub starts 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[21467]: cluster 2024-04-06T03:35:30.729874+0000 osd.2 (osd.2) 288 : cluster 0 2.3 scrub ok 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[29221]: cluster 2024-04-06T03:35:29.880977+0000 osd.3 (osd.3) 355 : cluster 0 5.14 scrub starts 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[29221]: cluster 2024-04-06T03:35:29.907019+0000 osd.3 (osd.3) 356 : cluster 0 5.14 scrub ok 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[29221]: cluster 2024-04-06T03:35:30.282977+0000 mgr.y (mgr.24368) 539 : cluster 0 pgmap v563: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 766 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[29221]: cluster 2024-04-06T03:35:30.724938+0000 osd.2 (osd.2) 287 : cluster 0 2.3 scrub starts 2024-04-06T03:35:31.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:31 smithi050 bash[29221]: cluster 2024-04-06T03:35:30.729874+0000 osd.2 (osd.2) 288 : cluster 0 2.3 scrub ok 2024-04-06T03:35:31.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:35:31.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:31.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:31.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:32.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:32.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:31 smithi153 bash[23852]: cluster 2024-04-06T03:35:29.880977+0000 osd.3 (osd.3) 355 : cluster 0 5.14 scrub starts 2024-04-06T03:35:32.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:31 smithi153 bash[23852]: cluster 2024-04-06T03:35:29.907019+0000 osd.3 (osd.3) 356 : cluster 0 5.14 scrub ok 2024-04-06T03:35:32.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:31 smithi153 bash[23852]: cluster 2024-04-06T03:35:30.282977+0000 mgr.y (mgr.24368) 539 : cluster 0 pgmap v563: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 766 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-04-06T03:35:32.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:31 smithi153 bash[23852]: cluster 2024-04-06T03:35:30.724938+0000 osd.2 (osd.2) 287 : cluster 0 2.3 scrub starts 2024-04-06T03:35:32.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:31 smithi153 bash[23852]: cluster 2024-04-06T03:35:30.729874+0000 osd.2 (osd.2) 288 : cluster 0 2.3 scrub ok 2024-04-06T03:35:32.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:32.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:32.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:32.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:32.297 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:32.396 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 45 16 62122 62106 86.2475 78.9375 0.0119398 0.0115898 2024-04-06T03:35:32.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:32.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:32.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:32.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:32.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:32.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:32.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-06T03:35:32.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:32.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:32 smithi050 bash[21467]: cluster 2024-04-06T03:35:31.203524+0000 osd.1 (osd.1) 296 : cluster 0 9.f scrub ok 2024-04-06T03:35:32.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:32 smithi050 bash[21467]: cluster 2024-04-06T03:35:31.329552+0000 osd.4 (osd.4) 333 : cluster 0 6.1d scrub starts 2024-04-06T03:35:32.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:32 smithi050 bash[21467]: cluster 2024-04-06T03:35:31.414565+0000 osd.4 (osd.4) 334 : cluster 0 6.1d scrub ok 2024-04-06T03:35:32.804 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:32 smithi050 bash[29221]: cluster 2024-04-06T03:35:31.203524+0000 osd.1 (osd.1) 296 : cluster 0 9.f scrub ok 2024-04-06T03:35:32.804 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:32 smithi050 bash[29221]: cluster 2024-04-06T03:35:31.329552+0000 osd.4 (osd.4) 333 : cluster 0 6.1d scrub starts 2024-04-06T03:35:32.805 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:32 smithi050 bash[29221]: cluster 2024-04-06T03:35:31.414565+0000 osd.4 (osd.4) 334 : cluster 0 6.1d scrub ok 2024-04-06T03:35:32.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:32.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:32.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:32.934 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:32 smithi153 bash[23852]: cluster 2024-04-06T03:35:31.203524+0000 osd.1 (osd.1) 296 : cluster 0 9.f scrub ok 2024-04-06T03:35:32.934 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:32 smithi153 bash[23852]: cluster 2024-04-06T03:35:31.329552+0000 osd.4 (osd.4) 333 : cluster 0 6.1d scrub starts 2024-04-06T03:35:32.934 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:32 smithi153 bash[23852]: cluster 2024-04-06T03:35:31.414565+0000 osd.4 (osd.4) 334 : cluster 0 6.1d scrub ok 2024-04-06T03:35:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:32.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:33.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:33.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:33.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:33.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:33.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:33.351 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:33.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:33.396 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 46 16 63311 63295 85.9879 74.3125 0.00387378 0.0116236 2024-04-06T03:35:33.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:33.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:33.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:33.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:33.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:33.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.284436+0000 mgr.y (mgr.24368) 540 : cluster 0 pgmap v564: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.310120+0000 osd.4 (osd.4) 335 : cluster 0 3.9 scrub starts 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.312896+0000 osd.4 (osd.4) 336 : cluster 0 3.9 scrub ok 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.543658+0000 osd.0 (osd.0) 334 : cluster 0 5.3 scrub starts 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.551840+0000 osd.0 (osd.0) 335 : cluster 0 5.3 scrub ok 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.658730+0000 osd.7 (osd.7) 367 : cluster 0 6.3 scrub starts 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.731243+0000 osd.7 (osd.7) 368 : cluster 0 6.3 scrub ok 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.792702+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-04-06T03:35:33.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.795280+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.856207+0000 osd.3 (osd.3) 357 : cluster 0 6.1f scrub starts 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.284436+0000 mgr.y (mgr.24368) 540 : cluster 0 pgmap v564: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.310120+0000 osd.4 (osd.4) 335 : cluster 0 3.9 scrub starts 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.312896+0000 osd.4 (osd.4) 336 : cluster 0 3.9 scrub ok 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.543658+0000 osd.0 (osd.0) 334 : cluster 0 5.3 scrub starts 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.551840+0000 osd.0 (osd.0) 335 : cluster 0 5.3 scrub ok 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.658730+0000 osd.7 (osd.7) 367 : cluster 0 6.3 scrub starts 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.731243+0000 osd.7 (osd.7) 368 : cluster 0 6.3 scrub ok 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.792702+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.795280+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-04-06T03:35:33.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:33 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.856207+0000 osd.3 (osd.3) 357 : cluster 0 6.1f scrub starts 2024-04-06T03:35:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:33.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:33.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:33.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:33.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:34.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.284436+0000 mgr.y (mgr.24368) 540 : cluster 0 pgmap v564: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-06T03:35:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.310120+0000 osd.4 (osd.4) 335 : cluster 0 3.9 scrub starts 2024-04-06T03:35:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.312896+0000 osd.4 (osd.4) 336 : cluster 0 3.9 scrub ok 2024-04-06T03:35:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.543658+0000 osd.0 (osd.0) 334 : cluster 0 5.3 scrub starts 2024-04-06T03:35:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.551840+0000 osd.0 (osd.0) 335 : cluster 0 5.3 scrub ok 2024-04-06T03:35:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.658730+0000 osd.7 (osd.7) 367 : cluster 0 6.3 scrub starts 2024-04-06T03:35:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.731243+0000 osd.7 (osd.7) 368 : cluster 0 6.3 scrub ok 2024-04-06T03:35:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.792702+0000 osd.2 (osd.2) 289 : cluster 0 6.14 scrub starts 2024-04-06T03:35:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.795280+0000 osd.2 (osd.2) 290 : cluster 0 6.14 scrub ok 2024-04-06T03:35:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:33 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.856207+0000 osd.3 (osd.3) 357 : cluster 0 6.1f scrub starts 2024-04-06T03:35:34.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:34.146 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:34.241 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:35:34.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:34.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:34.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:34.396 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 47 16 64539 64523 85.7912 76.75 0.0080918 0.0116505 2024-04-06T03:35:34.402 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:34.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:34.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:34.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:34.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:34.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:34.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:34.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:34.945 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:34.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:34.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:35.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: cluster 2024-04-06T03:35:32.979105+0000 osd.3 (osd.3) 358 : cluster 0 6.1f scrub ok 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: cluster 2024-04-06T03:35:33.264965+0000 osd.4 (osd.4) 337 : cluster 0 5.e deep-scrub starts 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: cluster 2024-04-06T03:35:33.269151+0000 osd.4 (osd.4) 338 : cluster 0 5.e deep-scrub ok 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: cluster 2024-04-06T03:35:33.555683+0000 osd.0 (osd.0) 336 : cluster 0 2.1f scrub starts 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: cluster 2024-04-06T03:35:33.557607+0000 osd.0 (osd.0) 337 : cluster 0 2.1f scrub ok 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: cluster 2024-04-06T03:35:34.285105+0000 mgr.y (mgr.24368) 541 : cluster 0 pgmap v565: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: audit 2024-04-06T03:35:34.502733+0000 mon.a (mon.0) 1370 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:35:35.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[21467]: audit 2024-04-06T03:35:34.504909+0000 mon.a (mon.0) 1371 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: cluster 2024-04-06T03:35:32.979105+0000 osd.3 (osd.3) 358 : cluster 0 6.1f scrub ok 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: cluster 2024-04-06T03:35:33.264965+0000 osd.4 (osd.4) 337 : cluster 0 5.e deep-scrub starts 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: cluster 2024-04-06T03:35:33.269151+0000 osd.4 (osd.4) 338 : cluster 0 5.e deep-scrub ok 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: cluster 2024-04-06T03:35:33.555683+0000 osd.0 (osd.0) 336 : cluster 0 2.1f scrub starts 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: cluster 2024-04-06T03:35:33.557607+0000 osd.0 (osd.0) 337 : cluster 0 2.1f scrub ok 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: cluster 2024-04-06T03:35:34.285105+0000 mgr.y (mgr.24368) 541 : cluster 0 pgmap v565: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: audit 2024-04-06T03:35:34.502733+0000 mon.a (mon.0) 1370 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:35:35.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:34 smithi050 bash[29221]: audit 2024-04-06T03:35:34.504909+0000 mon.a (mon.0) 1371 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:35.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:35.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:35.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:35.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: cluster 2024-04-06T03:35:32.979105+0000 osd.3 (osd.3) 358 : cluster 0 6.1f scrub ok 2024-04-06T03:35:35.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: cluster 2024-04-06T03:35:33.264965+0000 osd.4 (osd.4) 337 : cluster 0 5.e deep-scrub starts 2024-04-06T03:35:35.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: cluster 2024-04-06T03:35:33.269151+0000 osd.4 (osd.4) 338 : cluster 0 5.e deep-scrub ok 2024-04-06T03:35:35.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: cluster 2024-04-06T03:35:33.555683+0000 osd.0 (osd.0) 336 : cluster 0 2.1f scrub starts 2024-04-06T03:35:35.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: cluster 2024-04-06T03:35:33.557607+0000 osd.0 (osd.0) 337 : cluster 0 2.1f scrub ok 2024-04-06T03:35:35.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: cluster 2024-04-06T03:35:34.285105+0000 mgr.y (mgr.24368) 541 : cluster 0 pgmap v565: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 863 op/s 2024-04-06T03:35:35.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: audit 2024-04-06T03:35:34.502733+0000 mon.a (mon.0) 1370 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:35:35.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:34 smithi153 bash[23852]: audit 2024-04-06T03:35:34.504909+0000 mon.a (mon.0) 1371 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:35.327 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:35.396 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 48 16 65539 65523 85.3059 62.5 0.00267042 0.011708 2024-04-06T03:35:35.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:35.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:35.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:35.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:35.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.730 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:35.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:35.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:35.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:35.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:35.973 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:35 smithi153 bash[23852]: cluster 2024-04-06T03:35:34.562246+0000 osd.0 (osd.0) 338 : cluster 0 6.18 scrub starts 2024-04-06T03:35:35.973 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:35 smithi153 bash[23852]: cluster 2024-04-06T03:35:34.609707+0000 osd.0 (osd.0) 339 : cluster 0 6.18 scrub ok 2024-04-06T03:35:35.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:35.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:36.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:35 smithi050 bash[21467]: cluster 2024-04-06T03:35:34.562246+0000 osd.0 (osd.0) 338 : cluster 0 6.18 scrub starts 2024-04-06T03:35:36.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:35 smithi050 bash[21467]: cluster 2024-04-06T03:35:34.609707+0000 osd.0 (osd.0) 339 : cluster 0 6.18 scrub ok 2024-04-06T03:35:36.110 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:35 smithi050 bash[29221]: cluster 2024-04-06T03:35:34.562246+0000 osd.0 (osd.0) 338 : cluster 0 6.18 scrub starts 2024-04-06T03:35:36.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:35 smithi050 bash[29221]: cluster 2024-04-06T03:35:34.609707+0000 osd.0 (osd.0) 339 : cluster 0 6.18 scrub ok 2024-04-06T03:35:36.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:36.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:36.285 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:35:36.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:36.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 49 16 66375 66359 84.6291 52.25 0.0133519 0.0118103 2024-04-06T03:35:36.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:36.440 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:36.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.579 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:36.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:36.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.840 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:36.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:36.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:36.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[21467]: cluster 2024-04-06T03:35:35.277125+0000 osd.4 (osd.4) 339 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:35:36.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[21467]: cluster 2024-04-06T03:35:35.280097+0000 osd.4 (osd.4) 340 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:35:36.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[21467]: audit 2024-04-06T03:35:36.052713+0000 mon.a (mon.0) 1372 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:35:36.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.286465+0000 mgr.y (mgr.24368) 542 : cluster 0 pgmap v566: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-06T03:35:36.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[29221]: cluster 2024-04-06T03:35:35.277125+0000 osd.4 (osd.4) 339 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:35:36.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[29221]: cluster 2024-04-06T03:35:35.280097+0000 osd.4 (osd.4) 340 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:35:36.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[29221]: audit 2024-04-06T03:35:36.052713+0000 mon.a (mon.0) 1372 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:35:36.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:36 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.286465+0000 mgr.y (mgr.24368) 542 : cluster 0 pgmap v566: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-06T03:35:37.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:37.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:37.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:37.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:36 smithi153 bash[23852]: cluster 2024-04-06T03:35:35.277125+0000 osd.4 (osd.4) 339 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:35:37.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:36 smithi153 bash[23852]: cluster 2024-04-06T03:35:35.280097+0000 osd.4 (osd.4) 340 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:35:37.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:36 smithi153 bash[23852]: audit 2024-04-06T03:35:36.052713+0000 mon.a (mon.0) 1372 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:35:37.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:36 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.286465+0000 mgr.y (mgr.24368) 542 : cluster 0 pgmap v566: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-06T03:35:37.307 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:35:37.308 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:35:37.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:37.342 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:37.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 50 16 67198 67182 83.9651 51.4375 0.00428606 0.0118494 2024-04-06T03:35:37.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:37.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:37.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:37.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:37.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:37.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.738 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:37.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:37.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:37.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:37.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:37.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:37.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:38.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:38.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.603998+0000 osd.0 (osd.0) 340 : cluster 0 6.10 scrub starts 2024-04-06T03:35:38.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.671228+0000 osd.0 (osd.0) 341 : cluster 0 6.10 scrub ok 2024-04-06T03:35:38.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.716910+0000 osd.7 (osd.7) 369 : cluster 0 3.15 scrub starts 2024-04-06T03:35:38.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.721883+0000 osd.7 (osd.7) 370 : cluster 0 3.15 scrub ok 2024-04-06T03:35:38.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.761448+0000 osd.2 (osd.2) 291 : cluster 0 5.b scrub starts 2024-04-06T03:35:38.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.764863+0000 osd.2 (osd.2) 292 : cluster 0 5.b scrub ok 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: audit 2024-04-06T03:35:36.942674+0000 mon.a (mon.0) 1373 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: audit 2024-04-06T03:35:36.943767+0000 mon.a (mon.0) 1374 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[21467]: audit 2024-04-06T03:35:36.951546+0000 mon.a (mon.0) 1375 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.603998+0000 osd.0 (osd.0) 340 : cluster 0 6.10 scrub starts 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.671228+0000 osd.0 (osd.0) 341 : cluster 0 6.10 scrub ok 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.716910+0000 osd.7 (osd.7) 369 : cluster 0 3.15 scrub starts 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.721883+0000 osd.7 (osd.7) 370 : cluster 0 3.15 scrub ok 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.761448+0000 osd.2 (osd.2) 291 : cluster 0 5.b scrub starts 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.764863+0000 osd.2 (osd.2) 292 : cluster 0 5.b scrub ok 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: audit 2024-04-06T03:35:36.942674+0000 mon.a (mon.0) 1373 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: audit 2024-04-06T03:35:36.943767+0000 mon.a (mon.0) 1374 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:35:38.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:37 smithi050 bash[29221]: audit 2024-04-06T03:35:36.951546+0000 mon.a (mon.0) 1375 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:35:38.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:38.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:38.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.603998+0000 osd.0 (osd.0) 340 : cluster 0 6.10 scrub starts 2024-04-06T03:35:38.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.671228+0000 osd.0 (osd.0) 341 : cluster 0 6.10 scrub ok 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.716910+0000 osd.7 (osd.7) 369 : cluster 0 3.15 scrub starts 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.721883+0000 osd.7 (osd.7) 370 : cluster 0 3.15 scrub ok 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.761448+0000 osd.2 (osd.2) 291 : cluster 0 5.b scrub starts 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.764863+0000 osd.2 (osd.2) 292 : cluster 0 5.b scrub ok 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: audit 2024-04-06T03:35:36.942674+0000 mon.a (mon.0) 1373 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: audit 2024-04-06T03:35:36.943767+0000 mon.a (mon.0) 1374 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:35:38.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:37 smithi153 bash[23852]: audit 2024-04-06T03:35:36.951546+0000 mon.a (mon.0) 1375 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:35:38.257 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:38.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:38.387 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:38.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 51 16 68270 68254 83.6323 67 0.00640749 0.0119516 2024-04-06T03:35:38.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:38.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:38.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:38.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:38.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:38.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:38.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:38.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:39.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:39.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:39.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.907797+0000 osd.3 (osd.3) 359 : cluster 0 2.1e scrub starts 2024-04-06T03:35:39.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:36.914330+0000 osd.3 (osd.3) 360 : cluster 0 2.1e scrub ok 2024-04-06T03:35:39.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:37.323871+0000 osd.4 (osd.4) 341 : cluster 0 6.1e scrub starts 2024-04-06T03:35:39.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:37.346630+0000 osd.4 (osd.4) 342 : cluster 0 6.1e scrub ok 2024-04-06T03:35:39.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:37.628813+0000 osd.0 (osd.0) 342 : cluster 0 5.2 scrub starts 2024-04-06T03:35:39.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:37.636091+0000 osd.0 (osd.0) 343 : cluster 0 5.2 scrub ok 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:38.286894+0000 mgr.y (mgr.24368) 543 : cluster 0 pgmap v567: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:38.728259+0000 osd.2 (osd.2) 293 : cluster 0 5.a scrub starts 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[29221]: cluster 2024-04-06T03:35:38.729911+0000 osd.2 (osd.2) 294 : cluster 0 5.a scrub ok 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.907797+0000 osd.3 (osd.3) 359 : cluster 0 2.1e scrub starts 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:36.914330+0000 osd.3 (osd.3) 360 : cluster 0 2.1e scrub ok 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:37.323871+0000 osd.4 (osd.4) 341 : cluster 0 6.1e scrub starts 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:37.346630+0000 osd.4 (osd.4) 342 : cluster 0 6.1e scrub ok 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:37.628813+0000 osd.0 (osd.0) 342 : cluster 0 5.2 scrub starts 2024-04-06T03:35:39.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:37.636091+0000 osd.0 (osd.0) 343 : cluster 0 5.2 scrub ok 2024-04-06T03:35:39.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:38.286894+0000 mgr.y (mgr.24368) 543 : cluster 0 pgmap v567: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-06T03:35:39.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:38.728259+0000 osd.2 (osd.2) 293 : cluster 0 5.a scrub starts 2024-04-06T03:35:39.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:38 smithi050 bash[21467]: cluster 2024-04-06T03:35:38.729911+0000 osd.2 (osd.2) 294 : cluster 0 5.a scrub ok 2024-04-06T03:35:39.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:39.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.907797+0000 osd.3 (osd.3) 359 : cluster 0 2.1e scrub starts 2024-04-06T03:35:39.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:36.914330+0000 osd.3 (osd.3) 360 : cluster 0 2.1e scrub ok 2024-04-06T03:35:39.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:37.323871+0000 osd.4 (osd.4) 341 : cluster 0 6.1e scrub starts 2024-04-06T03:35:39.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:37.346630+0000 osd.4 (osd.4) 342 : cluster 0 6.1e scrub ok 2024-04-06T03:35:39.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:37.628813+0000 osd.0 (osd.0) 342 : cluster 0 5.2 scrub starts 2024-04-06T03:35:39.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:37.636091+0000 osd.0 (osd.0) 343 : cluster 0 5.2 scrub ok 2024-04-06T03:35:39.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:38.286894+0000 mgr.y (mgr.24368) 543 : cluster 0 pgmap v567: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-06T03:35:39.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:38.728259+0000 osd.2 (osd.2) 293 : cluster 0 5.a scrub starts 2024-04-06T03:35:39.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:38 smithi153 bash[23852]: cluster 2024-04-06T03:35:38.729911+0000 osd.2 (osd.2) 294 : cluster 0 5.a scrub ok 2024-04-06T03:35:39.307 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:39.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 52 16 69011 68995 82.9146 46.3125 0.00368442 0.0120431 2024-04-06T03:35:39.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:39.486 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:35:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:35:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:35:39.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:39.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:39.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:39.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:39.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:35:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:35:39.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:39.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:39.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.961 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:39.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:39.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:39 smithi153 bash[23852]: audit 2024-04-06T03:35:39.712551+0000 mgr.y (mgr.24368) 544 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:40.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:40.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:40.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:40.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:40.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:40.337 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:40.391 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:39 smithi050 bash[21467]: audit 2024-04-06T03:35:39.712551+0000 mgr.y (mgr.24368) 544 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:40.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:39 smithi050 bash[29221]: audit 2024-04-06T03:35:39.712551+0000 mgr.y (mgr.24368) 544 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:40.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 53 16 69675 69659 82.1331 41.5 0.156627 0.0121493 2024-04-06T03:35:40.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:40.472 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:40.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:40.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:40.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:40.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:40.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:40.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:40.865 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:40.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[21467]: cluster 2024-04-06T03:35:39.600643+0000 osd.0 (osd.0) 344 : cluster 0 3.1b scrub starts 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[21467]: cluster 2024-04-06T03:35:39.606732+0000 osd.0 (osd.0) 345 : cluster 0 3.1b scrub ok 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[21467]: cluster 2024-04-06T03:35:40.287579+0000 mgr.y (mgr.24368) 545 : cluster 0 pgmap v568: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 893 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[21467]: cluster 2024-04-06T03:35:40.742719+0000 osd.2 (osd.2) 295 : cluster 0 9.6 scrub starts 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[29221]: cluster 2024-04-06T03:35:39.600643+0000 osd.0 (osd.0) 344 : cluster 0 3.1b scrub starts 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[29221]: cluster 2024-04-06T03:35:39.606732+0000 osd.0 (osd.0) 345 : cluster 0 3.1b scrub ok 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[29221]: cluster 2024-04-06T03:35:40.287579+0000 mgr.y (mgr.24368) 545 : cluster 0 pgmap v568: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 893 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:35:40.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:40 smithi050 bash[29221]: cluster 2024-04-06T03:35:40.742719+0000 osd.2 (osd.2) 295 : cluster 0 9.6 scrub starts 2024-04-06T03:35:40.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:40.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:41.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:41.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:41.111 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:40 smithi153 bash[23852]: cluster 2024-04-06T03:35:39.600643+0000 osd.0 (osd.0) 344 : cluster 0 3.1b scrub starts 2024-04-06T03:35:41.111 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:40 smithi153 bash[23852]: cluster 2024-04-06T03:35:39.606732+0000 osd.0 (osd.0) 345 : cluster 0 3.1b scrub ok 2024-04-06T03:35:41.111 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:40 smithi153 bash[23852]: cluster 2024-04-06T03:35:40.287579+0000 mgr.y (mgr.24368) 545 : cluster 0 pgmap v568: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 893 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:35:41.111 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:40 smithi153 bash[23852]: cluster 2024-04-06T03:35:40.742719+0000 osd.2 (osd.2) 295 : cluster 0 9.6 scrub starts 2024-04-06T03:35:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:41.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:41.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:41.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:41.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:41.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 54 15 70535 70520 81.6086 53.8125 0.0162698 0.0122475 2024-04-06T03:35:41.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:41.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:41.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:41.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:41.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:41.760 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:41.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:41.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:41.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:42.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:42.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:42.170 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:42.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:42.307 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:42.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:42.398 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 55 16 71574 71558 81.3042 64.875 0.00863937 0.0122937 2024-04-06T03:35:42.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:42.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:42.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:42.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:42.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:42.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:42.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:42.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:42.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:42.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:42.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:43.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:40.897830+0000 osd.3 (osd.3) 361 : cluster 0 3.18 scrub starts 2024-04-06T03:35:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:40.912088+0000 osd.3 (osd.3) 362 : cluster 0 3.18 scrub ok 2024-04-06T03:35:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:41.803054+0000 osd.6 (osd.6) 93 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:41.803121+0000 osd.6 (osd.6) 94 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:41.874069+0000 mon.a (mon.0) 1376 : cluster 0 osdmap e248: 8 total, 8 up, 8 in 2024-04-06T03:35:43.063 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:42.293713+0000 mgr.y (mgr.24368) 546 : cluster 0 pgmap v570: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-06T03:35:43.064 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:42 smithi153 bash[23852]: cluster 2024-04-06T03:35:42.759694+0000 osd.2 (osd.2) 296 : cluster 0 9.6 scrub ok 2024-04-06T03:35:43.085 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:43.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:43.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:43.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:43.350 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:43.399 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 56 16 72321 72305 80.686 46.6875 0.0145014 0.0123886 2024-04-06T03:35:43.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:43.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:40.897830+0000 osd.3 (osd.3) 361 : cluster 0 3.18 scrub starts 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:40.912088+0000 osd.3 (osd.3) 362 : cluster 0 3.18 scrub ok 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:41.803054+0000 osd.6 (osd.6) 93 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:41.803121+0000 osd.6 (osd.6) 94 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:41.874069+0000 mon.a (mon.0) 1376 : cluster 0 osdmap e248: 8 total, 8 up, 8 in 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:42.293713+0000 mgr.y (mgr.24368) 546 : cluster 0 pgmap v570: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-06T03:35:43.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[21467]: cluster 2024-04-06T03:35:42.759694+0000 osd.2 (osd.2) 296 : cluster 0 9.6 scrub ok 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:40.897830+0000 osd.3 (osd.3) 361 : cluster 0 3.18 scrub starts 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:40.912088+0000 osd.3 (osd.3) 362 : cluster 0 3.18 scrub ok 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:41.803054+0000 osd.6 (osd.6) 93 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:41.803121+0000 osd.6 (osd.6) 94 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:41.874069+0000 mon.a (mon.0) 1376 : cluster 0 osdmap e248: 8 total, 8 up, 8 in 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:42.293713+0000 mgr.y (mgr.24368) 546 : cluster 0 pgmap v570: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-04-06T03:35:43.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:43 smithi050 bash[29221]: cluster 2024-04-06T03:35:42.759694+0000 osd.2 (osd.2) 296 : cluster 0 9.6 scrub ok 2024-04-06T03:35:43.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:43.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:43.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:43.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:43.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:43.872 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:43.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:43.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:44.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:44.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:44.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:44.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:44.265 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:44.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:44.400 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 57 16 73347 73331 80.3933 64.125 0.00879839 0.0124341 2024-04-06T03:35:44.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:44.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:44.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:44.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:44.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:44.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:44.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:44.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:44 smithi050 bash[21467]: cluster 2024-04-06T03:35:44.294248+0000 mgr.y (mgr.24368) 547 : cluster 0 pgmap v571: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 963 op/s 2024-04-06T03:35:44.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:44 smithi050 bash[29221]: cluster 2024-04-06T03:35:44.294248+0000 mgr.y (mgr.24368) 547 : cluster 0 pgmap v571: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 963 op/s 2024-04-06T03:35:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:44.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:45.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:45.040 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:44 smithi153 bash[23852]: cluster 2024-04-06T03:35:44.294248+0000 mgr.y (mgr.24368) 547 : cluster 0 pgmap v571: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 963 op/s 2024-04-06T03:35:45.050 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:45.152 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:45.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:45.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:45.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:45.400 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 58 16 74085 74069 79.8025 46.125 0.0180365 0.0125259 2024-04-06T03:35:45.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.438 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:45.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:45.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:45.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:45.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:45.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:45.808 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:45.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:45.941 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:45.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:46.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:45 smithi153 bash[23852]: cluster 2024-04-06T03:35:45.218469+0000 osd.3 (osd.3) 363 : cluster 0 3.b scrub starts 2024-04-06T03:35:46.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:45 smithi153 bash[23852]: cluster 2024-04-06T03:35:45.225190+0000 osd.3 (osd.3) 364 : cluster 0 3.b scrub ok 2024-04-06T03:35:46.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:45 smithi153 bash[23852]: cluster 2024-04-06T03:35:45.592674+0000 osd.0 (osd.0) 346 : cluster 0 3.6 scrub starts 2024-04-06T03:35:46.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:45 smithi153 bash[23852]: cluster 2024-04-06T03:35:45.595665+0000 osd.0 (osd.0) 347 : cluster 0 3.6 scrub ok 2024-04-06T03:35:46.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:46.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:46.222 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:46.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[21467]: cluster 2024-04-06T03:35:45.218469+0000 osd.3 (osd.3) 363 : cluster 0 3.b scrub starts 2024-04-06T03:35:46.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[21467]: cluster 2024-04-06T03:35:45.225190+0000 osd.3 (osd.3) 364 : cluster 0 3.b scrub ok 2024-04-06T03:35:46.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[21467]: cluster 2024-04-06T03:35:45.592674+0000 osd.0 (osd.0) 346 : cluster 0 3.6 scrub starts 2024-04-06T03:35:46.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[21467]: cluster 2024-04-06T03:35:45.595665+0000 osd.0 (osd.0) 347 : cluster 0 3.6 scrub ok 2024-04-06T03:35:46.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[29221]: cluster 2024-04-06T03:35:45.218469+0000 osd.3 (osd.3) 363 : cluster 0 3.b scrub starts 2024-04-06T03:35:46.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[29221]: cluster 2024-04-06T03:35:45.225190+0000 osd.3 (osd.3) 364 : cluster 0 3.b scrub ok 2024-04-06T03:35:46.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[29221]: cluster 2024-04-06T03:35:45.592674+0000 osd.0 (osd.0) 346 : cluster 0 3.6 scrub starts 2024-04-06T03:35:46.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:45 smithi050 bash[29221]: cluster 2024-04-06T03:35:45.595665+0000 osd.0 (osd.0) 347 : cluster 0 3.6 scrub ok 2024-04-06T03:35:46.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:46.361 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:46.400 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 59 16 75040 75024 79.4615 59.6875 0.0129741 0.0125777 2024-04-06T03:35:46.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:46.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:46.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:46.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:46.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:46.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:46.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:47.009 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:47.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:47.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:47.140 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:46 smithi050 bash[21467]: cluster 2024-04-06T03:35:45.583267+0000 osd.7 (osd.7) 371 : cluster 0 5.1a scrub starts 2024-04-06T03:35:47.140 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:46 smithi050 bash[21467]: cluster 2024-04-06T03:35:45.585103+0000 osd.7 (osd.7) 372 : cluster 0 5.1a scrub ok 2024-04-06T03:35:47.140 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:46 smithi050 bash[21467]: cluster 2024-04-06T03:35:46.294996+0000 mgr.y (mgr.24368) 548 : cluster 0 pgmap v572: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-06T03:35:47.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:46 smithi050 bash[29221]: cluster 2024-04-06T03:35:45.583267+0000 osd.7 (osd.7) 371 : cluster 0 5.1a scrub starts 2024-04-06T03:35:47.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:46 smithi050 bash[29221]: cluster 2024-04-06T03:35:45.585103+0000 osd.7 (osd.7) 372 : cluster 0 5.1a scrub ok 2024-04-06T03:35:47.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:46 smithi050 bash[29221]: cluster 2024-04-06T03:35:46.294996+0000 mgr.y (mgr.24368) 548 : cluster 0 pgmap v572: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-06T03:35:47.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:47.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:46 smithi153 bash[23852]: cluster 2024-04-06T03:35:45.583267+0000 osd.7 (osd.7) 371 : cluster 0 5.1a scrub starts 2024-04-06T03:35:47.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:46 smithi153 bash[23852]: cluster 2024-04-06T03:35:45.585103+0000 osd.7 (osd.7) 372 : cluster 0 5.1a scrub ok 2024-04-06T03:35:47.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:46 smithi153 bash[23852]: cluster 2024-04-06T03:35:46.294996+0000 mgr.y (mgr.24368) 548 : cluster 0 pgmap v572: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-06T03:35:47.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:47.400 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:35:47.406894+0000 min lat: 0.00108908 max lat: 0.428594 avg lat: 0.0126645 2024-04-06T03:35:47.400 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:35:47.400 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 60 16 75803 75787 78.9319 47.6875 0.0136178 0.0126645 2024-04-06T03:35:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:47.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:47.544 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:47.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:47.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:47.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:47.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:47.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:47.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:48.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:48.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:48.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:48.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:47 smithi153 bash[23852]: cluster 2024-04-06T03:35:46.707419+0000 osd.7 (osd.7) 373 : cluster 0 3.e scrub starts 2024-04-06T03:35:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:47 smithi153 bash[23852]: cluster 2024-04-06T03:35:46.723005+0000 osd.7 (osd.7) 374 : cluster 0 3.e scrub ok 2024-04-06T03:35:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:47 smithi153 bash[23852]: cluster 2024-04-06T03:35:47.299692+0000 osd.3 (osd.3) 365 : cluster 0 5.16 scrub starts 2024-04-06T03:35:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:47 smithi153 bash[23852]: cluster 2024-04-06T03:35:47.305582+0000 osd.3 (osd.3) 366 : cluster 0 5.16 scrub ok 2024-04-06T03:35:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:48.401 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 61 16 76469 76453 78.3202 41.625 0.005792 0.0127586 2024-04-06T03:35:48.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:48.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:48.593 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:48.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:48.726 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[21467]: cluster 2024-04-06T03:35:46.707419+0000 osd.7 (osd.7) 373 : cluster 0 3.e scrub starts 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[21467]: cluster 2024-04-06T03:35:46.723005+0000 osd.7 (osd.7) 374 : cluster 0 3.e scrub ok 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[21467]: cluster 2024-04-06T03:35:47.299692+0000 osd.3 (osd.3) 365 : cluster 0 5.16 scrub starts 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[21467]: cluster 2024-04-06T03:35:47.305582+0000 osd.3 (osd.3) 366 : cluster 0 5.16 scrub ok 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[29221]: cluster 2024-04-06T03:35:46.707419+0000 osd.7 (osd.7) 373 : cluster 0 3.e scrub starts 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[29221]: cluster 2024-04-06T03:35:46.723005+0000 osd.7 (osd.7) 374 : cluster 0 3.e scrub ok 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[29221]: cluster 2024-04-06T03:35:47.299692+0000 osd.3 (osd.3) 365 : cluster 0 5.16 scrub starts 2024-04-06T03:35:48.759 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:48 smithi050 bash[29221]: cluster 2024-04-06T03:35:47.305582+0000 osd.3 (osd.3) 366 : cluster 0 5.16 scrub ok 2024-04-06T03:35:48.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:48.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:48.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:48.981 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:48.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:49.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:49.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:49.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[29221]: cluster 2024-04-06T03:35:47.523214+0000 osd.7 (osd.7) 375 : cluster 0 6.1c scrub starts 2024-04-06T03:35:49.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[29221]: cluster 2024-04-06T03:35:47.525667+0000 osd.7 (osd.7) 376 : cluster 0 6.1c scrub ok 2024-04-06T03:35:49.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[29221]: cluster 2024-04-06T03:35:47.670050+0000 osd.1 (osd.1) 297 : cluster 0 9.0 deep-scrub starts 2024-04-06T03:35:49.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[29221]: cluster 2024-04-06T03:35:48.296378+0000 mgr.y (mgr.24368) 549 : cluster 0 pgmap v573: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-06T03:35:49.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[21467]: cluster 2024-04-06T03:35:47.523214+0000 osd.7 (osd.7) 375 : cluster 0 6.1c scrub starts 2024-04-06T03:35:49.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[21467]: cluster 2024-04-06T03:35:47.525667+0000 osd.7 (osd.7) 376 : cluster 0 6.1c scrub ok 2024-04-06T03:35:49.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[21467]: cluster 2024-04-06T03:35:47.670050+0000 osd.1 (osd.1) 297 : cluster 0 9.0 deep-scrub starts 2024-04-06T03:35:49.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[21467]: cluster 2024-04-06T03:35:48.296378+0000 mgr.y (mgr.24368) 549 : cluster 0 pgmap v573: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-06T03:35:49.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:49.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:49 smithi153 bash[23852]: cluster 2024-04-06T03:35:47.523214+0000 osd.7 (osd.7) 375 : cluster 0 6.1c scrub starts 2024-04-06T03:35:49.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:49 smithi153 bash[23852]: cluster 2024-04-06T03:35:47.525667+0000 osd.7 (osd.7) 376 : cluster 0 6.1c scrub ok 2024-04-06T03:35:49.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:49 smithi153 bash[23852]: cluster 2024-04-06T03:35:47.670050+0000 osd.1 (osd.1) 297 : cluster 0 9.0 deep-scrub starts 2024-04-06T03:35:49.234 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:49 smithi153 bash[23852]: cluster 2024-04-06T03:35:48.296378+0000 mgr.y (mgr.24368) 549 : cluster 0 pgmap v573: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 871 op/s 2024-04-06T03:35:49.245 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:49.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:49.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:49.401 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 62 16 76960 76944 77.5519 30.6875 0.00650487 0.0128452 2024-04-06T03:35:49.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:49.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:49.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:49.644 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:49.737 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:35:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:35:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:35:49.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:49.756 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:35:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:35:49.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:49.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:49.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:49.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:50.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:50.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:50.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:50.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:50.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:50.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:48.766347+0000 osd.1 (osd.1) 298 : cluster 0 9.4 scrub starts 2024-04-06T03:35:50.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:48.989015+0000 osd.0 (osd.0) 348 : cluster 0 3.1 scrub starts 2024-04-06T03:35:50.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:48.996659+0000 osd.0 (osd.0) 349 : cluster 0 3.1 scrub ok 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.240957+0000 osd.4 (osd.4) 343 : cluster 0 3.3 scrub starts 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.243103+0000 osd.4 (osd.4) 344 : cluster 0 3.3 scrub ok 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.332663+0000 osd.3 (osd.3) 367 : cluster 0 3.2 scrub starts 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.336755+0000 osd.3 (osd.3) 368 : cluster 0 3.2 scrub ok 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: audit 2024-04-06T03:35:49.527747+0000 mon.a (mon.0) 1377 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.629949+0000 osd.2 (osd.2) 297 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.630015+0000 osd.2 (osd.2) 298 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: audit 2024-04-06T03:35:49.720591+0000 mgr.y (mgr.24368) 550 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:50 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.874451+0000 osd.0 (osd.0) 350 : cluster 0 9.b scrub starts 2024-04-06T03:35:50.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:50.401 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 63 16 77645 77629 77.0004 42.8125 0.0102632 0.0129784 2024-04-06T03:35:50.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:50.439 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:48.766347+0000 osd.1 (osd.1) 298 : cluster 0 9.4 scrub starts 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:48.989015+0000 osd.0 (osd.0) 348 : cluster 0 3.1 scrub starts 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:48.996659+0000 osd.0 (osd.0) 349 : cluster 0 3.1 scrub ok 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.240957+0000 osd.4 (osd.4) 343 : cluster 0 3.3 scrub starts 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.243103+0000 osd.4 (osd.4) 344 : cluster 0 3.3 scrub ok 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.332663+0000 osd.3 (osd.3) 367 : cluster 0 3.2 scrub starts 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.336755+0000 osd.3 (osd.3) 368 : cluster 0 3.2 scrub ok 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: audit 2024-04-06T03:35:49.527747+0000 mon.a (mon.0) 1377 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.629949+0000 osd.2 (osd.2) 297 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:50.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.630015+0000 osd.2 (osd.2) 298 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: audit 2024-04-06T03:35:49.720591+0000 mgr.y (mgr.24368) 550 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.874451+0000 osd.0 (osd.0) 350 : cluster 0 9.b scrub starts 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:48.766347+0000 osd.1 (osd.1) 298 : cluster 0 9.4 scrub starts 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:48.989015+0000 osd.0 (osd.0) 348 : cluster 0 3.1 scrub starts 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:48.996659+0000 osd.0 (osd.0) 349 : cluster 0 3.1 scrub ok 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.240957+0000 osd.4 (osd.4) 343 : cluster 0 3.3 scrub starts 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.243103+0000 osd.4 (osd.4) 344 : cluster 0 3.3 scrub ok 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.332663+0000 osd.3 (osd.3) 367 : cluster 0 3.2 scrub starts 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.336755+0000 osd.3 (osd.3) 368 : cluster 0 3.2 scrub ok 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: audit 2024-04-06T03:35:49.527747+0000 mon.a (mon.0) 1377 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.629949+0000 osd.2 (osd.2) 297 : cluster 0 purged_snaps scrub starts 2024-04-06T03:35:50.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.630015+0000 osd.2 (osd.2) 298 : cluster 0 purged_snaps scrub ok 2024-04-06T03:35:50.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: audit 2024-04-06T03:35:49.720591+0000 mgr.y (mgr.24368) 550 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:35:50.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:50 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.874451+0000 osd.0 (osd.0) 350 : cluster 0 9.b scrub starts 2024-04-06T03:35:50.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:50.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:50.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:50.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:50.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:50.831 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:50.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:50.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:50.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:51.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:51.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:51.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:51.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:51 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.565069+0000 osd.7 (osd.7) 377 : cluster 0 3.d deep-scrub starts 2024-04-06T03:35:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:51 smithi153 bash[23852]: cluster 2024-04-06T03:35:49.576896+0000 osd.7 (osd.7) 378 : cluster 0 3.d deep-scrub ok 2024-04-06T03:35:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:51 smithi153 bash[23852]: cluster 2024-04-06T03:35:50.025765+0000 mon.a (mon.0) 1378 : cluster 0 osdmap e249: 8 total, 8 up, 8 in 2024-04-06T03:35:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:51 smithi153 bash[23852]: cluster 2024-04-06T03:35:50.296852+0000 mgr.y (mgr.24368) 551 : cluster 0 pgmap v575: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 607 B/s rd, 43 MiB/s wr, 685 op/s 2024-04-06T03:35:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:51 smithi153 bash[23852]: cluster 2024-04-06T03:35:50.375812+0000 osd.3 (osd.3) 369 : cluster 0 2.12 scrub starts 2024-04-06T03:35:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:51 smithi153 bash[23852]: cluster 2024-04-06T03:35:50.385805+0000 osd.3 (osd.3) 370 : cluster 0 2.12 scrub ok 2024-04-06T03:35:51.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:51.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:51.401 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 64 16 78219 78203 76.3578 35.875 0.013145 0.0130903 2024-04-06T03:35:51.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:51.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.565069+0000 osd.7 (osd.7) 377 : cluster 0 3.d deep-scrub starts 2024-04-06T03:35:51.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[21467]: cluster 2024-04-06T03:35:49.576896+0000 osd.7 (osd.7) 378 : cluster 0 3.d deep-scrub ok 2024-04-06T03:35:51.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[21467]: cluster 2024-04-06T03:35:50.025765+0000 mon.a (mon.0) 1378 : cluster 0 osdmap e249: 8 total, 8 up, 8 in 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[21467]: cluster 2024-04-06T03:35:50.296852+0000 mgr.y (mgr.24368) 551 : cluster 0 pgmap v575: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 607 B/s rd, 43 MiB/s wr, 685 op/s 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[21467]: cluster 2024-04-06T03:35:50.375812+0000 osd.3 (osd.3) 369 : cluster 0 2.12 scrub starts 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[21467]: cluster 2024-04-06T03:35:50.385805+0000 osd.3 (osd.3) 370 : cluster 0 2.12 scrub ok 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.565069+0000 osd.7 (osd.7) 377 : cluster 0 3.d deep-scrub starts 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[29221]: cluster 2024-04-06T03:35:49.576896+0000 osd.7 (osd.7) 378 : cluster 0 3.d deep-scrub ok 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[29221]: cluster 2024-04-06T03:35:50.025765+0000 mon.a (mon.0) 1378 : cluster 0 osdmap e249: 8 total, 8 up, 8 in 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[29221]: cluster 2024-04-06T03:35:50.296852+0000 mgr.y (mgr.24368) 551 : cluster 0 pgmap v575: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 607 B/s rd, 43 MiB/s wr, 685 op/s 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[29221]: cluster 2024-04-06T03:35:50.375812+0000 osd.3 (osd.3) 369 : cluster 0 2.12 scrub starts 2024-04-06T03:35:51.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:51 smithi050 bash[29221]: cluster 2024-04-06T03:35:50.385805+0000 osd.3 (osd.3) 370 : cluster 0 2.12 scrub ok 2024-04-06T03:35:51.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:51.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:51.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:51.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:51.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:51.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:51.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:51.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:52.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:52.067 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:52.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:52.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:52.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:52 smithi153 bash[23852]: cluster 2024-04-06T03:35:51.198147+0000 osd.4 (osd.4) 345 : cluster 0 3.1a scrub starts 2024-04-06T03:35:52.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:52 smithi153 bash[23852]: cluster 2024-04-06T03:35:51.202244+0000 osd.4 (osd.4) 346 : cluster 0 3.1a scrub ok 2024-04-06T03:35:52.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:52 smithi153 bash[23852]: cluster 2024-04-06T03:35:51.554899+0000 osd.0 (osd.0) 351 : cluster 0 9.b scrub ok 2024-04-06T03:35:52.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:52.316 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:52.401 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 65 16 79018 79002 75.9513 49.9375 0.00370948 0.0131261 2024-04-06T03:35:52.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:52.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:52.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:52.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:52 smithi050 bash[21467]: cluster 2024-04-06T03:35:51.198147+0000 osd.4 (osd.4) 345 : cluster 0 3.1a scrub starts 2024-04-06T03:35:52.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:52 smithi050 bash[21467]: cluster 2024-04-06T03:35:51.202244+0000 osd.4 (osd.4) 346 : cluster 0 3.1a scrub ok 2024-04-06T03:35:52.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:52 smithi050 bash[21467]: cluster 2024-04-06T03:35:51.554899+0000 osd.0 (osd.0) 351 : cluster 0 9.b scrub ok 2024-04-06T03:35:52.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:52 smithi050 bash[29221]: cluster 2024-04-06T03:35:51.198147+0000 osd.4 (osd.4) 345 : cluster 0 3.1a scrub starts 2024-04-06T03:35:52.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:52 smithi050 bash[29221]: cluster 2024-04-06T03:35:51.202244+0000 osd.4 (osd.4) 346 : cluster 0 3.1a scrub ok 2024-04-06T03:35:52.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:52 smithi050 bash[29221]: cluster 2024-04-06T03:35:51.554899+0000 osd.0 (osd.0) 351 : cluster 0 9.b scrub ok 2024-04-06T03:35:52.569 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:52.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:52.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:52.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:52.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:52.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:52.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:53.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:53.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:53.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:53.206 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:53 smithi153 bash[23852]: cluster 2024-04-06T03:35:51.573062+0000 osd.7 (osd.7) 379 : cluster 0 2.0 scrub starts 2024-04-06T03:35:53.206 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:53 smithi153 bash[23852]: cluster 2024-04-06T03:35:51.666623+0000 osd.7 (osd.7) 380 : cluster 0 2.0 scrub ok 2024-04-06T03:35:53.207 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:53 smithi153 bash[23852]: cluster 2024-04-06T03:35:51.692119+0000 osd.1 (osd.1) 299 : cluster 0 9.4 scrub ok 2024-04-06T03:35:53.207 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:53 smithi153 bash[23852]: cluster 2024-04-06T03:35:52.297593+0000 mgr.y (mgr.24368) 552 : cluster 0 pgmap v576: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 795 op/s 2024-04-06T03:35:53.207 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:53 smithi153 bash[23852]: cluster 2024-04-06T03:35:52.381132+0000 osd.3 (osd.3) 371 : cluster 0 4.0 scrub starts 2024-04-06T03:35:53.207 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:53 smithi153 bash[23852]: cluster 2024-04-06T03:35:52.382972+0000 osd.3 (osd.3) 372 : cluster 0 4.0 scrub ok 2024-04-06T03:35:53.229 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[21467]: cluster 2024-04-06T03:35:51.573062+0000 osd.7 (osd.7) 379 : cluster 0 2.0 scrub starts 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[21467]: cluster 2024-04-06T03:35:51.666623+0000 osd.7 (osd.7) 380 : cluster 0 2.0 scrub ok 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[21467]: cluster 2024-04-06T03:35:51.692119+0000 osd.1 (osd.1) 299 : cluster 0 9.4 scrub ok 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[21467]: cluster 2024-04-06T03:35:52.297593+0000 mgr.y (mgr.24368) 552 : cluster 0 pgmap v576: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 795 op/s 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[21467]: cluster 2024-04-06T03:35:52.381132+0000 osd.3 (osd.3) 371 : cluster 0 4.0 scrub starts 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[21467]: cluster 2024-04-06T03:35:52.382972+0000 osd.3 (osd.3) 372 : cluster 0 4.0 scrub ok 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[29221]: cluster 2024-04-06T03:35:51.573062+0000 osd.7 (osd.7) 379 : cluster 0 2.0 scrub starts 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[29221]: cluster 2024-04-06T03:35:51.666623+0000 osd.7 (osd.7) 380 : cluster 0 2.0 scrub ok 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[29221]: cluster 2024-04-06T03:35:51.692119+0000 osd.1 (osd.1) 299 : cluster 0 9.4 scrub ok 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[29221]: cluster 2024-04-06T03:35:52.297593+0000 mgr.y (mgr.24368) 552 : cluster 0 pgmap v576: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 795 op/s 2024-04-06T03:35:53.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[29221]: cluster 2024-04-06T03:35:52.381132+0000 osd.3 (osd.3) 371 : cluster 0 4.0 scrub starts 2024-04-06T03:35:53.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:53 smithi050 bash[29221]: cluster 2024-04-06T03:35:52.382972+0000 osd.3 (osd.3) 372 : cluster 0 4.0 scrub ok 2024-04-06T03:35:53.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:53.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:53.401 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 66 16 79543 79527 75.2976 32.8125 0.00782257 0.0132538 2024-04-06T03:35:53.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:53.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:53.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:53.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:53.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:53.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:53.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:35:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:35:53.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:53.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:53.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:53.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:53.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:54.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:54.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:54.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:54.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:54 smithi153 bash[23852]: cluster 2024-04-06T03:35:52.681934+0000 osd.7 (osd.7) 381 : cluster 0 2.8 scrub starts 2024-04-06T03:35:54.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:54 smithi153 bash[23852]: cluster 2024-04-06T03:35:52.688346+0000 osd.7 (osd.7) 382 : cluster 0 2.8 scrub ok 2024-04-06T03:35:54.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:54.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:54.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:54 smithi050 bash[21467]: cluster 2024-04-06T03:35:52.681934+0000 osd.7 (osd.7) 381 : cluster 0 2.8 scrub starts 2024-04-06T03:35:54.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:54 smithi050 bash[21467]: cluster 2024-04-06T03:35:52.688346+0000 osd.7 (osd.7) 382 : cluster 0 2.8 scrub ok 2024-04-06T03:35:54.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:54 smithi050 bash[29221]: cluster 2024-04-06T03:35:52.681934+0000 osd.7 (osd.7) 381 : cluster 0 2.8 scrub starts 2024-04-06T03:35:54.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:54 smithi050 bash[29221]: cluster 2024-04-06T03:35:52.688346+0000 osd.7 (osd.7) 382 : cluster 0 2.8 scrub ok 2024-04-06T03:35:54.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:54.403 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 67 16 80232 80216 74.8137 43.0625 0.0210211 0.0133582 2024-04-06T03:35:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:54.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:54.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:54.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:54.664 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:54.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:54.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:54.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:54.944 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:54.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:55.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:55.071 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:55.074 INFO:teuthology.orchestra.run.smithi050.stderr:set 3.4 pg_upmap_items mapping to [6->3,5->7,0->4] 2024-04-06T03:35:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:55.165 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[21467]: cluster 2024-04-06T03:35:54.035702+0000 osd.0 (osd.0) 352 : cluster 0 9.5 scrub starts 2024-04-06T03:35:55.165 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[21467]: cluster 2024-04-06T03:35:54.227415+0000 osd.4 (osd.4) 347 : cluster 0 2.f scrub starts 2024-04-06T03:35:55.165 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[21467]: cluster 2024-04-06T03:35:54.231956+0000 osd.4 (osd.4) 348 : cluster 0 2.f scrub ok 2024-04-06T03:35:55.165 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[21467]: cluster 2024-04-06T03:35:54.298274+0000 mgr.y (mgr.24368) 553 : cluster 0 pgmap v577: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 759 op/s 2024-04-06T03:35:55.166 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[21467]: audit 2024-04-06T03:35:54.722448+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.50:0/182096545' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.4", "id": [6, 3, 5, 7, 0, 4]}]: dispatch 2024-04-06T03:35:55.166 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[21467]: cluster 2024-04-06T03:35:54.969429+0000 osd.0 (osd.0) 353 : cluster 0 9.9 scrub starts 2024-04-06T03:35:55.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[29221]: cluster 2024-04-06T03:35:54.035702+0000 osd.0 (osd.0) 352 : cluster 0 9.5 scrub starts 2024-04-06T03:35:55.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[29221]: cluster 2024-04-06T03:35:54.227415+0000 osd.4 (osd.4) 347 : cluster 0 2.f scrub starts 2024-04-06T03:35:55.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[29221]: cluster 2024-04-06T03:35:54.231956+0000 osd.4 (osd.4) 348 : cluster 0 2.f scrub ok 2024-04-06T03:35:55.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[29221]: cluster 2024-04-06T03:35:54.298274+0000 mgr.y (mgr.24368) 553 : cluster 0 pgmap v577: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 759 op/s 2024-04-06T03:35:55.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[29221]: audit 2024-04-06T03:35:54.722448+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.50:0/182096545' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.4", "id": [6, 3, 5, 7, 0, 4]}]: dispatch 2024-04-06T03:35:55.167 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:55 smithi050 bash[29221]: cluster 2024-04-06T03:35:54.969429+0000 osd.0 (osd.0) 353 : cluster 0 9.9 scrub starts 2024-04-06T03:35:55.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:55.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:55 smithi153 bash[23852]: cluster 2024-04-06T03:35:54.035702+0000 osd.0 (osd.0) 352 : cluster 0 9.5 scrub starts 2024-04-06T03:35:55.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:55 smithi153 bash[23852]: cluster 2024-04-06T03:35:54.227415+0000 osd.4 (osd.4) 347 : cluster 0 2.f scrub starts 2024-04-06T03:35:55.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:55 smithi153 bash[23852]: cluster 2024-04-06T03:35:54.231956+0000 osd.4 (osd.4) 348 : cluster 0 2.f scrub ok 2024-04-06T03:35:55.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:55 smithi153 bash[23852]: cluster 2024-04-06T03:35:54.298274+0000 mgr.y (mgr.24368) 553 : cluster 0 pgmap v577: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 759 op/s 2024-04-06T03:35:55.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:55 smithi153 bash[23852]: audit 2024-04-06T03:35:54.722448+0000 mon.a (mon.0) 1379 : audit 1 from='client.? 172.21.15.50:0/182096545' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.4", "id": [6, 3, 5, 7, 0, 4]}]: dispatch 2024-04-06T03:35:55.185 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:55 smithi153 bash[23852]: cluster 2024-04-06T03:35:54.969429+0000 osd.0 (osd.0) 353 : cluster 0 9.9 scrub starts 2024-04-06T03:35:55.206 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:55.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:55.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:55.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 68 16 80687 80671 74.1317 28.4375 0.00552155 0.0134772 2024-04-06T03:35:55.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:55.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:55.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:55.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:55.689 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:55.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:55.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:55.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:55.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:55.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:55.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:56.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:56.097 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:56.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:56.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:56 smithi050 bash[21467]: audit 2024-04-06T03:35:55.060171+0000 mon.a (mon.0) 1380 : audit 1 from='client.? 172.21.15.50:0/182096545' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.4", "id": [6, 3, 5, 7, 0, 4]}]': finished 2024-04-06T03:35:56.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:56 smithi050 bash[21467]: cluster 2024-04-06T03:35:55.075190+0000 mon.a (mon.0) 1381 : cluster 0 osdmap e250: 8 total, 8 up, 8 in 2024-04-06T03:35:56.211 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:56 smithi050 bash[21467]: cluster 2024-04-06T03:35:55.812556+0000 osd.0 (osd.0) 354 : cluster 0 9.5 scrub ok 2024-04-06T03:35:56.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:56 smithi050 bash[29221]: audit 2024-04-06T03:35:55.060171+0000 mon.a (mon.0) 1380 : audit 1 from='client.? 172.21.15.50:0/182096545' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.4", "id": [6, 3, 5, 7, 0, 4]}]': finished 2024-04-06T03:35:56.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:56 smithi050 bash[29221]: cluster 2024-04-06T03:35:55.075190+0000 mon.a (mon.0) 1381 : cluster 0 osdmap e250: 8 total, 8 up, 8 in 2024-04-06T03:35:56.212 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:56 smithi050 bash[29221]: cluster 2024-04-06T03:35:55.812556+0000 osd.0 (osd.0) 354 : cluster 0 9.5 scrub ok 2024-04-06T03:35:56.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:56.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:56.345 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:56 smithi153 bash[23852]: audit 2024-04-06T03:35:55.060171+0000 mon.a (mon.0) 1380 : audit 1 from='client.? 172.21.15.50:0/182096545' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.4", "id": [6, 3, 5, 7, 0, 4]}]': finished 2024-04-06T03:35:56.345 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:56 smithi153 bash[23852]: cluster 2024-04-06T03:35:55.075190+0000 mon.a (mon.0) 1381 : cluster 0 osdmap e250: 8 total, 8 up, 8 in 2024-04-06T03:35:56.345 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:56 smithi153 bash[23852]: cluster 2024-04-06T03:35:55.812556+0000 osd.0 (osd.0) 354 : cluster 0 9.5 scrub ok 2024-04-06T03:35:56.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:56.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 69 16 81378 81362 73.6832 43.1875 0.00377073 0.0135613 2024-04-06T03:35:56.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:56.489 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:56.591 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:56.621 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:56.715 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:35:56.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:56.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:56.838 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:35:56.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:56.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:56.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:56.902 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:35:56.936 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:35:56.977 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:35:56.983 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:57.010 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:35:57.012 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:35:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:35:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:35:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:35:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:35:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:35:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:35:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:35:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:35:57.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:57.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:57.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:57.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:57 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.082630+0000 mon.a (mon.0) 1382 : cluster 0 osdmap e251: 8 total, 8 up, 8 in 2024-04-06T03:35:57.252 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:57 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.221681+0000 osd.0 (osd.0) 355 : cluster 0 9.9 scrub ok 2024-04-06T03:35:57.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:57 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.298844+0000 mgr.y (mgr.24368) 554 : cluster 0 pgmap v580: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 53 MiB/s wr, 845 op/s 2024-04-06T03:35:57.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:57 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.427878+0000 osd.3 (osd.3) 373 : cluster 0 3.7 scrub starts 2024-04-06T03:35:57.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:57 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.483917+0000 osd.3 (osd.3) 374 : cluster 0 3.7 scrub ok 2024-04-06T03:35:57.253 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:57 smithi153 bash[23852]: audit 2024-04-06T03:35:56.700457+0000 mon.c (mon.2) 75 : audit 0 from='client.? 172.21.15.50:0/2528941153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:35:57.270 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:57.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:57.397 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:57.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 70 15 81865 81850 73.0663 30.5 0.0284629 0.0136807 2024-04-06T03:35:57.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.082630+0000 mon.a (mon.0) 1382 : cluster 0 osdmap e251: 8 total, 8 up, 8 in 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.221681+0000 osd.0 (osd.0) 355 : cluster 0 9.9 scrub ok 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.298844+0000 mgr.y (mgr.24368) 554 : cluster 0 pgmap v580: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 53 MiB/s wr, 845 op/s 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.427878+0000 osd.3 (osd.3) 373 : cluster 0 3.7 scrub starts 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.483917+0000 osd.3 (osd.3) 374 : cluster 0 3.7 scrub ok 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[21467]: audit 2024-04-06T03:35:56.700457+0000 mon.c (mon.2) 75 : audit 0 from='client.? 172.21.15.50:0/2528941153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.082630+0000 mon.a (mon.0) 1382 : cluster 0 osdmap e251: 8 total, 8 up, 8 in 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.221681+0000 osd.0 (osd.0) 355 : cluster 0 9.9 scrub ok 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.298844+0000 mgr.y (mgr.24368) 554 : cluster 0 pgmap v580: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 53 MiB/s wr, 845 op/s 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.427878+0000 osd.3 (osd.3) 373 : cluster 0 3.7 scrub starts 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.483917+0000 osd.3 (osd.3) 374 : cluster 0 3.7 scrub ok 2024-04-06T03:35:57.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:57 smithi050 bash[29221]: audit 2024-04-06T03:35:56.700457+0000 mon.c (mon.2) 75 : audit 0 from='client.? 172.21.15.50:0/2528941153' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:35:57.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:57.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:57.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:57.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:57.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:57.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:57.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:57.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:57.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:57.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:58.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:58.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:58.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:58.170 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:58.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.923374+0000 osd.1 (osd.1) 300 : cluster 0 4.2 scrub starts 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[21467]: cluster 2024-04-06T03:35:56.926099+0000 osd.1 (osd.1) 301 : cluster 0 4.2 scrub ok 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[21467]: cluster 2024-04-06T03:35:57.238533+0000 osd.4 (osd.4) 349 : cluster 0 5.c scrub starts 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[21467]: cluster 2024-04-06T03:35:57.241546+0000 osd.4 (osd.4) 350 : cluster 0 5.c scrub ok 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[21467]: cluster 2024-04-06T03:35:57.497851+0000 osd.3 (osd.3) 375 : cluster 0 3.8 scrub starts 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[21467]: cluster 2024-04-06T03:35:57.530333+0000 osd.3 (osd.3) 376 : cluster 0 3.8 scrub ok 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.923374+0000 osd.1 (osd.1) 300 : cluster 0 4.2 scrub starts 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[29221]: cluster 2024-04-06T03:35:56.926099+0000 osd.1 (osd.1) 301 : cluster 0 4.2 scrub ok 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[29221]: cluster 2024-04-06T03:35:57.238533+0000 osd.4 (osd.4) 349 : cluster 0 5.c scrub starts 2024-04-06T03:35:58.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[29221]: cluster 2024-04-06T03:35:57.241546+0000 osd.4 (osd.4) 350 : cluster 0 5.c scrub ok 2024-04-06T03:35:58.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[29221]: cluster 2024-04-06T03:35:57.497851+0000 osd.3 (osd.3) 375 : cluster 0 3.8 scrub starts 2024-04-06T03:35:58.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:58 smithi050 bash[29221]: cluster 2024-04-06T03:35:57.530333+0000 osd.3 (osd.3) 376 : cluster 0 3.8 scrub ok 2024-04-06T03:35:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:58.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:58.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 71 16 82529 82513 72.6208 41.4375 0.0174833 0.0137652 2024-04-06T03:35:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:58.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:58.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:58 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.923374+0000 osd.1 (osd.1) 300 : cluster 0 4.2 scrub starts 2024-04-06T03:35:58.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:58 smithi153 bash[23852]: cluster 2024-04-06T03:35:56.926099+0000 osd.1 (osd.1) 301 : cluster 0 4.2 scrub ok 2024-04-06T03:35:58.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:58 smithi153 bash[23852]: cluster 2024-04-06T03:35:57.238533+0000 osd.4 (osd.4) 349 : cluster 0 5.c scrub starts 2024-04-06T03:35:58.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:58 smithi153 bash[23852]: cluster 2024-04-06T03:35:57.241546+0000 osd.4 (osd.4) 350 : cluster 0 5.c scrub ok 2024-04-06T03:35:58.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:58 smithi153 bash[23852]: cluster 2024-04-06T03:35:57.497851+0000 osd.3 (osd.3) 375 : cluster 0 3.8 scrub starts 2024-04-06T03:35:58.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:58 smithi153 bash[23852]: cluster 2024-04-06T03:35:57.530333+0000 osd.3 (osd.3) 376 : cluster 0 3.8 scrub ok 2024-04-06T03:35:58.554 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:58.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:58.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:35:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:35:58.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:58.819 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:35:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:35:58.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:58.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:58.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:59.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:59.073 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:35:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:35:59.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:59.210 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:59.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:59.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:35:59.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:35:59.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 72 16 83242 83226 72.2311 44.5625 0.0241872 0.0138339 2024-04-06T03:35:59.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:59.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:35:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:35:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:59 smithi153 bash[23852]: cluster 2024-04-06T03:35:57.597773+0000 osd.7 (osd.7) 383 : cluster 0 6.15 scrub starts 2024-04-06T03:35:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:59 smithi153 bash[23852]: cluster 2024-04-06T03:35:57.629349+0000 osd.7 (osd.7) 384 : cluster 0 6.15 scrub ok 2024-04-06T03:35:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:35:59 smithi153 bash[23852]: cluster 2024-04-06T03:35:58.299331+0000 mgr.y (mgr.24368) 555 : cluster 0 pgmap v581: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 663 op/s 2024-04-06T03:35:59.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[21467]: cluster 2024-04-06T03:35:57.597773+0000 osd.7 (osd.7) 383 : cluster 0 6.15 scrub starts 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[21467]: cluster 2024-04-06T03:35:57.629349+0000 osd.7 (osd.7) 384 : cluster 0 6.15 scrub ok 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[21467]: cluster 2024-04-06T03:35:58.299331+0000 mgr.y (mgr.24368) 555 : cluster 0 pgmap v581: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 663 op/s 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:35:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[29221]: cluster 2024-04-06T03:35:57.597773+0000 osd.7 (osd.7) 383 : cluster 0 6.15 scrub starts 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[29221]: cluster 2024-04-06T03:35:57.629349+0000 osd.7 (osd.7) 384 : cluster 0 6.15 scrub ok 2024-04-06T03:35:59.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:35:59 smithi050 bash[29221]: cluster 2024-04-06T03:35:58.299331+0000 mgr.y (mgr.24368) 555 : cluster 0 pgmap v581: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 663 op/s 2024-04-06T03:35:59.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:35:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:35:59.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:59.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:59.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:59.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:35:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:35:59.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:35:59.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:00.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:00.049 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:35:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:00.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:00.154 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:00.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:00 smithi050 bash[21467]: cluster 2024-04-06T03:35:59.495634+0000 osd.3 (osd.3) 377 : cluster 0 9.2 scrub starts 2024-04-06T03:36:00.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:00 smithi050 bash[21467]: audit 2024-04-06T03:35:59.728714+0000 mgr.y (mgr.24368) 556 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:00.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:00 smithi050 bash[29221]: cluster 2024-04-06T03:35:59.495634+0000 osd.3 (osd.3) 377 : cluster 0 9.2 scrub starts 2024-04-06T03:36:00.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:00 smithi050 bash[29221]: audit 2024-04-06T03:35:59.728714+0000 mgr.y (mgr.24368) 556 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:00.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:00.285 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:00.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:00.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 73 16 83932 83916 71.8324 43.125 0.010061 0.0138981 2024-04-06T03:36:00.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:00.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:00.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:00 smithi153 bash[23852]: cluster 2024-04-06T03:35:59.495634+0000 osd.3 (osd.3) 377 : cluster 0 9.2 scrub starts 2024-04-06T03:36:00.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:00 smithi153 bash[23852]: audit 2024-04-06T03:35:59.728714+0000 mgr.y (mgr.24368) 556 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:00.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:00.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:00.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:00.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:00.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:00.933 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:00.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:01.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:01.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:01.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:01.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:01 smithi153 bash[23852]: cluster 2024-04-06T03:36:00.238984+0000 osd.4 (osd.4) 351 : cluster 0 6.17 scrub starts 2024-04-06T03:36:01.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:01 smithi153 bash[23852]: cluster 2024-04-06T03:36:00.299996+0000 mgr.y (mgr.24368) 557 : cluster 0 pgmap v582: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 536 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:36:01.175 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:01 smithi153 bash[23852]: cluster 2024-04-06T03:36:00.330067+0000 osd.4 (osd.4) 352 : cluster 0 6.17 scrub ok 2024-04-06T03:36:01.191 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:01.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:01.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:01.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:01 smithi050 bash[21467]: cluster 2024-04-06T03:36:00.238984+0000 osd.4 (osd.4) 351 : cluster 0 6.17 scrub starts 2024-04-06T03:36:01.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:01 smithi050 bash[21467]: cluster 2024-04-06T03:36:00.299996+0000 mgr.y (mgr.24368) 557 : cluster 0 pgmap v582: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 536 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:36:01.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:01 smithi050 bash[21467]: cluster 2024-04-06T03:36:00.330067+0000 osd.4 (osd.4) 352 : cluster 0 6.17 scrub ok 2024-04-06T03:36:01.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:01 smithi050 bash[29221]: cluster 2024-04-06T03:36:00.238984+0000 osd.4 (osd.4) 351 : cluster 0 6.17 scrub starts 2024-04-06T03:36:01.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:01 smithi050 bash[29221]: cluster 2024-04-06T03:36:00.299996+0000 mgr.y (mgr.24368) 557 : cluster 0 pgmap v582: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 34 MiB/s wr, 536 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:36:01.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:01 smithi050 bash[29221]: cluster 2024-04-06T03:36:00.330067+0000 osd.4 (osd.4) 352 : cluster 0 6.17 scrub ok 2024-04-06T03:36:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:01.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 74 16 84539 84523 71.3743 37.9375 0.0151293 0.0140043 2024-04-06T03:36:01.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:01.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:01.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:01.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:01.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:01.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:01.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:01.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:02.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:02.033 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:02.097 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:02 smithi050 bash[21467]: cluster 2024-04-06T03:36:00.595945+0000 osd.7 (osd.7) 385 : cluster 0 3.11 scrub starts 2024-04-06T03:36:02.097 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:02 smithi050 bash[21467]: cluster 2024-04-06T03:36:00.603804+0000 osd.7 (osd.7) 386 : cluster 0 3.11 scrub ok 2024-04-06T03:36:02.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:02 smithi050 bash[21467]: cluster 2024-04-06T03:36:01.624322+0000 osd.3 (osd.3) 378 : cluster 0 9.1 scrub starts 2024-04-06T03:36:02.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:02.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:02 smithi050 bash[29221]: cluster 2024-04-06T03:36:00.595945+0000 osd.7 (osd.7) 385 : cluster 0 3.11 scrub starts 2024-04-06T03:36:02.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:02 smithi050 bash[29221]: cluster 2024-04-06T03:36:00.603804+0000 osd.7 (osd.7) 386 : cluster 0 3.11 scrub ok 2024-04-06T03:36:02.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:02 smithi050 bash[29221]: cluster 2024-04-06T03:36:01.624322+0000 osd.3 (osd.3) 378 : cluster 0 9.1 scrub starts 2024-04-06T03:36:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:02.178 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:36:02.178 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:36:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:02.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:02.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:02 smithi153 bash[23852]: cluster 2024-04-06T03:36:00.595945+0000 osd.7 (osd.7) 385 : cluster 0 3.11 scrub starts 2024-04-06T03:36:02.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:02 smithi153 bash[23852]: cluster 2024-04-06T03:36:00.603804+0000 osd.7 (osd.7) 386 : cluster 0 3.11 scrub ok 2024-04-06T03:36:02.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:02 smithi153 bash[23852]: cluster 2024-04-06T03:36:01.624322+0000 osd.3 (osd.3) 378 : cluster 0 9.1 scrub starts 2024-04-06T03:36:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:02.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 75 16 85255 85239 71.0193 44.75 0.0273995 0.0140735 2024-04-06T03:36:02.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:02.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:02.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:02.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:02.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:02.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:02.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:02.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:02.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:02.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:02.993 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:02.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:03.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:03.136 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:36:03.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:03.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:03.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:03.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[21467]: cluster 2024-04-06T03:36:01.579299+0000 osd.7 (osd.7) 387 : cluster 0 2.1d scrub starts 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[21467]: cluster 2024-04-06T03:36:01.583146+0000 osd.7 (osd.7) 388 : cluster 0 2.1d scrub ok 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[21467]: cluster 2024-04-06T03:36:02.079130+0000 osd.3 (osd.3) 379 : cluster 0 9.2 scrub ok 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[21467]: cluster 2024-04-06T03:36:02.300599+0000 mgr.y (mgr.24368) 558 : cluster 0 pgmap v583: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 660 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[21467]: audit 2024-04-06T03:36:02.569151+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.50:0/1264805334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[21467]: audit 2024-04-06T03:36:02.570234+0000 mon.a (mon.0) 1383 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[29221]: cluster 2024-04-06T03:36:01.579299+0000 osd.7 (osd.7) 387 : cluster 0 2.1d scrub starts 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[29221]: cluster 2024-04-06T03:36:01.583146+0000 osd.7 (osd.7) 388 : cluster 0 2.1d scrub ok 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[29221]: cluster 2024-04-06T03:36:02.079130+0000 osd.3 (osd.3) 379 : cluster 0 9.2 scrub ok 2024-04-06T03:36:03.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[29221]: cluster 2024-04-06T03:36:02.300599+0000 mgr.y (mgr.24368) 558 : cluster 0 pgmap v583: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 660 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:36:03.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[29221]: audit 2024-04-06T03:36:02.569151+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.50:0/1264805334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:36:03.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:03 smithi050 bash[29221]: audit 2024-04-06T03:36:02.570234+0000 mon.a (mon.0) 1383 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:36:03.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:03.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 76 16 86123 86107 70.7986 54.25 0.0108956 0.0141186 2024-04-06T03:36:03.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:03.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:03 smithi153 bash[23852]: cluster 2024-04-06T03:36:01.579299+0000 osd.7 (osd.7) 387 : cluster 0 2.1d scrub starts 2024-04-06T03:36:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:03 smithi153 bash[23852]: cluster 2024-04-06T03:36:01.583146+0000 osd.7 (osd.7) 388 : cluster 0 2.1d scrub ok 2024-04-06T03:36:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:03 smithi153 bash[23852]: cluster 2024-04-06T03:36:02.079130+0000 osd.3 (osd.3) 379 : cluster 0 9.2 scrub ok 2024-04-06T03:36:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:03 smithi153 bash[23852]: cluster 2024-04-06T03:36:02.300599+0000 mgr.y (mgr.24368) 558 : cluster 0 pgmap v583: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 41 MiB/s wr, 660 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:36:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:03 smithi153 bash[23852]: audit 2024-04-06T03:36:02.569151+0000 mon.c (mon.2) 76 : audit 1 from='client.? 172.21.15.50:0/1264805334' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:36:03.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:03 smithi153 bash[23852]: audit 2024-04-06T03:36:02.570234+0000 mon.a (mon.0) 1383 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:36:03.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:36:03.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:03.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:03.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:03.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:03.947 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:03.971 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:03.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:04.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:04.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:04.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:04.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:04 smithi153 bash[23852]: cluster 2024-04-06T03:36:02.613066+0000 osd.7 (osd.7) 389 : cluster 0 5.9 scrub starts 2024-04-06T03:36:04.213 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:04 smithi153 bash[23852]: cluster 2024-04-06T03:36:02.615090+0000 osd.7 (osd.7) 390 : cluster 0 5.9 scrub ok 2024-04-06T03:36:04.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:04 smithi153 bash[23852]: cluster 2024-04-06T03:36:03.100736+0000 mon.a (mon.0) 1384 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:04.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:04 smithi153 bash[23852]: audit 2024-04-06T03:36:03.112863+0000 mon.a (mon.0) 1385 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:36:04.214 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:04 smithi153 bash[23852]: cluster 2024-04-06T03:36:03.119173+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e252: 8 total, 8 up, 8 in 2024-04-06T03:36:04.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:04.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:04.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:04.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[21467]: cluster 2024-04-06T03:36:02.613066+0000 osd.7 (osd.7) 389 : cluster 0 5.9 scrub starts 2024-04-06T03:36:04.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[21467]: cluster 2024-04-06T03:36:02.615090+0000 osd.7 (osd.7) 390 : cluster 0 5.9 scrub ok 2024-04-06T03:36:04.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[21467]: cluster 2024-04-06T03:36:03.100736+0000 mon.a (mon.0) 1384 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:04.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[21467]: audit 2024-04-06T03:36:03.112863+0000 mon.a (mon.0) 1385 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:36:04.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[21467]: cluster 2024-04-06T03:36:03.119173+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e252: 8 total, 8 up, 8 in 2024-04-06T03:36:04.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[29221]: cluster 2024-04-06T03:36:02.613066+0000 osd.7 (osd.7) 389 : cluster 0 5.9 scrub starts 2024-04-06T03:36:04.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[29221]: cluster 2024-04-06T03:36:02.615090+0000 osd.7 (osd.7) 390 : cluster 0 5.9 scrub ok 2024-04-06T03:36:04.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[29221]: cluster 2024-04-06T03:36:03.100736+0000 mon.a (mon.0) 1384 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:04.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[29221]: audit 2024-04-06T03:36:03.112863+0000 mon.a (mon.0) 1385 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:36:04.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:04 smithi050 bash[29221]: cluster 2024-04-06T03:36:03.119173+0000 mon.a (mon.0) 1386 : cluster 0 osdmap e252: 8 total, 8 up, 8 in 2024-04-06T03:36:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:04.404 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 77 16 87053 87037 70.634 58.125 0.00835929 0.0141515 2024-04-06T03:36:04.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:04.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:04.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:04.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:04.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:04.765 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:04.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:04.898 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:04.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:05.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:05.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:05.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:05.158 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:05.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:05.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:05.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:05.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 78 16 87863 87847 70.3774 50.625 0.0192109 0.014203 2024-04-06T03:36:05.416 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:05.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:05.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:05.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:05.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:05.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:05.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:05 smithi153 bash[23852]: cluster 2024-04-06T03:36:04.304471+0000 mgr.y (mgr.24368) 559 : cluster 0 pgmap v585: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 622 B/s rd, 33 MiB/s wr, 522 op/s; 43 B/s, 1 objects/s recovering 2024-04-06T03:36:05.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:05 smithi153 bash[23852]: audit 2024-04-06T03:36:04.569454+0000 mon.a (mon.0) 1387 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:36:05.794 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:05 smithi153 bash[23852]: audit 2024-04-06T03:36:04.571599+0000 mon.a (mon.0) 1388 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:05.808 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:05.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:05.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:05.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:05.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:05 smithi050 bash[21467]: cluster 2024-04-06T03:36:04.304471+0000 mgr.y (mgr.24368) 559 : cluster 0 pgmap v585: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 622 B/s rd, 33 MiB/s wr, 522 op/s; 43 B/s, 1 objects/s recovering 2024-04-06T03:36:05.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:05 smithi050 bash[21467]: audit 2024-04-06T03:36:04.569454+0000 mon.a (mon.0) 1387 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:36:05.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:05 smithi050 bash[21467]: audit 2024-04-06T03:36:04.571599+0000 mon.a (mon.0) 1388 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:05.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:05 smithi050 bash[29221]: cluster 2024-04-06T03:36:04.304471+0000 mgr.y (mgr.24368) 559 : cluster 0 pgmap v585: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 622 B/s rd, 33 MiB/s wr, 522 op/s; 43 B/s, 1 objects/s recovering 2024-04-06T03:36:05.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:05 smithi050 bash[29221]: audit 2024-04-06T03:36:04.569454+0000 mon.a (mon.0) 1387 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:36:05.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:05 smithi050 bash[29221]: audit 2024-04-06T03:36:04.571599+0000 mon.a (mon.0) 1388 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:06.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:06.062 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:06.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:06.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:06.308 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:06.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:06.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 79 16 88698 88682 70.1471 52.1875 0.0136211 0.0142494 2024-04-06T03:36:06.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:36:06.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 0 1 2024-04-06T03:36:06.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:06.521 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:06.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-06T03:36:06.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:06.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:06.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:06.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:06.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:06.981 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:06.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:07.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:07.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:07.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:07.257 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:07.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:07.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:07.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:36:07.411435+0000 min lat: 0.00108908 max lat: 0.489088 avg lat: 0.0142933 2024-04-06T03:36:07.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:36:07.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 80 16 89532 89516 69.9218 52.125 0.00593608 0.0142933 2024-04-06T03:36:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:07.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:07.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:07.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:07.672 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:07.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:07.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:07 smithi050 bash[21467]: cluster 2024-04-06T03:36:06.305300+0000 mgr.y (mgr.24368) 560 : cluster 0 pgmap v586: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 713 op/s; 35 B/s, 1 objects/s recovering 2024-04-06T03:36:07.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:07 smithi050 bash[29221]: cluster 2024-04-06T03:36:06.305300+0000 mgr.y (mgr.24368) 560 : cluster 0 pgmap v586: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 713 op/s; 35 B/s, 1 objects/s recovering 2024-04-06T03:36:07.803 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:07.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:07.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:07 smithi153 bash[23852]: cluster 2024-04-06T03:36:06.305300+0000 mgr.y (mgr.24368) 560 : cluster 0 pgmap v586: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 713 op/s; 35 B/s, 1 objects/s recovering 2024-04-06T03:36:07.954 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:07.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:08.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:08.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:08.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:08.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:08.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:08.359 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:08.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 81 15 90458 90443 69.7738 57.9375 0.10041 0.0143205 2024-04-06T03:36:08.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:08.484 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:08.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:08.620 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:08.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:08.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:08.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:36:08.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:08.883 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:08.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:08.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:09.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:09.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:09.137 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:09.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:09.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:09.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:09.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:09.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 82 16 91346 91330 69.5989 55.4375 0.0171568 0.0143618 2024-04-06T03:36:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:09.484 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:09.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:09.510 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:36:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:36:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:36:09.528 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:09.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:09.658 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:09.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:09.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:36:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:09.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:09 smithi153 bash[23852]: cluster 2024-04-06T03:36:08.306642+0000 mgr.y (mgr.24368) 561 : cluster 0 pgmap v587: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 713 op/s; 35 B/s, 1 objects/s recovering 2024-04-06T03:36:09.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:09.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:09.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:09 smithi050 bash[21467]: cluster 2024-04-06T03:36:08.306642+0000 mgr.y (mgr.24368) 561 : cluster 0 pgmap v587: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 713 op/s; 35 B/s, 1 objects/s recovering 2024-04-06T03:36:09.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:09 smithi050 bash[29221]: cluster 2024-04-06T03:36:08.306642+0000 mgr.y (mgr.24368) 561 : cluster 0 pgmap v587: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 45 MiB/s wr, 713 op/s; 35 B/s, 1 objects/s recovering 2024-04-06T03:36:10.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:10.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:10.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:10.186 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:10.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:10.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:10.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 83 16 92263 92247 69.4509 57.3125 0.00841281 0.0143908 2024-04-06T03:36:10.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:10.502 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:10.504 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:10.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:10.577 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:10.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:10.703 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:10 smithi153 bash[23852]: audit 2024-04-06T03:36:09.736728+0000 mgr.y (mgr.24368) 562 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:10.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:10.830 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:10.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:10.948 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:10 smithi050 bash[21467]: audit 2024-04-06T03:36:09.736728+0000 mgr.y (mgr.24368) 562 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:10.948 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:10 smithi050 bash[29221]: audit 2024-04-06T03:36:09.736728+0000 mgr.y (mgr.24368) 562 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:10.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:10.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:11.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:11.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:11.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:11.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:11.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:11.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:11.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 84 16 92909 92893 69.1047 40.375 0.00740256 0.0144652 2024-04-06T03:36:11.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:11.494 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:11.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:11.628 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:11.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:11.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:11 smithi050 bash[21467]: cluster 2024-04-06T03:36:10.307455+0000 mgr.y (mgr.24368) 563 : cluster 0 pgmap v588: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 757 op/s 2024-04-06T03:36:11.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:11 smithi050 bash[29221]: cluster 2024-04-06T03:36:10.307455+0000 mgr.y (mgr.24368) 563 : cluster 0 pgmap v588: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 757 op/s 2024-04-06T03:36:11.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:11.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:11 smithi153 bash[23852]: cluster 2024-04-06T03:36:10.307455+0000 mgr.y (mgr.24368) 563 : cluster 0 pgmap v588: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 757 op/s 2024-04-06T03:36:11.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:11.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:11.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:12.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:12.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:12.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:12.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:12.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:12.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:12.402 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:12.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 85 16 93605 93589 68.8035 43.5 0.0175439 0.0145283 2024-04-06T03:36:12.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:12.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:12.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:12.654 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:12.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:12.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:12.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:12.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:12.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:13.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:13.051 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:13.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:13.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:13.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:13.287 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:13.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:13.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 86 16 94360 94344 68.5521 47.1875 0.00515002 0.0145676 2024-04-06T03:36:13.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:13.445 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:13.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:13.567 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:13.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:13.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:13.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:13.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:13.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:13.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:13 smithi153 bash[23852]: cluster 2024-04-06T03:36:12.308224+0000 mgr.y (mgr.24368) 564 : cluster 0 pgmap v589: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 52 MiB/s wr, 834 op/s 2024-04-06T03:36:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:13.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:13.969 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:13.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:13.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:13 smithi050 bash[21467]: cluster 2024-04-06T03:36:12.308224+0000 mgr.y (mgr.24368) 564 : cluster 0 pgmap v589: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 52 MiB/s wr, 834 op/s 2024-04-06T03:36:13.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:13 smithi050 bash[29221]: cluster 2024-04-06T03:36:12.308224+0000 mgr.y (mgr.24368) 564 : cluster 0 pgmap v589: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 52 MiB/s wr, 834 op/s 2024-04-06T03:36:14.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:14.103 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:14.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:14.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:14.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:14.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:14.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:14.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 87 16 95017 95001 68.2361 41.0625 0.0063157 0.0146483 2024-04-06T03:36:14.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:14.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:14.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:14.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:14.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:14.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:14.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:14.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:14.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:15.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:15.026 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:14 smithi153 bash[23852]: cluster 2024-04-06T03:36:14.308934+0000 mgr.y (mgr.24368) 565 : cluster 0 pgmap v590: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 914 B/s rd, 53 MiB/s wr, 849 op/s 2024-04-06T03:36:15.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:15.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:14 smithi050 bash[21467]: cluster 2024-04-06T03:36:14.308934+0000 mgr.y (mgr.24368) 565 : cluster 0 pgmap v590: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 914 B/s rd, 53 MiB/s wr, 849 op/s 2024-04-06T03:36:15.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:14 smithi050 bash[29221]: cluster 2024-04-06T03:36:14.308934+0000 mgr.y (mgr.24368) 565 : cluster 0 pgmap v590: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 914 B/s rd, 53 MiB/s wr, 849 op/s 2024-04-06T03:36:15.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:15.177 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:15.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:15.310 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:15.405 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 88 16 95653 95637 67.9123 39.75 0.00538832 0.0147151 2024-04-06T03:36:15.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:15.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:15.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:15.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:15.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:15.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:15.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:15.812 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:15.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:15.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:15.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:16.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:16.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:16.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:16.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:16.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:16.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 89 16 96320 96304 67.6177 41.6875 0.0105413 0.0147831 2024-04-06T03:36:16.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:16.480 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:16.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.618 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:16.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.747 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:16.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.877 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:16.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:16.997 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:16.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:17.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:17.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:17.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[21467]: cluster 2024-04-06T03:36:16.018228+0000 osd.0 (osd.0) 356 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:17.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[21467]: cluster 2024-04-06T03:36:16.018296+0000 osd.0 (osd.0) 357 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:17.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[21467]: cluster 2024-04-06T03:36:16.075758+0000 mon.a (mon.0) 1389 : cluster 0 osdmap e253: 8 total, 8 up, 8 in 2024-04-06T03:36:17.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[21467]: cluster 2024-04-06T03:36:16.118933+0000 osd.3 (osd.3) 380 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:17.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[21467]: cluster 2024-04-06T03:36:16.119004+0000 osd.3 (osd.3) 381 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[21467]: cluster 2024-04-06T03:36:16.309429+0000 mgr.y (mgr.24368) 566 : cluster 0 pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 49 MiB/s wr, 789 op/s 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[29221]: cluster 2024-04-06T03:36:16.018228+0000 osd.0 (osd.0) 356 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[29221]: cluster 2024-04-06T03:36:16.018296+0000 osd.0 (osd.0) 357 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[29221]: cluster 2024-04-06T03:36:16.075758+0000 mon.a (mon.0) 1389 : cluster 0 osdmap e253: 8 total, 8 up, 8 in 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[29221]: cluster 2024-04-06T03:36:16.118933+0000 osd.3 (osd.3) 380 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[29221]: cluster 2024-04-06T03:36:16.119004+0000 osd.3 (osd.3) 381 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:17.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:17 smithi050 bash[29221]: cluster 2024-04-06T03:36:16.309429+0000 mgr.y (mgr.24368) 566 : cluster 0 pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 49 MiB/s wr, 789 op/s 2024-04-06T03:36:17.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:17.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:17.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:17 smithi153 bash[23852]: cluster 2024-04-06T03:36:16.018228+0000 osd.0 (osd.0) 356 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:17.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:17 smithi153 bash[23852]: cluster 2024-04-06T03:36:16.018296+0000 osd.0 (osd.0) 357 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:17.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:17 smithi153 bash[23852]: cluster 2024-04-06T03:36:16.075758+0000 mon.a (mon.0) 1389 : cluster 0 osdmap e253: 8 total, 8 up, 8 in 2024-04-06T03:36:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:17 smithi153 bash[23852]: cluster 2024-04-06T03:36:16.118933+0000 osd.3 (osd.3) 380 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:17 smithi153 bash[23852]: cluster 2024-04-06T03:36:16.119004+0000 osd.3 (osd.3) 381 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:17 smithi153 bash[23852]: cluster 2024-04-06T03:36:16.309429+0000 mgr.y (mgr.24368) 566 : cluster 0 pgmap v592: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 49 MiB/s wr, 789 op/s 2024-04-06T03:36:17.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:17.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:17.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 90 16 96907 96891 67.274 36.6875 0.0348582 0.0148217 2024-04-06T03:36:17.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:17.512 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:17.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:17.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:17.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:17.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:17.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:17.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:17.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:18.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:18.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:18.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:18 smithi050 bash[21467]: cluster 2024-04-06T03:36:17.081216+0000 mon.a (mon.0) 1390 : cluster 0 osdmap e254: 8 total, 8 up, 8 in 2024-04-06T03:36:18.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:18 smithi050 bash[29221]: cluster 2024-04-06T03:36:17.081216+0000 mon.a (mon.0) 1390 : cluster 0 osdmap e254: 8 total, 8 up, 8 in 2024-04-06T03:36:18.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:18.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:18.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:18 smithi153 bash[23852]: cluster 2024-04-06T03:36:17.081216+0000 mon.a (mon.0) 1390 : cluster 0 osdmap e254: 8 total, 8 up, 8 in 2024-04-06T03:36:18.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:18.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total time run: 90.1561 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total writes made: 96907 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Write size: 65536 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object size: 65536 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Bandwidth (MB/sec): 67.18 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Bandwidth: 29.9274 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max bandwidth (MB/sec): 164.688 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min bandwidth (MB/sec): 28.4375 2024-04-06T03:36:18.406 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average IOPS: 1074 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev IOPS: 478.839 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max IOPS: 2635 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min IOPS: 455 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average Latency(s): 0.0148796 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Latency(s): 0.0282989 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max latency(s): 0.489088 2024-04-06T03:36:18.407 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min latency(s): 0.00108908 2024-04-06T03:36:18.416 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Cleaning up (deleting benchmark objects) 2024-04-06T03:36:18.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:18.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:18.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:18.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:18.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:18.804 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:18.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:18.939 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:18.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:19.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:19.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:19.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:19.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:19 smithi050 bash[21467]: cluster 2024-04-06T03:36:18.309850+0000 mgr.y (mgr.24368) 567 : cluster 0 pgmap v594: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 786 op/s 2024-04-06T03:36:19.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:19 smithi050 bash[29221]: cluster 2024-04-06T03:36:18.309850+0000 mgr.y (mgr.24368) 567 : cluster 0 pgmap v594: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 786 op/s 2024-04-06T03:36:19.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:19.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:19.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:19.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:19.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:19.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:19 smithi153 bash[23852]: cluster 2024-04-06T03:36:18.309850+0000 mgr.y (mgr.24368) 567 : cluster 0 pgmap v594: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 786 op/s 2024-04-06T03:36:19.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:19.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:19.611 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:36:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:36:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:36:19.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:19.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:19.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:19.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:19.900 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:19.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:20.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:20.014 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:36:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:20.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:20.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:20.141 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:20 smithi153 bash[23852]: cluster 2024-04-06T03:36:18.835216+0000 osd.1 (osd.1) 302 : cluster 0 9.0 deep-scrub ok 2024-04-06T03:36:20.141 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:20 smithi153 bash[23852]: audit 2024-04-06T03:36:19.594233+0000 mon.a (mon.0) 1391 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:20.141 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:20 smithi153 bash[23852]: audit 2024-04-06T03:36:19.743654+0000 mgr.y (mgr.24368) 568 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:20.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:20.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:20 smithi050 bash[21467]: cluster 2024-04-06T03:36:18.835216+0000 osd.1 (osd.1) 302 : cluster 0 9.0 deep-scrub ok 2024-04-06T03:36:20.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:20 smithi050 bash[21467]: audit 2024-04-06T03:36:19.594233+0000 mon.a (mon.0) 1391 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:20.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:20 smithi050 bash[21467]: audit 2024-04-06T03:36:19.743654+0000 mgr.y (mgr.24368) 568 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:20.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:20 smithi050 bash[29221]: cluster 2024-04-06T03:36:18.835216+0000 osd.1 (osd.1) 302 : cluster 0 9.0 deep-scrub ok 2024-04-06T03:36:20.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:20 smithi050 bash[29221]: audit 2024-04-06T03:36:19.594233+0000 mon.a (mon.0) 1391 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:20.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:20 smithi050 bash[29221]: audit 2024-04-06T03:36:19.743654+0000 mgr.y (mgr.24368) 568 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:20.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:20.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:20.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:20.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:20.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:20.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:20.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:20.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:20.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:20.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:20.933 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:20.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:21.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:21.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:21.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:21.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:21.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:21 smithi050 bash[21467]: cluster 2024-04-06T03:36:20.310936+0000 mgr.y (mgr.24368) 569 : cluster 0 pgmap v595: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 571 op/s 2024-04-06T03:36:21.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:21 smithi050 bash[29221]: cluster 2024-04-06T03:36:20.310936+0000 mgr.y (mgr.24368) 569 : cluster 0 pgmap v595: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 571 op/s 2024-04-06T03:36:21.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:21 smithi153 bash[23852]: cluster 2024-04-06T03:36:20.310936+0000 mgr.y (mgr.24368) 569 : cluster 0 pgmap v595: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 571 op/s 2024-04-06T03:36:21.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:21.330 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:21.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:21.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:21.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:21.598 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:21.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:21.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:21.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:21.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:21.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:21.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:22.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:22.124 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:22.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:22.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:22.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:22.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:22.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:22.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:22.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:22.782 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:22.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:22.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:22.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:23.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:23.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:23.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:23.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:23.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:23.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:23.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:23.302 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:23.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:23.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:23.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:23.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:23 smithi153 bash[23852]: cluster 2024-04-06T03:36:22.311708+0000 mgr.y (mgr.24368) 570 : cluster 0 pgmap v596: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 650 op/s 2024-04-06T03:36:23.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:23.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:23 smithi050 bash[21467]: cluster 2024-04-06T03:36:22.311708+0000 mgr.y (mgr.24368) 570 : cluster 0 pgmap v596: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 650 op/s 2024-04-06T03:36:23.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:23 smithi050 bash[29221]: cluster 2024-04-06T03:36:22.311708+0000 mgr.y (mgr.24368) 570 : cluster 0 pgmap v596: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 29 MiB/s wr, 650 op/s 2024-04-06T03:36:23.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:23.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:23.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:23.838 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:23.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:23.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:23.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:24.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:24.094 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:24.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:24.221 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:24.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:24.350 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:24.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:24.491 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:24.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:24.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:24.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:24.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:24.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:24.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:24.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:24.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:25.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:25.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:25.141 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:25.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:25.276 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:25.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:25.394 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:25 smithi153 bash[23852]: cluster 2024-04-06T03:36:24.312723+0000 mgr.y (mgr.24368) 571 : cluster 0 pgmap v597: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 MiB/s wr, 605 op/s 2024-04-06T03:36:25.417 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:25.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:25.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:25.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:25.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:25.696 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:25 smithi050 bash[21467]: cluster 2024-04-06T03:36:24.312723+0000 mgr.y (mgr.24368) 571 : cluster 0 pgmap v597: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 MiB/s wr, 605 op/s 2024-04-06T03:36:25.696 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:25 smithi050 bash[29221]: cluster 2024-04-06T03:36:24.312723+0000 mgr.y (mgr.24368) 571 : cluster 0 pgmap v597: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 MiB/s wr, 605 op/s 2024-04-06T03:36:25.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:25.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:25.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:25.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:25.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:26.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:26.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:26.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:26.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:26.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:26.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:26.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:26.477 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:26.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:26.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:26.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:26.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:26.802 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:36:26.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:26.854 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:36:26.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:26.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:26.906 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:36:26.948 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:36:26.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:26.988 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:36:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:27.026 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:36:27.062 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:36:27.105 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:36:27.114 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:36:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:36:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:36:27.114 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:36:27.115 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:36:27.115 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:36:27.115 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:36:27.115 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:36:27.139 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:27.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:27.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:27.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:27.395 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:27 smithi153 bash[23852]: cluster 2024-04-06T03:36:26.313480+0000 mgr.y (mgr.24368) 572 : cluster 0 pgmap v598: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.9 MiB/s wr, 721 op/s 2024-04-06T03:36:27.395 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:27 smithi153 bash[23852]: audit 2024-04-06T03:36:26.741608+0000 mon.a (mon.0) 1392 : audit 0 from='client.? 172.21.15.50:0/1987710688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:36:27.419 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:27.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:27 smithi050 bash[21467]: cluster 2024-04-06T03:36:26.313480+0000 mgr.y (mgr.24368) 572 : cluster 0 pgmap v598: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.9 MiB/s wr, 721 op/s 2024-04-06T03:36:27.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:27 smithi050 bash[21467]: audit 2024-04-06T03:36:26.741608+0000 mon.a (mon.0) 1392 : audit 0 from='client.? 172.21.15.50:0/1987710688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:36:27.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:27 smithi050 bash[29221]: cluster 2024-04-06T03:36:26.313480+0000 mgr.y (mgr.24368) 572 : cluster 0 pgmap v598: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 7.9 MiB/s wr, 721 op/s 2024-04-06T03:36:27.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:27 smithi050 bash[29221]: audit 2024-04-06T03:36:26.741608+0000 mon.a (mon.0) 1392 : audit 0 from='client.? 172.21.15.50:0/1987710688' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:36:27.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:27.550 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:27.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:27.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:27.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:27.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:27.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:27.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:28.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:28.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:28.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:28.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:28.332 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:28.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:28.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:28.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:28.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:28.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:28.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:28.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:28.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:28.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:28.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:28.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:29.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:29.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:29.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:29.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:29.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:29.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:29.388 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:36:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:36:29.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:29.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:29 smithi153 bash[23852]: cluster 2024-04-06T03:36:28.314193+0000 mgr.y (mgr.24368) 573 : cluster 0 pgmap v599: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1002 B/s rd, 7.1 MiB/s wr, 642 op/s 2024-04-06T03:36:29.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:29 smithi153 bash[23852]: cluster 2024-04-06T03:36:28.995212+0000 osd.1 (osd.1) 303 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:29.483 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:29 smithi153 bash[23852]: cluster 2024-04-06T03:36:28.995302+0000 osd.1 (osd.1) 304 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:29.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:29.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:29.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[21467]: cluster 2024-04-06T03:36:28.314193+0000 mgr.y (mgr.24368) 573 : cluster 0 pgmap v599: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1002 B/s rd, 7.1 MiB/s wr, 642 op/s 2024-04-06T03:36:29.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[21467]: cluster 2024-04-06T03:36:28.995212+0000 osd.1 (osd.1) 303 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:29.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[21467]: cluster 2024-04-06T03:36:28.995302+0000 osd.1 (osd.1) 304 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:29.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[29221]: cluster 2024-04-06T03:36:28.314193+0000 mgr.y (mgr.24368) 573 : cluster 0 pgmap v599: 148 pgs: 148 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1002 B/s rd, 7.1 MiB/s wr, 642 op/s 2024-04-06T03:36:29.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[29221]: cluster 2024-04-06T03:36:28.995212+0000 osd.1 (osd.1) 303 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:29.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:29 smithi050 bash[29221]: cluster 2024-04-06T03:36:28.995302+0000 osd.1 (osd.1) 304 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:29.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:29.766 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:29.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:29.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:29.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:30.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:30.016 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:36:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:30.032 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:30.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:30.158 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:30.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:30.287 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14745.0:163127 7.a 7:54d41af7:::benchmark_data_smithi050_68188_object66377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:28:27.826099+0000", 2024-04-06T03:36:30.357 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 482.53301341500003, 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49589186400000002, 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14745", 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1699098811", 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 163127 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.826099+0000", 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.826099+0000", 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.826101+0000", 2024-04-06T03:36:30.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6729999999999999e-06 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:27.826107+0000", 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5880000000000003e-06 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.266085+0000", 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43997776500000002 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.266110+0000", 2024-04-06T03:36:30.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5001999999999999e-05 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.273677+0000", 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075663249999999996 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.273887+0000", 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000210807 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.274109+0000", 2024-04-06T03:36:30.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000221287 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.291896+0000", 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017787605000000001 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.291978+0000", 2024-04-06T03:36:30.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999994e-05 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.321803+0000", 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029825199 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.321911+0000", 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010814 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:28:28.321991+0000", 2024-04-06T03:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9772999999999997e-05 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5601 8.f 8:fe631fbd:::benchmark_data_smithi050_86143_object5600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.646484+0000", 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 388.712628682, 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55694482300000003, 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.364 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5601 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646484+0000", 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646484+0000", 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646485+0000", 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9400000000000004e-07 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646507+0000", 2024-04-06T03:36:30.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2609999999999999e-05 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646516+0000", 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9509999999999995e-06 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646524+0000", 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8660000000000006e-06 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646621+0000", 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6482000000000002e-05 2024-04-06T03:36:30.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646735+0000", 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011393299999999999 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.646877+0000", 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014237599999999999 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.649778+0000", 2024-04-06T03:36:30.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029013670000000002 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.649792+0000", 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4211e-05 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203309+0000", 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55351640099999999 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203370+0000", 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1585000000000001e-05 2024-04-06T03:36:30.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203429+0000", 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8247e-05 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.374 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:5603 8.f 8:fa85aff8:::benchmark_data_smithi050_86143_object5602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-04-06T03:36:30.374 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:01.647076+0000", 2024-04-06T03:36:30.374 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 388.71203615000002, 2024-04-06T03:36:30.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.557147422, 2024-04-06T03:36:30.374 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.375 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.375 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:36:30.375 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:36:30.375 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5603 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647076+0000", 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647076+0000", 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647077+0000", 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3200000000000003e-07 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647100+0000", 2024-04-06T03:36:30.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2850999999999999e-05 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647113+0000", 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.259e-05 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.647121+0000", 2024-04-06T03:36:30.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1610000000000003e-06 2024-04-06T03:36:30.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648862+0000", 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017413870000000001 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.648957+0000", 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4593999999999996e-05 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.649091+0000", 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013390900000000001 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.650274+0000", 2024-04-06T03:36:30.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001182916 2024-04-06T03:36:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:01.650381+0000", 2024-04-06T03:36:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000106934 2024-04-06T03:36:30.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.203900+0000", 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55351910500000001 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.204162+0000", 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026212499999999999 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:02.204224+0000", 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1917999999999999e-05 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.385 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160887 8.f 8:f1bb2a93:::benchmark_data_smithi050_86143_object54576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.766383+0000", 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.59272910999999, 2024-04-06T03:36:30.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49886187300000001, 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:36:30.387 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160887 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766383+0000", 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766383+0000", 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766385+0000", 2024-04-06T03:36:30.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.658e-06 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766393+0000", 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7319999999999998e-06 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766416+0000", 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2775000000000001e-05 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766426+0000", 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0634e-05 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766492+0000", 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5969999999999993e-05 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766642+0000", 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149302 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766753+0000", 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000111517 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.773666+0000", 2024-04-06T03:36:30.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069130800000000003 2024-04-06T03:36:30.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.774956+0000", 2024-04-06T03:36:30.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001290254 2024-04-06T03:36:30.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.264960+0000", 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49000312899999998 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.265117+0000", 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015756200000000001 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.265245+0000", 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012826 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160902 8.f 8:facf42e0:::benchmark_data_smithi050_86143_object54591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.780359+0000", 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 235.57875378200001, 2024-04-06T03:36:30.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49317639699999999, 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160902 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780359+0000", 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780359+0000", 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780360+0000", 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.071e-06 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780365+0000", 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3190000000000002e-06 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780380+0000", 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.454e-05 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780389+0000", 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6059999999999992e-06 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780469+0000", 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9939999999999997e-05 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780610+0000", 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014072600000000001 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.780752+0000", 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000142037 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.787352+0000", 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065998369999999999 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.787606+0000", 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025455699999999998 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.273049+0000", 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48544235000000002 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.273187+0000", 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013863699999999999 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.273535+0000", 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034777700000000001 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:79866 9.8 9:1370e7dc:::benchmark_data_smithi050_105098_object79865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.402 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:53.800783+0000", 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 36.558329833000002, 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.211922056, 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 79866 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.403 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800783+0000", 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800783+0000", 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800784+0000", 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.305e-06 2024-04-06T03:36:30.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800805+0000", 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1194000000000001e-05 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800818+0000", 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800825+0000", 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1659999999999997e-06 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.800894+0000", 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9561999999999996e-05 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.801026+0000", 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013195700000000001 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.801154+0000", 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000127844 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.804588+0000", 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034339349999999999 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:53.804601+0000", 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2601e-05 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.012576+0000", 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.20797520899999999 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.012649+0000", 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3193000000000002e-05 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.012705+0000", 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5680000000000002e-05 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15135.0:80329 9.f e249/231 9:fe7d68e0:::benchmark_data_smithi050_105098_object80328:head v 249'5065, mlcod=249'5063)", 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.548479+0000", 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.810633203000002, 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34514785199999998, 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548479+0000", 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548479+0000", 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548480+0000", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.009e-06 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548505+0000", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4939e-05 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548522+0000", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6300999999999999e-05 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548532+0000", 2024-04-06T03:36:30.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0127e-05 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548892+0000", 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036074499999999999 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548916+0000", 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3139e-05 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.893499+0000", 2024-04-06T03:36:30.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34458311400000002 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.893627+0000", 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012847800000000001 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15135.0:80326 9.7 e249/231 9:ef4ce1e0:::benchmark_data_smithi050_105098_object80325:head v 249'4963, mlcod=249'4959)", 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.548576+0000", 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.810536759999998, 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34855613600000002, 2024-04-06T03:36:30.413 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548576+0000", 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548576+0000", 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548576+0000", 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548612+0000", 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5567e-05 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548621+0000", 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2129999999999997e-06 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548627+0000", 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.268e-06 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548972+0000", 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034509900000000002 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.548991+0000", 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8634000000000002e-05 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.897088+0000", 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.348097039 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.897132+0000", 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3853000000000002e-05 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:80325 9.1 9:8b2e76f3:::benchmark_data_smithi050_105098_object80324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.549562+0000", 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.809550408, 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35318060600000001, 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 80325 2024-04-06T03:36:30.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549562+0000", 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549562+0000", 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549564+0000", 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8050000000000001e-06 2024-04-06T03:36:30.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549593+0000", 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9230999999999998e-05 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549608+0000", 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4674999999999999e-05 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549617+0000", 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1339999999999993e-06 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549686+0000", 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8989e-05 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549836+0000", 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014951799999999999 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-04-06T03:36:30.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.549984+0000", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148088 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.553551+0000", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035678200000000002 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.555534+0000", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001982106 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.902474+0000", 2024-04-06T03:36:30.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34694028500000001 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.902629+0000", 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015483700000000001 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.902743+0000", 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011411799999999999 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:80333 9.2 9:4a71fd87:::benchmark_data_smithi050_105098_object80332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.551082+0000", 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.808030873, 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.357937002, 2024-04-06T03:36:30.425 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 80333 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.551082+0000", 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.551082+0000", 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.551083+0000", 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.051e-06 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.551115+0000", 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2591e-05 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.551125+0000", 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0117e-05 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.551134+0000", 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3699999999999995e-06 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.553625+0000", 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024911759999999999 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.553763+0000", 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138341 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.553911+0000", 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014805600000000001 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.898328+0000", 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34441643500000002 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.901914+0000", 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035866130000000002 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.908516+0000", 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066018450000000003 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.908972+0000", 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045575100000000001 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.909019+0000", 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6656000000000002e-05 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15135.0:80334 9.4 e249/231 9:2124837b:::benchmark_data_smithi050_105098_object80333:head v 249'5107, mlcod=249'5105)", 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.552681+0000", 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.806432041000001, 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34782971499999998, 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.552681+0000", 2024-04-06T03:36:30.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.552681+0000", 2024-04-06T03:36:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.552682+0000", 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1039999999999999e-06 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.552708+0000", 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6590000000000001e-05 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.552721+0000", 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2425e-05 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.552729+0000", 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1750000000000005e-06 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558873+0000", 2024-04-06T03:36:30.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061446060000000004 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558888+0000", 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4800000000000001e-05 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.900470+0000", 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34158148399999999 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.900510+0000", 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0531000000000002e-05 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:80337 9.2 9:4bffc471:::benchmark_data_smithi050_105098_object80336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.558112+0000", 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.801000901999998, 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.36026338000000002, 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.437 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 80337 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558112+0000", 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558112+0000", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558113+0000", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3689999999999999e-06 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558216+0000", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102608 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558229+0000", 2024-04-06T03:36:30.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.293e-05 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.558237+0000", 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.354e-06 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559930+0000", 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016932799999999999 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.893565+0000", 2024-04-06T03:36:30.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33363431199999999 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.893894+0000", 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032983699999999998 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.909032+0000", 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015137588 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.909079+0000", 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7426000000000003e-05 2024-04-06T03:36:30.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.918177+0000", 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0090973159999999994 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.918317+0000", 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013985000000000001 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.918375+0000", 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8510000000000001e-05 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15135.0:80333 9.2 e249/231 ondisk, result = 0)", 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.559483+0000", 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.799629340000003, 2024-04-06T03:36:30.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33885410300000002, 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559483+0000", 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559483+0000", 2024-04-06T03:36:30.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559484+0000", 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.23e-07 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559488+0000", 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2289999999999997e-06 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559500+0000", 2024-04-06T03:36:30.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1345999999999999e-05 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559507+0000", 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5850000000000002e-06 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.898313+0000", 2024-04-06T03:36:30.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33880539599999998 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.898321+0000", 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4200000000000007e-06 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.898337+0000", 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6303999999999998e-05 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:80340 9.1 9:8dd1db7c:::benchmark_data_smithi050_105098_object80339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.559602+0000", 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.799510302999998, 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.35127756100000002, 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 80340 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.448 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559602+0000", 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559602+0000", 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559603+0000", 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0330000000000001e-06 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559623+0000", 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9981000000000001e-05 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559634+0000", 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1134999999999999e-05 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559642+0000", 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4229999999999999e-06 2024-04-06T03:36:30.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559758+0000", 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011592800000000001 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559901+0000", 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014336899999999999 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 6,7", 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.894917+0000", 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33501539699999999 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.902782+0000", 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0078649779999999999 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.903542+0000", 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076079700000000001 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.910764+0000", 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0072212919999999998 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.910835+0000", 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1682999999999998e-05 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.910880+0000", 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4545000000000001e-05 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:80341 9.8 9:15413756:::benchmark_data_smithi050_105098_object80340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.559662+0000", 2024-04-06T03:36:30.453 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.799451046000001, 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34963111699999999, 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 80341 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559662+0000", 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559662+0000", 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559662+0000", 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6499999999999999e-07 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559692+0000", 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9655999999999999e-05 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559700+0000", 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1019999999999997e-06 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.559706+0000", 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0209999999999996e-06 2024-04-06T03:36:30.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.895103+0000", 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33539756300000001 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.895268+0000", 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016435700000000001 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.895440+0000", 2024-04-06T03:36:30.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017279599999999999 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.907010+0000", 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011569105999999999 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.907055+0000", 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5558e-05 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.908493+0000", 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001437432 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.909248+0000", 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075560500000000001 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.909293+0000", 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4456000000000003e-05 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15135.0:80339 9.3 e249/231 9:c46e5e02:::benchmark_data_smithi050_105098_object80338:head v 249'5117, mlcod=249'5115)", 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.560062+0000", 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.799050778999998, 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.34594660900000002, 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.560062+0000", 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.560062+0000", 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.560063+0000", 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.06e-07 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.560126+0000", 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3348000000000004e-05 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.894648+0000", 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33452184600000001 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.894673+0000", 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5406e-05 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.898797+0000", 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041237929999999997 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.898820+0000", 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3220000000000001e-05 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.905973+0000", 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071531980000000004 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.906008+0000", 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5092000000000002e-05 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:80351 9.c 9:3b4f2911:::benchmark_data_smithi050_105098_object80350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:54.573298+0000", 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.785814248000001, 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.33751803400000002, 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 80351 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.464 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.573298+0000", 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.573298+0000", 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.573300+0000", 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.593e-06 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.573323+0000", 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.334e-05 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.894637+0000", 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.32131386299999998 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.894652+0000", 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4752e-05 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.896029+0000", 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013767829999999999 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.896345+0000", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000316132 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,7", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.896646+0000", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000301172 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.902867+0000", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062205669999999998 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.908930+0000", 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060629719999999998 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.910684+0000", 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017545970000000001 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.910760+0000", 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5912000000000001e-05 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:54.910816+0000", 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6351000000000003e-05 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:81499 9.2 9:4f2d6e5f:::benchmark_data_smithi050_105098_object81498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:56.626984+0000", 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.732128193999998, 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.24909583699999999, 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.470 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 81499 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.626984+0000", 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.626984+0000", 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.626985+0000", 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1600000000000001e-07 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.627020+0000", 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5206999999999997e-05 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.627028+0000", 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6310000000000003e-06 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.627033+0000", 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4600000000000002e-06 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.627085+0000", 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2178000000000001e-05 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.627209+0000", 2024-04-06T03:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012322400000000001 2024-04-06T03:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.627348+0000", 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000139042 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.630227+0000", 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002879483 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.630494+0000", 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026693499999999999 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.875984+0000", 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.245489819 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.876039+0000", 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5154000000000001e-05 2024-04-06T03:36:30.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.876080+0000", 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1087999999999997e-05 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:81503 9.2 9:4f86108f:::benchmark_data_smithi050_105098_object81502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:56.632679+0000", 2024-04-06T03:36:30.477 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.726434021000003, 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.24351648200000001, 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 81503 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.632679+0000", 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.632679+0000", 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.632680+0000", 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1540000000000001e-06 2024-04-06T03:36:30.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.632704+0000", 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3896e-05 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.632716+0000", 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2765999999999999e-05 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.632725+0000", 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6090000000000001e-06 2024-04-06T03:36:30.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.633041+0000", 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031607500000000003 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.633164+0000", 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123368 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.633301+0000", 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013622299999999999 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.636127+0000", 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028261789999999998 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.636140+0000", 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.314e-05 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.876117+0000", 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.23997674699999999 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.876161+0000", 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4431999999999997e-05 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.876195+0000", 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3893000000000002e-05 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.483 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:81511 9.2 9:463e6c47:::benchmark_data_smithi050_105098_object81510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:56.643145+0000", 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.715967562000003, 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.236005621, 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 81511 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:36:30.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643145+0000", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643145+0000", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643146+0000", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.243e-06 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:36:30.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643171+0000", 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4873000000000001e-05 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643180+0000", 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0319999999999992e-06 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643186+0000", 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6679999999999998e-06 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643232+0000", 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5791999999999998e-05 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643318+0000", 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6539000000000006e-05 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,2", 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.643400+0000", 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2127999999999997e-05 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.645880+0000", 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024798929999999999 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.823856+0000", 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.17797613600000001 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:36:30.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.879046+0000", 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055189990000000001 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.879105+0000", 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9061000000000003e-05 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.879151+0000", 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5266000000000003e-05 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:36:30.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:36:30.490 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:36:30.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:36:30.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:30.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:30 smithi153 bash[23852]: cluster 2024-04-06T03:36:29.387560+0000 mon.a (mon.0) 1393 : cluster 0 osdmap e255: 8 total, 8 up, 8 in 2024-04-06T03:36:30.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:30 smithi153 bash[23852]: audit 2024-04-06T03:36:29.752796+0000 mgr.y (mgr.24368) 574 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:30.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:30 smithi153 bash[23852]: audit 2024-04-06T03:36:30.317397+0000 mon.a (mon.0) 1394 : audit 1 from='client.? 172.21.15.50:0/3877178286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:36:30.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:30.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:30 smithi050 bash[21467]: cluster 2024-04-06T03:36:29.387560+0000 mon.a (mon.0) 1393 : cluster 0 osdmap e255: 8 total, 8 up, 8 in 2024-04-06T03:36:30.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:30 smithi050 bash[21467]: audit 2024-04-06T03:36:29.752796+0000 mgr.y (mgr.24368) 574 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:30.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:30 smithi050 bash[21467]: audit 2024-04-06T03:36:30.317397+0000 mon.a (mon.0) 1394 : audit 1 from='client.? 172.21.15.50:0/3877178286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:36:30.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:30 smithi050 bash[29221]: cluster 2024-04-06T03:36:29.387560+0000 mon.a (mon.0) 1393 : cluster 0 osdmap e255: 8 total, 8 up, 8 in 2024-04-06T03:36:30.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:30 smithi050 bash[29221]: audit 2024-04-06T03:36:29.752796+0000 mgr.y (mgr.24368) 574 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:30.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:30 smithi050 bash[29221]: audit 2024-04-06T03:36:30.317397+0000 mon.a (mon.0) 1394 : audit 1 from='client.? 172.21.15.50:0/3877178286' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:36:30.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:30.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:30.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:30.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:30.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:30.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:30.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:31.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:31.056 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:31.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:31.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:31.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:31.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:31.421 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-06T03:36:31.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[21467]: cluster 2024-04-06T03:36:30.315359+0000 mgr.y (mgr.24368) 575 : cluster 0 pgmap v601: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 3.7 MiB/s wr, 794 op/s 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[21467]: cluster 2024-04-06T03:36:30.394509+0000 mon.a (mon.0) 1395 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[21467]: audit 2024-04-06T03:36:30.407659+0000 mon.a (mon.0) 1396 : audit 1 from='client.? 172.21.15.50:0/3877178286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[21467]: cluster 2024-04-06T03:36:30.414028+0000 mon.a (mon.0) 1397 : cluster 0 osdmap e256: 8 total, 8 up, 8 in 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[21467]: audit 2024-04-06T03:36:30.715615+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.50:0/2714708360' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[21467]: audit 2024-04-06T03:36:30.716051+0000 mon.a (mon.0) 1398 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:36:31.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[29221]: cluster 2024-04-06T03:36:30.315359+0000 mgr.y (mgr.24368) 575 : cluster 0 pgmap v601: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 3.7 MiB/s wr, 794 op/s 2024-04-06T03:36:31.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[29221]: cluster 2024-04-06T03:36:30.394509+0000 mon.a (mon.0) 1395 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:31.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[29221]: audit 2024-04-06T03:36:30.407659+0000 mon.a (mon.0) 1396 : audit 1 from='client.? 172.21.15.50:0/3877178286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:36:31.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[29221]: cluster 2024-04-06T03:36:30.414028+0000 mon.a (mon.0) 1397 : cluster 0 osdmap e256: 8 total, 8 up, 8 in 2024-04-06T03:36:31.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[29221]: audit 2024-04-06T03:36:30.715615+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.50:0/2714708360' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:36:31.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:31 smithi050 bash[29221]: audit 2024-04-06T03:36:30.716051+0000 mon.a (mon.0) 1398 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:36:31.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:31.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:31.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:31.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:31.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:31.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:31 smithi153 bash[23852]: cluster 2024-04-06T03:36:30.315359+0000 mgr.y (mgr.24368) 575 : cluster 0 pgmap v601: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 3.7 MiB/s wr, 794 op/s 2024-04-06T03:36:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:31 smithi153 bash[23852]: cluster 2024-04-06T03:36:30.394509+0000 mon.a (mon.0) 1395 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:31 smithi153 bash[23852]: audit 2024-04-06T03:36:30.407659+0000 mon.a (mon.0) 1396 : audit 1 from='client.? 172.21.15.50:0/3877178286' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:36:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:31 smithi153 bash[23852]: cluster 2024-04-06T03:36:30.414028+0000 mon.a (mon.0) 1397 : cluster 0 osdmap e256: 8 total, 8 up, 8 in 2024-04-06T03:36:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:31 smithi153 bash[23852]: audit 2024-04-06T03:36:30.715615+0000 mon.c (mon.2) 77 : audit 1 from='client.? 172.21.15.50:0/2714708360' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:36:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:31 smithi153 bash[23852]: audit 2024-04-06T03:36:30.716051+0000 mon.a (mon.0) 1398 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:36:31.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:31.863 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:31.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:31.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:31.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:32.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:36:32.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:32.121 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:32.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:32.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:32.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:32.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:32.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:32.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:32.509 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:32.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:32.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:32.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:32 smithi050 bash[29221]: audit 2024-04-06T03:36:31.411193+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:36:32.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:32 smithi050 bash[29221]: cluster 2024-04-06T03:36:31.415411+0000 mon.a (mon.0) 1400 : cluster 0 osdmap e257: 8 total, 8 up, 8 in 2024-04-06T03:36:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:32 smithi050 bash[21467]: audit 2024-04-06T03:36:31.411193+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:36:32.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:32 smithi050 bash[21467]: cluster 2024-04-06T03:36:31.415411+0000 mon.a (mon.0) 1400 : cluster 0 osdmap e257: 8 total, 8 up, 8 in 2024-04-06T03:36:32.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:32.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:32 smithi153 bash[23852]: audit 2024-04-06T03:36:31.411193+0000 mon.a (mon.0) 1399 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:36:32.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:32 smithi153 bash[23852]: cluster 2024-04-06T03:36:31.415411+0000 mon.a (mon.0) 1400 : cluster 0 osdmap e257: 8 total, 8 up, 8 in 2024-04-06T03:36:32.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:32.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:32.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:32.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:33.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:33.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:33.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:33.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:33.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:33.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:33.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:33.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:33.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:33 smithi050 bash[21467]: cluster 2024-04-06T03:36:32.315991+0000 mgr.y (mgr.24368) 576 : cluster 0 pgmap v604: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 797 op/s 2024-04-06T03:36:33.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:33 smithi050 bash[29221]: cluster 2024-04-06T03:36:32.315991+0000 mgr.y (mgr.24368) 576 : cluster 0 pgmap v604: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 797 op/s 2024-04-06T03:36:33.562 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:33.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:33.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:33.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:33 smithi153 bash[23852]: cluster 2024-04-06T03:36:32.315991+0000 mgr.y (mgr.24368) 576 : cluster 0 pgmap v604: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 797 op/s 2024-04-06T03:36:33.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:33.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:33.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:33.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:33.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:34.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:34.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:34.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:34.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:34.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:34.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:34.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-06T03:36:34.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:34.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:34.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 1 1 2024-04-06T03:36:34.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:34.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:34.782 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:34.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:34.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[21467]: cluster 2024-04-06T03:36:33.879875+0000 osd.5 (osd.5) 137 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[21467]: cluster 2024-04-06T03:36:33.879963+0000 osd.5 (osd.5) 138 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[21467]: cluster 2024-04-06T03:36:34.316669+0000 mgr.y (mgr.24368) 577 : cluster 0 pgmap v605: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[21467]: audit 2024-04-06T03:36:34.631497+0000 mon.a (mon.0) 1401 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[29221]: cluster 2024-04-06T03:36:33.879875+0000 osd.5 (osd.5) 137 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[29221]: cluster 2024-04-06T03:36:33.879963+0000 osd.5 (osd.5) 138 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[29221]: cluster 2024-04-06T03:36:34.316669+0000 mgr.y (mgr.24368) 577 : cluster 0 pgmap v605: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-06T03:36:34.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:34 smithi050 bash[29221]: audit 2024-04-06T03:36:34.631497+0000 mon.a (mon.0) 1401 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:34.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:34.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:35.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:34 smithi153 bash[23852]: cluster 2024-04-06T03:36:33.879875+0000 osd.5 (osd.5) 137 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:34 smithi153 bash[23852]: cluster 2024-04-06T03:36:33.879963+0000 osd.5 (osd.5) 138 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:34 smithi153 bash[23852]: cluster 2024-04-06T03:36:34.316669+0000 mgr.y (mgr.24368) 577 : cluster 0 pgmap v605: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-06T03:36:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:34 smithi153 bash[23852]: audit 2024-04-06T03:36:34.631497+0000 mon.a (mon.0) 1401 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:35.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:35.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:35.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:35.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:35.334 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:35.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:35.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:35.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:35.598 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:35.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:35.737 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:35.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:35.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:35.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:35.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:35.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:35 smithi050 bash[21467]: cluster 2024-04-06T03:36:34.863967+0000 mon.a (mon.0) 1402 : cluster 0 osdmap e258: 8 total, 8 up, 8 in 2024-04-06T03:36:35.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:35 smithi050 bash[29221]: cluster 2024-04-06T03:36:34.863967+0000 mon.a (mon.0) 1402 : cluster 0 osdmap e258: 8 total, 8 up, 8 in 2024-04-06T03:36:35.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:36.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.139 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:36.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:36.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:35 smithi153 bash[23852]: cluster 2024-04-06T03:36:34.863967+0000 mon.a (mon.0) 1402 : cluster 0 osdmap e258: 8 total, 8 up, 8 in 2024-04-06T03:36:36.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:36.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:36.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:36.391 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:36.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:36:36.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:36.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:36.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:36.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:36.805 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:36.814 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:36.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.928 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:36 smithi050 bash[21467]: cluster 2024-04-06T03:36:36.317238+0000 mgr.y (mgr.24368) 578 : cluster 0 pgmap v607: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-04-06T03:36:36.929 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:36 smithi050 bash[29221]: cluster 2024-04-06T03:36:36.317238+0000 mgr.y (mgr.24368) 578 : cluster 0 pgmap v607: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-04-06T03:36:36.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:36.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:37.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:37.056 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:36 smithi153 bash[23852]: cluster 2024-04-06T03:36:36.317238+0000 mgr.y (mgr.24368) 578 : cluster 0 pgmap v607: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-04-06T03:36:37.071 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:37.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:37.217 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:37.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:37.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:37.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:37.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:37.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:37.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:37.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:37.733 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:37.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:37.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:37.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:37.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:37.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:37.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:37 smithi153 bash[23852]: cluster 2024-04-06T03:36:36.347523+0000 osd.7 (osd.7) 391 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:37 smithi153 bash[23852]: cluster 2024-04-06T03:36:36.347585+0000 osd.7 (osd.7) 392 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:37 smithi153 bash[23852]: cluster 2024-04-06T03:36:36.861307+0000 mon.a (mon.0) 1403 : cluster 0 osdmap e259: 8 total, 8 up, 8 in 2024-04-06T03:36:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:37 smithi153 bash[23852]: audit 2024-04-06T03:36:37.045051+0000 mon.a (mon.0) 1404 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:36:38.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:38.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:38.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:38.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:38.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[29221]: cluster 2024-04-06T03:36:36.347523+0000 osd.7 (osd.7) 391 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[29221]: cluster 2024-04-06T03:36:36.347585+0000 osd.7 (osd.7) 392 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[29221]: cluster 2024-04-06T03:36:36.861307+0000 mon.a (mon.0) 1403 : cluster 0 osdmap e259: 8 total, 8 up, 8 in 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[29221]: audit 2024-04-06T03:36:37.045051+0000 mon.a (mon.0) 1404 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[21467]: cluster 2024-04-06T03:36:36.347523+0000 osd.7 (osd.7) 391 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[21467]: cluster 2024-04-06T03:36:36.347585+0000 osd.7 (osd.7) 392 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[21467]: cluster 2024-04-06T03:36:36.861307+0000 mon.a (mon.0) 1403 : cluster 0 osdmap e259: 8 total, 8 up, 8 in 2024-04-06T03:36:38.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:37 smithi050 bash[21467]: audit 2024-04-06T03:36:37.045051+0000 mon.a (mon.0) 1404 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:36:38.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:38.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:38.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:38.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:38.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:38.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:38.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:38.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:38.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:38.859 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:38.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:38.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:38.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:39.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[29221]: audit 2024-04-06T03:36:37.943045+0000 mon.a (mon.0) 1405 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[29221]: audit 2024-04-06T03:36:37.943815+0000 mon.a (mon.0) 1406 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[29221]: audit 2024-04-06T03:36:37.948087+0000 mon.a (mon.0) 1407 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[29221]: cluster 2024-04-06T03:36:38.318020+0000 mgr.y (mgr.24368) 579 : cluster 0 pgmap v609: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 690 op/s 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[21467]: audit 2024-04-06T03:36:37.943045+0000 mon.a (mon.0) 1405 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[21467]: audit 2024-04-06T03:36:37.943815+0000 mon.a (mon.0) 1406 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[21467]: audit 2024-04-06T03:36:37.948087+0000 mon.a (mon.0) 1407 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:36:39.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:38 smithi050 bash[21467]: cluster 2024-04-06T03:36:38.318020+0000 mgr.y (mgr.24368) 579 : cluster 0 pgmap v609: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 690 op/s 2024-04-06T03:36:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:39.239 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:36:39.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:39.268 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:38 smithi153 bash[23852]: audit 2024-04-06T03:36:37.943045+0000 mon.a (mon.0) 1405 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:36:39.268 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:38 smithi153 bash[23852]: audit 2024-04-06T03:36:37.943815+0000 mon.a (mon.0) 1406 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:36:39.268 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:38 smithi153 bash[23852]: audit 2024-04-06T03:36:37.948087+0000 mon.a (mon.0) 1407 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:36:39.268 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:38 smithi153 bash[23852]: cluster 2024-04-06T03:36:38.318020+0000 mgr.y (mgr.24368) 579 : cluster 0 pgmap v609: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 690 op/s 2024-04-06T03:36:39.275 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:39.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:39.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:39.486 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:36:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:36:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:36:39.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:39.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:39.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:39.660 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:39.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:39.796 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:39.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:39.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:36:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:39.922 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:39.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:40.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:40.036 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:39 smithi153 bash[23852]: audit 2024-04-06T03:36:39.760716+0000 mgr.y (mgr.24368) 580 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:40.051 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:40.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:40.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:40.181 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:40.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:39 smithi050 bash[21467]: audit 2024-04-06T03:36:39.760716+0000 mgr.y (mgr.24368) 580 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:40.236 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:39 smithi050 bash[29221]: audit 2024-04-06T03:36:39.760716+0000 mgr.y (mgr.24368) 580 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:40.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:40.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:40.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:40.449 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:40.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:40.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:40.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:40.702 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:40.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:40.831 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:40.890 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:36:40.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:40.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:40.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:40.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:41.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:41.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:41.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:41.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:40 smithi153 bash[23852]: cluster 2024-04-06T03:36:40.318867+0000 mgr.y (mgr.24368) 581 : cluster 0 pgmap v610: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-04-06T03:36:41.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:41.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:41.374 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:41.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:40 smithi050 bash[29221]: cluster 2024-04-06T03:36:40.318867+0000 mgr.y (mgr.24368) 581 : cluster 0 pgmap v610: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-04-06T03:36:41.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:40 smithi050 bash[21467]: cluster 2024-04-06T03:36:40.318867+0000 mgr.y (mgr.24368) 581 : cluster 0 pgmap v610: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 803 op/s 2024-04-06T03:36:41.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:41.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:41.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:41.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:41.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:41.755 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:41.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:41.883 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:41.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:41.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:42.017 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:42.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:42.123 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:42.150 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:42.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:42.287 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:42.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:42.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:42.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:42.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:42.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:42.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:42.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:42.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:42.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:42.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:43.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:43.050 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:43.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:43.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:43.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:43.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:43.432 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:43.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:43.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:43.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:43 smithi050 bash[21467]: cluster 2024-04-06T03:36:42.319565+0000 mgr.y (mgr.24368) 582 : cluster 0 pgmap v611: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s 2024-04-06T03:36:43.615 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:43 smithi050 bash[29221]: cluster 2024-04-06T03:36:42.319565+0000 mgr.y (mgr.24368) 582 : cluster 0 pgmap v611: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s 2024-04-06T03:36:43.625 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:43.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:43.753 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:43.800 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:43 smithi153 bash[23852]: cluster 2024-04-06T03:36:42.319565+0000 mgr.y (mgr.24368) 582 : cluster 0 pgmap v611: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 954 op/s 2024-04-06T03:36:43.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:43.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:43.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:43.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:43.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:44.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:44.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:44.137 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:44.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:44.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:44.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:44.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:44.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:44.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:44.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:44.669 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:44.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:44.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:44.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:44.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:44.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:45.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.050 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:45.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:45.184 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:45.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:45.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:45.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:45.417 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:36:45.418 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:36:45.418 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:36:45.418 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:36:45.419 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:45 smithi153 bash[23852]: cluster 2024-04-06T03:36:43.726813+0000 osd.6 (osd.6) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:45.420 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:45 smithi153 bash[23852]: cluster 2024-04-06T03:36:43.726901+0000 osd.6 (osd.6) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:45.420 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:45 smithi153 bash[23852]: cluster 2024-04-06T03:36:44.320267+0000 mgr.y (mgr.24368) 583 : cluster 0 pgmap v612: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 916 op/s 2024-04-06T03:36:45.420 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:45 smithi153 bash[23852]: cluster 2024-04-06T03:36:44.390194+0000 mon.a (mon.0) 1408 : cluster 0 osdmap e260: 8 total, 8 up, 8 in 2024-04-06T03:36:45.450 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:45.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:45.581 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:45.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[21467]: cluster 2024-04-06T03:36:43.726813+0000 osd.6 (osd.6) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[21467]: cluster 2024-04-06T03:36:43.726901+0000 osd.6 (osd.6) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[21467]: cluster 2024-04-06T03:36:44.320267+0000 mgr.y (mgr.24368) 583 : cluster 0 pgmap v612: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 916 op/s 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[21467]: cluster 2024-04-06T03:36:44.390194+0000 mon.a (mon.0) 1408 : cluster 0 osdmap e260: 8 total, 8 up, 8 in 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[29221]: cluster 2024-04-06T03:36:43.726813+0000 osd.6 (osd.6) 95 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[29221]: cluster 2024-04-06T03:36:43.726901+0000 osd.6 (osd.6) 96 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[29221]: cluster 2024-04-06T03:36:44.320267+0000 mgr.y (mgr.24368) 583 : cluster 0 pgmap v612: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 916 op/s 2024-04-06T03:36:45.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:45 smithi050 bash[29221]: cluster 2024-04-06T03:36:44.390194+0000 mon.a (mon.0) 1408 : cluster 0 osdmap e260: 8 total, 8 up, 8 in 2024-04-06T03:36:45.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.840 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:45.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:45.969 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:45.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:46.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:46.094 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:46.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:46.227 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:46.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-06T03:36:46.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:46.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:46.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:46.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:46.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:46.785 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:46.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:46.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:47.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:47.051 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:47.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:47.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:47.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:47.301 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:47.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:47.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:47.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:47 smithi050 bash[21467]: cluster 2024-04-06T03:36:46.323092+0000 mgr.y (mgr.24368) 584 : cluster 0 pgmap v614: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 880 op/s 2024-04-06T03:36:47.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:47 smithi050 bash[29221]: cluster 2024-04-06T03:36:46.323092+0000 mgr.y (mgr.24368) 584 : cluster 0 pgmap v614: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 880 op/s 2024-04-06T03:36:47.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:47 smithi153 bash[23852]: cluster 2024-04-06T03:36:46.323092+0000 mgr.y (mgr.24368) 584 : cluster 0 pgmap v614: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 880 op/s 2024-04-06T03:36:47.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:47.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:47.715 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:47.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:47.850 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:47.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:47.974 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:47.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:47.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:48.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:48.106 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:48.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:48.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:48.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:48.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:48.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:48.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:48.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:48.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:48.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:48.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:48.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:48.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:48.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:48.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:48.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:48.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:48.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:49.009 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:49.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:49.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:49.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:49.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:49.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:49.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:49.392 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:36:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:36:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:36:49.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:49.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:49.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:49 smithi153 bash[23852]: cluster 2024-04-06T03:36:48.323618+0000 mgr.y (mgr.24368) 585 : cluster 0 pgmap v615: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-04-06T03:36:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:49.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:49.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:49 smithi050 bash[21467]: cluster 2024-04-06T03:36:48.323618+0000 mgr.y (mgr.24368) 585 : cluster 0 pgmap v615: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-04-06T03:36:49.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:49 smithi050 bash[29221]: cluster 2024-04-06T03:36:48.323618+0000 mgr.y (mgr.24368) 585 : cluster 0 pgmap v615: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 833 op/s 2024-04-06T03:36:49.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:49.798 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:36:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:49.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:49.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:49.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:49.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:50.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:50.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:50.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:50.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:50.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:50.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:50.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:50.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:50.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:50.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:50.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:50 smithi050 bash[21467]: audit 2024-04-06T03:36:49.651829+0000 mon.a (mon.0) 1409 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:50.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:50 smithi050 bash[21467]: audit 2024-04-06T03:36:49.769070+0000 mgr.y (mgr.24368) 586 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:50.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:50 smithi050 bash[29221]: audit 2024-04-06T03:36:49.651829+0000 mon.a (mon.0) 1409 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:50.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:50 smithi050 bash[29221]: audit 2024-04-06T03:36:49.769070+0000 mgr.y (mgr.24368) 586 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:50.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:50.715 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:50 smithi153 bash[23852]: audit 2024-04-06T03:36:49.651829+0000 mon.a (mon.0) 1409 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:36:50.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:50 smithi153 bash[23852]: audit 2024-04-06T03:36:49.769070+0000 mgr.y (mgr.24368) 586 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:36:50.728 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:50.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:50.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:50.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:50.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:50.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:51.089 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:36:51.092 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:51.135 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:51.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:51.266 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:51.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:51.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:51.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:51.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:51.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:51.650 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:51 smithi050 bash[21467]: cluster 2024-04-06T03:36:50.324366+0000 mgr.y (mgr.24368) 587 : cluster 0 pgmap v616: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 887 op/s 2024-04-06T03:36:51.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:51 smithi050 bash[29221]: cluster 2024-04-06T03:36:50.324366+0000 mgr.y (mgr.24368) 587 : cluster 0 pgmap v616: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 887 op/s 2024-04-06T03:36:51.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:51.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:51 smithi153 bash[23852]: cluster 2024-04-06T03:36:50.324366+0000 mgr.y (mgr.24368) 587 : cluster 0 pgmap v616: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 887 op/s 2024-04-06T03:36:51.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:51.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:51.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:51.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:52.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:52.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:52.165 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:36:52.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:52.189 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:36:52.219 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:52.227 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:36:52.265 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:36:52.303 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:36:52.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:52.353 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:36:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:52.390 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:36:52.436 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:36:52.445 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:36:52.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:52.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:52.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:52 smithi153 bash[23852]: audit 2024-04-06T03:36:52.119652+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.50:0/2419532827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:36:52.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:52.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:52.667 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:52 smithi050 bash[21467]: audit 2024-04-06T03:36:52.119652+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.50:0/2419532827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:36:52.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:52 smithi050 bash[29221]: audit 2024-04-06T03:36:52.119652+0000 mon.c (mon.2) 78 : audit 0 from='client.? 172.21.15.50:0/2419532827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:36:52.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:52.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:52.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:52.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:52.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:52.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:53.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:53.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:53.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:53.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:53.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:53.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:53.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:53.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-04-06T03:36:53.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:53.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:53.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:53.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:53 smithi153 bash[23852]: cluster 2024-04-06T03:36:52.111073+0000 osd.4 (osd.4) 353 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:53.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:53 smithi153 bash[23852]: cluster 2024-04-06T03:36:52.111135+0000 osd.4 (osd.4) 354 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:53.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:53 smithi153 bash[23852]: cluster 2024-04-06T03:36:52.324970+0000 mgr.y (mgr.24368) 588 : cluster 0 pgmap v617: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:36:53.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:53 smithi153 bash[23852]: cluster 2024-04-06T03:36:52.429393+0000 mon.a (mon.0) 1410 : cluster 0 osdmap e261: 8 total, 8 up, 8 in 2024-04-06T03:36:53.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:53 smithi153 bash[23852]: audit 2024-04-06T03:36:53.256462+0000 mon.c (mon.2) 79 : audit 1 from='client.? 172.21.15.50:0/3818082266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:36:53.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:53 smithi153 bash[23852]: audit 2024-04-06T03:36:53.258422+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[21467]: cluster 2024-04-06T03:36:52.111073+0000 osd.4 (osd.4) 353 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[21467]: cluster 2024-04-06T03:36:52.111135+0000 osd.4 (osd.4) 354 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[21467]: cluster 2024-04-06T03:36:52.324970+0000 mgr.y (mgr.24368) 588 : cluster 0 pgmap v617: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[21467]: cluster 2024-04-06T03:36:52.429393+0000 mon.a (mon.0) 1410 : cluster 0 osdmap e261: 8 total, 8 up, 8 in 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[21467]: audit 2024-04-06T03:36:53.256462+0000 mon.c (mon.2) 79 : audit 1 from='client.? 172.21.15.50:0/3818082266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[21467]: audit 2024-04-06T03:36:53.258422+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:36:53.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[29221]: cluster 2024-04-06T03:36:52.111073+0000 osd.4 (osd.4) 353 : cluster 0 purged_snaps scrub starts 2024-04-06T03:36:53.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[29221]: cluster 2024-04-06T03:36:52.111135+0000 osd.4 (osd.4) 354 : cluster 0 purged_snaps scrub ok 2024-04-06T03:36:53.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[29221]: cluster 2024-04-06T03:36:52.324970+0000 mgr.y (mgr.24368) 588 : cluster 0 pgmap v617: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 805 op/s 2024-04-06T03:36:53.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[29221]: cluster 2024-04-06T03:36:52.429393+0000 mon.a (mon.0) 1410 : cluster 0 osdmap e261: 8 total, 8 up, 8 in 2024-04-06T03:36:53.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[29221]: audit 2024-04-06T03:36:53.256462+0000 mon.c (mon.2) 79 : audit 1 from='client.? 172.21.15.50:0/3818082266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:36:53.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:53 smithi050 bash[29221]: audit 2024-04-06T03:36:53.258422+0000 mon.a (mon.0) 1411 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:36:53.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:53.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:53.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:53.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:53.951 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:53.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:54.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:54.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:54.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:54.208 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:54.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:54.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:54.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:54.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:54 smithi153 bash[23852]: audit 2024-04-06T03:36:53.432666+0000 mon.a (mon.0) 1412 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:36:54.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:54 smithi153 bash[23852]: cluster 2024-04-06T03:36:53.436814+0000 mon.a (mon.0) 1413 : cluster 0 osdmap e262: 8 total, 8 up, 8 in 2024-04-06T03:36:54.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:54.625 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:54.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:54 smithi050 bash[21467]: audit 2024-04-06T03:36:53.432666+0000 mon.a (mon.0) 1412 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:36:54.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:54 smithi050 bash[21467]: cluster 2024-04-06T03:36:53.436814+0000 mon.a (mon.0) 1413 : cluster 0 osdmap e262: 8 total, 8 up, 8 in 2024-04-06T03:36:54.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:54 smithi050 bash[29221]: audit 2024-04-06T03:36:53.432666+0000 mon.a (mon.0) 1412 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:36:54.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:54 smithi050 bash[29221]: cluster 2024-04-06T03:36:53.436814+0000 mon.a (mon.0) 1413 : cluster 0 osdmap e262: 8 total, 8 up, 8 in 2024-04-06T03:36:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:54.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:54.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:54.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:54.992 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:55.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:55.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:55.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:55.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:55.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:55.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:55.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:55.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:55.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:55.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:55 smithi050 bash[29221]: cluster 2024-04-06T03:36:54.325823+0000 mgr.y (mgr.24368) 589 : cluster 0 pgmap v620: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 780 op/s 2024-04-06T03:36:55.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:55 smithi050 bash[21467]: cluster 2024-04-06T03:36:54.325823+0000 mgr.y (mgr.24368) 589 : cluster 0 pgmap v620: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 780 op/s 2024-04-06T03:36:55.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:55.744 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:55 smithi153 bash[23852]: cluster 2024-04-06T03:36:54.325823+0000 mgr.y (mgr.24368) 589 : cluster 0 pgmap v620: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 780 op/s 2024-04-06T03:36:55.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:55.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:55.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:55.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:55.974 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:36:55.974 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:36:56.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:56.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:56.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:56.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:56.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:56.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:56.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:56 smithi050 bash[29221]: audit 2024-04-06T03:36:55.482304+0000 mon.a (mon.0) 1414 : audit 1 from='client.? 172.21.15.50:0/2651844975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:36:56.455 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:36:56.484 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:56.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 2 1 2024-04-06T03:36:56.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:56.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:56.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:56 smithi050 bash[21467]: audit 2024-04-06T03:36:55.482304+0000 mon.a (mon.0) 1414 : audit 1 from='client.? 172.21.15.50:0/2651844975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:36:56.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:56.766 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:56 smithi153 bash[23852]: audit 2024-04-06T03:36:55.482304+0000 mon.a (mon.0) 1414 : audit 1 from='client.? 172.21.15.50:0/2651844975' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:36:56.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:56.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:56.924 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:56.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:57.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:57.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:57.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-06T03:36:57.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:57.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:57.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:57.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:57.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:57.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:57.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:57.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:57.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:57.700 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:57.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:57 smithi153 bash[23852]: cluster 2024-04-06T03:36:56.326656+0000 mgr.y (mgr.24368) 590 : cluster 0 pgmap v621: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:36:57.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:57 smithi153 bash[23852]: cluster 2024-04-06T03:36:56.439546+0000 mon.a (mon.0) 1415 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:57.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:57 smithi153 bash[23852]: audit 2024-04-06T03:36:56.448266+0000 mon.a (mon.0) 1416 : audit 1 from='client.? 172.21.15.50:0/2651844975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:36:57.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:57 smithi153 bash[23852]: cluster 2024-04-06T03:36:56.466570+0000 mon.a (mon.0) 1417 : cluster 0 osdmap e263: 8 total, 8 up, 8 in 2024-04-06T03:36:57.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:57 smithi153 bash[23852]: cluster 2024-04-06T03:36:56.768986+0000 osd.3 (osd.3) 382 : cluster 0 9.1 scrub starts 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[21467]: cluster 2024-04-06T03:36:56.326656+0000 mgr.y (mgr.24368) 590 : cluster 0 pgmap v621: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[21467]: cluster 2024-04-06T03:36:56.439546+0000 mon.a (mon.0) 1415 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[21467]: audit 2024-04-06T03:36:56.448266+0000 mon.a (mon.0) 1416 : audit 1 from='client.? 172.21.15.50:0/2651844975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[21467]: cluster 2024-04-06T03:36:56.466570+0000 mon.a (mon.0) 1417 : cluster 0 osdmap e263: 8 total, 8 up, 8 in 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[21467]: cluster 2024-04-06T03:36:56.768986+0000 osd.3 (osd.3) 382 : cluster 0 9.1 scrub starts 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[29221]: cluster 2024-04-06T03:36:56.326656+0000 mgr.y (mgr.24368) 590 : cluster 0 pgmap v621: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[29221]: cluster 2024-04-06T03:36:56.439546+0000 mon.a (mon.0) 1415 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[29221]: audit 2024-04-06T03:36:56.448266+0000 mon.a (mon.0) 1416 : audit 1 from='client.? 172.21.15.50:0/2651844975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:36:57.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[29221]: cluster 2024-04-06T03:36:56.466570+0000 mon.a (mon.0) 1417 : cluster 0 osdmap e263: 8 total, 8 up, 8 in 2024-04-06T03:36:57.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:57 smithi050 bash[29221]: cluster 2024-04-06T03:36:56.768986+0000 osd.3 (osd.3) 382 : cluster 0 9.1 scrub starts 2024-04-06T03:36:57.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:57.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:57.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:57.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:57.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:58.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:58.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:58.200 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:58.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:58.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:58.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:36:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:36:58.468 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:58.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:36:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:36:58.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:36:58.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:58.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:58 smithi050 bash[21467]: cluster 2024-04-06T03:36:57.782732+0000 osd.3 (osd.3) 383 : cluster 0 9.1 scrub ok 2024-04-06T03:36:58.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:58 smithi050 bash[29221]: cluster 2024-04-06T03:36:57.782732+0000 osd.3 (osd.3) 383 : cluster 0 9.1 scrub ok 2024-04-06T03:36:58.635 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:36:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:36:58.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:58.749 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:58 smithi153 bash[23852]: cluster 2024-04-06T03:36:57.782732+0000 osd.3 (osd.3) 383 : cluster 0 9.1 scrub ok 2024-04-06T03:36:58.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:58.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:58.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:58.908 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:58.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:59.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:59.047 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:59.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:59.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:36:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:36:59.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:59.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:36:59.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:59.440 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:36:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:36:59.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:59.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:59 smithi153 bash[23852]: cluster 2024-04-06T03:36:58.327033+0000 mgr.y (mgr.24368) 591 : cluster 0 pgmap v623: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 745 op/s 2024-04-06T03:36:59.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:36:59 smithi153 bash[23852]: cluster 2024-04-06T03:36:58.804676+0000 osd.3 (osd.3) 384 : cluster 0 9.c scrub starts 2024-04-06T03:36:59.577 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:36:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:36:59.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:59.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:59 smithi050 bash[21467]: cluster 2024-04-06T03:36:58.327033+0000 mgr.y (mgr.24368) 591 : cluster 0 pgmap v623: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 745 op/s 2024-04-06T03:36:59.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:36:59 smithi050 bash[21467]: cluster 2024-04-06T03:36:58.804676+0000 osd.3 (osd.3) 384 : cluster 0 9.c scrub starts 2024-04-06T03:36:59.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:59 smithi050 bash[29221]: cluster 2024-04-06T03:36:58.327033+0000 mgr.y (mgr.24368) 591 : cluster 0 pgmap v623: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 745 op/s 2024-04-06T03:36:59.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:36:59 smithi050 bash[29221]: cluster 2024-04-06T03:36:58.804676+0000 osd.3 (osd.3) 384 : cluster 0 9.c scrub starts 2024-04-06T03:36:59.691 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:36:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:36:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:36:59.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:59.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:59.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:36:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:36:59.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:59.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:36:59.962 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:36:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:36:59.984 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:36:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:36:59.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:00.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:00.113 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:00.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:00.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:00.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:00.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:00.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:00.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:00.493 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:00 smithi153 bash[23852]: cluster 2024-04-06T03:36:59.207825+0000 osd.2 (osd.2) 299 : cluster 0 2.17 scrub starts 2024-04-06T03:37:00.493 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:00 smithi153 bash[23852]: cluster 2024-04-06T03:36:59.209689+0000 osd.2 (osd.2) 300 : cluster 0 2.17 scrub ok 2024-04-06T03:37:00.493 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:00 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.356619+0000 osd.3 (osd.3) 385 : cluster 0 9.c scrub ok 2024-04-06T03:37:00.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:00.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:00.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:00 smithi050 bash[21467]: cluster 2024-04-06T03:36:59.207825+0000 osd.2 (osd.2) 299 : cluster 0 2.17 scrub starts 2024-04-06T03:37:00.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:00 smithi050 bash[21467]: cluster 2024-04-06T03:36:59.209689+0000 osd.2 (osd.2) 300 : cluster 0 2.17 scrub ok 2024-04-06T03:37:00.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:00 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.356619+0000 osd.3 (osd.3) 385 : cluster 0 9.c scrub ok 2024-04-06T03:37:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:00 smithi050 bash[29221]: cluster 2024-04-06T03:36:59.207825+0000 osd.2 (osd.2) 299 : cluster 0 2.17 scrub starts 2024-04-06T03:37:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:00 smithi050 bash[29221]: cluster 2024-04-06T03:36:59.209689+0000 osd.2 (osd.2) 300 : cluster 0 2.17 scrub ok 2024-04-06T03:37:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:00 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.356619+0000 osd.3 (osd.3) 385 : cluster 0 9.c scrub ok 2024-04-06T03:37:00.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:00.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:00.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:00.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:00.925 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:00.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:01.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:01.057 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:01.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:01.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:01.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:01.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:01.443 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:01.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:01.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:01.579 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:01.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:01.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: audit 2024-04-06T03:36:59.779365+0000 mgr.y (mgr.24368) 592 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.051222+0000 osd.4 (osd.4) 355 : cluster 0 5.11 scrub starts 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.052468+0000 osd.4 (osd.4) 356 : cluster 0 5.11 scrub ok 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.180853+0000 osd.1 (osd.1) 305 : cluster 0 2.6 scrub starts 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.186637+0000 osd.1 (osd.1) 306 : cluster 0 2.6 scrub ok 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.327759+0000 mgr.y (mgr.24368) 593 : cluster 0 pgmap v624: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 912 op/s 2024-04-06T03:37:01.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[21467]: cluster 2024-04-06T03:37:00.407157+0000 osd.7 (osd.7) 393 : cluster 0 9.a scrub starts 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: audit 2024-04-06T03:36:59.779365+0000 mgr.y (mgr.24368) 592 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.051222+0000 osd.4 (osd.4) 355 : cluster 0 5.11 scrub starts 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.052468+0000 osd.4 (osd.4) 356 : cluster 0 5.11 scrub ok 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.180853+0000 osd.1 (osd.1) 305 : cluster 0 2.6 scrub starts 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.186637+0000 osd.1 (osd.1) 306 : cluster 0 2.6 scrub ok 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.327759+0000 mgr.y (mgr.24368) 593 : cluster 0 pgmap v624: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 912 op/s 2024-04-06T03:37:01.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:01 smithi050 bash[29221]: cluster 2024-04-06T03:37:00.407157+0000 osd.7 (osd.7) 393 : cluster 0 9.a scrub starts 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: audit 2024-04-06T03:36:59.779365+0000 mgr.y (mgr.24368) 592 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.051222+0000 osd.4 (osd.4) 355 : cluster 0 5.11 scrub starts 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.052468+0000 osd.4 (osd.4) 356 : cluster 0 5.11 scrub ok 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.180853+0000 osd.1 (osd.1) 305 : cluster 0 2.6 scrub starts 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.186637+0000 osd.1 (osd.1) 306 : cluster 0 2.6 scrub ok 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.327759+0000 mgr.y (mgr.24368) 593 : cluster 0 pgmap v624: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 912 op/s 2024-04-06T03:37:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:01 smithi153 bash[23852]: cluster 2024-04-06T03:37:00.407157+0000 osd.7 (osd.7) 393 : cluster 0 9.a scrub starts 2024-04-06T03:37:01.815 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:37:01.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:01.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:01.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:01.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:01.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:02.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:02.115 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:02.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:02.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:02.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:02.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:02.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:02.468 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:02.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:02.520 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:02.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:02.635 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:02 smithi153 bash[23852]: cluster 2024-04-06T03:37:01.058938+0000 osd.4 (osd.4) 357 : cluster 0 5.18 scrub starts 2024-04-06T03:37:02.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:02 smithi153 bash[23852]: cluster 2024-04-06T03:37:01.060800+0000 osd.4 (osd.4) 358 : cluster 0 5.18 scrub ok 2024-04-06T03:37:02.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:02 smithi153 bash[23852]: cluster 2024-04-06T03:37:01.450102+0000 osd.7 (osd.7) 394 : cluster 0 9.a scrub ok 2024-04-06T03:37:02.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:02 smithi153 bash[23852]: cluster 2024-04-06T03:37:01.455205+0000 osd.7 (osd.7) 395 : cluster 0 9.7 scrub starts 2024-04-06T03:37:02.646 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:02.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:02.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:02.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[21467]: cluster 2024-04-06T03:37:01.058938+0000 osd.4 (osd.4) 357 : cluster 0 5.18 scrub starts 2024-04-06T03:37:02.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[21467]: cluster 2024-04-06T03:37:01.060800+0000 osd.4 (osd.4) 358 : cluster 0 5.18 scrub ok 2024-04-06T03:37:02.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[21467]: cluster 2024-04-06T03:37:01.450102+0000 osd.7 (osd.7) 394 : cluster 0 9.a scrub ok 2024-04-06T03:37:02.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[21467]: cluster 2024-04-06T03:37:01.455205+0000 osd.7 (osd.7) 395 : cluster 0 9.7 scrub starts 2024-04-06T03:37:02.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[29221]: cluster 2024-04-06T03:37:01.058938+0000 osd.4 (osd.4) 357 : cluster 0 5.18 scrub starts 2024-04-06T03:37:02.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[29221]: cluster 2024-04-06T03:37:01.060800+0000 osd.4 (osd.4) 358 : cluster 0 5.18 scrub ok 2024-04-06T03:37:02.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[29221]: cluster 2024-04-06T03:37:01.450102+0000 osd.7 (osd.7) 394 : cluster 0 9.a scrub ok 2024-04-06T03:37:02.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:02 smithi050 bash[29221]: cluster 2024-04-06T03:37:01.455205+0000 osd.7 (osd.7) 395 : cluster 0 9.7 scrub starts 2024-04-06T03:37:02.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:02.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:02.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:03.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:03.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:03.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:03.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:03.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:03.302 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:03.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:03.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:03.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:03.566 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:03.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:03.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:03.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[21467]: cluster 2024-04-06T03:37:02.151750+0000 osd.2 (osd.2) 301 : cluster 0 5.d scrub starts 2024-04-06T03:37:03.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[21467]: cluster 2024-04-06T03:37:02.153937+0000 osd.2 (osd.2) 302 : cluster 0 5.d scrub ok 2024-04-06T03:37:03.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[21467]: cluster 2024-04-06T03:37:02.328315+0000 mgr.y (mgr.24368) 594 : cluster 0 pgmap v625: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:37:03.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[21467]: cluster 2024-04-06T03:37:02.511463+0000 osd.7 (osd.7) 396 : cluster 0 9.7 scrub ok 2024-04-06T03:37:03.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[21467]: cluster 2024-04-06T03:37:03.350727+0000 osd.0 (osd.0) 358 : cluster 0 6.16 scrub starts 2024-04-06T03:37:03.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[21467]: cluster 2024-04-06T03:37:03.375561+0000 osd.0 (osd.0) 359 : cluster 0 6.16 scrub ok 2024-04-06T03:37:03.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[29221]: cluster 2024-04-06T03:37:02.151750+0000 osd.2 (osd.2) 301 : cluster 0 5.d scrub starts 2024-04-06T03:37:03.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[29221]: cluster 2024-04-06T03:37:02.153937+0000 osd.2 (osd.2) 302 : cluster 0 5.d scrub ok 2024-04-06T03:37:03.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[29221]: cluster 2024-04-06T03:37:02.328315+0000 mgr.y (mgr.24368) 594 : cluster 0 pgmap v625: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:37:03.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[29221]: cluster 2024-04-06T03:37:02.511463+0000 osd.7 (osd.7) 396 : cluster 0 9.7 scrub ok 2024-04-06T03:37:03.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[29221]: cluster 2024-04-06T03:37:03.350727+0000 osd.0 (osd.0) 358 : cluster 0 6.16 scrub starts 2024-04-06T03:37:03.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:03 smithi050 bash[29221]: cluster 2024-04-06T03:37:03.375561+0000 osd.0 (osd.0) 359 : cluster 0 6.16 scrub ok 2024-04-06T03:37:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:03 smithi153 bash[23852]: cluster 2024-04-06T03:37:02.151750+0000 osd.2 (osd.2) 301 : cluster 0 5.d scrub starts 2024-04-06T03:37:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:03 smithi153 bash[23852]: cluster 2024-04-06T03:37:02.153937+0000 osd.2 (osd.2) 302 : cluster 0 5.d scrub ok 2024-04-06T03:37:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:03 smithi153 bash[23852]: cluster 2024-04-06T03:37:02.328315+0000 mgr.y (mgr.24368) 594 : cluster 0 pgmap v625: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:37:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:03 smithi153 bash[23852]: cluster 2024-04-06T03:37:02.511463+0000 osd.7 (osd.7) 396 : cluster 0 9.7 scrub ok 2024-04-06T03:37:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:03 smithi153 bash[23852]: cluster 2024-04-06T03:37:03.350727+0000 osd.0 (osd.0) 358 : cluster 0 6.16 scrub starts 2024-04-06T03:37:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:03 smithi153 bash[23852]: cluster 2024-04-06T03:37:03.375561+0000 osd.0 (osd.0) 359 : cluster 0 6.16 scrub ok 2024-04-06T03:37:03.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:03.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:03.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:03.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:03.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:04.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:04.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:04.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:04.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:04.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:04.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:04.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:04 smithi153 bash[23852]: cluster 2024-04-06T03:37:03.107645+0000 osd.2 (osd.2) 303 : cluster 0 2.10 scrub starts 2024-04-06T03:37:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:04 smithi153 bash[23852]: cluster 2024-04-06T03:37:03.115652+0000 osd.2 (osd.2) 304 : cluster 0 2.10 scrub ok 2024-04-06T03:37:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:04.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:04.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:04.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:04.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:04 smithi050 bash[21467]: cluster 2024-04-06T03:37:03.107645+0000 osd.2 (osd.2) 303 : cluster 0 2.10 scrub starts 2024-04-06T03:37:04.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:04 smithi050 bash[21467]: cluster 2024-04-06T03:37:03.115652+0000 osd.2 (osd.2) 304 : cluster 0 2.10 scrub ok 2024-04-06T03:37:04.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:04 smithi050 bash[29221]: cluster 2024-04-06T03:37:03.107645+0000 osd.2 (osd.2) 303 : cluster 0 2.10 scrub starts 2024-04-06T03:37:04.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:04 smithi050 bash[29221]: cluster 2024-04-06T03:37:03.115652+0000 osd.2 (osd.2) 304 : cluster 0 2.10 scrub ok 2024-04-06T03:37:04.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:04.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:04.966 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:04.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:05.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:05.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:05.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:05.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:05.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:05.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:05.353 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:05.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:05.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:05.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:05.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:05 smithi050 bash[21467]: cluster 2024-04-06T03:37:04.329493+0000 mgr.y (mgr.24368) 595 : cluster 0 pgmap v626: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 941 op/s 2024-04-06T03:37:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:05 smithi050 bash[21467]: cluster 2024-04-06T03:37:04.524521+0000 osd.7 (osd.7) 397 : cluster 0 9.d scrub starts 2024-04-06T03:37:05.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:05 smithi050 bash[21467]: audit 2024-04-06T03:37:04.665533+0000 mon.a (mon.0) 1418 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:05.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:05 smithi050 bash[29221]: cluster 2024-04-06T03:37:04.329493+0000 mgr.y (mgr.24368) 595 : cluster 0 pgmap v626: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 941 op/s 2024-04-06T03:37:05.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:05 smithi050 bash[29221]: cluster 2024-04-06T03:37:04.524521+0000 osd.7 (osd.7) 397 : cluster 0 9.d scrub starts 2024-04-06T03:37:05.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:05 smithi050 bash[29221]: audit 2024-04-06T03:37:04.665533+0000 mon.a (mon.0) 1418 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:05.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:05.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:05.900 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:05.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:06.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:06.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:05 smithi153 bash[23852]: cluster 2024-04-06T03:37:04.329493+0000 mgr.y (mgr.24368) 595 : cluster 0 pgmap v626: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 941 op/s 2024-04-06T03:37:06.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:05 smithi153 bash[23852]: cluster 2024-04-06T03:37:04.524521+0000 osd.7 (osd.7) 397 : cluster 0 9.d scrub starts 2024-04-06T03:37:06.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:05 smithi153 bash[23852]: audit 2024-04-06T03:37:04.665533+0000 mon.a (mon.0) 1418 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:06.026 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:06.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:06.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:06.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:06.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:06.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:06.425 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:06.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:06.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:06.553 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:06.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:06.666 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:06 smithi153 bash[23852]: cluster 2024-04-06T03:37:05.404825+0000 osd.7 (osd.7) 398 : cluster 0 9.d scrub ok 2024-04-06T03:37:06.681 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:06.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:06.802 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:06.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:06.915 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:06 smithi050 bash[21467]: cluster 2024-04-06T03:37:05.404825+0000 osd.7 (osd.7) 398 : cluster 0 9.d scrub ok 2024-04-06T03:37:06.915 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:06 smithi050 bash[29221]: cluster 2024-04-06T03:37:05.404825+0000 osd.7 (osd.7) 398 : cluster 0 9.d scrub ok 2024-04-06T03:37:06.929 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:06.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:07.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:07.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:07.174 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:07.206 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:07.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:07.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:07.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:07.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:07.471 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:07.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:07.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:07.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:07.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:07.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:07 smithi153 bash[23852]: cluster 2024-04-06T03:37:06.330495+0000 mgr.y (mgr.24368) 596 : cluster 0 pgmap v627: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:07.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:07 smithi153 bash[23852]: cluster 2024-04-06T03:37:06.517855+0000 osd.7 (osd.7) 399 : cluster 0 9.3 scrub starts 2024-04-06T03:37:07.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:07.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:07 smithi050 bash[21467]: cluster 2024-04-06T03:37:06.330495+0000 mgr.y (mgr.24368) 596 : cluster 0 pgmap v627: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:07.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:07 smithi050 bash[21467]: cluster 2024-04-06T03:37:06.517855+0000 osd.7 (osd.7) 399 : cluster 0 9.3 scrub starts 2024-04-06T03:37:07.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:07 smithi050 bash[29221]: cluster 2024-04-06T03:37:06.330495+0000 mgr.y (mgr.24368) 596 : cluster 0 pgmap v627: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:07.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:07 smithi050 bash[29221]: cluster 2024-04-06T03:37:06.517855+0000 osd.7 (osd.7) 399 : cluster 0 9.3 scrub starts 2024-04-06T03:37:07.869 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:37:07.869 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:37:07.880 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:07.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:07.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:08.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:08.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:08.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:08.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:08.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:08.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:08.388 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:08.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:08.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:08.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:08.680 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:08.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:06.925701+0000 osd.4 (osd.4) 359 : cluster 0 3.14 scrub starts 2024-04-06T03:37:08.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:06.930189+0000 osd.4 (osd.4) 360 : cluster 0 3.14 scrub ok 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.010887+0000 osd.2 (osd.2) 305 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.010989+0000 osd.2 (osd.2) 306 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.026479+0000 osd.3 (osd.3) 386 : cluster 0 9.8 scrub starts 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.409271+0000 osd.0 (osd.0) 360 : cluster 0 3.12 scrub starts 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.414374+0000 osd.0 (osd.0) 361 : cluster 0 3.12 scrub ok 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.514039+0000 mon.a (mon.0) 1419 : cluster 0 osdmap e264: 8 total, 8 up, 8 in 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.527981+0000 osd.7 (osd.7) 400 : cluster 0 9.e scrub starts 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.858112+0000 osd.7 (osd.7) 401 : cluster 0 9.3 scrub ok 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[21467]: cluster 2024-04-06T03:37:08.136936+0000 osd.1 (osd.1) 307 : cluster 0 3.19 scrub starts 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:06.925701+0000 osd.4 (osd.4) 359 : cluster 0 3.14 scrub starts 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:06.930189+0000 osd.4 (osd.4) 360 : cluster 0 3.14 scrub ok 2024-04-06T03:37:08.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.010887+0000 osd.2 (osd.2) 305 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.010989+0000 osd.2 (osd.2) 306 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.026479+0000 osd.3 (osd.3) 386 : cluster 0 9.8 scrub starts 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.409271+0000 osd.0 (osd.0) 360 : cluster 0 3.12 scrub starts 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.414374+0000 osd.0 (osd.0) 361 : cluster 0 3.12 scrub ok 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.514039+0000 mon.a (mon.0) 1419 : cluster 0 osdmap e264: 8 total, 8 up, 8 in 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.527981+0000 osd.7 (osd.7) 400 : cluster 0 9.e scrub starts 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.858112+0000 osd.7 (osd.7) 401 : cluster 0 9.3 scrub ok 2024-04-06T03:37:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:08 smithi050 bash[29221]: cluster 2024-04-06T03:37:08.136936+0000 osd.1 (osd.1) 307 : cluster 0 3.19 scrub starts 2024-04-06T03:37:08.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:06.925701+0000 osd.4 (osd.4) 359 : cluster 0 3.14 scrub starts 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:06.930189+0000 osd.4 (osd.4) 360 : cluster 0 3.14 scrub ok 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.010887+0000 osd.2 (osd.2) 305 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.010989+0000 osd.2 (osd.2) 306 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.026479+0000 osd.3 (osd.3) 386 : cluster 0 9.8 scrub starts 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.409271+0000 osd.0 (osd.0) 360 : cluster 0 3.12 scrub starts 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.414374+0000 osd.0 (osd.0) 361 : cluster 0 3.12 scrub ok 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.514039+0000 mon.a (mon.0) 1419 : cluster 0 osdmap e264: 8 total, 8 up, 8 in 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.527981+0000 osd.7 (osd.7) 400 : cluster 0 9.e scrub starts 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.858112+0000 osd.7 (osd.7) 401 : cluster 0 9.3 scrub ok 2024-04-06T03:37:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:08 smithi153 bash[23852]: cluster 2024-04-06T03:37:08.136936+0000 osd.1 (osd.1) 307 : cluster 0 3.19 scrub starts 2024-04-06T03:37:08.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:08.926 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:08.949 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:08.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:08.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:09.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:09.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:09.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:09.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:09.318 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:09.344 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:09.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:09.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:09.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-06T03:37:09.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:09.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:09.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:09.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.968140+0000 osd.4 (osd.4) 361 : cluster 0 3.1c scrub starts 2024-04-06T03:37:09.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21467]: cluster 2024-04-06T03:37:07.970998+0000 osd.4 (osd.4) 362 : cluster 0 3.1c scrub ok 2024-04-06T03:37:09.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21467]: cluster 2024-04-06T03:37:08.159336+0000 osd.1 (osd.1) 308 : cluster 0 3.19 scrub ok 2024-04-06T03:37:09.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21467]: cluster 2024-04-06T03:37:08.294143+0000 osd.3 (osd.3) 387 : cluster 0 9.8 scrub ok 2024-04-06T03:37:09.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21467]: cluster 2024-04-06T03:37:08.332747+0000 mgr.y (mgr.24368) 597 : cluster 0 pgmap v629: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:09.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21467]: cluster 2024-04-06T03:37:08.674967+0000 osd.7 (osd.7) 402 : cluster 0 9.e scrub ok 2024-04-06T03:37:09.720 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:37:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:37:09.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.968140+0000 osd.4 (osd.4) 361 : cluster 0 3.1c scrub starts 2024-04-06T03:37:09.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[29221]: cluster 2024-04-06T03:37:07.970998+0000 osd.4 (osd.4) 362 : cluster 0 3.1c scrub ok 2024-04-06T03:37:09.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[29221]: cluster 2024-04-06T03:37:08.159336+0000 osd.1 (osd.1) 308 : cluster 0 3.19 scrub ok 2024-04-06T03:37:09.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[29221]: cluster 2024-04-06T03:37:08.294143+0000 osd.3 (osd.3) 387 : cluster 0 9.8 scrub ok 2024-04-06T03:37:09.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[29221]: cluster 2024-04-06T03:37:08.332747+0000 mgr.y (mgr.24368) 597 : cluster 0 pgmap v629: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:09.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:09 smithi050 bash[29221]: cluster 2024-04-06T03:37:08.674967+0000 osd.7 (osd.7) 402 : cluster 0 9.e scrub ok 2024-04-06T03:37:09.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:09.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.968140+0000 osd.4 (osd.4) 361 : cluster 0 3.1c scrub starts 2024-04-06T03:37:09.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[23852]: cluster 2024-04-06T03:37:07.970998+0000 osd.4 (osd.4) 362 : cluster 0 3.1c scrub ok 2024-04-06T03:37:09.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[23852]: cluster 2024-04-06T03:37:08.159336+0000 osd.1 (osd.1) 308 : cluster 0 3.19 scrub ok 2024-04-06T03:37:09.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[23852]: cluster 2024-04-06T03:37:08.294143+0000 osd.3 (osd.3) 387 : cluster 0 9.8 scrub ok 2024-04-06T03:37:09.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[23852]: cluster 2024-04-06T03:37:08.332747+0000 mgr.y (mgr.24368) 597 : cluster 0 pgmap v629: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:09.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[23852]: cluster 2024-04-06T03:37:08.674967+0000 osd.7 (osd.7) 402 : cluster 0 9.e scrub ok 2024-04-06T03:37:09.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:09.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:09.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:09.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:10.008 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:10.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:37:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:37:10.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:10.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:10.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:10.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:10.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:10.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:10.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:10.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[21467]: cluster 2024-04-06T03:37:09.004515+0000 osd.3 (osd.3) 388 : cluster 0 5.17 scrub starts 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[21467]: cluster 2024-04-06T03:37:09.019365+0000 osd.3 (osd.3) 389 : cluster 0 5.17 scrub ok 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[21467]: cluster 2024-04-06T03:37:09.418166+0000 osd.0 (osd.0) 362 : cluster 0 6.0 scrub starts 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[21467]: cluster 2024-04-06T03:37:09.463932+0000 osd.0 (osd.0) 363 : cluster 0 6.0 scrub ok 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[21467]: cluster 2024-04-06T03:37:09.536459+0000 osd.7 (osd.7) 403 : cluster 0 6.8 scrub starts 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[21467]: cluster 2024-04-06T03:37:09.544113+0000 osd.7 (osd.7) 404 : cluster 0 6.8 scrub ok 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[29221]: cluster 2024-04-06T03:37:09.004515+0000 osd.3 (osd.3) 388 : cluster 0 5.17 scrub starts 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[29221]: cluster 2024-04-06T03:37:09.019365+0000 osd.3 (osd.3) 389 : cluster 0 5.17 scrub ok 2024-04-06T03:37:10.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[29221]: cluster 2024-04-06T03:37:09.418166+0000 osd.0 (osd.0) 362 : cluster 0 6.0 scrub starts 2024-04-06T03:37:10.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[29221]: cluster 2024-04-06T03:37:09.463932+0000 osd.0 (osd.0) 363 : cluster 0 6.0 scrub ok 2024-04-06T03:37:10.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[29221]: cluster 2024-04-06T03:37:09.536459+0000 osd.7 (osd.7) 403 : cluster 0 6.8 scrub starts 2024-04-06T03:37:10.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:10 smithi050 bash[29221]: cluster 2024-04-06T03:37:09.544113+0000 osd.7 (osd.7) 404 : cluster 0 6.8 scrub ok 2024-04-06T03:37:10.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:10.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:10 smithi153 bash[23852]: cluster 2024-04-06T03:37:09.004515+0000 osd.3 (osd.3) 388 : cluster 0 5.17 scrub starts 2024-04-06T03:37:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:10 smithi153 bash[23852]: cluster 2024-04-06T03:37:09.019365+0000 osd.3 (osd.3) 389 : cluster 0 5.17 scrub ok 2024-04-06T03:37:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:10 smithi153 bash[23852]: cluster 2024-04-06T03:37:09.418166+0000 osd.0 (osd.0) 362 : cluster 0 6.0 scrub starts 2024-04-06T03:37:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:10 smithi153 bash[23852]: cluster 2024-04-06T03:37:09.463932+0000 osd.0 (osd.0) 363 : cluster 0 6.0 scrub ok 2024-04-06T03:37:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:10 smithi153 bash[23852]: cluster 2024-04-06T03:37:09.536459+0000 osd.7 (osd.7) 403 : cluster 0 6.8 scrub starts 2024-04-06T03:37:10.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:10 smithi153 bash[23852]: cluster 2024-04-06T03:37:09.544113+0000 osd.7 (osd.7) 404 : cluster 0 6.8 scrub ok 2024-04-06T03:37:10.812 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:10.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:10.940 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:10.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:11.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:11.070 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:11.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:11.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:11.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:11.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:11.474 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:11.580 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:11 smithi153 bash[23852]: audit 2024-04-06T03:37:09.788064+0000 mgr.y (mgr.24368) 598 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:11.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:11 smithi153 bash[23852]: cluster 2024-04-06T03:37:10.012686+0000 osd.2 (osd.2) 307 : cluster 0 2.c scrub starts 2024-04-06T03:37:11.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:11 smithi153 bash[23852]: cluster 2024-04-06T03:37:10.014007+0000 osd.2 (osd.2) 308 : cluster 0 2.c scrub ok 2024-04-06T03:37:11.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:11 smithi153 bash[23852]: cluster 2024-04-06T03:37:10.333525+0000 mgr.y (mgr.24368) 599 : cluster 0 pgmap v630: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:37:11.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:11 smithi153 bash[23852]: cluster 2024-04-06T03:37:10.895085+0000 osd.4 (osd.4) 363 : cluster 0 5.1f scrub starts 2024-04-06T03:37:11.589 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:11 smithi153 bash[23852]: cluster 2024-04-06T03:37:10.899526+0000 osd.4 (osd.4) 364 : cluster 0 5.1f scrub ok 2024-04-06T03:37:11.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:11.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.736 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:11.841 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:11.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[21467]: audit 2024-04-06T03:37:09.788064+0000 mgr.y (mgr.24368) 598 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[21467]: cluster 2024-04-06T03:37:10.012686+0000 osd.2 (osd.2) 307 : cluster 0 2.c scrub starts 2024-04-06T03:37:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[21467]: cluster 2024-04-06T03:37:10.014007+0000 osd.2 (osd.2) 308 : cluster 0 2.c scrub ok 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[21467]: cluster 2024-04-06T03:37:10.333525+0000 mgr.y (mgr.24368) 599 : cluster 0 pgmap v630: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[21467]: cluster 2024-04-06T03:37:10.895085+0000 osd.4 (osd.4) 363 : cluster 0 5.1f scrub starts 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[21467]: cluster 2024-04-06T03:37:10.899526+0000 osd.4 (osd.4) 364 : cluster 0 5.1f scrub ok 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[29221]: audit 2024-04-06T03:37:09.788064+0000 mgr.y (mgr.24368) 598 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[29221]: cluster 2024-04-06T03:37:10.012686+0000 osd.2 (osd.2) 307 : cluster 0 2.c scrub starts 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[29221]: cluster 2024-04-06T03:37:10.014007+0000 osd.2 (osd.2) 308 : cluster 0 2.c scrub ok 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[29221]: cluster 2024-04-06T03:37:10.333525+0000 mgr.y (mgr.24368) 599 : cluster 0 pgmap v630: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[29221]: cluster 2024-04-06T03:37:10.895085+0000 osd.4 (osd.4) 363 : cluster 0 5.1f scrub starts 2024-04-06T03:37:11.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:11 smithi050 bash[29221]: cluster 2024-04-06T03:37:10.899526+0000 osd.4 (osd.4) 364 : cluster 0 5.1f scrub ok 2024-04-06T03:37:11.993 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:11.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:12.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:12.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:12.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:12.256 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:12.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:12.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:12.524 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:12.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:12.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:12.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[21467]: cluster 2024-04-06T03:37:10.999827+0000 osd.2 (osd.2) 309 : cluster 0 6.f scrub starts 2024-04-06T03:37:12.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.095967+0000 osd.2 (osd.2) 310 : cluster 0 6.f scrub ok 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.473553+0000 osd.7 (osd.7) 405 : cluster 0 3.f scrub starts 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.478267+0000 osd.0 (osd.0) 364 : cluster 0 6.d scrub starts 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.478918+0000 osd.7 (osd.7) 406 : cluster 0 3.f scrub ok 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.636564+0000 osd.0 (osd.0) 365 : cluster 0 6.d scrub ok 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[29221]: cluster 2024-04-06T03:37:10.999827+0000 osd.2 (osd.2) 309 : cluster 0 6.f scrub starts 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.095967+0000 osd.2 (osd.2) 310 : cluster 0 6.f scrub ok 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.473553+0000 osd.7 (osd.7) 405 : cluster 0 3.f scrub starts 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.478267+0000 osd.0 (osd.0) 364 : cluster 0 6.d scrub starts 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.478918+0000 osd.7 (osd.7) 406 : cluster 0 3.f scrub ok 2024-04-06T03:37:12.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:12 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.636564+0000 osd.0 (osd.0) 365 : cluster 0 6.d scrub ok 2024-04-06T03:37:12.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:12.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:12.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:12 smithi153 bash[23852]: cluster 2024-04-06T03:37:10.999827+0000 osd.2 (osd.2) 309 : cluster 0 6.f scrub starts 2024-04-06T03:37:12.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:12 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.095967+0000 osd.2 (osd.2) 310 : cluster 0 6.f scrub ok 2024-04-06T03:37:12.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:12 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.473553+0000 osd.7 (osd.7) 405 : cluster 0 3.f scrub starts 2024-04-06T03:37:12.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:12 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.478267+0000 osd.0 (osd.0) 364 : cluster 0 6.d scrub starts 2024-04-06T03:37:12.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:12 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.478918+0000 osd.7 (osd.7) 406 : cluster 0 3.f scrub ok 2024-04-06T03:37:12.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:12 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.636564+0000 osd.0 (osd.0) 365 : cluster 0 6.d scrub ok 2024-04-06T03:37:12.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:12.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:12.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:12.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:13.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:13.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:13.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:13.206 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:13.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:13.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:13.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:13.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:13.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:13.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:13.599 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:13.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:13.728 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:13.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.971762+0000 osd.2 (osd.2) 311 : cluster 0 2.e scrub starts 2024-04-06T03:37:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:11.975259+0000 osd.2 (osd.2) 312 : cluster 0 2.e scrub ok 2024-04-06T03:37:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.084138+0000 osd.3 (osd.3) 390 : cluster 0 6.b scrub starts 2024-04-06T03:37:13.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.205861+0000 osd.3 (osd.3) 391 : cluster 0 6.b scrub ok 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.334190+0000 mgr.y (mgr.24368) 600 : cluster 0 pgmap v631: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 875 op/s 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.439197+0000 osd.7 (osd.7) 407 : cluster 0 3.16 scrub starts 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.448280+0000 osd.7 (osd.7) 408 : cluster 0 3.16 scrub ok 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.486157+0000 osd.0 (osd.0) 366 : cluster 0 3.17 scrub starts 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.501034+0000 osd.0 (osd.0) 367 : cluster 0 3.17 scrub ok 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.971762+0000 osd.2 (osd.2) 311 : cluster 0 2.e scrub starts 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:11.975259+0000 osd.2 (osd.2) 312 : cluster 0 2.e scrub ok 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.084138+0000 osd.3 (osd.3) 390 : cluster 0 6.b scrub starts 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.205861+0000 osd.3 (osd.3) 391 : cluster 0 6.b scrub ok 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.334190+0000 mgr.y (mgr.24368) 600 : cluster 0 pgmap v631: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 875 op/s 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.439197+0000 osd.7 (osd.7) 407 : cluster 0 3.16 scrub starts 2024-04-06T03:37:13.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.448280+0000 osd.7 (osd.7) 408 : cluster 0 3.16 scrub ok 2024-04-06T03:37:13.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.486157+0000 osd.0 (osd.0) 366 : cluster 0 3.17 scrub starts 2024-04-06T03:37:13.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:13 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.501034+0000 osd.0 (osd.0) 367 : cluster 0 3.17 scrub ok 2024-04-06T03:37:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.971762+0000 osd.2 (osd.2) 311 : cluster 0 2.e scrub starts 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:11.975259+0000 osd.2 (osd.2) 312 : cluster 0 2.e scrub ok 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.084138+0000 osd.3 (osd.3) 390 : cluster 0 6.b scrub starts 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.205861+0000 osd.3 (osd.3) 391 : cluster 0 6.b scrub ok 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.334190+0000 mgr.y (mgr.24368) 600 : cluster 0 pgmap v631: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 875 op/s 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.439197+0000 osd.7 (osd.7) 407 : cluster 0 3.16 scrub starts 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.448280+0000 osd.7 (osd.7) 408 : cluster 0 3.16 scrub ok 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.486157+0000 osd.0 (osd.0) 366 : cluster 0 3.17 scrub starts 2024-04-06T03:37:13.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:13 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.501034+0000 osd.0 (osd.0) 367 : cluster 0 3.17 scrub ok 2024-04-06T03:37:13.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:13.862 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:13.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:13.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:13.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:14.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:14.118 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:14.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:14.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:14.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:14.303 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:37:14.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:14.391 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:14.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:14.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:14.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.952983+0000 osd.2 (osd.2) 313 : cluster 0 2.1 scrub starts 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:12.956567+0000 osd.2 (osd.2) 314 : cluster 0 2.1 scrub ok 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:13.061056+0000 osd.3 (osd.3) 392 : cluster 0 6.6 scrub starts 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:13.078866+0000 osd.3 (osd.3) 393 : cluster 0 6.6 scrub ok 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:13.425700+0000 osd.7 (osd.7) 409 : cluster 0 5.19 scrub starts 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:13.435638+0000 osd.0 (osd.0) 368 : cluster 0 2.13 scrub starts 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:13.436047+0000 osd.7 (osd.7) 410 : cluster 0 5.19 scrub ok 2024-04-06T03:37:14.632 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:14 smithi153 bash[23852]: cluster 2024-04-06T03:37:13.437050+0000 osd.0 (osd.0) 369 : cluster 0 2.13 scrub ok 2024-04-06T03:37:14.646 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.952983+0000 osd.2 (osd.2) 313 : cluster 0 2.1 scrub starts 2024-04-06T03:37:14.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:12.956567+0000 osd.2 (osd.2) 314 : cluster 0 2.1 scrub ok 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:13.061056+0000 osd.3 (osd.3) 392 : cluster 0 6.6 scrub starts 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:13.078866+0000 osd.3 (osd.3) 393 : cluster 0 6.6 scrub ok 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:13.425700+0000 osd.7 (osd.7) 409 : cluster 0 5.19 scrub starts 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:13.435638+0000 osd.0 (osd.0) 368 : cluster 0 2.13 scrub starts 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:13.436047+0000 osd.7 (osd.7) 410 : cluster 0 5.19 scrub ok 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[21467]: cluster 2024-04-06T03:37:13.437050+0000 osd.0 (osd.0) 369 : cluster 0 2.13 scrub ok 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.952983+0000 osd.2 (osd.2) 313 : cluster 0 2.1 scrub starts 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:12.956567+0000 osd.2 (osd.2) 314 : cluster 0 2.1 scrub ok 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:13.061056+0000 osd.3 (osd.3) 392 : cluster 0 6.6 scrub starts 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:13.078866+0000 osd.3 (osd.3) 393 : cluster 0 6.6 scrub ok 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:13.425700+0000 osd.7 (osd.7) 409 : cluster 0 5.19 scrub starts 2024-04-06T03:37:14.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:13.435638+0000 osd.0 (osd.0) 368 : cluster 0 2.13 scrub starts 2024-04-06T03:37:14.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:13.436047+0000 osd.7 (osd.7) 410 : cluster 0 5.19 scrub ok 2024-04-06T03:37:14.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:14 smithi050 bash[29221]: cluster 2024-04-06T03:37:13.437050+0000 osd.0 (osd.0) 369 : cluster 0 2.13 scrub ok 2024-04-06T03:37:14.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:14.775 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:14.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:14.920 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:14.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:15.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:15.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:15.166 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:15.184 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:15.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:15.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:15.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:15.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:15.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:15.576 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:15.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.018143+0000 osd.3 (osd.3) 394 : cluster 0 6.7 scrub starts 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.116648+0000 osd.3 (osd.3) 395 : cluster 0 6.7 scrub ok 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.334730+0000 mgr.y (mgr.24368) 601 : cluster 0 pgmap v632: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.376603+0000 osd.7 (osd.7) 411 : cluster 0 2.b scrub starts 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.380230+0000 osd.7 (osd.7) 412 : cluster 0 2.b scrub ok 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.893513+0000 osd.4 (osd.4) 365 : cluster 0 5.f scrub starts 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[29221]: cluster 2024-04-06T03:37:14.900925+0000 osd.4 (osd.4) 366 : cluster 0 5.f scrub ok 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.018143+0000 osd.3 (osd.3) 394 : cluster 0 6.7 scrub starts 2024-04-06T03:37:15.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.116648+0000 osd.3 (osd.3) 395 : cluster 0 6.7 scrub ok 2024-04-06T03:37:15.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.334730+0000 mgr.y (mgr.24368) 601 : cluster 0 pgmap v632: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:37:15.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.376603+0000 osd.7 (osd.7) 411 : cluster 0 2.b scrub starts 2024-04-06T03:37:15.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.380230+0000 osd.7 (osd.7) 412 : cluster 0 2.b scrub ok 2024-04-06T03:37:15.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.893513+0000 osd.4 (osd.4) 365 : cluster 0 5.f scrub starts 2024-04-06T03:37:15.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:15 smithi050 bash[21467]: cluster 2024-04-06T03:37:14.900925+0000 osd.4 (osd.4) 366 : cluster 0 5.f scrub ok 2024-04-06T03:37:15.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:15.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:15.819 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.018143+0000 osd.3 (osd.3) 394 : cluster 0 6.7 scrub starts 2024-04-06T03:37:15.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.116648+0000 osd.3 (osd.3) 395 : cluster 0 6.7 scrub ok 2024-04-06T03:37:15.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.334730+0000 mgr.y (mgr.24368) 601 : cluster 0 pgmap v632: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:37:15.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.376603+0000 osd.7 (osd.7) 411 : cluster 0 2.b scrub starts 2024-04-06T03:37:15.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.380230+0000 osd.7 (osd.7) 412 : cluster 0 2.b scrub ok 2024-04-06T03:37:15.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.893513+0000 osd.4 (osd.4) 365 : cluster 0 5.f scrub starts 2024-04-06T03:37:15.820 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:15 smithi153 bash[23852]: cluster 2024-04-06T03:37:14.900925+0000 osd.4 (osd.4) 366 : cluster 0 5.f scrub ok 2024-04-06T03:37:15.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:15.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:15.969 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:15.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:16.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:16.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:16.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:16.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:16.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:16.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:16.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:16.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:16.614 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:16.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:16.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[21467]: cluster 2024-04-06T03:37:15.087366+0000 osd.3 (osd.3) 396 : cluster 0 6.c scrub starts 2024-04-06T03:37:16.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[21467]: cluster 2024-04-06T03:37:15.095028+0000 osd.3 (osd.3) 397 : cluster 0 6.c scrub ok 2024-04-06T03:37:16.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.077083+0000 osd.1 (osd.1) 309 : cluster 0 2.15 scrub starts 2024-04-06T03:37:16.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.077981+0000 osd.1 (osd.1) 310 : cluster 0 2.15 scrub ok 2024-04-06T03:37:16.652 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[29221]: cluster 2024-04-06T03:37:15.087366+0000 osd.3 (osd.3) 396 : cluster 0 6.c scrub starts 2024-04-06T03:37:16.652 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[29221]: cluster 2024-04-06T03:37:15.095028+0000 osd.3 (osd.3) 397 : cluster 0 6.c scrub ok 2024-04-06T03:37:16.652 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.077083+0000 osd.1 (osd.1) 309 : cluster 0 2.15 scrub starts 2024-04-06T03:37:16.652 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:16 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.077981+0000 osd.1 (osd.1) 310 : cluster 0 2.15 scrub ok 2024-04-06T03:37:16.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:16.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:16.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:16.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:16 smithi153 bash[23852]: cluster 2024-04-06T03:37:15.087366+0000 osd.3 (osd.3) 396 : cluster 0 6.c scrub starts 2024-04-06T03:37:16.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:16 smithi153 bash[23852]: cluster 2024-04-06T03:37:15.095028+0000 osd.3 (osd.3) 397 : cluster 0 6.c scrub ok 2024-04-06T03:37:16.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:16 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.077083+0000 osd.1 (osd.1) 309 : cluster 0 2.15 scrub starts 2024-04-06T03:37:16.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:16 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.077981+0000 osd.1 (osd.1) 310 : cluster 0 2.15 scrub ok 2024-04-06T03:37:16.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:16.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:16.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:17.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:17.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:17.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.188 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:37:17.193 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:17.231 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:37:17.270 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:37:17.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:17.324 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:37:17.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:17.369 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:37:17.411 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:37:17.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.450 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:37:17.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:17.501 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:37:17.511 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:37:17.512 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:37:17.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.569 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:37:17.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:17.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:17.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.035480+0000 osd.3 (osd.3) 398 : cluster 0 5.13 scrub starts 2024-04-06T03:37:17.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.037894+0000 osd.3 (osd.3) 399 : cluster 0 5.13 scrub ok 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.335495+0000 mgr.y (mgr.24368) 602 : cluster 0 pgmap v633: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.839158+0000 osd.4 (osd.4) 367 : cluster 0 2.1a scrub starts 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:16.840091+0000 osd.4 (osd.4) 368 : cluster 0 2.1a scrub ok 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: audit 2024-04-06T03:37:17.085198+0000 mon.b (mon.1) 41 : audit 0 from='client.? 172.21.15.50:0/1690164481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.105736+0000 osd.1 (osd.1) 311 : cluster 0 5.1d scrub starts 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.116914+0000 osd.1 (osd.1) 312 : cluster 0 5.1d scrub ok 2024-04-06T03:37:17.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:17 smithi153 bash[23852]: audit 2024-04-06T03:37:17.357736+0000 mon.a (mon.0) 1420 : audit 1 from='client.? 172.21.15.50:0/3224370358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:37:17.725 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:17.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:17.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.035480+0000 osd.3 (osd.3) 398 : cluster 0 5.13 scrub starts 2024-04-06T03:37:17.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.037894+0000 osd.3 (osd.3) 399 : cluster 0 5.13 scrub ok 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.335495+0000 mgr.y (mgr.24368) 602 : cluster 0 pgmap v633: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.839158+0000 osd.4 (osd.4) 367 : cluster 0 2.1a scrub starts 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:16.840091+0000 osd.4 (osd.4) 368 : cluster 0 2.1a scrub ok 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: audit 2024-04-06T03:37:17.085198+0000 mon.b (mon.1) 41 : audit 0 from='client.? 172.21.15.50:0/1690164481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.105736+0000 osd.1 (osd.1) 311 : cluster 0 5.1d scrub starts 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.116914+0000 osd.1 (osd.1) 312 : cluster 0 5.1d scrub ok 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[21467]: audit 2024-04-06T03:37:17.357736+0000 mon.a (mon.0) 1420 : audit 1 from='client.? 172.21.15.50:0/3224370358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.035480+0000 osd.3 (osd.3) 398 : cluster 0 5.13 scrub starts 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.037894+0000 osd.3 (osd.3) 399 : cluster 0 5.13 scrub ok 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.335495+0000 mgr.y (mgr.24368) 602 : cluster 0 pgmap v633: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.839158+0000 osd.4 (osd.4) 367 : cluster 0 2.1a scrub starts 2024-04-06T03:37:17.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:16.840091+0000 osd.4 (osd.4) 368 : cluster 0 2.1a scrub ok 2024-04-06T03:37:17.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: audit 2024-04-06T03:37:17.085198+0000 mon.b (mon.1) 41 : audit 0 from='client.? 172.21.15.50:0/1690164481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:37:17.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.105736+0000 osd.1 (osd.1) 311 : cluster 0 5.1d scrub starts 2024-04-06T03:37:17.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.116914+0000 osd.1 (osd.1) 312 : cluster 0 5.1d scrub ok 2024-04-06T03:37:17.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:17 smithi050 bash[29221]: audit 2024-04-06T03:37:17.357736+0000 mon.a (mon.0) 1420 : audit 1 from='client.? 172.21.15.50:0/3224370358' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:37:17.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:17.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:17.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.985 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:17.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:18.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:18.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:18.219 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:18.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:18.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:18.586 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-06T03:37:18.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:18.657 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:18.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.378812+0000 osd.7 (osd.7) 413 : cluster 0 2.5 scrub starts 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.382253+0000 osd.7 (osd.7) 414 : cluster 0 2.5 scrub ok 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.455177+0000 osd.0 (osd.0) 370 : cluster 0 6.9 scrub starts 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.547288+0000 osd.0 (osd.0) 371 : cluster 0 6.9 scrub ok 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.561527+0000 mon.a (mon.0) 1421 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.561563+0000 mon.a (mon.0) 1422 : cluster 1 Cluster is now healthy 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: audit 2024-04-06T03:37:17.563768+0000 mon.a (mon.0) 1423 : audit 1 from='client.? 172.21.15.50:0/3224370358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: cluster 2024-04-06T03:37:17.572138+0000 mon.a (mon.0) 1424 : cluster 0 osdmap e265: 8 total, 8 up, 8 in 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: audit 2024-04-06T03:37:17.658461+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.50:0/1791008846' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:37:18.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:18 smithi153 bash[23852]: audit 2024-04-06T03:37:17.659024+0000 mon.a (mon.0) 1425 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:37:18.780 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:18.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:18.908 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:18.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:18.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.378812+0000 osd.7 (osd.7) 413 : cluster 0 2.5 scrub starts 2024-04-06T03:37:18.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.382253+0000 osd.7 (osd.7) 414 : cluster 0 2.5 scrub ok 2024-04-06T03:37:18.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.455177+0000 osd.0 (osd.0) 370 : cluster 0 6.9 scrub starts 2024-04-06T03:37:18.922 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.547288+0000 osd.0 (osd.0) 371 : cluster 0 6.9 scrub ok 2024-04-06T03:37:19.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:19.040 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:19.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:19.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:19.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:19.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.561527+0000 mon.a (mon.0) 1421 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:37:19.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.561563+0000 mon.a (mon.0) 1422 : cluster 1 Cluster is now healthy 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: audit 2024-04-06T03:37:17.563768+0000 mon.a (mon.0) 1423 : audit 1 from='client.? 172.21.15.50:0/3224370358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: cluster 2024-04-06T03:37:17.572138+0000 mon.a (mon.0) 1424 : cluster 0 osdmap e265: 8 total, 8 up, 8 in 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: audit 2024-04-06T03:37:17.658461+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.50:0/1791008846' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[21467]: audit 2024-04-06T03:37:17.659024+0000 mon.a (mon.0) 1425 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.378812+0000 osd.7 (osd.7) 413 : cluster 0 2.5 scrub starts 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.382253+0000 osd.7 (osd.7) 414 : cluster 0 2.5 scrub ok 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.455177+0000 osd.0 (osd.0) 370 : cluster 0 6.9 scrub starts 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.547288+0000 osd.0 (osd.0) 371 : cluster 0 6.9 scrub ok 2024-04-06T03:37:19.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.561527+0000 mon.a (mon.0) 1421 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:37:19.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.561563+0000 mon.a (mon.0) 1422 : cluster 1 Cluster is now healthy 2024-04-06T03:37:19.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: audit 2024-04-06T03:37:17.563768+0000 mon.a (mon.0) 1423 : audit 1 from='client.? 172.21.15.50:0/3224370358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:37:19.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: cluster 2024-04-06T03:37:17.572138+0000 mon.a (mon.0) 1424 : cluster 0 osdmap e265: 8 total, 8 up, 8 in 2024-04-06T03:37:19.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: audit 2024-04-06T03:37:17.658461+0000 mon.c (mon.2) 80 : audit 1 from='client.? 172.21.15.50:0/1791008846' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:37:19.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:18 smithi050 bash[29221]: audit 2024-04-06T03:37:17.659024+0000 mon.a (mon.0) 1425 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:37:19.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:19.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:19.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:19.429 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:19.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:19.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:19.580 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:37:19.580 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:37:19.580 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:37:19.580 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:37:19.637 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:37:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:37:19.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:19.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:19.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.335952+0000 mgr.y (mgr.24368) 603 : cluster 0 pgmap v635: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-04-06T03:37:19.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.370184+0000 osd.7 (osd.7) 415 : cluster 0 5.4 scrub starts 2024-04-06T03:37:19.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.371999+0000 osd.7 (osd.7) 416 : cluster 0 5.4 scrub ok 2024-04-06T03:37:19.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.486778+0000 osd.0 (osd.0) 372 : cluster 0 5.1e scrub starts 2024-04-06T03:37:19.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.488025+0000 osd.0 (osd.0) 373 : cluster 0 5.1e scrub ok 2024-04-06T03:37:19.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: audit 2024-04-06T03:37:18.577978+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:37:19.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.581034+0000 mon.a (mon.0) 1427 : cluster 0 osdmap e266: 8 total, 8 up, 8 in 2024-04-06T03:37:19.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.842226+0000 osd.4 (osd.4) 369 : cluster 0 3.1d scrub starts 2024-04-06T03:37:19.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[23852]: cluster 2024-04-06T03:37:18.846474+0000 osd.4 (osd.4) 370 : cluster 0 3.1d scrub ok 2024-04-06T03:37:19.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:19.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.335952+0000 mgr.y (mgr.24368) 603 : cluster 0 pgmap v635: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-04-06T03:37:19.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.370184+0000 osd.7 (osd.7) 415 : cluster 0 5.4 scrub starts 2024-04-06T03:37:19.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.371999+0000 osd.7 (osd.7) 416 : cluster 0 5.4 scrub ok 2024-04-06T03:37:19.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.486778+0000 osd.0 (osd.0) 372 : cluster 0 5.1e scrub starts 2024-04-06T03:37:19.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.488025+0000 osd.0 (osd.0) 373 : cluster 0 5.1e scrub ok 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: audit 2024-04-06T03:37:18.577978+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.581034+0000 mon.a (mon.0) 1427 : cluster 0 osdmap e266: 8 total, 8 up, 8 in 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.842226+0000 osd.4 (osd.4) 369 : cluster 0 3.1d scrub starts 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[21467]: cluster 2024-04-06T03:37:18.846474+0000 osd.4 (osd.4) 370 : cluster 0 3.1d scrub ok 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.335952+0000 mgr.y (mgr.24368) 603 : cluster 0 pgmap v635: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.370184+0000 osd.7 (osd.7) 415 : cluster 0 5.4 scrub starts 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.371999+0000 osd.7 (osd.7) 416 : cluster 0 5.4 scrub ok 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.486778+0000 osd.0 (osd.0) 372 : cluster 0 5.1e scrub starts 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.488025+0000 osd.0 (osd.0) 373 : cluster 0 5.1e scrub ok 2024-04-06T03:37:19.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: audit 2024-04-06T03:37:18.577978+0000 mon.a (mon.0) 1426 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:37:19.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.581034+0000 mon.a (mon.0) 1427 : cluster 0 osdmap e266: 8 total, 8 up, 8 in 2024-04-06T03:37:19.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.842226+0000 osd.4 (osd.4) 369 : cluster 0 3.1d scrub starts 2024-04-06T03:37:19.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:19 smithi050 bash[29221]: cluster 2024-04-06T03:37:18.846474+0000 osd.4 (osd.4) 370 : cluster 0 3.1d scrub ok 2024-04-06T03:37:19.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:19.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:19.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:20.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:20.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:37:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:37:20.062 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:20.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:20.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:20.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:20.316 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:20.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:20.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:20.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:20.527 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-06T03:37:20.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:20.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:20.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:20.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:20.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:20.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:20.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:20.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:20 smithi153 bash[23852]: cluster 2024-04-06T03:37:19.446765+0000 osd.0 (osd.0) 374 : cluster 0 2.4 scrub starts 2024-04-06T03:37:20.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:20 smithi153 bash[23852]: cluster 2024-04-06T03:37:19.458429+0000 osd.0 (osd.0) 375 : cluster 0 2.4 scrub ok 2024-04-06T03:37:20.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:20 smithi153 bash[23852]: audit 2024-04-06T03:37:19.690689+0000 mon.a (mon.0) 1428 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:20.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:20 smithi153 bash[23852]: audit 2024-04-06T03:37:19.691547+0000 mon.a (mon.0) 1429 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:20.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:20 smithi153 bash[23852]: cluster 2024-04-06T03:37:20.271464+0000 osd.1 (osd.1) 313 : cluster 0 6.1 scrub starts 2024-04-06T03:37:20.981 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:20.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:20.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[21467]: cluster 2024-04-06T03:37:19.446765+0000 osd.0 (osd.0) 374 : cluster 0 2.4 scrub starts 2024-04-06T03:37:20.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[21467]: cluster 2024-04-06T03:37:19.458429+0000 osd.0 (osd.0) 375 : cluster 0 2.4 scrub ok 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[21467]: audit 2024-04-06T03:37:19.690689+0000 mon.a (mon.0) 1428 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[21467]: audit 2024-04-06T03:37:19.691547+0000 mon.a (mon.0) 1429 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[21467]: cluster 2024-04-06T03:37:20.271464+0000 osd.1 (osd.1) 313 : cluster 0 6.1 scrub starts 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[29221]: cluster 2024-04-06T03:37:19.446765+0000 osd.0 (osd.0) 374 : cluster 0 2.4 scrub starts 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[29221]: cluster 2024-04-06T03:37:19.458429+0000 osd.0 (osd.0) 375 : cluster 0 2.4 scrub ok 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[29221]: audit 2024-04-06T03:37:19.690689+0000 mon.a (mon.0) 1428 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[29221]: audit 2024-04-06T03:37:19.691547+0000 mon.a (mon.0) 1429 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:20.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:20 smithi050 bash[29221]: cluster 2024-04-06T03:37:20.271464+0000 osd.1 (osd.1) 313 : cluster 0 6.1 scrub starts 2024-04-06T03:37:21.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:21.112 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:21.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:21.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:21.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:21.477 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:21.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:21.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:21.629 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:21.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:21.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:21.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:21 smithi153 bash[23852]: audit 2024-04-06T03:37:19.796104+0000 mgr.y (mgr.24368) 604 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:21.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:21 smithi153 bash[23852]: cluster 2024-04-06T03:37:20.149931+0000 osd.2 (osd.2) 315 : cluster 0 2.16 scrub starts 2024-04-06T03:37:21.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:21 smithi153 bash[23852]: cluster 2024-04-06T03:37:20.152285+0000 osd.2 (osd.2) 316 : cluster 0 2.16 scrub ok 2024-04-06T03:37:21.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:21 smithi153 bash[23852]: cluster 2024-04-06T03:37:20.336877+0000 mgr.y (mgr.24368) 605 : cluster 0 pgmap v637: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 915 op/s 2024-04-06T03:37:21.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:21 smithi153 bash[23852]: cluster 2024-04-06T03:37:20.403255+0000 osd.1 (osd.1) 314 : cluster 0 6.1 scrub ok 2024-04-06T03:37:21.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:21.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 3 1 2024-04-06T03:37:21.894 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:21.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:21.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[21467]: audit 2024-04-06T03:37:19.796104+0000 mgr.y (mgr.24368) 604 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[21467]: cluster 2024-04-06T03:37:20.149931+0000 osd.2 (osd.2) 315 : cluster 0 2.16 scrub starts 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[21467]: cluster 2024-04-06T03:37:20.152285+0000 osd.2 (osd.2) 316 : cluster 0 2.16 scrub ok 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[21467]: cluster 2024-04-06T03:37:20.336877+0000 mgr.y (mgr.24368) 605 : cluster 0 pgmap v637: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 915 op/s 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[21467]: cluster 2024-04-06T03:37:20.403255+0000 osd.1 (osd.1) 314 : cluster 0 6.1 scrub ok 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[29221]: audit 2024-04-06T03:37:19.796104+0000 mgr.y (mgr.24368) 604 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[29221]: cluster 2024-04-06T03:37:20.149931+0000 osd.2 (osd.2) 315 : cluster 0 2.16 scrub starts 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[29221]: cluster 2024-04-06T03:37:20.152285+0000 osd.2 (osd.2) 316 : cluster 0 2.16 scrub ok 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[29221]: cluster 2024-04-06T03:37:20.336877+0000 mgr.y (mgr.24368) 605 : cluster 0 pgmap v637: 148 pgs: 148 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 915 op/s 2024-04-06T03:37:21.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:21 smithi050 bash[29221]: cluster 2024-04-06T03:37:20.403255+0000 osd.1 (osd.1) 314 : cluster 0 6.1 scrub ok 2024-04-06T03:37:22.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:22.025 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:22.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:22.149 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:22.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:22.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:22.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:22.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:22.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:22.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:22.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:22.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:22.704 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:22.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:22.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:22.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:22.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:23.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:23.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:23.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:23.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:23.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:23.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:23.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:23.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:37:23.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.791 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:23.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:37:23.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[21467]: cluster 2024-04-06T03:37:22.291835+0000 osd.1 (osd.1) 315 : cluster 0 2.9 scrub starts 2024-04-06T03:37:23.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[21467]: cluster 2024-04-06T03:37:22.300710+0000 osd.1 (osd.1) 316 : cluster 0 2.9 scrub ok 2024-04-06T03:37:23.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[21467]: cluster 2024-04-06T03:37:22.337520+0000 mgr.y (mgr.24368) 606 : cluster 0 pgmap v638: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:37:23.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[21467]: cluster 2024-04-06T03:37:22.536269+0000 osd.0 (osd.0) 376 : cluster 0 3.10 scrub starts 2024-04-06T03:37:23.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[21467]: cluster 2024-04-06T03:37:22.538319+0000 osd.0 (osd.0) 377 : cluster 0 3.10 scrub ok 2024-04-06T03:37:23.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[29221]: cluster 2024-04-06T03:37:22.291835+0000 osd.1 (osd.1) 315 : cluster 0 2.9 scrub starts 2024-04-06T03:37:23.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[29221]: cluster 2024-04-06T03:37:22.300710+0000 osd.1 (osd.1) 316 : cluster 0 2.9 scrub ok 2024-04-06T03:37:23.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[29221]: cluster 2024-04-06T03:37:22.337520+0000 mgr.y (mgr.24368) 606 : cluster 0 pgmap v638: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:37:23.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[29221]: cluster 2024-04-06T03:37:22.536269+0000 osd.0 (osd.0) 376 : cluster 0 3.10 scrub starts 2024-04-06T03:37:23.905 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:23 smithi050 bash[29221]: cluster 2024-04-06T03:37:22.538319+0000 osd.0 (osd.0) 377 : cluster 0 3.10 scrub ok 2024-04-06T03:37:23.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:23.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:24.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:24.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:23 smithi153 bash[23852]: cluster 2024-04-06T03:37:22.291835+0000 osd.1 (osd.1) 315 : cluster 0 2.9 scrub starts 2024-04-06T03:37:24.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:23 smithi153 bash[23852]: cluster 2024-04-06T03:37:22.300710+0000 osd.1 (osd.1) 316 : cluster 0 2.9 scrub ok 2024-04-06T03:37:24.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:23 smithi153 bash[23852]: cluster 2024-04-06T03:37:22.337520+0000 mgr.y (mgr.24368) 606 : cluster 0 pgmap v638: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:37:24.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:23 smithi153 bash[23852]: cluster 2024-04-06T03:37:22.536269+0000 osd.0 (osd.0) 376 : cluster 0 3.10 scrub starts 2024-04-06T03:37:24.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:23 smithi153 bash[23852]: cluster 2024-04-06T03:37:22.538319+0000 osd.0 (osd.0) 377 : cluster 0 3.10 scrub ok 2024-04-06T03:37:24.065 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:24.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:24.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:24.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:24.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:24.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:24.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:24.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:24.596 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:24.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:24.737 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:24.841 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:24.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:24.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:24.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:24.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:24 smithi153 bash[23852]: cluster 2024-04-06T03:37:23.133651+0000 osd.2 (osd.2) 317 : cluster 0 3.0 scrub starts 2024-04-06T03:37:24.977 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:24 smithi153 bash[23852]: cluster 2024-04-06T03:37:23.140067+0000 osd.2 (osd.2) 318 : cluster 0 3.0 scrub ok 2024-04-06T03:37:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:24 smithi050 bash[21467]: cluster 2024-04-06T03:37:23.133651+0000 osd.2 (osd.2) 317 : cluster 0 3.0 scrub starts 2024-04-06T03:37:24.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:24 smithi050 bash[21467]: cluster 2024-04-06T03:37:23.140067+0000 osd.2 (osd.2) 318 : cluster 0 3.0 scrub ok 2024-04-06T03:37:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:24 smithi050 bash[29221]: cluster 2024-04-06T03:37:23.133651+0000 osd.2 (osd.2) 317 : cluster 0 3.0 scrub starts 2024-04-06T03:37:24.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:24 smithi050 bash[29221]: cluster 2024-04-06T03:37:23.140067+0000 osd.2 (osd.2) 318 : cluster 0 3.0 scrub ok 2024-04-06T03:37:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:25.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:25.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:25.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:25.347 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:37:25.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:25.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:25.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:25.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:25.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:25.669 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:25.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.320653+0000 osd.1 (osd.1) 317 : cluster 0 2.2 scrub starts 2024-04-06T03:37:25.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.327229+0000 osd.1 (osd.1) 318 : cluster 0 2.2 scrub ok 2024-04-06T03:37:25.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.338186+0000 mgr.y (mgr.24368) 607 : cluster 0 pgmap v639: 148 pgs: 148 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-04-06T03:37:25.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.551904+0000 osd.0 (osd.0) 378 : cluster 0 6.a scrub starts 2024-04-06T03:37:25.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.601124+0000 osd.0 (osd.0) 379 : cluster 0 6.a scrub ok 2024-04-06T03:37:25.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.793877+0000 osd.4 (osd.4) 371 : cluster 0 6.1a scrub starts 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[21467]: cluster 2024-04-06T03:37:24.823926+0000 osd.4 (osd.4) 372 : cluster 0 6.1a scrub ok 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.320653+0000 osd.1 (osd.1) 317 : cluster 0 2.2 scrub starts 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.327229+0000 osd.1 (osd.1) 318 : cluster 0 2.2 scrub ok 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.338186+0000 mgr.y (mgr.24368) 607 : cluster 0 pgmap v639: 148 pgs: 148 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.551904+0000 osd.0 (osd.0) 378 : cluster 0 6.a scrub starts 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.601124+0000 osd.0 (osd.0) 379 : cluster 0 6.a scrub ok 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.793877+0000 osd.4 (osd.4) 371 : cluster 0 6.1a scrub starts 2024-04-06T03:37:25.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:25 smithi050 bash[29221]: cluster 2024-04-06T03:37:24.823926+0000 osd.4 (osd.4) 372 : cluster 0 6.1a scrub ok 2024-04-06T03:37:25.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:25.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:25.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:25.936 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:25.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:26.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.320653+0000 osd.1 (osd.1) 317 : cluster 0 2.2 scrub starts 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.327229+0000 osd.1 (osd.1) 318 : cluster 0 2.2 scrub ok 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.338186+0000 mgr.y (mgr.24368) 607 : cluster 0 pgmap v639: 148 pgs: 148 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.551904+0000 osd.0 (osd.0) 378 : cluster 0 6.a scrub starts 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.601124+0000 osd.0 (osd.0) 379 : cluster 0 6.a scrub ok 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.793877+0000 osd.4 (osd.4) 371 : cluster 0 6.1a scrub starts 2024-04-06T03:37:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:25 smithi153 bash[23852]: cluster 2024-04-06T03:37:24.823926+0000 osd.4 (osd.4) 372 : cluster 0 6.1a scrub ok 2024-04-06T03:37:26.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:26.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:26.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:26.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:26.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:26.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:26.453 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:26.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:26.578 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:26.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:26.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:26.721 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:26.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:26.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:26.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:26.948 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:26 smithi153 bash[23852]: cluster 2024-04-06T03:37:25.751242+0000 osd.4 (osd.4) 373 : cluster 0 2.18 scrub starts 2024-04-06T03:37:26.948 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:26 smithi153 bash[23852]: cluster 2024-04-06T03:37:25.752937+0000 osd.4 (osd.4) 374 : cluster 0 2.18 scrub ok 2024-04-06T03:37:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:26.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:26.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:26 smithi050 bash[21467]: cluster 2024-04-06T03:37:25.751242+0000 osd.4 (osd.4) 373 : cluster 0 2.18 scrub starts 2024-04-06T03:37:26.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:26 smithi050 bash[21467]: cluster 2024-04-06T03:37:25.752937+0000 osd.4 (osd.4) 374 : cluster 0 2.18 scrub ok 2024-04-06T03:37:26.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:26 smithi050 bash[29221]: cluster 2024-04-06T03:37:25.751242+0000 osd.4 (osd.4) 373 : cluster 0 2.18 scrub starts 2024-04-06T03:37:26.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:26 smithi050 bash[29221]: cluster 2024-04-06T03:37:25.752937+0000 osd.4 (osd.4) 374 : cluster 0 2.18 scrub ok 2024-04-06T03:37:27.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:27.092 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:27.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:27.225 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:27.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.363 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:27.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:27.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:27.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:27.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:27.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:27.625 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:27.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:27.738 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:27.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:27.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:27.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:27.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:27.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:27.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.082737+0000 osd.3 (osd.3) 400 : cluster 0 3.5 deep-scrub starts 2024-04-06T03:37:27.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.091708+0000 osd.3 (osd.3) 401 : cluster 0 3.5 deep-scrub ok 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.125336+0000 osd.2 (osd.2) 319 : cluster 0 6.14 scrub starts 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.130302+0000 osd.2 (osd.2) 320 : cluster 0 6.14 scrub ok 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.339051+0000 mgr.y (mgr.24368) 608 : cluster 0 pgmap v640: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.665123+0000 osd.0 (osd.0) 380 : cluster 0 5.3 scrub starts 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[21467]: cluster 2024-04-06T03:37:26.668258+0000 osd.0 (osd.0) 381 : cluster 0 5.3 scrub ok 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.082737+0000 osd.3 (osd.3) 400 : cluster 0 3.5 deep-scrub starts 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.091708+0000 osd.3 (osd.3) 401 : cluster 0 3.5 deep-scrub ok 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.125336+0000 osd.2 (osd.2) 319 : cluster 0 6.14 scrub starts 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.130302+0000 osd.2 (osd.2) 320 : cluster 0 6.14 scrub ok 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.339051+0000 mgr.y (mgr.24368) 608 : cluster 0 pgmap v640: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:37:27.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.665123+0000 osd.0 (osd.0) 380 : cluster 0 5.3 scrub starts 2024-04-06T03:37:27.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:27 smithi050 bash[29221]: cluster 2024-04-06T03:37:26.668258+0000 osd.0 (osd.0) 381 : cluster 0 5.3 scrub ok 2024-04-06T03:37:28.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.082737+0000 osd.3 (osd.3) 400 : cluster 0 3.5 deep-scrub starts 2024-04-06T03:37:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.091708+0000 osd.3 (osd.3) 401 : cluster 0 3.5 deep-scrub ok 2024-04-06T03:37:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.125336+0000 osd.2 (osd.2) 319 : cluster 0 6.14 scrub starts 2024-04-06T03:37:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.130302+0000 osd.2 (osd.2) 320 : cluster 0 6.14 scrub ok 2024-04-06T03:37:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.339051+0000 mgr.y (mgr.24368) 608 : cluster 0 pgmap v640: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:37:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.665123+0000 osd.0 (osd.0) 380 : cluster 0 5.3 scrub starts 2024-04-06T03:37:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:27 smithi153 bash[23852]: cluster 2024-04-06T03:37:26.668258+0000 osd.0 (osd.0) 381 : cluster 0 5.3 scrub ok 2024-04-06T03:37:28.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:28.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:28.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:28.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:28.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:28.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:28.396 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:28.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:28.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:28.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:28.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:28.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:28.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:28.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:28.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:28.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[21467]: cluster 2024-04-06T03:37:27.145232+0000 osd.2 (osd.2) 321 : cluster 0 3.4 scrub starts 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[21467]: cluster 2024-04-06T03:37:27.149874+0000 osd.2 (osd.2) 322 : cluster 0 3.4 scrub ok 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[21467]: cluster 2024-04-06T03:37:27.530724+0000 osd.7 (osd.7) 417 : cluster 0 6.5 scrub starts 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[21467]: cluster 2024-04-06T03:37:27.577813+0000 osd.7 (osd.7) 418 : cluster 0 6.5 scrub ok 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[21467]: cluster 2024-04-06T03:37:27.701856+0000 osd.4 (osd.4) 375 : cluster 0 6.2 scrub starts 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[21467]: cluster 2024-04-06T03:37:27.705188+0000 osd.4 (osd.4) 376 : cluster 0 6.2 scrub ok 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[29221]: cluster 2024-04-06T03:37:27.145232+0000 osd.2 (osd.2) 321 : cluster 0 3.4 scrub starts 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[29221]: cluster 2024-04-06T03:37:27.149874+0000 osd.2 (osd.2) 322 : cluster 0 3.4 scrub ok 2024-04-06T03:37:28.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[29221]: cluster 2024-04-06T03:37:27.530724+0000 osd.7 (osd.7) 417 : cluster 0 6.5 scrub starts 2024-04-06T03:37:28.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[29221]: cluster 2024-04-06T03:37:27.577813+0000 osd.7 (osd.7) 418 : cluster 0 6.5 scrub ok 2024-04-06T03:37:28.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[29221]: cluster 2024-04-06T03:37:27.701856+0000 osd.4 (osd.4) 375 : cluster 0 6.2 scrub starts 2024-04-06T03:37:28.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:28 smithi050 bash[29221]: cluster 2024-04-06T03:37:27.705188+0000 osd.4 (osd.4) 376 : cluster 0 6.2 scrub ok 2024-04-06T03:37:29.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:28 smithi153 bash[23852]: cluster 2024-04-06T03:37:27.145232+0000 osd.2 (osd.2) 321 : cluster 0 3.4 scrub starts 2024-04-06T03:37:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:28 smithi153 bash[23852]: cluster 2024-04-06T03:37:27.149874+0000 osd.2 (osd.2) 322 : cluster 0 3.4 scrub ok 2024-04-06T03:37:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:28 smithi153 bash[23852]: cluster 2024-04-06T03:37:27.530724+0000 osd.7 (osd.7) 417 : cluster 0 6.5 scrub starts 2024-04-06T03:37:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:28 smithi153 bash[23852]: cluster 2024-04-06T03:37:27.577813+0000 osd.7 (osd.7) 418 : cluster 0 6.5 scrub ok 2024-04-06T03:37:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:28 smithi153 bash[23852]: cluster 2024-04-06T03:37:27.701856+0000 osd.4 (osd.4) 375 : cluster 0 6.2 scrub starts 2024-04-06T03:37:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:28 smithi153 bash[23852]: cluster 2024-04-06T03:37:27.705188+0000 osd.4 (osd.4) 376 : cluster 0 6.2 scrub ok 2024-04-06T03:37:29.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:29.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:29.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:29.286 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:29.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:29.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:29.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:29.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:29.567 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:37:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:37:29.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:29.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:29.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:29.854 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:29.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:29.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.058262+0000 osd.3 (osd.3) 402 : cluster 0 6.13 scrub starts 2024-04-06T03:37:29.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.115672+0000 osd.2 (osd.2) 323 : cluster 0 5.8 scrub starts 2024-04-06T03:37:29.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.132446+0000 osd.2 (osd.2) 324 : cluster 0 5.8 scrub ok 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.165901+0000 osd.3 (osd.3) 403 : cluster 0 6.13 scrub ok 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.339396+0000 mgr.y (mgr.24368) 609 : cluster 0 pgmap v641: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.543868+0000 osd.0 (osd.0) 382 : cluster 0 6.10 scrub starts 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.564742+0000 osd.0 (osd.0) 383 : cluster 0 6.10 scrub ok 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.608022+0000 osd.7 (osd.7) 419 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.609904+0000 osd.7 (osd.7) 420 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.741870+0000 osd.4 (osd.4) 377 : cluster 0 2.11 scrub starts 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[29221]: cluster 2024-04-06T03:37:28.743101+0000 osd.4 (osd.4) 378 : cluster 0 2.11 scrub ok 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.058262+0000 osd.3 (osd.3) 402 : cluster 0 6.13 scrub starts 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.115672+0000 osd.2 (osd.2) 323 : cluster 0 5.8 scrub starts 2024-04-06T03:37:29.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.132446+0000 osd.2 (osd.2) 324 : cluster 0 5.8 scrub ok 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.165901+0000 osd.3 (osd.3) 403 : cluster 0 6.13 scrub ok 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.339396+0000 mgr.y (mgr.24368) 609 : cluster 0 pgmap v641: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.543868+0000 osd.0 (osd.0) 382 : cluster 0 6.10 scrub starts 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.564742+0000 osd.0 (osd.0) 383 : cluster 0 6.10 scrub ok 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.608022+0000 osd.7 (osd.7) 419 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.609904+0000 osd.7 (osd.7) 420 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.741870+0000 osd.4 (osd.4) 377 : cluster 0 2.11 scrub starts 2024-04-06T03:37:29.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:29 smithi050 bash[21467]: cluster 2024-04-06T03:37:28.743101+0000 osd.4 (osd.4) 378 : cluster 0 2.11 scrub ok 2024-04-06T03:37:29.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:29.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:30.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:37:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.058262+0000 osd.3 (osd.3) 402 : cluster 0 6.13 scrub starts 2024-04-06T03:37:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.115672+0000 osd.2 (osd.2) 323 : cluster 0 5.8 scrub starts 2024-04-06T03:37:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.132446+0000 osd.2 (osd.2) 324 : cluster 0 5.8 scrub ok 2024-04-06T03:37:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.165901+0000 osd.3 (osd.3) 403 : cluster 0 6.13 scrub ok 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.339396+0000 mgr.y (mgr.24368) 609 : cluster 0 pgmap v641: 148 pgs: 148 active+clean; 2.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.543868+0000 osd.0 (osd.0) 382 : cluster 0 6.10 scrub starts 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.564742+0000 osd.0 (osd.0) 383 : cluster 0 6.10 scrub ok 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.608022+0000 osd.7 (osd.7) 419 : cluster 0 2.1c deep-scrub starts 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.609904+0000 osd.7 (osd.7) 420 : cluster 0 2.1c deep-scrub ok 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.741870+0000 osd.4 (osd.4) 377 : cluster 0 2.11 scrub starts 2024-04-06T03:37:30.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:29 smithi153 bash[23852]: cluster 2024-04-06T03:37:28.743101+0000 osd.4 (osd.4) 378 : cluster 0 2.11 scrub ok 2024-04-06T03:37:30.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:30.110 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:30.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:30.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:30.275 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:37:30.275 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:37:30.275 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:37:30.275 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:37:30.275 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:37:30.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:30.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:30.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:30.495 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:30.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:30.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:30.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:30.753 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.081963+0000 osd.2 (osd.2) 325 : cluster 0 5.6 scrub starts 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.084250+0000 osd.2 (osd.2) 326 : cluster 0 5.6 scrub ok 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.091186+0000 osd.3 (osd.3) 404 : cluster 0 3.c deep-scrub starts 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.093753+0000 osd.3 (osd.3) 405 : cluster 0 3.c deep-scrub ok 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.499496+0000 osd.0 (osd.0) 384 : cluster 0 5.1b scrub starts 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.502332+0000 osd.0 (osd.0) 385 : cluster 0 5.1b scrub ok 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.732316+0000 osd.4 (osd.4) 379 : cluster 0 5.1c scrub starts 2024-04-06T03:37:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:30 smithi153 bash[23852]: cluster 2024-04-06T03:37:29.739798+0000 osd.4 (osd.4) 380 : cluster 0 5.1c scrub ok 2024-04-06T03:37:30.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:30.878 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:30.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:30.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.081963+0000 osd.2 (osd.2) 325 : cluster 0 5.6 scrub starts 2024-04-06T03:37:30.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.084250+0000 osd.2 (osd.2) 326 : cluster 0 5.6 scrub ok 2024-04-06T03:37:30.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.091186+0000 osd.3 (osd.3) 404 : cluster 0 3.c deep-scrub starts 2024-04-06T03:37:30.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.093753+0000 osd.3 (osd.3) 405 : cluster 0 3.c deep-scrub ok 2024-04-06T03:37:30.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.499496+0000 osd.0 (osd.0) 384 : cluster 0 5.1b scrub starts 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.502332+0000 osd.0 (osd.0) 385 : cluster 0 5.1b scrub ok 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.732316+0000 osd.4 (osd.4) 379 : cluster 0 5.1c scrub starts 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[21467]: cluster 2024-04-06T03:37:29.739798+0000 osd.4 (osd.4) 380 : cluster 0 5.1c scrub ok 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.081963+0000 osd.2 (osd.2) 325 : cluster 0 5.6 scrub starts 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.084250+0000 osd.2 (osd.2) 326 : cluster 0 5.6 scrub ok 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.091186+0000 osd.3 (osd.3) 404 : cluster 0 3.c deep-scrub starts 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.093753+0000 osd.3 (osd.3) 405 : cluster 0 3.c deep-scrub ok 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.499496+0000 osd.0 (osd.0) 384 : cluster 0 5.1b scrub starts 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.502332+0000 osd.0 (osd.0) 385 : cluster 0 5.1b scrub ok 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.732316+0000 osd.4 (osd.4) 379 : cluster 0 5.1c scrub starts 2024-04-06T03:37:30.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:30 smithi050 bash[29221]: cluster 2024-04-06T03:37:29.739798+0000 osd.4 (osd.4) 380 : cluster 0 5.1c scrub ok 2024-04-06T03:37:30.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:31.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:31.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:31.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:31.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:31.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-06T03:37:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:31.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:31.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:31.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:31.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:31.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.667 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:31.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:31.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: audit 2024-04-06T03:37:29.805266+0000 mgr.y (mgr.24368) 610 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:31.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.248896+0000 osd.3 (osd.3) 406 : cluster 0 2.19 scrub starts 2024-04-06T03:37:31.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.256055+0000 osd.3 (osd.3) 407 : cluster 0 2.19 scrub ok 2024-04-06T03:37:31.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.340097+0000 mgr.y (mgr.24368) 611 : cluster 0 pgmap v642: 148 pgs: 148 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 975 op/s 2024-04-06T03:37:31.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.484364+0000 osd.0 (osd.0) 386 : cluster 0 3.1f scrub starts 2024-04-06T03:37:31.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.487681+0000 osd.0 (osd.0) 387 : cluster 0 3.1f scrub ok 2024-04-06T03:37:31.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.548814+0000 osd.7 (osd.7) 421 : cluster 0 3.13 deep-scrub starts 2024-04-06T03:37:31.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:31 smithi153 bash[23852]: cluster 2024-04-06T03:37:30.552691+0000 osd.7 (osd.7) 422 : cluster 0 3.13 deep-scrub ok 2024-04-06T03:37:31.929 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:31.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: audit 2024-04-06T03:37:29.805266+0000 mgr.y (mgr.24368) 610 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.248896+0000 osd.3 (osd.3) 406 : cluster 0 2.19 scrub starts 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.256055+0000 osd.3 (osd.3) 407 : cluster 0 2.19 scrub ok 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.340097+0000 mgr.y (mgr.24368) 611 : cluster 0 pgmap v642: 148 pgs: 148 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 975 op/s 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.484364+0000 osd.0 (osd.0) 386 : cluster 0 3.1f scrub starts 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.487681+0000 osd.0 (osd.0) 387 : cluster 0 3.1f scrub ok 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.548814+0000 osd.7 (osd.7) 421 : cluster 0 3.13 deep-scrub starts 2024-04-06T03:37:31.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[21467]: cluster 2024-04-06T03:37:30.552691+0000 osd.7 (osd.7) 422 : cluster 0 3.13 deep-scrub ok 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: audit 2024-04-06T03:37:29.805266+0000 mgr.y (mgr.24368) 610 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.248896+0000 osd.3 (osd.3) 406 : cluster 0 2.19 scrub starts 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.256055+0000 osd.3 (osd.3) 407 : cluster 0 2.19 scrub ok 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.340097+0000 mgr.y (mgr.24368) 611 : cluster 0 pgmap v642: 148 pgs: 148 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 975 op/s 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.484364+0000 osd.0 (osd.0) 386 : cluster 0 3.1f scrub starts 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.487681+0000 osd.0 (osd.0) 387 : cluster 0 3.1f scrub ok 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.548814+0000 osd.7 (osd.7) 421 : cluster 0 3.13 deep-scrub starts 2024-04-06T03:37:31.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:31 smithi050 bash[29221]: cluster 2024-04-06T03:37:30.552691+0000 osd.7 (osd.7) 422 : cluster 0 3.13 deep-scrub ok 2024-04-06T03:37:32.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:32.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:32.200 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:32.303 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:32.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:32.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:32.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:32.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:32.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:32.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:32.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:32.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:32.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:32.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:32.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:32 smithi153 bash[23852]: cluster 2024-04-06T03:37:31.506388+0000 osd.0 (osd.0) 388 : cluster 0 5.5 scrub starts 2024-04-06T03:37:32.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:32 smithi153 bash[23852]: cluster 2024-04-06T03:37:31.509439+0000 osd.0 (osd.0) 389 : cluster 0 5.5 scrub ok 2024-04-06T03:37:32.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:32 smithi153 bash[23852]: cluster 2024-04-06T03:37:31.509879+0000 osd.7 (osd.7) 423 : cluster 0 6.3 deep-scrub starts 2024-04-06T03:37:32.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:32 smithi153 bash[23852]: cluster 2024-04-06T03:37:31.563707+0000 osd.7 (osd.7) 424 : cluster 0 6.3 deep-scrub ok 2024-04-06T03:37:32.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:32 smithi153 bash[23852]: cluster 2024-04-06T03:37:31.729598+0000 osd.4 (osd.4) 381 : cluster 0 4.1 scrub starts 2024-04-06T03:37:32.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:32 smithi153 bash[23852]: cluster 2024-04-06T03:37:31.732349+0000 osd.4 (osd.4) 382 : cluster 0 4.1 scrub ok 2024-04-06T03:37:32.978 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:32.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:33.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:33.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:33.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:33.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[21467]: cluster 2024-04-06T03:37:31.506388+0000 osd.0 (osd.0) 388 : cluster 0 5.5 scrub starts 2024-04-06T03:37:33.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[21467]: cluster 2024-04-06T03:37:31.509439+0000 osd.0 (osd.0) 389 : cluster 0 5.5 scrub ok 2024-04-06T03:37:33.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[21467]: cluster 2024-04-06T03:37:31.509879+0000 osd.7 (osd.7) 423 : cluster 0 6.3 deep-scrub starts 2024-04-06T03:37:33.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[21467]: cluster 2024-04-06T03:37:31.563707+0000 osd.7 (osd.7) 424 : cluster 0 6.3 deep-scrub ok 2024-04-06T03:37:33.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[21467]: cluster 2024-04-06T03:37:31.729598+0000 osd.4 (osd.4) 381 : cluster 0 4.1 scrub starts 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[21467]: cluster 2024-04-06T03:37:31.732349+0000 osd.4 (osd.4) 382 : cluster 0 4.1 scrub ok 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[29221]: cluster 2024-04-06T03:37:31.506388+0000 osd.0 (osd.0) 388 : cluster 0 5.5 scrub starts 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[29221]: cluster 2024-04-06T03:37:31.509439+0000 osd.0 (osd.0) 389 : cluster 0 5.5 scrub ok 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[29221]: cluster 2024-04-06T03:37:31.509879+0000 osd.7 (osd.7) 423 : cluster 0 6.3 deep-scrub starts 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[29221]: cluster 2024-04-06T03:37:31.563707+0000 osd.7 (osd.7) 424 : cluster 0 6.3 deep-scrub ok 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[29221]: cluster 2024-04-06T03:37:31.729598+0000 osd.4 (osd.4) 381 : cluster 0 4.1 scrub starts 2024-04-06T03:37:33.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:32 smithi050 bash[29221]: cluster 2024-04-06T03:37:31.732349+0000 osd.4 (osd.4) 382 : cluster 0 4.1 scrub ok 2024-04-06T03:37:33.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:33.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:33.373 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:33.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:33.504 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:33.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:33.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:33.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:33.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:33.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.052739+0000 osd.2 (osd.2) 327 : cluster 0 9.6 scrub starts 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.290848+0000 osd.1 (osd.1) 319 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.290900+0000 osd.1 (osd.1) 320 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.340778+0000 mgr.y (mgr.24368) 612 : cluster 0 pgmap v643: 148 pgs: 148 active+clean; 2.0 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 899 op/s 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.468754+0000 osd.7 (osd.7) 425 : cluster 0 6.12 scrub starts 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.481401+0000 osd.7 (osd.7) 426 : cluster 0 6.12 scrub ok 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.527151+0000 osd.0 (osd.0) 390 : cluster 0 5.15 scrub starts 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.536896+0000 osd.0 (osd.0) 391 : cluster 0 5.15 scrub ok 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.673372+0000 osd.2 (osd.2) 328 : cluster 0 9.6 scrub ok 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.705726+0000 osd.4 (osd.4) 383 : cluster 0 3.a scrub starts 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.711923+0000 osd.4 (osd.4) 384 : cluster 0 3.a scrub ok 2024-04-06T03:37:33.882 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:33 smithi153 bash[23852]: cluster 2024-04-06T03:37:32.764869+0000 mon.a (mon.0) 1430 : cluster 0 osdmap e267: 8 total, 8 up, 8 in 2024-04-06T03:37:33.901 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:33.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:33.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.052739+0000 osd.2 (osd.2) 327 : cluster 0 9.6 scrub starts 2024-04-06T03:37:33.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.290848+0000 osd.1 (osd.1) 319 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:33.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.290900+0000 osd.1 (osd.1) 320 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:33.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.340778+0000 mgr.y (mgr.24368) 612 : cluster 0 pgmap v643: 148 pgs: 148 active+clean; 2.0 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 899 op/s 2024-04-06T03:37:33.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.468754+0000 osd.7 (osd.7) 425 : cluster 0 6.12 scrub starts 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.481401+0000 osd.7 (osd.7) 426 : cluster 0 6.12 scrub ok 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.527151+0000 osd.0 (osd.0) 390 : cluster 0 5.15 scrub starts 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.536896+0000 osd.0 (osd.0) 391 : cluster 0 5.15 scrub ok 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.673372+0000 osd.2 (osd.2) 328 : cluster 0 9.6 scrub ok 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.705726+0000 osd.4 (osd.4) 383 : cluster 0 3.a scrub starts 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.711923+0000 osd.4 (osd.4) 384 : cluster 0 3.a scrub ok 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[29221]: cluster 2024-04-06T03:37:32.764869+0000 mon.a (mon.0) 1430 : cluster 0 osdmap e267: 8 total, 8 up, 8 in 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.052739+0000 osd.2 (osd.2) 327 : cluster 0 9.6 scrub starts 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.290848+0000 osd.1 (osd.1) 319 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.290900+0000 osd.1 (osd.1) 320 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:33.981 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.340778+0000 mgr.y (mgr.24368) 612 : cluster 0 pgmap v643: 148 pgs: 148 active+clean; 2.0 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 899 op/s 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.468754+0000 osd.7 (osd.7) 425 : cluster 0 6.12 scrub starts 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.481401+0000 osd.7 (osd.7) 426 : cluster 0 6.12 scrub ok 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.527151+0000 osd.0 (osd.0) 390 : cluster 0 5.15 scrub starts 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.536896+0000 osd.0 (osd.0) 391 : cluster 0 5.15 scrub ok 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.673372+0000 osd.2 (osd.2) 328 : cluster 0 9.6 scrub ok 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.705726+0000 osd.4 (osd.4) 383 : cluster 0 3.a scrub starts 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.711923+0000 osd.4 (osd.4) 384 : cluster 0 3.a scrub ok 2024-04-06T03:37:33.982 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:33 smithi050 bash[21467]: cluster 2024-04-06T03:37:32.764869+0000 mon.a (mon.0) 1430 : cluster 0 osdmap e267: 8 total, 8 up, 8 in 2024-04-06T03:37:34.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:34.038 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:34.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:34.164 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:34.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:34.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:34.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:34.424 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:34.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:34.562 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:34.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:34.688 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:34.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:34.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:34.824 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:34.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:34.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:34.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:34 smithi153 bash[23852]: cluster 2024-04-06T03:37:33.011230+0000 osd.2 (osd.2) 329 : cluster 0 5.a deep-scrub starts 2024-04-06T03:37:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:34 smithi153 bash[23852]: cluster 2024-04-06T03:37:33.012709+0000 osd.2 (osd.2) 330 : cluster 0 5.a deep-scrub ok 2024-04-06T03:37:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:34 smithi153 bash[23852]: cluster 2024-04-06T03:37:33.577378+0000 osd.0 (osd.0) 392 : cluster 0 6.18 scrub starts 2024-04-06T03:37:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:34 smithi153 bash[23852]: cluster 2024-04-06T03:37:33.643710+0000 osd.0 (osd.0) 393 : cluster 0 6.18 scrub ok 2024-04-06T03:37:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:34 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.341422+0000 mgr.y (mgr.24368) 613 : cluster 0 pgmap v645: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 918 op/s 2024-04-06T03:37:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:34 smithi153 bash[23852]: audit 2024-04-06T03:37:34.703498+0000 mon.a (mon.0) 1431 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:35.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:35.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:35.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:35.222 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[21467]: cluster 2024-04-06T03:37:33.011230+0000 osd.2 (osd.2) 329 : cluster 0 5.a deep-scrub starts 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[21467]: cluster 2024-04-06T03:37:33.012709+0000 osd.2 (osd.2) 330 : cluster 0 5.a deep-scrub ok 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[21467]: cluster 2024-04-06T03:37:33.577378+0000 osd.0 (osd.0) 392 : cluster 0 6.18 scrub starts 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[21467]: cluster 2024-04-06T03:37:33.643710+0000 osd.0 (osd.0) 393 : cluster 0 6.18 scrub ok 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.341422+0000 mgr.y (mgr.24368) 613 : cluster 0 pgmap v645: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 918 op/s 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[21467]: audit 2024-04-06T03:37:34.703498+0000 mon.a (mon.0) 1431 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[29221]: cluster 2024-04-06T03:37:33.011230+0000 osd.2 (osd.2) 329 : cluster 0 5.a deep-scrub starts 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[29221]: cluster 2024-04-06T03:37:33.012709+0000 osd.2 (osd.2) 330 : cluster 0 5.a deep-scrub ok 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[29221]: cluster 2024-04-06T03:37:33.577378+0000 osd.0 (osd.0) 392 : cluster 0 6.18 scrub starts 2024-04-06T03:37:35.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[29221]: cluster 2024-04-06T03:37:33.643710+0000 osd.0 (osd.0) 393 : cluster 0 6.18 scrub ok 2024-04-06T03:37:35.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.341422+0000 mgr.y (mgr.24368) 613 : cluster 0 pgmap v645: 148 pgs: 148 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 918 op/s 2024-04-06T03:37:35.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:34 smithi050 bash[29221]: audit 2024-04-06T03:37:34.703498+0000 mon.a (mon.0) 1431 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:35.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:35.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:35.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:35.485 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:35.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:35.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:35.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:35.736 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:35.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:35.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:35.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:35.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:35.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:35 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.264830+0000 osd.3 (osd.3) 408 : cluster 0 5.14 scrub starts 2024-04-06T03:37:35.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:35 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.277434+0000 osd.3 (osd.3) 409 : cluster 0 5.14 scrub ok 2024-04-06T03:37:35.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:35 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.352563+0000 osd.1 (osd.1) 321 : cluster 0 5.12 scrub starts 2024-04-06T03:37:35.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:35 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.354278+0000 osd.1 (osd.1) 322 : cluster 0 5.12 scrub ok 2024-04-06T03:37:35.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:35 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.530531+0000 osd.7 (osd.7) 427 : cluster 0 2.7 scrub starts 2024-04-06T03:37:35.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:35 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.532774+0000 osd.7 (osd.7) 428 : cluster 0 2.7 scrub ok 2024-04-06T03:37:36.012 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:36.062 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:37:36.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.141 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.264830+0000 osd.3 (osd.3) 408 : cluster 0 5.14 scrub starts 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.277434+0000 osd.3 (osd.3) 409 : cluster 0 5.14 scrub ok 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.352563+0000 osd.1 (osd.1) 321 : cluster 0 5.12 scrub starts 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.354278+0000 osd.1 (osd.1) 322 : cluster 0 5.12 scrub ok 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.530531+0000 osd.7 (osd.7) 427 : cluster 0 2.7 scrub starts 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.532774+0000 osd.7 (osd.7) 428 : cluster 0 2.7 scrub ok 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.264830+0000 osd.3 (osd.3) 408 : cluster 0 5.14 scrub starts 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.277434+0000 osd.3 (osd.3) 409 : cluster 0 5.14 scrub ok 2024-04-06T03:37:36.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.352563+0000 osd.1 (osd.1) 321 : cluster 0 5.12 scrub starts 2024-04-06T03:37:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.354278+0000 osd.1 (osd.1) 322 : cluster 0 5.12 scrub ok 2024-04-06T03:37:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.530531+0000 osd.7 (osd.7) 427 : cluster 0 2.7 scrub starts 2024-04-06T03:37:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.532774+0000 osd.7 (osd.7) 428 : cluster 0 2.7 scrub ok 2024-04-06T03:37:36.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:36.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:36.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:36.397 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:36.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:36.523 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:36.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:36.660 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:36.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.785 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:36.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:36 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.955619+0000 osd.2 (osd.2) 331 : cluster 0 2.3 scrub starts 2024-04-06T03:37:36.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:36 smithi153 bash[23852]: cluster 2024-04-06T03:37:34.956820+0000 osd.2 (osd.2) 332 : cluster 0 2.3 scrub ok 2024-04-06T03:37:36.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:36 smithi153 bash[23852]: cluster 2024-04-06T03:37:35.554616+0000 osd.7 (osd.7) 429 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:37:36.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:36 smithi153 bash[23852]: cluster 2024-04-06T03:37:35.558620+0000 osd.7 (osd.7) 430 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:37:36.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:36 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.342130+0000 mgr.y (mgr.24368) 614 : cluster 0 pgmap v646: 148 pgs: 148 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 800 op/s 2024-04-06T03:37:36.913 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:36.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:37.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:37.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:37.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:36 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.955619+0000 osd.2 (osd.2) 331 : cluster 0 2.3 scrub starts 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:34.956820+0000 osd.2 (osd.2) 332 : cluster 0 2.3 scrub ok 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:35.554616+0000 osd.7 (osd.7) 429 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:35.558620+0000 osd.7 (osd.7) 430 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.342130+0000 mgr.y (mgr.24368) 614 : cluster 0 pgmap v646: 148 pgs: 148 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 800 op/s 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.955619+0000 osd.2 (osd.2) 331 : cluster 0 2.3 scrub starts 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:34.956820+0000 osd.2 (osd.2) 332 : cluster 0 2.3 scrub ok 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:35.554616+0000 osd.7 (osd.7) 429 : cluster 0 5.10 deep-scrub starts 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:35.558620+0000 osd.7 (osd.7) 430 : cluster 0 5.10 deep-scrub ok 2024-04-06T03:37:37.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.342130+0000 mgr.y (mgr.24368) 614 : cluster 0 pgmap v646: 148 pgs: 148 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 800 op/s 2024-04-06T03:37:37.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:37.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:37.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:37.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:37.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:37.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:37.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:37.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:37.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:37.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:37.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:37.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:37.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:37.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:38.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:38.083 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:37 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.294056+0000 osd.3 (osd.3) 410 : cluster 0 6.1f scrub starts 2024-04-06T03:37:38.083 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:37 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.392040+0000 osd.1 (osd.1) 323 : cluster 0 6.19 scrub starts 2024-04-06T03:37:38.083 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:37 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.460143+0000 osd.3 (osd.3) 411 : cluster 0 6.1f scrub ok 2024-04-06T03:37:38.083 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:37 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.488382+0000 osd.1 (osd.1) 324 : cluster 0 6.19 scrub ok 2024-04-06T03:37:38.083 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:37 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.521017+0000 osd.0 (osd.0) 394 : cluster 0 2.1f scrub starts 2024-04-06T03:37:38.083 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:37 smithi153 bash[23852]: cluster 2024-04-06T03:37:36.522344+0000 osd.0 (osd.0) 395 : cluster 0 2.1f scrub ok 2024-04-06T03:37:38.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:38.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:38.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.294056+0000 osd.3 (osd.3) 410 : cluster 0 6.1f scrub starts 2024-04-06T03:37:38.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.392040+0000 osd.1 (osd.1) 323 : cluster 0 6.19 scrub starts 2024-04-06T03:37:38.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.460143+0000 osd.3 (osd.3) 411 : cluster 0 6.1f scrub ok 2024-04-06T03:37:38.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.488382+0000 osd.1 (osd.1) 324 : cluster 0 6.19 scrub ok 2024-04-06T03:37:38.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.521017+0000 osd.0 (osd.0) 394 : cluster 0 2.1f scrub starts 2024-04-06T03:37:38.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[21467]: cluster 2024-04-06T03:37:36.522344+0000 osd.0 (osd.0) 395 : cluster 0 2.1f scrub ok 2024-04-06T03:37:38.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.294056+0000 osd.3 (osd.3) 410 : cluster 0 6.1f scrub starts 2024-04-06T03:37:38.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.392040+0000 osd.1 (osd.1) 323 : cluster 0 6.19 scrub starts 2024-04-06T03:37:38.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.460143+0000 osd.3 (osd.3) 411 : cluster 0 6.1f scrub ok 2024-04-06T03:37:38.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.488382+0000 osd.1 (osd.1) 324 : cluster 0 6.19 scrub ok 2024-04-06T03:37:38.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.521017+0000 osd.0 (osd.0) 394 : cluster 0 2.1f scrub starts 2024-04-06T03:37:38.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:37 smithi050 bash[29221]: cluster 2024-04-06T03:37:36.522344+0000 osd.0 (osd.0) 395 : cluster 0 2.1f scrub ok 2024-04-06T03:37:38.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:38.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:38.376 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:38.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:38.508 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:38.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:38.657 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:38.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:38.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:38.902 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:38.923 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:38.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:39.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:39.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:39.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:39.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:37.510421+0000 osd.1 (osd.1) 325 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:37:39.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:37.513287+0000 osd.1 (osd.1) 326 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:37:39.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:37.540363+0000 osd.7 (osd.7) 431 : cluster 0 3.e scrub starts 2024-04-06T03:37:39.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:37.543005+0000 osd.7 (osd.7) 432 : cluster 0 3.e scrub ok 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:37.652241+0000 osd.4 (osd.4) 385 : cluster 0 6.e scrub starts 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:37.704504+0000 osd.4 (osd.4) 386 : cluster 0 6.e scrub ok 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: audit 2024-04-06T03:37:37.991562+0000 mon.a (mon.0) 1432 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: cluster 2024-04-06T03:37:38.342656+0000 mgr.y (mgr.24368) 615 : cluster 0 pgmap v647: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 934 op/s 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: audit 2024-04-06T03:37:38.854807+0000 mon.a (mon.0) 1433 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: audit 2024-04-06T03:37:38.855899+0000 mon.a (mon.0) 1434 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:37:39.170 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:38 smithi153 bash[23852]: audit 2024-04-06T03:37:38.866871+0000 mon.a (mon.0) 1435 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:39.186 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:37.510421+0000 osd.1 (osd.1) 325 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:37.513287+0000 osd.1 (osd.1) 326 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:37.540363+0000 osd.7 (osd.7) 431 : cluster 0 3.e scrub starts 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:37.543005+0000 osd.7 (osd.7) 432 : cluster 0 3.e scrub ok 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:37.652241+0000 osd.4 (osd.4) 385 : cluster 0 6.e scrub starts 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:37.704504+0000 osd.4 (osd.4) 386 : cluster 0 6.e scrub ok 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: audit 2024-04-06T03:37:37.991562+0000 mon.a (mon.0) 1432 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: cluster 2024-04-06T03:37:38.342656+0000 mgr.y (mgr.24368) 615 : cluster 0 pgmap v647: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 934 op/s 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: audit 2024-04-06T03:37:38.854807+0000 mon.a (mon.0) 1433 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: audit 2024-04-06T03:37:38.855899+0000 mon.a (mon.0) 1434 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:37:39.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:38 smithi050 bash[21467]: audit 2024-04-06T03:37:38.866871+0000 mon.a (mon.0) 1435 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:39.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:39.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:39.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:39.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:39.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:39.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:39.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:39.715 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:39.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:37.510421+0000 osd.1 (osd.1) 325 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:37:39.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:37.513287+0000 osd.1 (osd.1) 326 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:37:39.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:37.540363+0000 osd.7 (osd.7) 431 : cluster 0 3.e scrub starts 2024-04-06T03:37:39.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:37.543005+0000 osd.7 (osd.7) 432 : cluster 0 3.e scrub ok 2024-04-06T03:37:39.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:37.652241+0000 osd.4 (osd.4) 385 : cluster 0 6.e scrub starts 2024-04-06T03:37:39.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:37.704504+0000 osd.4 (osd.4) 386 : cluster 0 6.e scrub ok 2024-04-06T03:37:39.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: audit 2024-04-06T03:37:37.991562+0000 mon.a (mon.0) 1432 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:37:39.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:38.342656+0000 mgr.y (mgr.24368) 615 : cluster 0 pgmap v647: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 934 op/s 2024-04-06T03:37:39.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: audit 2024-04-06T03:37:38.854807+0000 mon.a (mon.0) 1433 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:37:39.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: audit 2024-04-06T03:37:38.855899+0000 mon.a (mon.0) 1434 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:37:39.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: audit 2024-04-06T03:37:38.866871+0000 mon.a (mon.0) 1435 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:39.738 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:37:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:37:39.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:39.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:39.947 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:39.981 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:39.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:40.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:37:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:37:40.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:39 smithi153 bash[23852]: cluster 2024-04-06T03:37:38.577382+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub starts 2024-04-06T03:37:40.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:39 smithi153 bash[23852]: cluster 2024-04-06T03:37:38.582490+0000 osd.0 (osd.0) 397 : cluster 0 3.1b scrub ok 2024-04-06T03:37:40.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:39 smithi153 bash[23852]: cluster 2024-04-06T03:37:38.914745+0000 osd.2 (osd.2) 333 : cluster 0 5.b scrub starts 2024-04-06T03:37:40.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:39 smithi153 bash[23852]: cluster 2024-04-06T03:37:38.921265+0000 osd.2 (osd.2) 334 : cluster 0 5.b scrub ok 2024-04-06T03:37:40.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:40.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:40.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[21467]: cluster 2024-04-06T03:37:38.577382+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub starts 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[21467]: cluster 2024-04-06T03:37:38.582490+0000 osd.0 (osd.0) 397 : cluster 0 3.1b scrub ok 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[21467]: cluster 2024-04-06T03:37:38.914745+0000 osd.2 (osd.2) 333 : cluster 0 5.b scrub starts 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[21467]: cluster 2024-04-06T03:37:38.921265+0000 osd.2 (osd.2) 334 : cluster 0 5.b scrub ok 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:38.577382+0000 osd.0 (osd.0) 396 : cluster 0 3.1b scrub starts 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:38.582490+0000 osd.0 (osd.0) 397 : cluster 0 3.1b scrub ok 2024-04-06T03:37:40.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:38.914745+0000 osd.2 (osd.2) 333 : cluster 0 5.b scrub starts 2024-04-06T03:37:40.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:39 smithi050 bash[29221]: cluster 2024-04-06T03:37:38.921265+0000 osd.2 (osd.2) 334 : cluster 0 5.b scrub ok 2024-04-06T03:37:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:40.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:40.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:40.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:40.496 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:40.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:40.629 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:40.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:40.759 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:40.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:40.895 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:40.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:41.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:41.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:39.517917+0000 osd.0 (osd.0) 398 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:41.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:39.517986+0000 osd.0 (osd.0) 399 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:41.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:39.552526+0000 osd.7 (osd.7) 433 : cluster 0 3.15 scrub starts 2024-04-06T03:37:41.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:39.557633+0000 osd.7 (osd.7) 434 : cluster 0 3.15 scrub ok 2024-04-06T03:37:41.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: audit 2024-04-06T03:37:39.819676+0000 mgr.y (mgr.24368) 616 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:41.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:39.893440+0000 mon.a (mon.0) 1436 : cluster 0 osdmap e268: 8 total, 8 up, 8 in 2024-04-06T03:37:41.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:40.237394+0000 osd.3 (osd.3) 412 : cluster 0 3.1e scrub starts 2024-04-06T03:37:41.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:40.241000+0000 osd.3 (osd.3) 413 : cluster 0 3.1e scrub ok 2024-04-06T03:37:41.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:40 smithi153 bash[23852]: cluster 2024-04-06T03:37:40.344047+0000 mgr.y (mgr.24368) 617 : cluster 0 pgmap v649: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s 2024-04-06T03:37:41.025 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:41.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:41.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:41.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:39.517917+0000 osd.0 (osd.0) 398 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:41.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:39.517986+0000 osd.0 (osd.0) 399 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:41.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:39.552526+0000 osd.7 (osd.7) 433 : cluster 0 3.15 scrub starts 2024-04-06T03:37:41.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:39.557633+0000 osd.7 (osd.7) 434 : cluster 0 3.15 scrub ok 2024-04-06T03:37:41.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: audit 2024-04-06T03:37:39.819676+0000 mgr.y (mgr.24368) 616 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:41.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:39.893440+0000 mon.a (mon.0) 1436 : cluster 0 osdmap e268: 8 total, 8 up, 8 in 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:40.237394+0000 osd.3 (osd.3) 412 : cluster 0 3.1e scrub starts 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:40.241000+0000 osd.3 (osd.3) 413 : cluster 0 3.1e scrub ok 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[21467]: cluster 2024-04-06T03:37:40.344047+0000 mgr.y (mgr.24368) 617 : cluster 0 pgmap v649: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:39.517917+0000 osd.0 (osd.0) 398 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:39.517986+0000 osd.0 (osd.0) 399 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:39.552526+0000 osd.7 (osd.7) 433 : cluster 0 3.15 scrub starts 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:39.557633+0000 osd.7 (osd.7) 434 : cluster 0 3.15 scrub ok 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: audit 2024-04-06T03:37:39.819676+0000 mgr.y (mgr.24368) 616 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:39.893440+0000 mon.a (mon.0) 1436 : cluster 0 osdmap e268: 8 total, 8 up, 8 in 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:40.237394+0000 osd.3 (osd.3) 412 : cluster 0 3.1e scrub starts 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:40.241000+0000 osd.3 (osd.3) 413 : cluster 0 3.1e scrub ok 2024-04-06T03:37:41.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:40 smithi050 bash[29221]: cluster 2024-04-06T03:37:40.344047+0000 mgr.y (mgr.24368) 617 : cluster 0 pgmap v649: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s 2024-04-06T03:37:41.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:41.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:41.303 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:37:41.303 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:37:41.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:37:41.303 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:15560 8.8 e188/182 8:10cdbd71:::benchmark_data_smithi050_86143_object15559:head v 188'998, mlcod=188'996)", 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:06.844623+0000", 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 454.456700205, 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.058464901999999999, 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844623+0000", 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844623+0000", 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844623+0000", 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0999999999999999e-07 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844678+0000", 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5173999999999997e-05 2024-04-06T03:37:41.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844686+0000", 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8730000000000007e-06 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844691+0000", 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6530000000000002e-06 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844732+0000", 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1519e-05 2024-04-06T03:37:41.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844744+0000", 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1781e-05 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.902945+0000", 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.058200997999999997 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.903087+0000", 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014239400000000001 2024-04-06T03:37:41.307 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:15557 8.a e188/182 8:5c96f801:::benchmark_data_smithi050_86143_object15556:head v 188'939, mlcod=188'937)", 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:06.844899+0000", 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 454.45642383299997, 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.054316320000000001, 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844899+0000", 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844899+0000", 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844900+0000", 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7499999999999999e-07 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844947+0000", 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7713000000000002e-05 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844960+0000", 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2506e-05 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.844967+0000", 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4760000000000001e-06 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.845009+0000", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1573999999999999e-05 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.845022+0000", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2809000000000001e-05 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.899104+0000", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.054082296000000002 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.899215+0000", 2024-04-06T03:37:41.311 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011107099999999999 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:15564 8.8 e188/182 8:12170825:::benchmark_data_smithi050_86143_object15563:head v 188'999, mlcod=188'996)", 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:06.845754+0000", 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 454.45556840799998, 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.058513281, 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.845754+0000", 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.845754+0000", 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.845755+0000", 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5000000000000002e-07 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.845835+0000", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9764000000000004e-05 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.898320+0000", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.052484909000000003 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.898345+0000", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5593999999999999e-05 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.899196+0000", 2024-04-06T03:37:41.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00085025900000000002 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.899216+0000", 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0596999999999999e-05 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.904228+0000", 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0050122129999999997 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.904268+0000", 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9295000000000002e-05 2024-04-06T03:37:41.315 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:15568 8.d e188/182 8:b48a4d11:::benchmark_data_smithi050_86143_object15567:head v 188'1009, mlcod=188'1007)", 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:06.846254+0000", 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 454.45506922300001, 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.057945681999999998, 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.846254+0000", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.846254+0000", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.846254+0000", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.92e-07 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.846280+0000", 2024-04-06T03:37:41.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5839000000000001e-05 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.897396+0000", 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.051115682000000003 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.897420+0000", 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3986999999999999e-05 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.897565+0000", 2024-04-06T03:37:41.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014561399999999999 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.897617+0000", 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1829000000000001e-05 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.904086+0000", 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006468255 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:06.904199+0000", 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000113684 2024-04-06T03:37:41.319 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:74162 8.a e199/182 8:58c2a27e:::benchmark_data_smithi050_86143_object74161:head v 199'4629, mlcod=199'4623)", 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:53.050009+0000", 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 408.25131419899998, 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.44251473800000002, 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.050009+0000", 2024-04-06T03:37:41.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.050009+0000", 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.050009+0000", 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6899999999999998e-07 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.050067+0000", 2024-04-06T03:37:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8436000000000001e-05 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.050075+0000", 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5240000000000003e-06 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.050080+0000", 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5900000000000001e-06 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481179+0000", 2024-04-06T03:37:41.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43109993499999999 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481242+0000", 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2985999999999996e-05 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.492459+0000", 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.01121639 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.492523+0000", 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4407999999999997e-05 2024-04-06T03:37:41.323 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:74166 8.9 e199/182 8:9a57aa0f:::benchmark_data_smithi050_86143_object74165:head v 199'4594, mlcod=199'4592)", 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:53.051095+0000", 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 408.25022757400001, 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.436249524, 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.051095+0000", 2024-04-06T03:37:41.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.051095+0000", 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.051097+0000", 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.663e-06 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.051148+0000", 2024-04-06T03:37:41.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1078000000000001e-05 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481140+0000", 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.42999170599999997 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481191+0000", 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1665000000000001e-05 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481302+0000", 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111085 2024-04-06T03:37:41.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481327+0000", 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4709e-05 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.487262+0000", 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0059345999999999999 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.487345+0000", 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3017999999999997e-05 2024-04-06T03:37:41.327 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:74163 8.a e199/182 8:5789611e:::benchmark_data_smithi050_86143_object74162:head v 199'4630, mlcod=199'4625)", 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:53.053470+0000", 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 408.24785294499998, 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.44400604199999999, 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053470+0000", 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053470+0000", 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053470+0000", 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.06e-07 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053576+0000", 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000106081 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481166+0000", 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.42758934999999998 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481193+0000", 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7025999999999998e-05 2024-04-06T03:37:41.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.483900+0000", 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027071450000000002 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.483915+0000", 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5313999999999999e-05 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.497416+0000", 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.013500717000000001 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.497476+0000", 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9902999999999999e-05 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:74169 8.0 e199/182 8:0a78b792:::benchmark_data_smithi050_86143_object74168:head v 199'4580, mlcod=199'4578)", 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:53.053514+0000", 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 408.24780855199998, 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.44115484900000002, 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.332 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053514+0000", 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053514+0000", 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053515+0000", 2024-04-06T03:37:41.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6700000000000003e-07 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.053590+0000", 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5282000000000002e-05 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481143+0000", 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.42755236499999999 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481201+0000", 2024-04-06T03:37:41.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8944999999999999e-05 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481297+0000", 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5851000000000001e-05 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.481319+0000", 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1937999999999999e-05 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.494633+0000", 2024-04-06T03:37:41.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.013313620999999999 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:53.494669+0000", 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6180000000000003e-05 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:83617 8.0 e199/182 8:0d665aba:::benchmark_data_smithi050_86143_object83616:head v 199'5153, mlcod=199'5149)", 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:31:03.448883+0000", 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 397.85243971599999, 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.27099036799999998, 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.448883+0000", 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.448883+0000", 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.448884+0000", 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4199999999999995e-07 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.448916+0000", 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2706000000000002e-05 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708513+0000", 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.25959660000000001 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708550+0000", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6974999999999998e-05 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708773+0000", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022324899999999999 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708830+0000", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7287999999999998e-05 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.719796+0000", 2024-04-06T03:37:41.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010965282999999999 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.719873+0000", 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7625000000000003e-05 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:83610 8.3 e199/182 8:cd49666e:::benchmark_data_smithi050_86143_object83609:head v 199'5183, mlcod=199'5181)", 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:31:03.449226+0000", 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 397.85209661099998, 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.270724311, 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.340 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.449226+0000", 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.449226+0000", 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.449227+0000", 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7800000000000002e-07 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.449338+0000", 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111018 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708518+0000", 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.25918031600000002 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.342 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708552+0000", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.3562999999999997e-05 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708720+0000", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016835600000000001 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.708783+0000", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2952000000000003e-05 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.719768+0000", 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010984850000000001 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:31:03.719950+0000", 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000182778 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:126610 9.7 e260/231 9:e82ff979:::benchmark_data_smithi050_105098_object29700:head v 260'7771, mlcod=260'7767)", 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:52.144462+0000", 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.156860680999998, 2024-04-06T03:37:41.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045288175, 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144462+0000", 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144462+0000", 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144463+0000", 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6300000000000005e-07 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144468+0000", 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7339999999999996e-06 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144478+0000", 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8400000000000007e-06 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144486+0000", 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.272e-06 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144536+0000", 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0339e-05 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144556+0000", 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9763e-05 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.189665+0000", 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045109690000000001 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.189750+0000", 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4874000000000001e-05 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:126609 9.b e260/231 9:d63f7ecc:::benchmark_data_smithi050_105098_object29699:head v 260'7942, mlcod=260'7940)", 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:37:41.348 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.155367935000001, 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.047109446999999999, 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.06e-07 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145960+0000", 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9479999999999999e-06 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145968+0000", 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.028e-06 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145973+0000", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0000000000000004e-06 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184892+0000", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.038918992999999999 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184922+0000", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9802999999999999e-05 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.192971+0000", 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0080485069999999999 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.193064+0000", 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3561999999999996e-05 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:126611 9.5 e260/231 9:a229340f:::benchmark_data_smithi050_105098_object29701:head v 260'8046, mlcod=260'8044)", 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 49.154846978000002, 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.047985633999999999, 2024-04-06T03:37:41.352 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3200000000000005e-07 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146482+0000", 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.434e-06 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184884+0000", 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.038401804999999997 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184897+0000", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3607e-05 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184969+0000", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1616e-05 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.185238+0000", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026927299999999999 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.192198+0000", 2024-04-06T03:37:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006959659 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.194461+0000", 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0022636079999999999 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:150343 9.6 e265/231 9:698074d9:::benchmark_data_smithi050_105098_object53433:head v 265'9487, mlcod=265'9477)", 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:18.135008+0000", 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 23.166314625999998, 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.043565305999999998, 2024-04-06T03:37:41.356 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.135008+0000", 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.135008+0000", 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.135010+0000", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8500000000000001e-06 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.135028+0000", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7609e-05 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.135069+0000", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1236000000000003e-05 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.135091+0000", 2024-04-06T03:37:41.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2473e-05 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.149531+0000", 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.014439309000000001 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.149569+0000", 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8878999999999998e-05 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.178499+0000", 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.028929059 2024-04-06T03:37:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:18.178573+0000", 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4890999999999998e-05 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158006 9.b e266/231 9:d95ff99c:::benchmark_data_smithi050_105098_object61096:head v 266'9938, mlcod=266'9935)", 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.269568+0000", 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.031754769999999, 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38518448100000002, 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.360 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269568+0000", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269568+0000", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269569+0000", 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5099999999999999e-07 2024-04-06T03:37:41.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269576+0000", 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3690000000000001e-06 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269585+0000", 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2380000000000003e-06 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269592+0000", 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9299999999999997e-06 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269645+0000", 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2774e-05 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269666+0000", 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0922999999999999e-05 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654688+0000", 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.385021944 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654752+0000", 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4652000000000004e-05 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158007 9.b e266/231 9:d021809f:::benchmark_data_smithi050_105098_object61097:head v 266'9939, mlcod=266'9936)", 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.270463+0000", 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.030859481, 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.39470748900000002, 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270463+0000", 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270463+0000", 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270464+0000", 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.37e-07 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270469+0000", 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8409999999999999e-06 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270478+0000", 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6390000000000004e-06 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270485+0000", 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8369999999999999e-06 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654855+0000", 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38436940200000003 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654898+0000", 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3735000000000001e-05 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.665106+0000", 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010207941999999999 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.665171+0000", 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4455999999999994e-05 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158010 9.7 e266/231 9:eb52c8f4:::benchmark_data_smithi050_105098_object61100:head v 266'9750, mlcod=266'9748)", 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.270494+0000", 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.030828856999999, 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38428609699999999, 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270494+0000", 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270494+0000", 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270495+0000", 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3200000000000004e-07 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270502+0000", 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0119999999999999e-06 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270512+0000", 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0539e-05 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270520+0000", 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8769999999999993e-06 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270566+0000", 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5643999999999999e-05 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270584+0000", 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8638999999999999e-05 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654721+0000", 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.384136431 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654780+0000", 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9123000000000003e-05 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158012 9.9 e266/231 9:9f35b491:::benchmark_data_smithi050_105098_object61102:head v 266'9951, mlcod=266'9949)", 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.270818+0000", 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.030504426, 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.39316358299999998, 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.372 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270818+0000", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270818+0000", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270819+0000", 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2399999999999998e-07 2024-04-06T03:37:41.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270826+0000", 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3390000000000004e-06 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270836+0000", 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0135e-05 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270843+0000", 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5509999999999996e-06 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654797+0000", 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.383953716 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654874+0000", 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7322000000000003e-05 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.663935+0000", 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0090613699999999991 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.663982+0000", 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6526000000000001e-05 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158016 9.a e266/231 9:5a69f0d8:::benchmark_data_smithi050_105098_object61106:head v 266'9932, mlcod=266'9929)", 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.272633+0000", 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.028690128999999, 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.396393147, 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.376 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272633+0000", 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272633+0000", 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272634+0000", 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5499999999999996e-07 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272640+0000", 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8689999999999998e-06 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654752+0000", 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.382111068 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654773+0000", 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1979000000000001e-05 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.655063+0000", 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028948599999999999 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.655122+0000", 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8545999999999998e-05 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.668903+0000", 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.013781513 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.669026+0000", 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000122731 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158023 9.8 e266/231 9:12f66281:::benchmark_data_smithi050_105098_object61113:head v 266'9691, mlcod=266'9689)", 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.283140+0000", 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 15.018182777, 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38472973399999999, 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283140+0000", 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283140+0000", 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283141+0000", 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1969999999999999e-06 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283150+0000", 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3149999999999998e-06 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654662+0000", 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.37151177299999999 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654694+0000", 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2109e-05 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654862+0000", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000167388 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654891+0000", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9600000000000001e-05 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.667814+0000", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012922849 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:37:41.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.667870+0000", 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5503e-05 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:37:41.384 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:37:41.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:41.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:41.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:41.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:41.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:41.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:41.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:41.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:41.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:41.940 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:41.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:41.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:41 smithi050 bash[21467]: cluster 2024-04-06T03:37:40.500754+0000 osd.0 (osd.0) 400 : cluster 0 5.2 scrub starts 2024-04-06T03:37:41.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:41 smithi050 bash[21467]: cluster 2024-04-06T03:37:40.503894+0000 osd.0 (osd.0) 401 : cluster 0 5.2 scrub ok 2024-04-06T03:37:41.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:41 smithi050 bash[29221]: cluster 2024-04-06T03:37:40.500754+0000 osd.0 (osd.0) 400 : cluster 0 5.2 scrub starts 2024-04-06T03:37:41.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:41 smithi050 bash[29221]: cluster 2024-04-06T03:37:40.503894+0000 osd.0 (osd.0) 401 : cluster 0 5.2 scrub ok 2024-04-06T03:37:42.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:42.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:42.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:42.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:41 smithi153 bash[23852]: cluster 2024-04-06T03:37:40.500754+0000 osd.0 (osd.0) 400 : cluster 0 5.2 scrub starts 2024-04-06T03:37:42.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:41 smithi153 bash[23852]: cluster 2024-04-06T03:37:40.503894+0000 osd.0 (osd.0) 401 : cluster 0 5.2 scrub ok 2024-04-06T03:37:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.204 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:42.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:42.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:42.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:42.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-06T03:37:42.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:42.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:42.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:42.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:42.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:42.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:42.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:42.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:42.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:42.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:42 smithi153 bash[23852]: cluster 2024-04-06T03:37:41.538735+0000 osd.0 (osd.0) 402 : cluster 0 9.5 scrub starts 2024-04-06T03:37:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:42 smithi153 bash[23852]: cluster 2024-04-06T03:37:41.653312+0000 osd.4 (osd.4) 387 : cluster 0 5.1 scrub starts 2024-04-06T03:37:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:42 smithi153 bash[23852]: cluster 2024-04-06T03:37:41.654157+0000 osd.4 (osd.4) 388 : cluster 0 5.1 scrub ok 2024-04-06T03:37:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:42 smithi153 bash[23852]: cluster 2024-04-06T03:37:42.062710+0000 osd.0 (osd.0) 403 : cluster 0 9.5 scrub ok 2024-04-06T03:37:42.984 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:42 smithi153 bash[23852]: cluster 2024-04-06T03:37:42.344876+0000 mgr.y (mgr.24368) 618 : cluster 0 pgmap v650: 148 pgs: 148 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:37:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[21467]: cluster 2024-04-06T03:37:41.538735+0000 osd.0 (osd.0) 402 : cluster 0 9.5 scrub starts 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[21467]: cluster 2024-04-06T03:37:41.653312+0000 osd.4 (osd.4) 387 : cluster 0 5.1 scrub starts 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[21467]: cluster 2024-04-06T03:37:41.654157+0000 osd.4 (osd.4) 388 : cluster 0 5.1 scrub ok 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[21467]: cluster 2024-04-06T03:37:42.062710+0000 osd.0 (osd.0) 403 : cluster 0 9.5 scrub ok 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[21467]: cluster 2024-04-06T03:37:42.344876+0000 mgr.y (mgr.24368) 618 : cluster 0 pgmap v650: 148 pgs: 148 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[29221]: cluster 2024-04-06T03:37:41.538735+0000 osd.0 (osd.0) 402 : cluster 0 9.5 scrub starts 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[29221]: cluster 2024-04-06T03:37:41.653312+0000 osd.4 (osd.4) 387 : cluster 0 5.1 scrub starts 2024-04-06T03:37:43.101 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[29221]: cluster 2024-04-06T03:37:41.654157+0000 osd.4 (osd.4) 388 : cluster 0 5.1 scrub ok 2024-04-06T03:37:43.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[29221]: cluster 2024-04-06T03:37:42.062710+0000 osd.0 (osd.0) 403 : cluster 0 9.5 scrub ok 2024-04-06T03:37:43.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:42 smithi050 bash[29221]: cluster 2024-04-06T03:37:42.344876+0000 mgr.y (mgr.24368) 618 : cluster 0 pgmap v650: 148 pgs: 148 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-04-06T03:37:43.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.143 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:43.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:43.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:43.448 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:37:43.512 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:37:43.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:43.542 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:43.561 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:37:43.602 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:37:43.643 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:37:43.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:43.664 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:43.676 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:37:43.710 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:37:43.751 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:37:43.763 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:37:43.763 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:37:43.763 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:37:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:37:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:37:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:37:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:37:43.764 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:37:43.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:43.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:43.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:43.968 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:37:44.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.083 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:44.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: cluster 2024-04-06T03:37:42.670945+0000 osd.4 (osd.4) 389 : cluster 0 5.e scrub starts 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: cluster 2024-04-06T03:37:42.672464+0000 osd.4 (osd.4) 390 : cluster 0 5.e scrub ok 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.202984+0000 osd.3 (osd.3) 414 : cluster 0 2.14 scrub starts 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.208156+0000 osd.3 (osd.3) 415 : cluster 0 2.14 scrub ok 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.248908+0000 osd.3 (osd.3) 416 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.248971+0000 osd.3 (osd.3) 417 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: audit 2024-04-06T03:37:43.387258+0000 mon.a (mon.0) 1437 : audit 0 from='client.? 172.21.15.50:0/2625438651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: audit 2024-04-06T03:37:43.448181+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.50:0/3425766413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:37:44.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:43 smithi153 bash[23852]: audit 2024-04-06T03:37:43.449115+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:37:44.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: cluster 2024-04-06T03:37:42.670945+0000 osd.4 (osd.4) 389 : cluster 0 5.e scrub starts 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: cluster 2024-04-06T03:37:42.672464+0000 osd.4 (osd.4) 390 : cluster 0 5.e scrub ok 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.202984+0000 osd.3 (osd.3) 414 : cluster 0 2.14 scrub starts 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.208156+0000 osd.3 (osd.3) 415 : cluster 0 2.14 scrub ok 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.248908+0000 osd.3 (osd.3) 416 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.248971+0000 osd.3 (osd.3) 417 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: audit 2024-04-06T03:37:43.387258+0000 mon.a (mon.0) 1437 : audit 0 from='client.? 172.21.15.50:0/2625438651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: audit 2024-04-06T03:37:43.448181+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.50:0/3425766413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:37:44.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[21467]: audit 2024-04-06T03:37:43.449115+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: cluster 2024-04-06T03:37:42.670945+0000 osd.4 (osd.4) 389 : cluster 0 5.e scrub starts 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: cluster 2024-04-06T03:37:42.672464+0000 osd.4 (osd.4) 390 : cluster 0 5.e scrub ok 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.202984+0000 osd.3 (osd.3) 414 : cluster 0 2.14 scrub starts 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.208156+0000 osd.3 (osd.3) 415 : cluster 0 2.14 scrub ok 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.248908+0000 osd.3 (osd.3) 416 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.248971+0000 osd.3 (osd.3) 417 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: audit 2024-04-06T03:37:43.387258+0000 mon.a (mon.0) 1437 : audit 0 from='client.? 172.21.15.50:0/2625438651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: audit 2024-04-06T03:37:43.448181+0000 mon.c (mon.2) 81 : audit 1 from='client.? 172.21.15.50:0/3425766413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:37:44.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:43 smithi050 bash[29221]: audit 2024-04-06T03:37:43.449115+0000 mon.a (mon.0) 1438 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:37:44.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:44.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:44.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:44.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:44.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:44.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:44.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:44.884 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:44.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:44.959 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-06T03:37:44.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:44.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.490679+0000 osd.1 (osd.1) 327 : cluster 0 2.d scrub starts 2024-04-06T03:37:44.998 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.497781+0000 osd.1 (osd.1) 328 : cluster 0 2.d scrub ok 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.543050+0000 osd.7 (osd.7) 435 : cluster 0 1.0 scrub starts 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.549998+0000 osd.7 (osd.7) 436 : cluster 0 1.0 scrub ok 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.929971+0000 mon.a (mon.0) 1439 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: audit 2024-04-06T03:37:43.945130+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:43.955522+0000 mon.a (mon.0) 1441 : cluster 0 osdmap e269: 8 total, 8 up, 8 in 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: audit 2024-04-06T03:37:43.985201+0000 mon.a (mon.0) 1442 : audit 1 from='client.? 172.21.15.50:0/3304597538' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:37:44.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:44 smithi153 bash[23852]: cluster 2024-04-06T03:37:44.345532+0000 mgr.y (mgr.24368) 619 : cluster 0 pgmap v652: 148 pgs: 148 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:45.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:45.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:45.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.490679+0000 osd.1 (osd.1) 327 : cluster 0 2.d scrub starts 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.497781+0000 osd.1 (osd.1) 328 : cluster 0 2.d scrub ok 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.543050+0000 osd.7 (osd.7) 435 : cluster 0 1.0 scrub starts 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.549998+0000 osd.7 (osd.7) 436 : cluster 0 1.0 scrub ok 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.929971+0000 mon.a (mon.0) 1439 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: audit 2024-04-06T03:37:43.945130+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:43.955522+0000 mon.a (mon.0) 1441 : cluster 0 osdmap e269: 8 total, 8 up, 8 in 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: audit 2024-04-06T03:37:43.985201+0000 mon.a (mon.0) 1442 : audit 1 from='client.? 172.21.15.50:0/3304597538' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:37:45.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[21467]: cluster 2024-04-06T03:37:44.345532+0000 mgr.y (mgr.24368) 619 : cluster 0 pgmap v652: 148 pgs: 148 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.490679+0000 osd.1 (osd.1) 327 : cluster 0 2.d scrub starts 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.497781+0000 osd.1 (osd.1) 328 : cluster 0 2.d scrub ok 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.543050+0000 osd.7 (osd.7) 435 : cluster 0 1.0 scrub starts 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.549998+0000 osd.7 (osd.7) 436 : cluster 0 1.0 scrub ok 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.929971+0000 mon.a (mon.0) 1439 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: audit 2024-04-06T03:37:43.945130+0000 mon.a (mon.0) 1440 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:43.955522+0000 mon.a (mon.0) 1441 : cluster 0 osdmap e269: 8 total, 8 up, 8 in 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: audit 2024-04-06T03:37:43.985201+0000 mon.a (mon.0) 1442 : audit 1 from='client.? 172.21.15.50:0/3304597538' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:37:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:44 smithi050 bash[29221]: cluster 2024-04-06T03:37:44.345532+0000 mgr.y (mgr.24368) 619 : cluster 0 pgmap v652: 148 pgs: 148 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 908 op/s 2024-04-06T03:37:45.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:45.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:45.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:45.425 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:45.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:45.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:45.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:45.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:45.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:45.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:45.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:45.978 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:45.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:46.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:46.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:46.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:46.218 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:45 smithi153 bash[23852]: audit 2024-04-06T03:37:44.946747+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.50:0/3304597538' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:37:46.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:45 smithi153 bash[23852]: cluster 2024-04-06T03:37:44.951401+0000 mon.a (mon.0) 1444 : cluster 0 osdmap e270: 8 total, 8 up, 8 in 2024-04-06T03:37:46.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:45 smithi050 bash[21467]: audit 2024-04-06T03:37:44.946747+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.50:0/3304597538' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:37:46.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:45 smithi050 bash[21467]: cluster 2024-04-06T03:37:44.951401+0000 mon.a (mon.0) 1444 : cluster 0 osdmap e270: 8 total, 8 up, 8 in 2024-04-06T03:37:46.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:45 smithi050 bash[29221]: audit 2024-04-06T03:37:44.946747+0000 mon.a (mon.0) 1443 : audit 1 from='client.? 172.21.15.50:0/3304597538' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:37:46.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:45 smithi050 bash[29221]: cluster 2024-04-06T03:37:44.951401+0000 mon.a (mon.0) 1444 : cluster 0 osdmap e270: 8 total, 8 up, 8 in 2024-04-06T03:37:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.240 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:46.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:46.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:46.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:46.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:46.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:46.638 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:46.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:46.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:46.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:46.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:46.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:46.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:47.004 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:46 smithi153 bash[23852]: cluster 2024-04-06T03:37:46.345992+0000 mgr.y (mgr.24368) 620 : cluster 0 pgmap v654: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 793 B/s rd, 0 B/s wr, 771 op/s 2024-04-06T03:37:47.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:47.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:47.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:47.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:47.155 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:37:47.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:47.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:47.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:46 smithi050 bash[21467]: cluster 2024-04-06T03:37:46.345992+0000 mgr.y (mgr.24368) 620 : cluster 0 pgmap v654: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 793 B/s rd, 0 B/s wr, 771 op/s 2024-04-06T03:37:47.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:46 smithi050 bash[29221]: cluster 2024-04-06T03:37:46.345992+0000 mgr.y (mgr.24368) 620 : cluster 0 pgmap v654: 148 pgs: 148 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 793 B/s rd, 0 B/s wr, 771 op/s 2024-04-06T03:37:47.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:47.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:47.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:47.535 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:47.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:47.661 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:47.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:47.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:47.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:47.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:47.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:47.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:48.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:48.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:48.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:48.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:48.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:48.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:48.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:48.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:48.439 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:48.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:48.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:48.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:48.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:48.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:48.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:48.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 4 1 2024-04-06T03:37:49.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:49.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:49.300 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:49.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:49.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:49.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:49.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:49 smithi050 bash[21467]: cluster 2024-04-06T03:37:48.346604+0000 mgr.y (mgr.24368) 621 : cluster 0 pgmap v655: 148 pgs: 148 active+clean; 1.1 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 904 op/s 2024-04-06T03:37:49.487 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:49 smithi050 bash[29221]: cluster 2024-04-06T03:37:48.346604+0000 mgr.y (mgr.24368) 621 : cluster 0 pgmap v655: 148 pgs: 148 active+clean; 1.1 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 904 op/s 2024-04-06T03:37:49.487 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:37:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:37:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:37:49.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:49.572 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:49 smithi153 bash[23852]: cluster 2024-04-06T03:37:48.346604+0000 mgr.y (mgr.24368) 621 : cluster 0 pgmap v655: 148 pgs: 148 active+clean; 1.1 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 904 op/s 2024-04-06T03:37:49.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:49.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:49.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:49.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:49.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:49.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:49.948 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:37:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:37:49.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.003 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:50.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:50.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:50.139 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:50.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:50.274 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:50.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:50.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:50.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:50.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:50.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:50.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:50.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:50.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:50.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:37:50.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:37:50.935 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:50.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:50.957 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:50 smithi050 bash[29221]: audit 2024-04-06T03:37:49.727914+0000 mon.a (mon.0) 1445 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:50.957 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:50 smithi050 bash[29221]: audit 2024-04-06T03:37:49.731568+0000 mon.a (mon.0) 1446 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:50.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:50 smithi050 bash[21467]: audit 2024-04-06T03:37:49.727914+0000 mon.a (mon.0) 1445 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:50.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:50 smithi050 bash[21467]: audit 2024-04-06T03:37:49.731568+0000 mon.a (mon.0) 1446 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:50.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:50 smithi153 bash[23852]: audit 2024-04-06T03:37:49.727914+0000 mon.a (mon.0) 1445 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:37:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:50 smithi153 bash[23852]: audit 2024-04-06T03:37:49.731568+0000 mon.a (mon.0) 1446 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:37:51.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:51.080 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:51.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:51.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:51.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:51.345 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:51.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:51.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:51.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:51.610 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:51.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:51.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:51.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:51.859 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:51 smithi153 bash[23852]: audit 2024-04-06T03:37:49.831269+0000 mgr.y (mgr.24368) 622 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:51.859 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:51 smithi153 bash[23852]: cluster 2024-04-06T03:37:49.960827+0000 osd.5 (osd.5) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:51.859 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:51 smithi153 bash[23852]: cluster 2024-04-06T03:37:49.960880+0000 osd.5 (osd.5) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:51.859 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:51 smithi153 bash[23852]: cluster 2024-04-06T03:37:50.347342+0000 mgr.y (mgr.24368) 623 : cluster 0 pgmap v656: 148 pgs: 148 active+clean; 938 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-04-06T03:37:51.859 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:51 smithi153 bash[23852]: cluster 2024-04-06T03:37:50.739605+0000 mon.a (mon.0) 1447 : cluster 0 osdmap e271: 8 total, 8 up, 8 in 2024-04-06T03:37:51.872 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:51.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:51.976 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[21467]: audit 2024-04-06T03:37:49.831269+0000 mgr.y (mgr.24368) 622 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[21467]: cluster 2024-04-06T03:37:49.960827+0000 osd.5 (osd.5) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[21467]: cluster 2024-04-06T03:37:49.960880+0000 osd.5 (osd.5) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[21467]: cluster 2024-04-06T03:37:50.347342+0000 mgr.y (mgr.24368) 623 : cluster 0 pgmap v656: 148 pgs: 148 active+clean; 938 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[21467]: cluster 2024-04-06T03:37:50.739605+0000 mon.a (mon.0) 1447 : cluster 0 osdmap e271: 8 total, 8 up, 8 in 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[29221]: audit 2024-04-06T03:37:49.831269+0000 mgr.y (mgr.24368) 622 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:37:51.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[29221]: cluster 2024-04-06T03:37:49.960827+0000 osd.5 (osd.5) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:37:51.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[29221]: cluster 2024-04-06T03:37:49.960880+0000 osd.5 (osd.5) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:37:51.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[29221]: cluster 2024-04-06T03:37:50.347342+0000 mgr.y (mgr.24368) 623 : cluster 0 pgmap v656: 148 pgs: 148 active+clean; 938 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-04-06T03:37:51.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:51 smithi050 bash[29221]: cluster 2024-04-06T03:37:50.739605+0000 mon.a (mon.0) 1447 : cluster 0 osdmap e271: 8 total, 8 up, 8 in 2024-04-06T03:37:52.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:52.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:52.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:52.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:52.273 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:52.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:52.367 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:37:52.367 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:37:52.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:52.405 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:52.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:52.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:52.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:52.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:52.664 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:52.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:52.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:52.796 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:52.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:52.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:52.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:53.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:53.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:53.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:53.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:53.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.348 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:53.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-06T03:37:53.488 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:53.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:53.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:53.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.754 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:53.830 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:53.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:53 smithi050 bash[29221]: cluster 2024-04-06T03:37:52.348013+0000 mgr.y (mgr.24368) 624 : cluster 0 pgmap v658: 148 pgs: 148 active+clean; 811 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-04-06T03:37:53.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:53 smithi050 bash[21467]: cluster 2024-04-06T03:37:52.348013+0000 mgr.y (mgr.24368) 624 : cluster 0 pgmap v658: 148 pgs: 148 active+clean; 811 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-04-06T03:37:53.884 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:53.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:53.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:53.999 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:53 smithi153 bash[23852]: cluster 2024-04-06T03:37:52.348013+0000 mgr.y (mgr.24368) 624 : cluster 0 pgmap v658: 148 pgs: 148 active+clean; 811 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-04-06T03:37:54.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:54.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:54.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:54.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:54.295 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:54.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:54.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:54.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:54.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.703 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:54.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:54.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:54.851 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:37:54.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:54.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:54.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:55.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:54 smithi153 bash[23852]: cluster 2024-04-06T03:37:54.348626+0000 mgr.y (mgr.24368) 625 : cluster 0 pgmap v659: 148 pgs: 148 active+clean; 750 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 846 op/s 2024-04-06T03:37:55.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:55.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:55.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:55.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:55.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:54 smithi050 bash[21467]: cluster 2024-04-06T03:37:54.348626+0000 mgr.y (mgr.24368) 625 : cluster 0 pgmap v659: 148 pgs: 148 active+clean; 750 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 846 op/s 2024-04-06T03:37:55.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:54 smithi050 bash[29221]: cluster 2024-04-06T03:37:54.348626+0000 mgr.y (mgr.24368) 625 : cluster 0 pgmap v659: 148 pgs: 148 active+clean; 750 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 846 op/s 2024-04-06T03:37:55.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:55.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:55.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:55.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:55.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:55.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:55.766 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:55.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:55.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:55.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:55.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:56.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:56.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:56.163 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:56.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:56.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:56.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:56.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:56.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:56.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:56.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:56.845 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:56.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:56.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:56.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:57.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:57.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:57.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:57.225 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:57.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:57.358 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:57.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:57.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:37:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:37:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:57 smithi153 bash[23852]: cluster 2024-04-06T03:37:56.349121+0000 mgr.y (mgr.24368) 626 : cluster 0 pgmap v660: 148 pgs: 148 active+clean; 620 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:37:57.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:57.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:57.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:57.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:57.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:57 smithi050 bash[21467]: cluster 2024-04-06T03:37:56.349121+0000 mgr.y (mgr.24368) 626 : cluster 0 pgmap v660: 148 pgs: 148 active+clean; 620 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:37:57.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:57 smithi050 bash[29221]: cluster 2024-04-06T03:37:56.349121+0000 mgr.y (mgr.24368) 626 : cluster 0 pgmap v660: 148 pgs: 148 active+clean; 620 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:37:57.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:57.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:57.890 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:57.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:57.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:58.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:58.143 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:37:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:37:58.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:58.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:58.352 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:37:58.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:58.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:58.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:58.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:58.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:58.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:58.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:58.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:58.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:58.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:59.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:59.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:59.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:59.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:37:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:37:59.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:59.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:37:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:37:59.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:59.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:37:59.563 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:59.586 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:37:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:37:59.650 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:37:59 smithi153 bash[23852]: cluster 2024-04-06T03:37:58.349884+0000 mgr.y (mgr.24368) 627 : cluster 0 pgmap v661: 148 pgs: 148 active+clean; 481 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-04-06T03:37:59.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:59.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:37:59 smithi050 bash[21467]: cluster 2024-04-06T03:37:58.349884+0000 mgr.y (mgr.24368) 627 : cluster 0 pgmap v661: 148 pgs: 148 active+clean; 481 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-04-06T03:37:59.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:37:59 smithi050 bash[29221]: cluster 2024-04-06T03:37:58.349884+0000 mgr.y (mgr.24368) 627 : cluster 0 pgmap v661: 148 pgs: 148 active+clean; 481 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-04-06T03:37:59.700 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:37:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:37:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:37:59.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:37:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:37:59.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:59.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:37:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:37:59.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:37:59.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:37:59.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:37:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:37:59.981 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:37:59 smithi153 bash[51781]: logger=sqlstore.transactions t=2024-04-06T03:37:59.831312455Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-06T03:37:59.992 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:00.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:00.133 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:00.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:00.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:00.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:00.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:00.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:00.559 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:00.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:00.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:00.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:00.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:00.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:00.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:01.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:01.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:01.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:01.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:01.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:01.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:01.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:01.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:01.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:01.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:01.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:01.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:01.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:01 smithi050 bash[21467]: audit 2024-04-06T03:37:59.841660+0000 mgr.y (mgr.24368) 628 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:01.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:01 smithi050 bash[21467]: cluster 2024-04-06T03:38:00.350524+0000 mgr.y (mgr.24368) 629 : cluster 0 pgmap v662: 148 pgs: 148 active+clean; 308 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:38:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:01 smithi050 bash[29221]: audit 2024-04-06T03:37:59.841660+0000 mgr.y (mgr.24368) 628 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:01.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:01 smithi050 bash[29221]: cluster 2024-04-06T03:38:00.350524+0000 mgr.y (mgr.24368) 629 : cluster 0 pgmap v662: 148 pgs: 148 active+clean; 308 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:38:01.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:01.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:01 smithi153 bash[23852]: audit 2024-04-06T03:37:59.841660+0000 mgr.y (mgr.24368) 628 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:01 smithi153 bash[23852]: cluster 2024-04-06T03:38:00.350524+0000 mgr.y (mgr.24368) 629 : cluster 0 pgmap v662: 148 pgs: 148 active+clean; 308 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:38:01.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:01.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:01.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:01.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:02.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:02.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:02.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:02.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:02.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:02.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:02.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:02.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:02.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:02.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:02.654 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:02.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:02.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:02.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:02.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:03.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:03.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:03.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:03.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:03.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:03.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:03.320 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:03.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:03.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:03.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:03.596 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:03.610 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:38:03.610 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:38:03.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:03 smithi153 bash[23852]: cluster 2024-04-06T03:38:02.351141+0000 mgr.y (mgr.24368) 630 : cluster 0 pgmap v663: 148 pgs: 148 active+clean; 173 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:38:03.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:03.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:03.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:03 smithi050 bash[21467]: cluster 2024-04-06T03:38:02.351141+0000 mgr.y (mgr.24368) 630 : cluster 0 pgmap v663: 148 pgs: 148 active+clean; 173 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:38:03.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:03 smithi050 bash[29221]: cluster 2024-04-06T03:38:02.351141+0000 mgr.y (mgr.24368) 630 : cluster 0 pgmap v663: 148 pgs: 148 active+clean; 173 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:38:03.738 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Removed 96907 objects 2024-04-06T03:38:03.739 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Clean up completed and total clean up time :105.321 2024-04-06T03:38:03.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:03.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:03.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:03.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:03.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:04.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:04.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:04.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:04.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:04.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:04.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:04.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:04.507 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:04.525 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-06T03:38:04.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:04.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:04.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:04 smithi050 bash[21467]: cluster 2024-04-06T03:38:02.709925+0000 osd.6 (osd.6) 97 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:04.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:04 smithi050 bash[21467]: cluster 2024-04-06T03:38:02.709992+0000 osd.6 (osd.6) 98 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:04.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:04 smithi050 bash[21467]: cluster 2024-04-06T03:38:03.427562+0000 mon.a (mon.0) 1448 : cluster 0 osdmap e272: 8 total, 8 up, 8 in 2024-04-06T03:38:04.653 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:04 smithi050 bash[29221]: cluster 2024-04-06T03:38:02.709925+0000 osd.6 (osd.6) 97 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:04.653 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:04 smithi050 bash[29221]: cluster 2024-04-06T03:38:02.709992+0000 osd.6 (osd.6) 98 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:04.653 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:04 smithi050 bash[29221]: cluster 2024-04-06T03:38:03.427562+0000 mon.a (mon.0) 1448 : cluster 0 osdmap e272: 8 total, 8 up, 8 in 2024-04-06T03:38:04.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:04.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:04 smithi153 bash[23852]: cluster 2024-04-06T03:38:02.709925+0000 osd.6 (osd.6) 97 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:04 smithi153 bash[23852]: cluster 2024-04-06T03:38:02.709992+0000 osd.6 (osd.6) 98 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:04 smithi153 bash[23852]: cluster 2024-04-06T03:38:03.427562+0000 mon.a (mon.0) 1448 : cluster 0 osdmap e272: 8 total, 8 up, 8 in 2024-04-06T03:38:04.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:04.894 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:04.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:04.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:05.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:05.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:05.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:05.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:05.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:05.420 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:05.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:05.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:05 smithi050 bash[21467]: cluster 2024-04-06T03:38:04.352101+0000 mgr.y (mgr.24368) 631 : cluster 0 pgmap v665: 148 pgs: 148 active+clean; 127 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 997 op/s 2024-04-06T03:38:05.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:05 smithi050 bash[21467]: audit 2024-04-06T03:38:04.749555+0000 mon.a (mon.0) 1449 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:05.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:05 smithi050 bash[29221]: cluster 2024-04-06T03:38:04.352101+0000 mgr.y (mgr.24368) 631 : cluster 0 pgmap v665: 148 pgs: 148 active+clean; 127 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 997 op/s 2024-04-06T03:38:05.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:05 smithi050 bash[29221]: audit 2024-04-06T03:38:04.749555+0000 mon.a (mon.0) 1449 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:05.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:05.668 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:05 smithi153 bash[23852]: cluster 2024-04-06T03:38:04.352101+0000 mgr.y (mgr.24368) 631 : cluster 0 pgmap v665: 148 pgs: 148 active+clean; 127 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 997 op/s 2024-04-06T03:38:05.668 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:05 smithi153 bash[23852]: audit 2024-04-06T03:38:04.749555+0000 mon.a (mon.0) 1449 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:05.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:05.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:05.819 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:05.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:05.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:05.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:06.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:06.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:06.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:06.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:06.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:06.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:06.340 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:06.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:06.470 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:06.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:06.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:06.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:06.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:06.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:06.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:06.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:06.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:06.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:06.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:07.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:07.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:07.257 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:07.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:07.382 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:07.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:07.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:07.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:07.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:07.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:07.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:07 smithi050 bash[29221]: cluster 2024-04-06T03:38:06.353125+0000 mgr.y (mgr.24368) 632 : cluster 0 pgmap v666: 148 pgs: 148 active+clean; 46 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 918 op/s 2024-04-06T03:38:07.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:07 smithi050 bash[29221]: cluster 2024-04-06T03:38:06.855829+0000 osd.7 (osd.7) 437 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:07.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:07 smithi050 bash[29221]: cluster 2024-04-06T03:38:06.855944+0000 osd.7 (osd.7) 438 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:07.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:07 smithi050 bash[21467]: cluster 2024-04-06T03:38:06.353125+0000 mgr.y (mgr.24368) 632 : cluster 0 pgmap v666: 148 pgs: 148 active+clean; 46 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 918 op/s 2024-04-06T03:38:07.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:07 smithi050 bash[21467]: cluster 2024-04-06T03:38:06.855829+0000 osd.7 (osd.7) 437 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:07.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:07 smithi050 bash[21467]: cluster 2024-04-06T03:38:06.855944+0000 osd.7 (osd.7) 438 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:07.750 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:07 smithi153 bash[23852]: cluster 2024-04-06T03:38:06.353125+0000 mgr.y (mgr.24368) 632 : cluster 0 pgmap v666: 148 pgs: 148 active+clean; 46 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 918 op/s 2024-04-06T03:38:07.751 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:07 smithi153 bash[23852]: cluster 2024-04-06T03:38:06.855829+0000 osd.7 (osd.7) 437 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:07.751 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:07 smithi153 bash[23852]: cluster 2024-04-06T03:38:06.855944+0000 osd.7 (osd.7) 438 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:07.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:07.779 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:38:07.800 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:38:07.822 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:38:07.843 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:38:07.864 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:38:07.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:07.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:07.889 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:38:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:07.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:07.916 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:38:07.941 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:38:07.942 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:38:07.942 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:38:07.942 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:38:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:38:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:38:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:38:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:38:07.943 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:38:07.991 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:08.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:08.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.147 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:08.250 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:08.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:08.408 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:08.454 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-06T03:38:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:38:08.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:08 smithi153 bash[23852]: cluster 2024-04-06T03:38:07.453147+0000 mon.a (mon.0) 1450 : cluster 0 osdmap e273: 8 total, 8 up, 8 in 2024-04-06T03:38:08.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:08 smithi153 bash[23852]: audit 2024-04-06T03:38:07.744796+0000 mon.a (mon.0) 1451 : audit 0 from='client.? 172.21.15.50:0/303753233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:08.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:08 smithi153 bash[23852]: audit 2024-04-06T03:38:07.934201+0000 mon.a (mon.0) 1452 : audit 1 from='client.? 172.21.15.50:0/2840656290' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:38:08.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:08 smithi153 bash[23852]: audit 2024-04-06T03:38:07.943520+0000 mon.c (mon.2) 82 : audit 1 from='client.? 172.21.15.50:0/902466716' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:08.522 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:08 smithi153 bash[23852]: audit 2024-04-06T03:38:07.943779+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:08.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:08.687 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-04-06T03:38:08.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-04-06T03:38:08.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[29221]: cluster 2024-04-06T03:38:07.453147+0000 mon.a (mon.0) 1450 : cluster 0 osdmap e273: 8 total, 8 up, 8 in 2024-04-06T03:38:08.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[29221]: audit 2024-04-06T03:38:07.744796+0000 mon.a (mon.0) 1451 : audit 0 from='client.? 172.21.15.50:0/303753233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:08.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[29221]: audit 2024-04-06T03:38:07.934201+0000 mon.a (mon.0) 1452 : audit 1 from='client.? 172.21.15.50:0/2840656290' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[29221]: audit 2024-04-06T03:38:07.943520+0000 mon.c (mon.2) 82 : audit 1 from='client.? 172.21.15.50:0/902466716' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[29221]: audit 2024-04-06T03:38:07.943779+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[21467]: cluster 2024-04-06T03:38:07.453147+0000 mon.a (mon.0) 1450 : cluster 0 osdmap e273: 8 total, 8 up, 8 in 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[21467]: audit 2024-04-06T03:38:07.744796+0000 mon.a (mon.0) 1451 : audit 0 from='client.? 172.21.15.50:0/303753233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[21467]: audit 2024-04-06T03:38:07.934201+0000 mon.a (mon.0) 1452 : audit 1 from='client.? 172.21.15.50:0/2840656290' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[21467]: audit 2024-04-06T03:38:07.943520+0000 mon.c (mon.2) 82 : audit 1 from='client.? 172.21.15.50:0/902466716' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:08.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:08 smithi050 bash[21467]: audit 2024-04-06T03:38:07.943779+0000 mon.a (mon.0) 1453 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:08.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:08.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:08.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:08.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:08.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:09.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:09.047 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:09.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:09.175 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:09.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:09.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:09.307 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:38:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:09.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:09.421 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:38:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:38:09.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:09.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:09 smithi153 bash[23852]: cluster 2024-04-06T03:38:08.353941+0000 mgr.y (mgr.24368) 633 : cluster 0 pgmap v668: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 616 op/s 2024-04-06T03:38:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:09 smithi153 bash[23852]: cluster 2024-04-06T03:38:08.446782+0000 mon.a (mon.0) 1454 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:38:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:09 smithi153 bash[23852]: audit 2024-04-06T03:38:08.448977+0000 mon.a (mon.0) 1455 : audit 1 from='client.? 172.21.15.50:0/2840656290' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:38:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:09 smithi153 bash[23852]: audit 2024-04-06T03:38:08.449118+0000 mon.a (mon.0) 1456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:38:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:09 smithi153 bash[23852]: cluster 2024-04-06T03:38:08.454644+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e274: 8 total, 8 up, 8 in 2024-04-06T03:38:09.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:09.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:09.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[21467]: cluster 2024-04-06T03:38:08.353941+0000 mgr.y (mgr.24368) 633 : cluster 0 pgmap v668: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 616 op/s 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[21467]: cluster 2024-04-06T03:38:08.446782+0000 mon.a (mon.0) 1454 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[21467]: audit 2024-04-06T03:38:08.448977+0000 mon.a (mon.0) 1455 : audit 1 from='client.? 172.21.15.50:0/2840656290' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[21467]: audit 2024-04-06T03:38:08.449118+0000 mon.a (mon.0) 1456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[21467]: cluster 2024-04-06T03:38:08.454644+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e274: 8 total, 8 up, 8 in 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[29221]: cluster 2024-04-06T03:38:08.353941+0000 mgr.y (mgr.24368) 633 : cluster 0 pgmap v668: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 616 op/s 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[29221]: cluster 2024-04-06T03:38:08.446782+0000 mon.a (mon.0) 1454 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[29221]: audit 2024-04-06T03:38:08.448977+0000 mon.a (mon.0) 1455 : audit 1 from='client.? 172.21.15.50:0/2840656290' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[29221]: audit 2024-04-06T03:38:08.449118+0000 mon.a (mon.0) 1456 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:38:09.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:09 smithi050 bash[29221]: cluster 2024-04-06T03:38:08.454644+0000 mon.a (mon.0) 1457 : cluster 0 osdmap e274: 8 total, 8 up, 8 in 2024-04-06T03:38:09.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:09.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:09.829 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:09.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:09.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:09.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:10.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:10.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:10.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:10.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:10.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:38:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:38:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:10.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:10.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:10.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:10.474 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:10.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:10.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:10.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 5 1 2024-04-06T03:38:10.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:10.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:10.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:10.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:10.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:10.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:10.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:10.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:11.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:38:11.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:11.117 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:11.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:11.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:11.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:11.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:11.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:11.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:11.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:11.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:11.635 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:11.637 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:11 smithi153 bash[23852]: audit 2024-04-06T03:38:09.851572+0000 mgr.y (mgr.24368) 634 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:11.637 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:11 smithi153 bash[23852]: cluster 2024-04-06T03:38:10.354769+0000 mgr.y (mgr.24368) 635 : cluster 0 pgmap v670: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 399 op/s 2024-04-06T03:38:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:11.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:11 smithi050 bash[21467]: audit 2024-04-06T03:38:09.851572+0000 mgr.y (mgr.24368) 634 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:11.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:11 smithi050 bash[21467]: cluster 2024-04-06T03:38:10.354769+0000 mgr.y (mgr.24368) 635 : cluster 0 pgmap v670: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 399 op/s 2024-04-06T03:38:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:11 smithi050 bash[29221]: audit 2024-04-06T03:38:09.851572+0000 mgr.y (mgr.24368) 634 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:11.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:11 smithi050 bash[29221]: cluster 2024-04-06T03:38:10.354769+0000 mgr.y (mgr.24368) 635 : cluster 0 pgmap v670: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 399 op/s 2024-04-06T03:38:11.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:11.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:11.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:11.897 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:11.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:12.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:12.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:12.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:12.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:12.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:12.288 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:12.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.422 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:12.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:12.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:12.632 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:38:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:38:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:38:12.632 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17650 8.5 e188/182 8:a5b71103:::benchmark_data_smithi050_86143_object17649:head v 188'1100, mlcod=188'1098)", 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.618177+0000", 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 484.01114197700002, 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15274811899999999, 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618177+0000", 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618177+0000", 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618178+0000", 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2800000000000005e-07 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618205+0000", 2024-04-06T03:38:12.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6792999999999999e-05 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618215+0000", 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0599e-05 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618221+0000", 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0090000000000004e-06 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618268+0000", 2024-04-06T03:38:12.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6851999999999998e-05 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.618289+0000", 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0798e-05 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619210+0000", 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000920838 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770925+0000", 2024-04-06T03:38:12.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.151715302 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17641 8.9 e188/182 8:9cd1a4c4:::benchmark_data_smithi050_86143_object17640:head v 188'1118, mlcod=188'1115)", 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 484.00936851500001, 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15268031300000001, 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.619951+0000", 2024-04-06T03:38:12.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3499999999999995e-07 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.620026+0000", 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5055999999999994e-05 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770855+0000", 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.150828727 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770876+0000", 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1024999999999999e-05 2024-04-06T03:38:12.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770972+0000", 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6123000000000001e-05 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771008+0000", 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5932000000000001e-05 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.772580+0000", 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001571467 2024-04-06T03:38:12.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.772631+0000", 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1248000000000001e-05 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17654 8.5 e188/182 8:ae505e77:::benchmark_data_smithi050_86143_object17653:head v 188'1101, mlcod=188'1098)", 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.622691+0000", 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 484.0066276, 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.15084618, 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.641 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622691+0000", 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622691+0000", 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622692+0000", 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0049999999999999e-06 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.622721+0000", 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8861999999999999e-05 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770859+0000", 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.14813772 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770875+0000", 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6096999999999999e-05 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770978+0000", 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000103112 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771008+0000", 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9629999999999999e-05 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773472+0000", 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024642190000000001 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773538+0000", 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5535000000000002e-05 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.14928.0:17669 8.3 e188/182 8:c5e49b00:::benchmark_data_smithi050_86143_object17668:head v 188'1081, mlcod=188'1079)", 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:30:08.634617+0000", 2024-04-06T03:38:12.645 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 483.99470156500001, 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.138758459, 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634617+0000", 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634617+0000", 2024-04-06T03:38:12.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634619+0000", 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.883e-06 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.634687+0000", 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7918999999999998e-05 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770893+0000", 2024-04-06T03:38:12.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.136205936 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.770905+0000", 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.223e-05 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771008+0000", 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000102527 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.771030+0000", 2024-04-06T03:38:12.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2368e-05 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773273+0000", 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0022429630000000002 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:30:08.773376+0000", 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000102633 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:76256 9.0 e248/231 9:03b5bf3c:::benchmark_data_smithi050_105098_object76255:head v 248'4725, mlcod=248'4720)", 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:48.185765+0000", 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 144.44355440000001, 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.035694731, 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.185765+0000", 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.185765+0000", 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.185767+0000", 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0999999999999998e-06 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.185817+0000", 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0476999999999998e-05 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.185838+0000", 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0645999999999999e-05 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.185852+0000", 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3967e-05 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.202103+0000", 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.016251333 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.202124+0000", 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0652000000000001e-05 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.221283+0000", 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.019158692000000001 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:48.221459+0000", 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017686399999999999 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(osd.1.0:66889 9.0 e249/231 9:0150ed92:::benchmark_data_smithi050_105098_object10022:head v 249'5046, mlcod=249'5038)", 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:50.707402+0000", 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 141.92191694900001, 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.04101204, 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.707402+0000", 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.707402+0000", 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.707403+0000", 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.24e-06 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.707414+0000", 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0334e-05 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.707443+0000", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9017e-05 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.707456+0000", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3624e-05 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.713725+0000", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0062690840000000003 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.713763+0000", 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7239e-05 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.748362+0000", 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.034599613000000001 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.748414+0000", 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1888999999999997e-05 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(osd.1.0:66890 9.0 e249/231 9:0151394c:::benchmark_data_smithi050_105098_object44468:head v 249'5047, mlcod=249'5038)", 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:50.709645+0000", 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 141.91967402099999, 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.038805421999999999, 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.709645+0000", 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.709645+0000", 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.709646+0000", 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3340000000000001e-06 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.709657+0000", 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0971999999999999e-05 2024-04-06T03:38:12.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.709676+0000", 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8372999999999998e-05 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.709688+0000", 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2174000000000001e-05 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.718468+0000", 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0087802350000000008 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.718529+0000", 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1045999999999998e-05 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.748419+0000", 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.029889563000000001 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.748450+0000", 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1724999999999999e-05 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(osd.1.0:66891 9.0 e249/231 9:0151d1ff:::benchmark_data_smithi050_105098_object32187:head v 249'5048, mlcod=249'5038)", 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:50.711690+0000", 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 141.917629132, 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.038809574999999999, 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.711690+0000", 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.711690+0000", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.711691+0000", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5120000000000001e-06 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.711705+0000", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4052e-05 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.711725+0000", 2024-04-06T03:38:12.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9690999999999998e-05 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.711738+0000", 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3074000000000001e-05 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.721907+0000", 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010168740000000001 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.721941+0000", 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4178999999999999e-05 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.750455+0000", 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.028513495 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.750499+0000", 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4832e-05 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(osd.1.0:66893 9.0 e249/231 9:01520edd:::benchmark_data_smithi050_105098_object2895:head v 249'5049, mlcod=249'5038)", 2024-04-06T03:38:12.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:50.714465+0000", 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 141.91485429100001, 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.036065945000000002, 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.714465+0000", 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.714465+0000", 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.714466+0000", 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3039999999999999e-06 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.714476+0000", 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0016e-05 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.714492+0000", 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5732000000000001e-05 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.714504+0000", 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2208999999999999e-05 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.725781+0000", 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.011276738999999999 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.725809+0000", 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8629e-05 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.750503+0000", 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.024693853000000002 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.750531+0000", 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7463000000000001e-05 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:77903 9.0 e249/231 9:090d879f:::benchmark_data_smithi050_105098_object77902:head v 249'5098, mlcod=249'5096)", 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:50.960073+0000", 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 141.66924597100001, 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.035411884999999997, 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960073+0000", 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960073+0000", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960074+0000", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4e-07 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960121+0000", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7379999999999997e-05 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960134+0000", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2140000000000001e-05 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960140+0000", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7009999999999999e-06 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960212+0000", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2181000000000006e-05 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.960234+0000", 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.141e-05 2024-04-06T03:38:12.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.995455+0000", 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.035221373 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:50.995485+0000", 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.976e-05 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(osd.1.0:68021 9.0 e250/231 9:041daef0:::benchmark_data_smithi050_105098_object21455:head v 250'5796, mlcod=250'5777)", 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:55.372775+0000", 2024-04-06T03:38:12.673 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 137.25654446300001, 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.035452809000000002, 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.372775+0000", 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.372775+0000", 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.372777+0000", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9180000000000001e-06 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.372796+0000", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9054000000000001e-05 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.372821+0000", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5653e-05 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.372838+0000", 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7062999999999999e-05 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.387707+0000", 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.014868386000000001 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.387738+0000", 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0948000000000003e-05 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.408162+0000", 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.020424670999999998 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:55.408227+0000", 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5116000000000005e-05 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.677 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:81565 9.0 e251/231 9:0ae2ee9b:::benchmark_data_smithi050_105098_object81564:head v 251'5984, mlcod=251'5982)", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:35:56.927922+0000", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 135.70139736499999, 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017969503000000001, 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.927922+0000", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.927922+0000", 2024-04-06T03:38:12.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.927923+0000", 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2729999999999999e-06 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.927977+0000", 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4033000000000003e-05 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.927995+0000", 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7957000000000001e-05 2024-04-06T03:38:12.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.928006+0000", 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1535000000000001e-05 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.934583+0000", 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0065767630000000002 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.934611+0000", 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7659999999999999e-05 2024-04-06T03:38:12.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.945751+0000", 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.011140499 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:35:56.945891+0000", 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013978300000000001 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:115048 9.b e259/231 9:d2cf8fdc:::benchmark_data_smithi050_105098_object18138:head v 259'7248, mlcod=259'7243)", 2024-04-06T03:38:12.681 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:38.327800+0000", 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 94.301519278000001, 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018581555999999999, 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.327800+0000", 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.327800+0000", 2024-04-06T03:38:12.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.327802+0000", 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.103e-06 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.327819+0000", 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.738e-05 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.327860+0000", 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0441999999999997e-05 2024-04-06T03:38:12.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.327881+0000", 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1287000000000001e-05 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.333771+0000", 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0058900089999999999 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.333911+0000", 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014039200000000001 2024-04-06T03:38:12.684 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.346169+0000", 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012257593000000001 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:38.346381+0000", 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00021235000000000001 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:118287 9.1 e259/231 9:8752c863:::benchmark_data_smithi050_105098_object21377:head v 259'7484, mlcod=259'7477)", 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:41.966992+0000", 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 90.662327340000004, 2024-04-06T03:38:12.685 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018315175, 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.966992+0000", 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.966992+0000", 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.686 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.966993+0000", 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1699999999999997e-07 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.967004+0000", 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1538e-05 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.967026+0000", 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1535999999999998e-05 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.687 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.967036+0000", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0096e-05 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.972417+0000", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0053814969999999998 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.972443+0000", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5519000000000001e-05 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.985265+0000", 2024-04-06T03:38:12.688 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012822513000000001 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:41.985307+0000", 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1559e-05 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:123710 9.1 e260/231 9:877a2dab:::benchmark_data_smithi050_105098_object26800:head v 260'7827, mlcod=260'7825)", 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:48.882453+0000", 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 83.746866531999999, 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018562597, 2024-04-06T03:38:12.689 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882453+0000", 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882453+0000", 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.690 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882454+0000", 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.252e-06 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882465+0000", 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0783e-05 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882486+0000", 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1841e-05 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.691 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882502+0000", 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5522000000000001e-05 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882617+0000", 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000115092 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.882663+0000", 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5769000000000002e-05 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.692 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.900941+0000", 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018277872000000001 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:48.901015+0000", 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4466000000000002e-05 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:125481 9.1 e260/231 9:8e109ba5:::benchmark_data_smithi050_105098_object28571:head v 260'7933, mlcod=260'7931)", 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:51.035999+0000", 2024-04-06T03:38:12.693 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 81.593320306999999, 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017327665999999999, 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.035999+0000", 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.035999+0000", 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.694 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.036000+0000", 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.733e-06 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.036016+0000", 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5319999999999999e-05 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.036057+0000", 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1152999999999997e-05 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.695 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.036080+0000", 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2609999999999999e-05 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.036224+0000", 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000144764 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.036280+0000", 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5708999999999998e-05 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.696 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.053201+0000", 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.016920727999999999 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:51.053326+0000", 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000125649 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:159404 9.b e266/231 9:dad824d1:::benchmark_data_smithi050_105098_object62494:head v 266'10033, mlcod=266'10031)", 2024-04-06T03:38:12.697 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:28.107989+0000", 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 44.521329575000003, 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.023832169, 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.107989+0000", 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.107989+0000", 2024-04-06T03:38:12.698 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.107991+0000", 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.091e-06 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.107999+0000", 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9899999999999997e-06 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.108013+0000", 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4080000000000001e-05 2024-04-06T03:38:12.699 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.108021+0000", 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7819999999999999e-06 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.108080+0000", 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8334000000000001e-05 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.108100+0000", 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0203e-05 2024-04-06T03:38:12.700 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.131785+0000", 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.02368549 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:28.131822+0000", 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6198999999999997e-05 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.701 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:161893 9.0 e266/231 9:0ef89e9a:::benchmark_data_smithi050_105098_object64983:head v 266'13933, mlcod=266'13931)", 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:31.222074+0000", 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 41.407244957000003, 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018956331, 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222074+0000", 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.702 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222074+0000", 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222075+0000", 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.54e-07 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222082+0000", 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2099999999999996e-06 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.703 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222095+0000", 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2746e-05 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222102+0000", 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.233e-06 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222159+0000", 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6656000000000001e-05 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.704 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.222180+0000", 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1483000000000001e-05 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.240959+0000", 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018778751999999999 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:31.241030+0000", 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1496999999999997e-05 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.705 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:166770 9.b e267/231 9:d9ae1541:::benchmark_data_smithi050_105098_object69860:head v 267'10521, mlcod=267'10519)", 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:35.859797+0000", 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 36.769522168000002, 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.018479705999999999, 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.859797+0000", 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.706 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.859797+0000", 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.859799+0000", 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6089999999999999e-06 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.859817+0000", 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7453e-05 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.707 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.859864+0000", 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6780000000000003e-05 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.859885+0000", 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1359e-05 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.860091+0000", 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00020573100000000001 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.708 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.860157+0000", 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6001999999999996e-05 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.878104+0000", 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017947498999999999 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:35.878277+0000", 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000172273 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.709 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:167617 9.1 e267/231 9:80d780cf:::benchmark_data_smithi050_105098_object70707:head v 267'10554, mlcod=267'10548)", 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:36.652764+0000", 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 35.976555464999997, 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.019038718999999999, 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.652764+0000", 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.710 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.652764+0000", 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.652764+0000", 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6700000000000003e-07 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.652772+0000", 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3289999999999996e-06 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.711 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.652786+0000", 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4613999999999999e-05 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.652799+0000", 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2968e-05 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.661219+0000", 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0084201740000000008 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.712 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.661241+0000", 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1277e-05 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.671690+0000", 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010449175999999999 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:36.671802+0000", 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112514 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.713 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:38:12.714 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:38:12.714 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:38:12.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:12.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:38:12.738 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:12.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:12.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:12.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:12.995 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:12.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:13.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:13.124 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:13.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:13.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:13.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:13.468 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:13.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:13 smithi050 bash[21467]: cluster 2024-04-06T03:38:12.355940+0000 mgr.y (mgr.24368) 636 : cluster 0 pgmap v671: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 253 op/s 2024-04-06T03:38:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:13 smithi050 bash[29221]: cluster 2024-04-06T03:38:12.355940+0000 mgr.y (mgr.24368) 636 : cluster 0 pgmap v671: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 253 op/s 2024-04-06T03:38:13.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:13.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-06T03:38:13.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:13.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:13.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.766 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:13 smithi153 bash[23852]: cluster 2024-04-06T03:38:12.355940+0000 mgr.y (mgr.24368) 636 : cluster 0 pgmap v671: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 253 op/s 2024-04-06T03:38:13.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:13.895 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:13.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:13.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:14.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:14.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:14.151 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:14.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:14.283 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:14.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:14.412 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:14.489 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:14.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:14.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:14.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:14.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:14.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:14.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:14.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:14.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:15.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:15.043 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:14 smithi050 bash[21467]: cluster 2024-04-06T03:38:14.357032+0000 mgr.y (mgr.24368) 637 : cluster 0 pgmap v672: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T03:38:15.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:14 smithi050 bash[29221]: cluster 2024-04-06T03:38:14.357032+0000 mgr.y (mgr.24368) 637 : cluster 0 pgmap v672: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T03:38:15.057 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:15.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:15.189 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:15.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:14 smithi153 bash[23852]: cluster 2024-04-06T03:38:14.357032+0000 mgr.y (mgr.24368) 637 : cluster 0 pgmap v672: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-06T03:38:15.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:15.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:15.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:15.508 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:15.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:15.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:15.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:15.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:15.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:15.846 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:15.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:15.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:15.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:16.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:16.108 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:16.211 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:16.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:16.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:16.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:16.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:16.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:16.522 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:16.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:16.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:16.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:16.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:16.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:16.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:16.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:16.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:17.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:17.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:17.137 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:17.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:17.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:17.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:17.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:17.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:17.528 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:17.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:17.643 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:17 smithi153 bash[23852]: cluster 2024-04-06T03:38:16.358099+0000 mgr.y (mgr.24368) 638 : cluster 0 pgmap v673: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-04-06T03:38:17.660 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:17.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:17 smithi050 bash[21467]: cluster 2024-04-06T03:38:16.358099+0000 mgr.y (mgr.24368) 638 : cluster 0 pgmap v673: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-04-06T03:38:17.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:17 smithi050 bash[29221]: cluster 2024-04-06T03:38:16.358099+0000 mgr.y (mgr.24368) 638 : cluster 0 pgmap v673: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-04-06T03:38:17.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:17.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:17.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:17.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:17.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:18.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:18.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:18.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:18.189 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:18.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:18.297 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:38:18.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:18.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:18.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:18.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:18.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:18.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:18.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:18.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:18.843 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:18.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:18.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:18.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:19.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:19.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:19.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:19.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:19.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:19.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:19.414 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:38:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:38:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:38:19.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:19.489 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:19.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:19.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:19.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:19 smithi050 bash[21467]: cluster 2024-04-06T03:38:18.358875+0000 mgr.y (mgr.24368) 639 : cluster 0 pgmap v674: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:19.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:19 smithi050 bash[29221]: cluster 2024-04-06T03:38:18.358875+0000 mgr.y (mgr.24368) 639 : cluster 0 pgmap v674: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:19.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:19.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:19 smithi153 bash[23852]: cluster 2024-04-06T03:38:18.358875+0000 mgr.y (mgr.24368) 639 : cluster 0 pgmap v674: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:19.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:19.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:19.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:19.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:19.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:19.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:20.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:20.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:38:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:38:20.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:20.139 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:20.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:20.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:20.268 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:20.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:20.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:20.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:20.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:20.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:20 smithi153 bash[23852]: cluster 2024-04-06T03:38:19.773664+0000 osd.4 (osd.4) 391 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:20 smithi153 bash[23852]: cluster 2024-04-06T03:38:19.773787+0000 osd.4 (osd.4) 392 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:20 smithi153 bash[23852]: audit 2024-04-06T03:38:19.779028+0000 mon.a (mon.0) 1458 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:20.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:20 smithi050 bash[21467]: cluster 2024-04-06T03:38:19.773664+0000 osd.4 (osd.4) 391 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:20.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:20 smithi050 bash[21467]: cluster 2024-04-06T03:38:19.773787+0000 osd.4 (osd.4) 392 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:20.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:20 smithi050 bash[21467]: audit 2024-04-06T03:38:19.779028+0000 mon.a (mon.0) 1458 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:20.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:20 smithi050 bash[29221]: cluster 2024-04-06T03:38:19.773664+0000 osd.4 (osd.4) 391 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:20.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:20 smithi050 bash[29221]: cluster 2024-04-06T03:38:19.773787+0000 osd.4 (osd.4) 392 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:20.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:20 smithi050 bash[29221]: audit 2024-04-06T03:38:19.779028+0000 mon.a (mon.0) 1458 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:20.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:20.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:20.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:20.785 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:20.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:20.916 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:20.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:21.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:21.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:21.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:21.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:21.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:21.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:21.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:21.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:21.541 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:38:21.542 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:38:21.542 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:38:21.542 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:38:21.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:21 smithi050 bash[21467]: audit 2024-04-06T03:38:19.862653+0000 mgr.y (mgr.24368) 640 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:21.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:21 smithi050 bash[21467]: cluster 2024-04-06T03:38:20.359887+0000 mgr.y (mgr.24368) 641 : cluster 0 pgmap v675: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-06T03:38:21.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:21 smithi050 bash[21467]: cluster 2024-04-06T03:38:20.429284+0000 mon.a (mon.0) 1459 : cluster 0 osdmap e275: 8 total, 8 up, 8 in 2024-04-06T03:38:21.551 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:21 smithi050 bash[29221]: audit 2024-04-06T03:38:19.862653+0000 mgr.y (mgr.24368) 640 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:21.551 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:21 smithi050 bash[29221]: cluster 2024-04-06T03:38:20.359887+0000 mgr.y (mgr.24368) 641 : cluster 0 pgmap v675: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-06T03:38:21.551 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:21 smithi050 bash[29221]: cluster 2024-04-06T03:38:20.429284+0000 mon.a (mon.0) 1459 : cluster 0 osdmap e275: 8 total, 8 up, 8 in 2024-04-06T03:38:21.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:21.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:21 smithi153 bash[23852]: audit 2024-04-06T03:38:19.862653+0000 mgr.y (mgr.24368) 640 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:21.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:21 smithi153 bash[23852]: cluster 2024-04-06T03:38:20.359887+0000 mgr.y (mgr.24368) 641 : cluster 0 pgmap v675: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-06T03:38:21.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:21 smithi153 bash[23852]: cluster 2024-04-06T03:38:20.429284+0000 mon.a (mon.0) 1459 : cluster 0 osdmap e275: 8 total, 8 up, 8 in 2024-04-06T03:38:21.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:21.693 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:21.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:21.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:21.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:21.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:21.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:22.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:22.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:22.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:22.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:22.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:22.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-06T03:38:22.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:22.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:22.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:22.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:22.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:22.738 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:22.841 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:22.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:22.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:22.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:23.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:23.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:23.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:23.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:23.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:23.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:23.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:23.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:23 smithi153 bash[23852]: cluster 2024-04-06T03:38:22.361000+0000 mgr.y (mgr.24368) 642 : cluster 0 pgmap v677: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:23.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:23.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:23 smithi050 bash[21467]: cluster 2024-04-06T03:38:22.361000+0000 mgr.y (mgr.24368) 642 : cluster 0 pgmap v677: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:23.651 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:23 smithi050 bash[29221]: cluster 2024-04-06T03:38:22.361000+0000 mgr.y (mgr.24368) 642 : cluster 0 pgmap v677: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:23.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:23.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:23.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:23.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:23.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:24.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:24.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:24.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:24.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:24.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:24.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:24.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:24.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:24.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:24.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:24.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:24.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:24.820 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:24.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:24.951 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:24.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:25.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:25.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:25.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:25.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:25.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:25.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:25.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:25.447 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_2' removed 2024-04-06T03:38:25.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:25.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:25.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:25.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:25.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:25.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:25 smithi050 bash[29221]: cluster 2024-04-06T03:38:24.362062+0000 mgr.y (mgr.24368) 643 : cluster 0 pgmap v678: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:25.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:25 smithi050 bash[29221]: audit 2024-04-06T03:38:24.972145+0000 mon.a (mon.0) 1460 : audit 1 from='client.? 172.21.15.50:0/1222440469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:38:25.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:25 smithi050 bash[21467]: cluster 2024-04-06T03:38:24.362062+0000 mgr.y (mgr.24368) 643 : cluster 0 pgmap v678: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:25.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:25 smithi050 bash[21467]: audit 2024-04-06T03:38:24.972145+0000 mon.a (mon.0) 1460 : audit 1 from='client.? 172.21.15.50:0/1222440469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:38:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:25 smithi153 bash[23852]: cluster 2024-04-06T03:38:24.362062+0000 mgr.y (mgr.24368) 643 : cluster 0 pgmap v678: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:38:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:25 smithi153 bash[23852]: audit 2024-04-06T03:38:24.972145+0000 mon.a (mon.0) 1460 : audit 1 from='client.? 172.21.15.50:0/1222440469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:38:25.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:25.859 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:25.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:25.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:25.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:26.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:26.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:26.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:26.254 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:26.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:26.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:26.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:26.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:26.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:26.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:26 smithi050 bash[21467]: audit 2024-04-06T03:38:25.443982+0000 mon.a (mon.0) 1461 : audit 1 from='client.? 172.21.15.50:0/1222440469' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:38:26.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:26 smithi050 bash[21467]: cluster 2024-04-06T03:38:25.451867+0000 mon.a (mon.0) 1462 : cluster 0 osdmap e276: 8 total, 8 up, 8 in 2024-04-06T03:38:26.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:26 smithi050 bash[29221]: audit 2024-04-06T03:38:25.443982+0000 mon.a (mon.0) 1461 : audit 1 from='client.? 172.21.15.50:0/1222440469' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:38:26.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:26 smithi050 bash[29221]: cluster 2024-04-06T03:38:25.451867+0000 mon.a (mon.0) 1462 : cluster 0 osdmap e276: 8 total, 8 up, 8 in 2024-04-06T03:38:26.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:26 smithi153 bash[23852]: audit 2024-04-06T03:38:25.443982+0000 mon.a (mon.0) 1461 : audit 1 from='client.? 172.21.15.50:0/1222440469' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:38:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:26 smithi153 bash[23852]: cluster 2024-04-06T03:38:25.451867+0000 mon.a (mon.0) 1462 : cluster 0 osdmap e276: 8 total, 8 up, 8 in 2024-04-06T03:38:26.879 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:26.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:27.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:27.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:27.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:27.179 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:38:27.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:27.302 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:27.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:27.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:27.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:27.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:27 smithi153 bash[23852]: cluster 2024-04-06T03:38:26.362908+0000 mgr.y (mgr.24368) 644 : cluster 0 pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:38:27.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:27.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:27.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:27.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:27 smithi050 bash[21467]: cluster 2024-04-06T03:38:26.362908+0000 mgr.y (mgr.24368) 644 : cluster 0 pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:38:27.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:27 smithi050 bash[29221]: cluster 2024-04-06T03:38:26.362908+0000 mgr.y (mgr.24368) 644 : cluster 0 pgmap v680: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:38:27.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:27.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:27.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:27.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:27.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:27.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:27.974 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-06T03:38:27.975 INFO:tasks.radosbench:Beginning radosbench... 2024-04-06T03:38:27.975 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-04-06T03:38:27.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool create unique_pool_3 16 2024-04-06T03:38:28.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:28.085 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:28.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:28.216 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:28.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:28.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:28.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:28.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:28.469 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:38:28.490 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:38:28.511 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:38:28.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:28 smithi050 bash[21467]: audit 2024-04-06T03:38:28.445929+0000 mon.c (mon.2) 83 : audit 0 from='client.? 172.21.15.50:0/1245477132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:28.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:28 smithi050 bash[29221]: audit 2024-04-06T03:38:28.445929+0000 mon.c (mon.2) 83 : audit 0 from='client.? 172.21.15.50:0/1245477132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:28.534 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:38:28.559 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:38:28.569 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:28.580 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:28 smithi153 bash[23852]: audit 2024-04-06T03:38:28.445929+0000 mon.c (mon.2) 83 : audit 0 from='client.? 172.21.15.50:0/1245477132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:28.585 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:38:28.596 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:28.610 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:38:28.633 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:38:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:38:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:38:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:38:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:38:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:38:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:38:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:38:28.635 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:38:28.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:28.725 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:28.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:28.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:28.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:28.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:28.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:29.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:29.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:29.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:29.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:29.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:29.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:29.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:29.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:38:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:38:29.478 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[29221]: cluster 2024-04-06T03:38:28.363561+0000 mgr.y (mgr.24368) 645 : cluster 0 pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:38:29.478 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[29221]: audit 2024-04-06T03:38:28.935148+0000 mon.c (mon.2) 84 : audit 1 from='client.? 172.21.15.50:0/2724874781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:38:29.478 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[29221]: audit 2024-04-06T03:38:28.935470+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:38:29.478 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[29221]: audit 2024-04-06T03:38:28.982383+0000 mon.c (mon.2) 85 : audit 1 from='client.? 172.21.15.50:0/630857508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:38:29.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[29221]: audit 2024-04-06T03:38:28.982778+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:38:29.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[21467]: cluster 2024-04-06T03:38:28.363561+0000 mgr.y (mgr.24368) 645 : cluster 0 pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:38:29.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[21467]: audit 2024-04-06T03:38:28.935148+0000 mon.c (mon.2) 84 : audit 1 from='client.? 172.21.15.50:0/2724874781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:38:29.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[21467]: audit 2024-04-06T03:38:28.935470+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:38:29.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[21467]: audit 2024-04-06T03:38:28.982383+0000 mon.c (mon.2) 85 : audit 1 from='client.? 172.21.15.50:0/630857508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:38:29.479 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:29 smithi050 bash[21467]: audit 2024-04-06T03:38:28.982778+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:38:29.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:29 smithi153 bash[23852]: cluster 2024-04-06T03:38:28.363561+0000 mgr.y (mgr.24368) 645 : cluster 0 pgmap v681: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:38:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:29 smithi153 bash[23852]: audit 2024-04-06T03:38:28.935148+0000 mon.c (mon.2) 84 : audit 1 from='client.? 172.21.15.50:0/2724874781' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:38:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:29 smithi153 bash[23852]: audit 2024-04-06T03:38:28.935470+0000 mon.a (mon.0) 1463 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:38:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:29 smithi153 bash[23852]: audit 2024-04-06T03:38:28.982383+0000 mon.c (mon.2) 85 : audit 1 from='client.? 172.21.15.50:0/630857508' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:38:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:29 smithi153 bash[23852]: audit 2024-04-06T03:38:28.982778+0000 mon.a (mon.0) 1464 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:38:29.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:29.625 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:29.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:29.755 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:29.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:38:29.879 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-04-06T03:38:29.883 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:29.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:29.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:30.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:38:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:38:30.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:30.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:30.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:30.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:30.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:30.397 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:30.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:30.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:30 smithi153 bash[23852]: cluster 2024-04-06T03:38:29.866563+0000 mon.a (mon.0) 1465 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:38:30.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:30 smithi153 bash[23852]: audit 2024-04-06T03:38:29.868970+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:38:30.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:30 smithi153 bash[23852]: audit 2024-04-06T03:38:29.869151+0000 mon.a (mon.0) 1467 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:38:30.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:30 smithi153 bash[23852]: cluster 2024-04-06T03:38:29.877565+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e277: 8 total, 8 up, 8 in 2024-04-06T03:38:30.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:30 smithi153 bash[23852]: cluster 2024-04-06T03:38:29.925003+0000 osd.2 (osd.2) 335 : cluster 0 2.17 scrub starts 2024-04-06T03:38:30.509 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:30 smithi153 bash[23852]: cluster 2024-04-06T03:38:29.927473+0000 osd.2 (osd.2) 336 : cluster 0 2.17 scrub ok 2024-04-06T03:38:30.511 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:38:30.512 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:38:30.520 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:30.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:30.645 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:30.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[29221]: cluster 2024-04-06T03:38:29.866563+0000 mon.a (mon.0) 1465 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:38:30.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[29221]: audit 2024-04-06T03:38:29.868970+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:38:30.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[29221]: audit 2024-04-06T03:38:29.869151+0000 mon.a (mon.0) 1467 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:38:30.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[29221]: cluster 2024-04-06T03:38:29.877565+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e277: 8 total, 8 up, 8 in 2024-04-06T03:38:30.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[29221]: cluster 2024-04-06T03:38:29.925003+0000 osd.2 (osd.2) 335 : cluster 0 2.17 scrub starts 2024-04-06T03:38:30.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[29221]: cluster 2024-04-06T03:38:29.927473+0000 osd.2 (osd.2) 336 : cluster 0 2.17 scrub ok 2024-04-06T03:38:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[21467]: cluster 2024-04-06T03:38:29.866563+0000 mon.a (mon.0) 1465 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:38:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[21467]: audit 2024-04-06T03:38:29.868970+0000 mon.a (mon.0) 1466 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:38:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[21467]: audit 2024-04-06T03:38:29.869151+0000 mon.a (mon.0) 1467 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:38:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[21467]: cluster 2024-04-06T03:38:29.877565+0000 mon.a (mon.0) 1468 : cluster 0 osdmap e277: 8 total, 8 up, 8 in 2024-04-06T03:38:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[21467]: cluster 2024-04-06T03:38:29.925003+0000 osd.2 (osd.2) 335 : cluster 0 2.17 scrub starts 2024-04-06T03:38:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:30 smithi050 bash[21467]: cluster 2024-04-06T03:38:29.927473+0000 osd.2 (osd.2) 336 : cluster 0 2.17 scrub ok 2024-04-06T03:38:30.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:30.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:30.904 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:30.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:31.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:31.034 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:31.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:31.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 6 1 2024-04-06T03:38:31.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:31.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:31.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:31.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:31.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-06T03:38:31.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:31.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:31.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:31.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:31.671 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:31.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: audit 2024-04-06T03:38:29.867453+0000 mgr.y (mgr.24368) 646 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:31.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.364181+0000 mgr.y (mgr.24368) 647 : cluster 0 pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.720243+0000 osd.0 (osd.0) 404 : cluster 0 3.1 scrub starts 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.724982+0000 osd.0 (osd.0) 405 : cluster 0 3.1 scrub ok 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.766751+0000 osd.4 (osd.4) 393 : cluster 0 3.1c scrub starts 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.770294+0000 osd.4 (osd.4) 394 : cluster 0 3.1c scrub ok 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.955943+0000 osd.2 (osd.2) 337 : cluster 0 2.10 scrub starts 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.248631+0000 osd.3 (osd.3) 418 : cluster 0 5.17 scrub starts 2024-04-06T03:38:31.673 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:31 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.250102+0000 osd.3 (osd.3) 419 : cluster 0 5.17 scrub ok 2024-04-06T03:38:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:31.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: audit 2024-04-06T03:38:29.867453+0000 mgr.y (mgr.24368) 646 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:31.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.364181+0000 mgr.y (mgr.24368) 647 : cluster 0 pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:38:31.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.720243+0000 osd.0 (osd.0) 404 : cluster 0 3.1 scrub starts 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.724982+0000 osd.0 (osd.0) 405 : cluster 0 3.1 scrub ok 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.766751+0000 osd.4 (osd.4) 393 : cluster 0 3.1c scrub starts 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.770294+0000 osd.4 (osd.4) 394 : cluster 0 3.1c scrub ok 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.955943+0000 osd.2 (osd.2) 337 : cluster 0 2.10 scrub starts 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.248631+0000 osd.3 (osd.3) 418 : cluster 0 5.17 scrub starts 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.250102+0000 osd.3 (osd.3) 419 : cluster 0 5.17 scrub ok 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: audit 2024-04-06T03:38:29.867453+0000 mgr.y (mgr.24368) 646 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.364181+0000 mgr.y (mgr.24368) 647 : cluster 0 pgmap v683: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.720243+0000 osd.0 (osd.0) 404 : cluster 0 3.1 scrub starts 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.724982+0000 osd.0 (osd.0) 405 : cluster 0 3.1 scrub ok 2024-04-06T03:38:31.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.766751+0000 osd.4 (osd.4) 393 : cluster 0 3.1c scrub starts 2024-04-06T03:38:31.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.770294+0000 osd.4 (osd.4) 394 : cluster 0 3.1c scrub ok 2024-04-06T03:38:31.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.955943+0000 osd.2 (osd.2) 337 : cluster 0 2.10 scrub starts 2024-04-06T03:38:31.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.248631+0000 osd.3 (osd.3) 418 : cluster 0 5.17 scrub starts 2024-04-06T03:38:31.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:31 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.250102+0000 osd.3 (osd.3) 419 : cluster 0 5.17 scrub ok 2024-04-06T03:38:31.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:38:31.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:31.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:31.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:31.976 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:31.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:32.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:32.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:32.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:32.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:32.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:32.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:32.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:32.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:32.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:32.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:32.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:32.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:32.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:32.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:32.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:32.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:30.958493+0000 osd.2 (osd.2) 338 : cluster 0 2.10 scrub ok 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.315150+0000 osd.2 (osd.2) 339 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.315219+0000 osd.2 (osd.2) 340 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.677612+0000 osd.0 (osd.0) 406 : cluster 0 3.6 scrub starts 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.679731+0000 osd.0 (osd.0) 407 : cluster 0 3.6 scrub ok 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.815866+0000 osd.4 (osd.4) 395 : cluster 0 3.14 scrub starts 2024-04-06T03:38:33.008 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.820158+0000 osd.4 (osd.4) 396 : cluster 0 3.14 scrub ok 2024-04-06T03:38:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:31.875996+0000 mon.a (mon.0) 1469 : cluster 0 osdmap e278: 8 total, 8 up, 8 in 2024-04-06T03:38:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:32.267110+0000 osd.7 (osd.7) 439 : cluster 0 3.16 scrub starts 2024-04-06T03:38:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:32.273572+0000 osd.7 (osd.7) 440 : cluster 0 3.16 scrub ok 2024-04-06T03:38:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:32.278904+0000 osd.3 (osd.3) 420 : cluster 0 2.1b scrub starts 2024-04-06T03:38:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:32.280554+0000 osd.3 (osd.3) 421 : cluster 0 2.1b scrub ok 2024-04-06T03:38:33.009 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:32 smithi153 bash[23852]: cluster 2024-04-06T03:38:32.364833+0000 mgr.y (mgr.24368) 648 : cluster 0 pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:38:33.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:33.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:33.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:38:33.140 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:30.958493+0000 osd.2 (osd.2) 338 : cluster 0 2.10 scrub ok 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.315150+0000 osd.2 (osd.2) 339 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.315219+0000 osd.2 (osd.2) 340 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.677612+0000 osd.0 (osd.0) 406 : cluster 0 3.6 scrub starts 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.679731+0000 osd.0 (osd.0) 407 : cluster 0 3.6 scrub ok 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.815866+0000 osd.4 (osd.4) 395 : cluster 0 3.14 scrub starts 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.820158+0000 osd.4 (osd.4) 396 : cluster 0 3.14 scrub ok 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:31.875996+0000 mon.a (mon.0) 1469 : cluster 0 osdmap e278: 8 total, 8 up, 8 in 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:32.267110+0000 osd.7 (osd.7) 439 : cluster 0 3.16 scrub starts 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:32.273572+0000 osd.7 (osd.7) 440 : cluster 0 3.16 scrub ok 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:32.278904+0000 osd.3 (osd.3) 420 : cluster 0 2.1b scrub starts 2024-04-06T03:38:33.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:32.280554+0000 osd.3 (osd.3) 421 : cluster 0 2.1b scrub ok 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[29221]: cluster 2024-04-06T03:38:32.364833+0000 mgr.y (mgr.24368) 648 : cluster 0 pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:30.958493+0000 osd.2 (osd.2) 338 : cluster 0 2.10 scrub ok 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.315150+0000 osd.2 (osd.2) 339 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.315219+0000 osd.2 (osd.2) 340 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.677612+0000 osd.0 (osd.0) 406 : cluster 0 3.6 scrub starts 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.679731+0000 osd.0 (osd.0) 407 : cluster 0 3.6 scrub ok 2024-04-06T03:38:33.142 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.815866+0000 osd.4 (osd.4) 395 : cluster 0 3.14 scrub starts 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.820158+0000 osd.4 (osd.4) 396 : cluster 0 3.14 scrub ok 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:31.875996+0000 mon.a (mon.0) 1469 : cluster 0 osdmap e278: 8 total, 8 up, 8 in 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:32.267110+0000 osd.7 (osd.7) 439 : cluster 0 3.16 scrub starts 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:32.273572+0000 osd.7 (osd.7) 440 : cluster 0 3.16 scrub ok 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:32.278904+0000 osd.3 (osd.3) 420 : cluster 0 2.1b scrub starts 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:32.280554+0000 osd.3 (osd.3) 421 : cluster 0 2.1b scrub ok 2024-04-06T03:38:33.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:32 smithi050 bash[21467]: cluster 2024-04-06T03:38:32.364833+0000 mgr.y (mgr.24368) 648 : cluster 0 pgmap v685: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:38:33.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:33.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:33.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:33.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:33.419 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:33.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:33.548 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:33.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:33.659 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:33.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:33.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:33.811 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:33.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:33.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:33.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:33 smithi050 bash[21467]: cluster 2024-04-06T03:38:33.267058+0000 osd.3 (osd.3) 422 : cluster 0 6.1b scrub starts 2024-04-06T03:38:33.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:33 smithi050 bash[29221]: cluster 2024-04-06T03:38:33.267058+0000 osd.3 (osd.3) 422 : cluster 0 6.1b scrub starts 2024-04-06T03:38:33.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:33.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:34.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:33 smithi153 bash[23852]: cluster 2024-04-06T03:38:33.267058+0000 osd.3 (osd.3) 422 : cluster 0 6.1b scrub starts 2024-04-06T03:38:34.071 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:34.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:34.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:34.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:34.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:34.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:34.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:34.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:34.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:34.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:34.724 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:34.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:34.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:34.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:34.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:34 smithi153 bash[23852]: cluster 2024-04-06T03:38:33.397231+0000 osd.3 (osd.3) 423 : cluster 0 6.1b scrub ok 2024-04-06T03:38:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:34 smithi153 bash[23852]: cluster 2024-04-06T03:38:34.233108+0000 osd.3 (osd.3) 424 : cluster 0 6.11 scrub starts 2024-04-06T03:38:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:34 smithi153 bash[23852]: cluster 2024-04-06T03:38:34.251162+0000 osd.3 (osd.3) 425 : cluster 0 6.11 scrub ok 2024-04-06T03:38:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:34 smithi153 bash[23852]: cluster 2024-04-06T03:38:34.365580+0000 mgr.y (mgr.24368) 649 : cluster 0 pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:35.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:34 smithi153 bash[23852]: audit 2024-04-06T03:38:34.796362+0000 mon.a (mon.0) 1470 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:35.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:35.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[29221]: cluster 2024-04-06T03:38:33.397231+0000 osd.3 (osd.3) 423 : cluster 0 6.1b scrub ok 2024-04-06T03:38:35.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[29221]: cluster 2024-04-06T03:38:34.233108+0000 osd.3 (osd.3) 424 : cluster 0 6.11 scrub starts 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[29221]: cluster 2024-04-06T03:38:34.251162+0000 osd.3 (osd.3) 425 : cluster 0 6.11 scrub ok 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[29221]: cluster 2024-04-06T03:38:34.365580+0000 mgr.y (mgr.24368) 649 : cluster 0 pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[29221]: audit 2024-04-06T03:38:34.796362+0000 mon.a (mon.0) 1470 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[21467]: cluster 2024-04-06T03:38:33.397231+0000 osd.3 (osd.3) 423 : cluster 0 6.1b scrub ok 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[21467]: cluster 2024-04-06T03:38:34.233108+0000 osd.3 (osd.3) 424 : cluster 0 6.11 scrub starts 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[21467]: cluster 2024-04-06T03:38:34.251162+0000 osd.3 (osd.3) 425 : cluster 0 6.11 scrub ok 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[21467]: cluster 2024-04-06T03:38:34.365580+0000 mgr.y (mgr.24368) 649 : cluster 0 pgmap v686: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:35.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:34 smithi050 bash[21467]: audit 2024-04-06T03:38:34.796362+0000 mon.a (mon.0) 1470 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:35.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:35.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:35.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:35.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:35.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:35.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:35.495 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:35.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:35.625 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:35.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:35.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:35.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:35.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:35.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:35.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:36.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:36.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:35 smithi153 bash[23852]: cluster 2024-04-06T03:38:34.906417+0000 osd.1 (osd.1) 329 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:36.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:35 smithi153 bash[23852]: cluster 2024-04-06T03:38:34.906515+0000 osd.1 (osd.1) 330 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:36.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:35 smithi153 bash[23852]: cluster 2024-04-06T03:38:35.190540+0000 osd.7 (osd.7) 441 : cluster 0 3.11 scrub starts 2024-04-06T03:38:36.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:35 smithi153 bash[23852]: cluster 2024-04-06T03:38:35.195611+0000 osd.7 (osd.7) 442 : cluster 0 3.11 scrub ok 2024-04-06T03:38:36.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:36.143 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:36.203 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:38:36.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[21467]: cluster 2024-04-06T03:38:34.906417+0000 osd.1 (osd.1) 329 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[21467]: cluster 2024-04-06T03:38:34.906515+0000 osd.1 (osd.1) 330 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[21467]: cluster 2024-04-06T03:38:35.190540+0000 osd.7 (osd.7) 441 : cluster 0 3.11 scrub starts 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[21467]: cluster 2024-04-06T03:38:35.195611+0000 osd.7 (osd.7) 442 : cluster 0 3.11 scrub ok 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[29221]: cluster 2024-04-06T03:38:34.906417+0000 osd.1 (osd.1) 329 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[29221]: cluster 2024-04-06T03:38:34.906515+0000 osd.1 (osd.1) 330 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[29221]: cluster 2024-04-06T03:38:35.190540+0000 osd.7 (osd.7) 441 : cluster 0 3.11 scrub starts 2024-04-06T03:38:36.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:35 smithi050 bash[29221]: cluster 2024-04-06T03:38:35.195611+0000 osd.7 (osd.7) 442 : cluster 0 3.11 scrub ok 2024-04-06T03:38:36.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:36.269 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:36.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:36.399 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:36.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:36.527 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:36.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:36.633 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:36.633 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:36.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:36.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:36.788 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:36.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:36.918 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:36.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:37.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:37.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:34.984711+0000 osd.2 (osd.2) 341 : cluster 0 6.f scrub starts 2024-04-06T03:38:37.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:35.085710+0000 osd.2 (osd.2) 342 : cluster 0 6.f scrub ok 2024-04-06T03:38:37.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:35.725394+0000 osd.1 (osd.1) 331 : cluster 0 5.7 scrub starts 2024-04-06T03:38:37.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:35.752978+0000 osd.1 (osd.1) 332 : cluster 0 5.7 scrub ok 2024-04-06T03:38:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:35.905106+0000 mon.a (mon.0) 1471 : cluster 0 osdmap e279: 8 total, 8 up, 8 in 2024-04-06T03:38:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.148327+0000 osd.7 (osd.7) 443 : cluster 0 6.15 scrub starts 2024-04-06T03:38:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.191030+0000 osd.7 (osd.7) 444 : cluster 0 6.15 scrub ok 2024-04-06T03:38:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.267475+0000 osd.3 (osd.3) 426 : cluster 0 2.a scrub starts 2024-04-06T03:38:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.268783+0000 osd.3 (osd.3) 427 : cluster 0 2.a scrub ok 2024-04-06T03:38:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:36 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.366215+0000 mgr.y (mgr.24368) 650 : cluster 0 pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-06T03:38:37.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:37.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:37.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:34.984711+0000 osd.2 (osd.2) 341 : cluster 0 6.f scrub starts 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:35.085710+0000 osd.2 (osd.2) 342 : cluster 0 6.f scrub ok 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:35.725394+0000 osd.1 (osd.1) 331 : cluster 0 5.7 scrub starts 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:35.752978+0000 osd.1 (osd.1) 332 : cluster 0 5.7 scrub ok 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:35.905106+0000 mon.a (mon.0) 1471 : cluster 0 osdmap e279: 8 total, 8 up, 8 in 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.148327+0000 osd.7 (osd.7) 443 : cluster 0 6.15 scrub starts 2024-04-06T03:38:37.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.191030+0000 osd.7 (osd.7) 444 : cluster 0 6.15 scrub ok 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.267475+0000 osd.3 (osd.3) 426 : cluster 0 2.a scrub starts 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.268783+0000 osd.3 (osd.3) 427 : cluster 0 2.a scrub ok 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.366215+0000 mgr.y (mgr.24368) 650 : cluster 0 pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:34.984711+0000 osd.2 (osd.2) 341 : cluster 0 6.f scrub starts 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:35.085710+0000 osd.2 (osd.2) 342 : cluster 0 6.f scrub ok 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:35.725394+0000 osd.1 (osd.1) 331 : cluster 0 5.7 scrub starts 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:35.752978+0000 osd.1 (osd.1) 332 : cluster 0 5.7 scrub ok 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:35.905106+0000 mon.a (mon.0) 1471 : cluster 0 osdmap e279: 8 total, 8 up, 8 in 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.148327+0000 osd.7 (osd.7) 443 : cluster 0 6.15 scrub starts 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.191030+0000 osd.7 (osd.7) 444 : cluster 0 6.15 scrub ok 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.267475+0000 osd.3 (osd.3) 426 : cluster 0 2.a scrub starts 2024-04-06T03:38:37.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.268783+0000 osd.3 (osd.3) 427 : cluster 0 2.a scrub ok 2024-04-06T03:38:37.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:36 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.366215+0000 mgr.y (mgr.24368) 650 : cluster 0 pgmap v688: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-06T03:38:37.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:37.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:37.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:37.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:37.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:37.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:37.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:37.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:37.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:37.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:37.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:37 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.722318+0000 osd.0 (osd.0) 408 : cluster 0 3.17 scrub starts 2024-04-06T03:38:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:37 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.725146+0000 osd.0 (osd.0) 409 : cluster 0 3.17 scrub ok 2024-04-06T03:38:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:37 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.171206+0000 osd.7 (osd.7) 445 : cluster 0 2.1d scrub starts 2024-04-06T03:38:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:37 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.173081+0000 osd.7 (osd.7) 446 : cluster 0 2.1d scrub ok 2024-04-06T03:38:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:37 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.298375+0000 osd.3 (osd.3) 428 : cluster 0 5.16 scrub starts 2024-04-06T03:38:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:37 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.300966+0000 osd.3 (osd.3) 429 : cluster 0 5.16 scrub ok 2024-04-06T03:38:37.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:37.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:38.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.722318+0000 osd.0 (osd.0) 408 : cluster 0 3.17 scrub starts 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.725146+0000 osd.0 (osd.0) 409 : cluster 0 3.17 scrub ok 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.171206+0000 osd.7 (osd.7) 445 : cluster 0 2.1d scrub starts 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.173081+0000 osd.7 (osd.7) 446 : cluster 0 2.1d scrub ok 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.298375+0000 osd.3 (osd.3) 428 : cluster 0 5.16 scrub starts 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.300966+0000 osd.3 (osd.3) 429 : cluster 0 5.16 scrub ok 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.722318+0000 osd.0 (osd.0) 408 : cluster 0 3.17 scrub starts 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.725146+0000 osd.0 (osd.0) 409 : cluster 0 3.17 scrub ok 2024-04-06T03:38:38.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.171206+0000 osd.7 (osd.7) 445 : cluster 0 2.1d scrub starts 2024-04-06T03:38:38.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.173081+0000 osd.7 (osd.7) 446 : cluster 0 2.1d scrub ok 2024-04-06T03:38:38.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.298375+0000 osd.3 (osd.3) 428 : cluster 0 5.16 scrub starts 2024-04-06T03:38:38.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:37 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.300966+0000 osd.3 (osd.3) 429 : cluster 0 5.16 scrub ok 2024-04-06T03:38:38.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:38.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:38.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:38.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:38.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:38.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:38.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:38.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:38.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:38.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:38.741 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:38.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:38.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:38.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:38.975 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:38.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.973602+0000 osd.2 (osd.2) 343 : cluster 0 2.c scrub starts 2024-04-06T03:38:38.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:36.976003+0000 osd.2 (osd.2) 344 : cluster 0 2.c scrub ok 2024-04-06T03:38:38.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.718349+0000 osd.0 (osd.0) 410 : cluster 0 6.0 scrub starts 2024-04-06T03:38:38.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.764164+0000 osd.0 (osd.0) 411 : cluster 0 6.0 scrub ok 2024-04-06T03:38:38.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.777084+0000 osd.1 (osd.1) 333 : cluster 0 4.2 scrub starts 2024-04-06T03:38:38.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.780053+0000 osd.1 (osd.1) 334 : cluster 0 4.2 scrub ok 2024-04-06T03:38:38.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:38.148598+0000 osd.7 (osd.7) 447 : cluster 0 5.9 scrub starts 2024-04-06T03:38:38.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:38.151181+0000 osd.7 (osd.7) 448 : cluster 0 5.9 scrub ok 2024-04-06T03:38:38.987 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:38 smithi153 bash[23852]: cluster 2024-04-06T03:38:38.367117+0000 mgr.y (mgr.24368) 651 : cluster 0 pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:39.000 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:39.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:39.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:39.204 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:39.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:39.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.973602+0000 osd.2 (osd.2) 343 : cluster 0 2.c scrub starts 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:36.976003+0000 osd.2 (osd.2) 344 : cluster 0 2.c scrub ok 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.718349+0000 osd.0 (osd.0) 410 : cluster 0 6.0 scrub starts 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.764164+0000 osd.0 (osd.0) 411 : cluster 0 6.0 scrub ok 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.777084+0000 osd.1 (osd.1) 333 : cluster 0 4.2 scrub starts 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.780053+0000 osd.1 (osd.1) 334 : cluster 0 4.2 scrub ok 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:38.148598+0000 osd.7 (osd.7) 447 : cluster 0 5.9 scrub starts 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:38.151181+0000 osd.7 (osd.7) 448 : cluster 0 5.9 scrub ok 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[21467]: cluster 2024-04-06T03:38:38.367117+0000 mgr.y (mgr.24368) 651 : cluster 0 pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.973602+0000 osd.2 (osd.2) 343 : cluster 0 2.c scrub starts 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:36.976003+0000 osd.2 (osd.2) 344 : cluster 0 2.c scrub ok 2024-04-06T03:38:39.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.718349+0000 osd.0 (osd.0) 410 : cluster 0 6.0 scrub starts 2024-04-06T03:38:39.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.764164+0000 osd.0 (osd.0) 411 : cluster 0 6.0 scrub ok 2024-04-06T03:38:39.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.777084+0000 osd.1 (osd.1) 333 : cluster 0 4.2 scrub starts 2024-04-06T03:38:39.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.780053+0000 osd.1 (osd.1) 334 : cluster 0 4.2 scrub ok 2024-04-06T03:38:39.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:38.148598+0000 osd.7 (osd.7) 447 : cluster 0 5.9 scrub starts 2024-04-06T03:38:39.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:38.151181+0000 osd.7 (osd.7) 448 : cluster 0 5.9 scrub ok 2024-04-06T03:38:39.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:38 smithi050 bash[29221]: cluster 2024-04-06T03:38:38.367117+0000 mgr.y (mgr.24368) 651 : cluster 0 pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:39.250 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:39.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:39.379 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:39.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:39.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:39.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:39.611 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:38:39.611 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:38:39.612 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:38:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:38:39.624 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:39.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:39.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:39.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:39.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:39.944 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_3' created 2024-04-06T03:38:39.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:39.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:40.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:40.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.953750+0000 osd.4 (osd.4) 397 : cluster 0 3.9 scrub starts 2024-04-06T03:38:40.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: cluster 2024-04-06T03:38:37.957270+0000 osd.4 (osd.4) 398 : cluster 0 3.9 scrub ok 2024-04-06T03:38:40.146 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: cluster 2024-04-06T03:38:38.747756+0000 osd.1 (osd.1) 335 : cluster 0 2.6 scrub starts 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: cluster 2024-04-06T03:38:38.749324+0000 osd.1 (osd.1) 336 : cluster 0 2.6 scrub ok 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: audit 2024-04-06T03:38:38.956000+0000 mon.a (mon.0) 1472 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.134700+0000 osd.7 (osd.7) 449 : cluster 0 5.1a scrub starts 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.136448+0000 osd.7 (osd.7) 450 : cluster 0 5.1a scrub ok 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: audit 2024-04-06T03:38:39.209497+0000 mon.a (mon.0) 1473 : audit 1 from='client.? 172.21.15.50:0/1873574127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: audit 2024-04-06T03:38:39.661999+0000 mon.a (mon.0) 1474 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: audit 2024-04-06T03:38:39.662989+0000 mon.a (mon.0) 1475 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[29221]: audit 2024-04-06T03:38:39.669074+0000 mon.a (mon.0) 1476 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.953750+0000 osd.4 (osd.4) 397 : cluster 0 3.9 scrub starts 2024-04-06T03:38:40.147 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: cluster 2024-04-06T03:38:37.957270+0000 osd.4 (osd.4) 398 : cluster 0 3.9 scrub ok 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: cluster 2024-04-06T03:38:38.747756+0000 osd.1 (osd.1) 335 : cluster 0 2.6 scrub starts 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: cluster 2024-04-06T03:38:38.749324+0000 osd.1 (osd.1) 336 : cluster 0 2.6 scrub ok 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: audit 2024-04-06T03:38:38.956000+0000 mon.a (mon.0) 1472 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.134700+0000 osd.7 (osd.7) 449 : cluster 0 5.1a scrub starts 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.136448+0000 osd.7 (osd.7) 450 : cluster 0 5.1a scrub ok 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: audit 2024-04-06T03:38:39.209497+0000 mon.a (mon.0) 1473 : audit 1 from='client.? 172.21.15.50:0/1873574127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: audit 2024-04-06T03:38:39.661999+0000 mon.a (mon.0) 1474 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: audit 2024-04-06T03:38:39.662989+0000 mon.a (mon.0) 1475 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:38:40.148 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:39 smithi050 bash[21467]: audit 2024-04-06T03:38:39.669074+0000 mon.a (mon.0) 1476 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:38:40.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:40.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:38:40.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.953750+0000 osd.4 (osd.4) 397 : cluster 0 3.9 scrub starts 2024-04-06T03:38:40.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: cluster 2024-04-06T03:38:37.957270+0000 osd.4 (osd.4) 398 : cluster 0 3.9 scrub ok 2024-04-06T03:38:40.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: cluster 2024-04-06T03:38:38.747756+0000 osd.1 (osd.1) 335 : cluster 0 2.6 scrub starts 2024-04-06T03:38:40.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: cluster 2024-04-06T03:38:38.749324+0000 osd.1 (osd.1) 336 : cluster 0 2.6 scrub ok 2024-04-06T03:38:40.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: audit 2024-04-06T03:38:38.956000+0000 mon.a (mon.0) 1472 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:38:40.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.134700+0000 osd.7 (osd.7) 449 : cluster 0 5.1a scrub starts 2024-04-06T03:38:40.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.136448+0000 osd.7 (osd.7) 450 : cluster 0 5.1a scrub ok 2024-04-06T03:38:40.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: audit 2024-04-06T03:38:39.209497+0000 mon.a (mon.0) 1473 : audit 1 from='client.? 172.21.15.50:0/1873574127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-04-06T03:38:40.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: audit 2024-04-06T03:38:39.661999+0000 mon.a (mon.0) 1474 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:38:40.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: audit 2024-04-06T03:38:39.662989+0000 mon.a (mon.0) 1475 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:38:40.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:39 smithi153 bash[23852]: audit 2024-04-06T03:38:39.669074+0000 mon.a (mon.0) 1476 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:38:40.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:40.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-06T03:38:40.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:40.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:40.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:40.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:40.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:40.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:40.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:40.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:41.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.707884+0000 osd.1 (osd.1) 337 : cluster 0 3.19 scrub starts 2024-04-06T03:38:41.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.710120+0000 osd.1 (osd.1) 338 : cluster 0 3.19 scrub ok 2024-04-06T03:38:41.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.747732+0000 osd.0 (osd.0) 412 : cluster 0 6.16 scrub starts 2024-04-06T03:38:41.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.773784+0000 osd.0 (osd.0) 413 : cluster 0 6.16 scrub ok 2024-04-06T03:38:41.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: audit 2024-04-06T03:38:39.875070+0000 mgr.y (mgr.24368) 652 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:41.005 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: audit 2024-04-06T03:38:39.941385+0000 mon.a (mon.0) 1477 : audit 1 from='client.? 172.21.15.50:0/1873574127' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:39.944864+0000 mon.a (mon.0) 1478 : cluster 0 osdmap e280: 8 total, 8 up, 8 in 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.104186+0000 osd.7 (osd.7) 451 : cluster 0 6.1c scrub starts 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.107794+0000 osd.7 (osd.7) 452 : cluster 0 6.1c scrub ok 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.367521+0000 mgr.y (mgr.24368) 653 : cluster 0 pgmap v691: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.707884+0000 osd.1 (osd.1) 337 : cluster 0 3.19 scrub starts 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.710120+0000 osd.1 (osd.1) 338 : cluster 0 3.19 scrub ok 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.747732+0000 osd.0 (osd.0) 412 : cluster 0 6.16 scrub starts 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.773784+0000 osd.0 (osd.0) 413 : cluster 0 6.16 scrub ok 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: audit 2024-04-06T03:38:39.875070+0000 mgr.y (mgr.24368) 652 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: audit 2024-04-06T03:38:39.941385+0000 mon.a (mon.0) 1477 : audit 1 from='client.? 172.21.15.50:0/1873574127' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:39.944864+0000 mon.a (mon.0) 1478 : cluster 0 osdmap e280: 8 total, 8 up, 8 in 2024-04-06T03:38:41.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.104186+0000 osd.7 (osd.7) 451 : cluster 0 6.1c scrub starts 2024-04-06T03:38:41.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.107794+0000 osd.7 (osd.7) 452 : cluster 0 6.1c scrub ok 2024-04-06T03:38:41.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:40 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.367521+0000 mgr.y (mgr.24368) 653 : cluster 0 pgmap v691: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:38:41.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:41.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:41.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:41.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:41.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:41.301 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.707884+0000 osd.1 (osd.1) 337 : cluster 0 3.19 scrub starts 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.710120+0000 osd.1 (osd.1) 338 : cluster 0 3.19 scrub ok 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.747732+0000 osd.0 (osd.0) 412 : cluster 0 6.16 scrub starts 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.773784+0000 osd.0 (osd.0) 413 : cluster 0 6.16 scrub ok 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: audit 2024-04-06T03:38:39.875070+0000 mgr.y (mgr.24368) 652 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: audit 2024-04-06T03:38:39.941385+0000 mon.a (mon.0) 1477 : audit 1 from='client.? 172.21.15.50:0/1873574127' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:39.944864+0000 mon.a (mon.0) 1478 : cluster 0 osdmap e280: 8 total, 8 up, 8 in 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.104186+0000 osd.7 (osd.7) 451 : cluster 0 6.1c scrub starts 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.107794+0000 osd.7 (osd.7) 452 : cluster 0 6.1c scrub ok 2024-04-06T03:38:41.302 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:40 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.367521+0000 mgr.y (mgr.24368) 653 : cluster 0 pgmap v691: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:38:41.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:41.412 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:41.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:41.544 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:41.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:41.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:41.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:41.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:41.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:41.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:41.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:42.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.941573+0000 mon.a (mon.0) 1479 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.948721+0000 osd.4 (osd.4) 399 : cluster 0 2.f scrub starts 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.956231+0000 osd.4 (osd.4) 400 : cluster 0 2.f scrub ok 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.961515+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e281: 8 total, 8 up, 8 in 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[21467]: cluster 2024-04-06T03:38:41.074139+0000 osd.7 (osd.7) 453 : cluster 0 2.0 scrub starts 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[21467]: cluster 2024-04-06T03:38:41.076597+0000 osd.7 (osd.7) 454 : cluster 0 2.0 scrub ok 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.941573+0000 mon.a (mon.0) 1479 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.948721+0000 osd.4 (osd.4) 399 : cluster 0 2.f scrub starts 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.956231+0000 osd.4 (osd.4) 400 : cluster 0 2.f scrub ok 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.961515+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e281: 8 total, 8 up, 8 in 2024-04-06T03:38:42.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[29221]: cluster 2024-04-06T03:38:41.074139+0000 osd.7 (osd.7) 453 : cluster 0 2.0 scrub starts 2024-04-06T03:38:42.049 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:41 smithi050 bash[29221]: cluster 2024-04-06T03:38:41.076597+0000 osd.7 (osd.7) 454 : cluster 0 2.0 scrub ok 2024-04-06T03:38:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:42.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:42.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:41 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.941573+0000 mon.a (mon.0) 1479 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:38:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:41 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.948721+0000 osd.4 (osd.4) 399 : cluster 0 2.f scrub starts 2024-04-06T03:38:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:41 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.956231+0000 osd.4 (osd.4) 400 : cluster 0 2.f scrub ok 2024-04-06T03:38:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:41 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.961515+0000 mon.a (mon.0) 1480 : cluster 0 osdmap e281: 8 total, 8 up, 8 in 2024-04-06T03:38:42.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:41 smithi153 bash[23852]: cluster 2024-04-06T03:38:41.074139+0000 osd.7 (osd.7) 453 : cluster 0 2.0 scrub starts 2024-04-06T03:38:42.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:41 smithi153 bash[23852]: cluster 2024-04-06T03:38:41.076597+0000 osd.7 (osd.7) 454 : cluster 0 2.0 scrub ok 2024-04-06T03:38:42.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:42.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:42.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:42.457 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:42.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:42.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:42.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:42.720 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:42.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:42.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:42.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:42.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:43.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:43.116 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:43.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.977920+0000 osd.2 (osd.2) 345 : cluster 0 5.d scrub starts 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:40.981110+0000 osd.2 (osd.2) 346 : cluster 0 5.d scrub ok 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:41.931307+0000 osd.2 (osd.2) 347 : cluster 0 2.e scrub starts 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:41.932881+0000 osd.2 (osd.2) 348 : cluster 0 2.e scrub ok 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:41.956846+0000 mon.a (mon.0) 1481 : cluster 0 osdmap e282: 8 total, 8 up, 8 in 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:42.256586+0000 osd.3 (osd.3) 430 : cluster 0 5.0 scrub starts 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:42.258667+0000 osd.3 (osd.3) 431 : cluster 0 5.0 scrub ok 2024-04-06T03:38:43.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[29221]: cluster 2024-04-06T03:38:42.367995+0000 mgr.y (mgr.24368) 654 : cluster 0 pgmap v694: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.977920+0000 osd.2 (osd.2) 345 : cluster 0 5.d scrub starts 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:40.981110+0000 osd.2 (osd.2) 346 : cluster 0 5.d scrub ok 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:41.931307+0000 osd.2 (osd.2) 347 : cluster 0 2.e scrub starts 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:41.932881+0000 osd.2 (osd.2) 348 : cluster 0 2.e scrub ok 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:41.956846+0000 mon.a (mon.0) 1481 : cluster 0 osdmap e282: 8 total, 8 up, 8 in 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:42.256586+0000 osd.3 (osd.3) 430 : cluster 0 5.0 scrub starts 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:42.258667+0000 osd.3 (osd.3) 431 : cluster 0 5.0 scrub ok 2024-04-06T03:38:43.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:42 smithi050 bash[21467]: cluster 2024-04-06T03:38:42.367995+0000 mgr.y (mgr.24368) 654 : cluster 0 pgmap v694: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:38:43.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.977920+0000 osd.2 (osd.2) 345 : cluster 0 5.d scrub starts 2024-04-06T03:38:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:40.981110+0000 osd.2 (osd.2) 346 : cluster 0 5.d scrub ok 2024-04-06T03:38:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:41.931307+0000 osd.2 (osd.2) 347 : cluster 0 2.e scrub starts 2024-04-06T03:38:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:41.932881+0000 osd.2 (osd.2) 348 : cluster 0 2.e scrub ok 2024-04-06T03:38:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:41.956846+0000 mon.a (mon.0) 1481 : cluster 0 osdmap e282: 8 total, 8 up, 8 in 2024-04-06T03:38:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:42.256586+0000 osd.3 (osd.3) 430 : cluster 0 5.0 scrub starts 2024-04-06T03:38:43.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:42.258667+0000 osd.3 (osd.3) 431 : cluster 0 5.0 scrub ok 2024-04-06T03:38:43.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:42 smithi153 bash[23852]: cluster 2024-04-06T03:38:42.367995+0000 mgr.y (mgr.24368) 654 : cluster 0 pgmap v694: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:38:43.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:43.380 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:43.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:43.511 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:43.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:43.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:43.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:43.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:43.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:43.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:44.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:44.032 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:44.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:44.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:44.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:44.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:44.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:44.428 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:44.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:44.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:44.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:44.687 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:44.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:44.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:44.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:44.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:44.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:45.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:45.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:45.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[21467]: cluster 2024-04-06T03:38:42.975474+0000 osd.2 (osd.2) 349 : cluster 0 3.0 scrub starts 2024-04-06T03:38:45.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[21467]: cluster 2024-04-06T03:38:42.981377+0000 osd.2 (osd.2) 350 : cluster 0 3.0 scrub ok 2024-04-06T03:38:45.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[21467]: cluster 2024-04-06T03:38:43.798158+0000 osd.1 (osd.1) 339 : cluster 0 5.1d scrub starts 2024-04-06T03:38:45.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[21467]: cluster 2024-04-06T03:38:43.800547+0000 osd.1 (osd.1) 340 : cluster 0 5.1d scrub ok 2024-04-06T03:38:45.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.368990+0000 mgr.y (mgr.24368) 655 : cluster 0 pgmap v695: 148 pgs: 4 unknown, 144 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:45.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[29221]: cluster 2024-04-06T03:38:42.975474+0000 osd.2 (osd.2) 349 : cluster 0 3.0 scrub starts 2024-04-06T03:38:45.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[29221]: cluster 2024-04-06T03:38:42.981377+0000 osd.2 (osd.2) 350 : cluster 0 3.0 scrub ok 2024-04-06T03:38:45.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[29221]: cluster 2024-04-06T03:38:43.798158+0000 osd.1 (osd.1) 339 : cluster 0 5.1d scrub starts 2024-04-06T03:38:45.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[29221]: cluster 2024-04-06T03:38:43.800547+0000 osd.1 (osd.1) 340 : cluster 0 5.1d scrub ok 2024-04-06T03:38:45.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:44 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.368990+0000 mgr.y (mgr.24368) 655 : cluster 0 pgmap v695: 148 pgs: 4 unknown, 144 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:45.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:45.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:44 smithi153 bash[23852]: cluster 2024-04-06T03:38:42.975474+0000 osd.2 (osd.2) 349 : cluster 0 3.0 scrub starts 2024-04-06T03:38:45.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:44 smithi153 bash[23852]: cluster 2024-04-06T03:38:42.981377+0000 osd.2 (osd.2) 350 : cluster 0 3.0 scrub ok 2024-04-06T03:38:45.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:44 smithi153 bash[23852]: cluster 2024-04-06T03:38:43.798158+0000 osd.1 (osd.1) 339 : cluster 0 5.1d scrub starts 2024-04-06T03:38:45.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:44 smithi153 bash[23852]: cluster 2024-04-06T03:38:43.800547+0000 osd.1 (osd.1) 340 : cluster 0 5.1d scrub ok 2024-04-06T03:38:45.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:44 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.368990+0000 mgr.y (mgr.24368) 655 : cluster 0 pgmap v695: 148 pgs: 4 unknown, 144 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:45.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:45.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-04-06T03:38:45.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:45.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:45.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:45.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:45.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:45.593 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:45.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:45.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:45.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:45.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:45.856 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:45.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:45.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.073537+0000 osd.4 (osd.4) 401 : cluster 0 6.1e scrub starts 2024-04-06T03:38:45.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.104742+0000 osd.4 (osd.4) 402 : cluster 0 6.1e scrub ok 2024-04-06T03:38:45.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.737163+0000 osd.0 (osd.0) 414 : cluster 0 2.13 scrub starts 2024-04-06T03:38:45.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.738908+0000 osd.0 (osd.0) 415 : cluster 0 2.13 scrub ok 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.789849+0000 osd.1 (osd.1) 341 : cluster 0 2.15 scrub starts 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:44.792071+0000 osd.1 (osd.1) 342 : cluster 0 2.15 scrub ok 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:45.004951+0000 osd.7 (osd.7) 455 : cluster 0 3.d scrub starts 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[29221]: cluster 2024-04-06T03:38:45.009693+0000 osd.7 (osd.7) 456 : cluster 0 3.d scrub ok 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.073537+0000 osd.4 (osd.4) 401 : cluster 0 6.1e scrub starts 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.104742+0000 osd.4 (osd.4) 402 : cluster 0 6.1e scrub ok 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.737163+0000 osd.0 (osd.0) 414 : cluster 0 2.13 scrub starts 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.738908+0000 osd.0 (osd.0) 415 : cluster 0 2.13 scrub ok 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.789849+0000 osd.1 (osd.1) 341 : cluster 0 2.15 scrub starts 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:44.792071+0000 osd.1 (osd.1) 342 : cluster 0 2.15 scrub ok 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:45.004951+0000 osd.7 (osd.7) 455 : cluster 0 3.d scrub starts 2024-04-06T03:38:45.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:45 smithi050 bash[21467]: cluster 2024-04-06T03:38:45.009693+0000 osd.7 (osd.7) 456 : cluster 0 3.d scrub ok 2024-04-06T03:38:45.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:45.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:46.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.073537+0000 osd.4 (osd.4) 401 : cluster 0 6.1e scrub starts 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.104742+0000 osd.4 (osd.4) 402 : cluster 0 6.1e scrub ok 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.737163+0000 osd.0 (osd.0) 414 : cluster 0 2.13 scrub starts 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.738908+0000 osd.0 (osd.0) 415 : cluster 0 2.13 scrub ok 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.789849+0000 osd.1 (osd.1) 341 : cluster 0 2.15 scrub starts 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:44.792071+0000 osd.1 (osd.1) 342 : cluster 0 2.15 scrub ok 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:45.004951+0000 osd.7 (osd.7) 455 : cluster 0 3.d scrub starts 2024-04-06T03:38:46.102 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:45 smithi153 bash[23852]: cluster 2024-04-06T03:38:45.009693+0000 osd.7 (osd.7) 456 : cluster 0 3.d scrub ok 2024-04-06T03:38:46.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:46.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:46.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:46.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:46.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:46.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:46.510 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:46.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:46.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:46.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:46.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:46.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:46.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:46.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[21467]: cluster 2024-04-06T03:38:45.597361+0000 osd.0 (osd.0) 416 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:46.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[21467]: cluster 2024-04-06T03:38:45.597476+0000 osd.0 (osd.0) 417 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:46.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[21467]: cluster 2024-04-06T03:38:45.900738+0000 mon.a (mon.0) 1482 : cluster 0 osdmap e283: 8 total, 8 up, 8 in 2024-04-06T03:38:46.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[21467]: cluster 2024-04-06T03:38:46.369729+0000 mgr.y (mgr.24368) 656 : cluster 0 pgmap v697: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[29221]: cluster 2024-04-06T03:38:45.597361+0000 osd.0 (osd.0) 416 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:46.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[29221]: cluster 2024-04-06T03:38:45.597476+0000 osd.0 (osd.0) 417 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:46.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[29221]: cluster 2024-04-06T03:38:45.900738+0000 mon.a (mon.0) 1482 : cluster 0 osdmap e283: 8 total, 8 up, 8 in 2024-04-06T03:38:46.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:46 smithi050 bash[29221]: cluster 2024-04-06T03:38:46.369729+0000 mgr.y (mgr.24368) 656 : cluster 0 pgmap v697: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:47.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:47.035 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:46 smithi153 bash[23852]: cluster 2024-04-06T03:38:45.597361+0000 osd.0 (osd.0) 416 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:46 smithi153 bash[23852]: cluster 2024-04-06T03:38:45.597476+0000 osd.0 (osd.0) 417 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:46 smithi153 bash[23852]: cluster 2024-04-06T03:38:45.900738+0000 mon.a (mon.0) 1482 : cluster 0 osdmap e283: 8 total, 8 up, 8 in 2024-04-06T03:38:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:46 smithi153 bash[23852]: cluster 2024-04-06T03:38:46.369729+0000 mgr.y (mgr.24368) 656 : cluster 0 pgmap v697: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:47.139 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:47.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:47.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:47.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:47.399 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:47.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:47.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:47.562 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:38:47.583 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:38:47.603 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:38:47.624 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:38:47.650 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:38:47.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:47.675 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:38:47.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:47.701 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:38:47.726 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:38:47.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:38:47.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:47.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:47.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:47.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:47.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:47.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:48.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:48.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:47 smithi050 bash[21467]: cluster 2024-04-06T03:38:47.200087+0000 osd.3 (osd.3) 432 : cluster 0 3.18 scrub starts 2024-04-06T03:38:48.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:47 smithi050 bash[21467]: cluster 2024-04-06T03:38:47.204647+0000 osd.3 (osd.3) 433 : cluster 0 3.18 scrub ok 2024-04-06T03:38:48.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:47 smithi050 bash[21467]: audit 2024-04-06T03:38:47.539479+0000 mon.c (mon.2) 86 : audit 0 from='client.? 172.21.15.50:0/2326965821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:48.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:47 smithi050 bash[29221]: cluster 2024-04-06T03:38:47.200087+0000 osd.3 (osd.3) 432 : cluster 0 3.18 scrub starts 2024-04-06T03:38:48.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:47 smithi050 bash[29221]: cluster 2024-04-06T03:38:47.204647+0000 osd.3 (osd.3) 433 : cluster 0 3.18 scrub ok 2024-04-06T03:38:48.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:47 smithi050 bash[29221]: audit 2024-04-06T03:38:47.539479+0000 mon.c (mon.2) 86 : audit 0 from='client.? 172.21.15.50:0/2326965821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:48.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:48.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:48.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:47 smithi153 bash[23852]: cluster 2024-04-06T03:38:47.200087+0000 osd.3 (osd.3) 432 : cluster 0 3.18 scrub starts 2024-04-06T03:38:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:47 smithi153 bash[23852]: cluster 2024-04-06T03:38:47.204647+0000 osd.3 (osd.3) 433 : cluster 0 3.18 scrub ok 2024-04-06T03:38:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:47 smithi153 bash[23852]: audit 2024-04-06T03:38:47.539479+0000 mon.c (mon.2) 86 : audit 0 from='client.? 172.21.15.50:0/2326965821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:38:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:48.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:48.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:48.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:48.585 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:48.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:48.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:48.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:48.846 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:48.949 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:48.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:48.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:48.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:48 smithi050 bash[21467]: cluster 2024-04-06T03:38:48.370854+0000 mgr.y (mgr.24368) 657 : cluster 0 pgmap v698: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:38:48.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:48 smithi050 bash[29221]: cluster 2024-04-06T03:38:48.370854+0000 mgr.y (mgr.24368) 657 : cluster 0 pgmap v698: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:38:49.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:49.090 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:48 smithi153 bash[23852]: cluster 2024-04-06T03:38:48.370854+0000 mgr.y (mgr.24368) 657 : cluster 0 pgmap v698: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:38:49.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:49.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:49.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:49.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:49.361 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:49.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.495 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:49.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:49.520 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:38:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:38:49.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:49.624 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:49.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:49.755 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:49.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:49.886 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:49.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:49.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:49.927 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-06T03:38:49.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:50.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:50.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:50.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:50.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:50.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[29221]: audit 2024-04-06T03:38:49.150083+0000 mon.a (mon.0) 1483 : audit 1 from='client.? 172.21.15.50:0/4138806668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:38:50.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[29221]: audit 2024-04-06T03:38:49.834124+0000 mon.a (mon.0) 1484 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:38:50.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[29221]: audit 2024-04-06T03:38:49.835645+0000 mon.a (mon.0) 1485 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:50.191 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[21467]: audit 2024-04-06T03:38:49.150083+0000 mon.a (mon.0) 1483 : audit 1 from='client.? 172.21.15.50:0/4138806668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:38:50.191 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[21467]: audit 2024-04-06T03:38:49.834124+0000 mon.a (mon.0) 1484 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:38:50.191 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:49 smithi050 bash[21467]: audit 2024-04-06T03:38:49.835645+0000 mon.a (mon.0) 1485 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:50.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:50.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:50.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:38:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:38:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:49 smithi153 bash[23852]: audit 2024-04-06T03:38:49.150083+0000 mon.a (mon.0) 1483 : audit 1 from='client.? 172.21.15.50:0/4138806668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:38:50.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:49 smithi153 bash[23852]: audit 2024-04-06T03:38:49.834124+0000 mon.a (mon.0) 1484 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:38:50.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:49 smithi153 bash[23852]: audit 2024-04-06T03:38:49.835645+0000 mon.a (mon.0) 1485 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:38:50.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:50.393 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:50.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:50.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:50.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:50.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:50.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:50.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:50.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:50.913 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:50.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:50.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[21467]: cluster 2024-04-06T03:38:49.027272+0000 osd.2 (osd.2) 351 : cluster 0 5.6 scrub starts 2024-04-06T03:38:50.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[21467]: cluster 2024-04-06T03:38:49.029787+0000 osd.2 (osd.2) 352 : cluster 0 5.6 scrub ok 2024-04-06T03:38:50.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[21467]: audit 2024-04-06T03:38:49.883089+0000 mgr.y (mgr.24368) 658 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:50.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[21467]: audit 2024-04-06T03:38:49.922855+0000 mon.a (mon.0) 1486 : audit 1 from='client.? 172.21.15.50:0/4138806668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:38:51.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:51.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:51.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:51.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[21467]: cluster 2024-04-06T03:38:49.927839+0000 mon.a (mon.0) 1487 : cluster 0 osdmap e284: 8 total, 8 up, 8 in 2024-04-06T03:38:51.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[21467]: cluster 2024-04-06T03:38:50.371354+0000 mgr.y (mgr.24368) 659 : cluster 0 pgmap v700: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:38:51.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[29221]: cluster 2024-04-06T03:38:49.027272+0000 osd.2 (osd.2) 351 : cluster 0 5.6 scrub starts 2024-04-06T03:38:51.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[29221]: cluster 2024-04-06T03:38:49.029787+0000 osd.2 (osd.2) 352 : cluster 0 5.6 scrub ok 2024-04-06T03:38:51.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[29221]: audit 2024-04-06T03:38:49.883089+0000 mgr.y (mgr.24368) 658 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:51.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[29221]: audit 2024-04-06T03:38:49.922855+0000 mon.a (mon.0) 1486 : audit 1 from='client.? 172.21.15.50:0/4138806668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:38:51.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[29221]: cluster 2024-04-06T03:38:49.927839+0000 mon.a (mon.0) 1487 : cluster 0 osdmap e284: 8 total, 8 up, 8 in 2024-04-06T03:38:51.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:50 smithi050 bash[29221]: cluster 2024-04-06T03:38:50.371354+0000 mgr.y (mgr.24368) 659 : cluster 0 pgmap v700: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:38:51.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:51.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:51.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:50 smithi153 bash[23852]: cluster 2024-04-06T03:38:49.027272+0000 osd.2 (osd.2) 351 : cluster 0 5.6 scrub starts 2024-04-06T03:38:51.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:50 smithi153 bash[23852]: cluster 2024-04-06T03:38:49.029787+0000 osd.2 (osd.2) 352 : cluster 0 5.6 scrub ok 2024-04-06T03:38:51.289 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:50 smithi153 bash[23852]: audit 2024-04-06T03:38:49.883089+0000 mgr.y (mgr.24368) 658 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:38:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:50 smithi153 bash[23852]: audit 2024-04-06T03:38:49.922855+0000 mon.a (mon.0) 1486 : audit 1 from='client.? 172.21.15.50:0/4138806668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:38:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:50 smithi153 bash[23852]: cluster 2024-04-06T03:38:49.927839+0000 mon.a (mon.0) 1487 : cluster 0 osdmap e284: 8 total, 8 up, 8 in 2024-04-06T03:38:51.290 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:50 smithi153 bash[23852]: cluster 2024-04-06T03:38:50.371354+0000 mgr.y (mgr.24368) 659 : cluster 0 pgmap v700: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:38:51.307 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:51.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:51.437 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:51.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:51.568 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:51.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:51.697 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:51.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:51.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:51.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 7 1 2024-04-06T03:38:51.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:51.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:51.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:52.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:52.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:52.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[21467]: cluster 2024-04-06T03:38:50.415408+0000 osd.5 (osd.5) 141 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[21467]: cluster 2024-04-06T03:38:50.415512+0000 osd.5 (osd.5) 142 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[21467]: cluster 2024-04-06T03:38:50.751266+0000 osd.1 (osd.1) 343 : cluster 0 2.9 scrub starts 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[21467]: cluster 2024-04-06T03:38:50.753385+0000 osd.1 (osd.1) 344 : cluster 0 2.9 scrub ok 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[21467]: cluster 2024-04-06T03:38:50.940313+0000 mon.a (mon.0) 1488 : cluster 0 osdmap e285: 8 total, 8 up, 8 in 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[29221]: cluster 2024-04-06T03:38:50.415408+0000 osd.5 (osd.5) 141 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[29221]: cluster 2024-04-06T03:38:50.415512+0000 osd.5 (osd.5) 142 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[29221]: cluster 2024-04-06T03:38:50.751266+0000 osd.1 (osd.1) 343 : cluster 0 2.9 scrub starts 2024-04-06T03:38:52.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[29221]: cluster 2024-04-06T03:38:50.753385+0000 osd.1 (osd.1) 344 : cluster 0 2.9 scrub ok 2024-04-06T03:38:52.200 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:51 smithi050 bash[29221]: cluster 2024-04-06T03:38:50.940313+0000 mon.a (mon.0) 1488 : cluster 0 osdmap e285: 8 total, 8 up, 8 in 2024-04-06T03:38:52.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:52.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:51 smithi153 bash[23852]: cluster 2024-04-06T03:38:50.415408+0000 osd.5 (osd.5) 141 : cluster 0 purged_snaps scrub starts 2024-04-06T03:38:52.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:51 smithi153 bash[23852]: cluster 2024-04-06T03:38:50.415512+0000 osd.5 (osd.5) 142 : cluster 0 purged_snaps scrub ok 2024-04-06T03:38:52.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:51 smithi153 bash[23852]: cluster 2024-04-06T03:38:50.751266+0000 osd.1 (osd.1) 343 : cluster 0 2.9 scrub starts 2024-04-06T03:38:52.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:51 smithi153 bash[23852]: cluster 2024-04-06T03:38:50.753385+0000 osd.1 (osd.1) 344 : cluster 0 2.9 scrub ok 2024-04-06T03:38:52.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:51 smithi153 bash[23852]: cluster 2024-04-06T03:38:50.940313+0000 mon.a (mon.0) 1488 : cluster 0 osdmap e285: 8 total, 8 up, 8 in 2024-04-06T03:38:52.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:52.344 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:52.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:52.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:52.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:52.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:52.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:52.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:38:52.736 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:52.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:52.868 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:52.943 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:38:52.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:52.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: cluster 2024-04-06T03:38:51.172087+0000 osd.4 (osd.4) 403 : cluster 0 6.1d scrub starts 2024-04-06T03:38:52.979 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: cluster 2024-04-06T03:38:51.251478+0000 osd.4 (osd.4) 404 : cluster 0 6.1d scrub ok 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: audit 2024-04-06T03:38:52.221373+0000 mon.c (mon.2) 87 : audit 1 from='client.? 172.21.15.50:0/1974146556' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: audit 2024-04-06T03:38:52.221741+0000 mon.a (mon.0) 1489 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.263935+0000 osd.3 (osd.3) 434 : cluster 0 2.1e scrub starts 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.265533+0000 osd.3 (osd.3) 435 : cluster 0 2.1e scrub ok 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.371996+0000 mgr.y (mgr.24368) 660 : cluster 0 pgmap v702: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: cluster 2024-04-06T03:38:51.172087+0000 osd.4 (osd.4) 403 : cluster 0 6.1d scrub starts 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: cluster 2024-04-06T03:38:51.251478+0000 osd.4 (osd.4) 404 : cluster 0 6.1d scrub ok 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: audit 2024-04-06T03:38:52.221373+0000 mon.c (mon.2) 87 : audit 1 from='client.? 172.21.15.50:0/1974146556' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: audit 2024-04-06T03:38:52.221741+0000 mon.a (mon.0) 1489 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.263935+0000 osd.3 (osd.3) 434 : cluster 0 2.1e scrub starts 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.265533+0000 osd.3 (osd.3) 435 : cluster 0 2.1e scrub ok 2024-04-06T03:38:52.980 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:52 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.371996+0000 mgr.y (mgr.24368) 660 : cluster 0 pgmap v702: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:38:52.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:52.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:53.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:53.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:53.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:53.245 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:53.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: cluster 2024-04-06T03:38:51.172087+0000 osd.4 (osd.4) 403 : cluster 0 6.1d scrub starts 2024-04-06T03:38:53.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: cluster 2024-04-06T03:38:51.251478+0000 osd.4 (osd.4) 404 : cluster 0 6.1d scrub ok 2024-04-06T03:38:53.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: audit 2024-04-06T03:38:52.221373+0000 mon.c (mon.2) 87 : audit 1 from='client.? 172.21.15.50:0/1974146556' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:53.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: audit 2024-04-06T03:38:52.221741+0000 mon.a (mon.0) 1489 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:38:53.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.263935+0000 osd.3 (osd.3) 434 : cluster 0 2.1e scrub starts 2024-04-06T03:38:53.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.265533+0000 osd.3 (osd.3) 435 : cluster 0 2.1e scrub ok 2024-04-06T03:38:53.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:52 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.371996+0000 mgr.y (mgr.24368) 660 : cluster 0 pgmap v702: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:38:53.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:53.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:53.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:53.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:53.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:53.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:53.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:53.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:53.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:53.906 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:53.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:54.009 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.162515+0000 osd.4 (osd.4) 405 : cluster 0 6.17 scrub starts 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.303747+0000 osd.4 (osd.4) 406 : cluster 0 6.17 scrub ok 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.933074+0000 mon.a (mon.0) 1490 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: audit 2024-04-06T03:38:52.940076+0000 mon.a (mon.0) 1491 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: cluster 2024-04-06T03:38:52.944527+0000 mon.a (mon.0) 1492 : cluster 0 osdmap e286: 8 total, 8 up, 8 in 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: cluster 2024-04-06T03:38:53.197870+0000 osd.4 (osd.4) 407 : cluster 0 3.3 scrub starts 2024-04-06T03:38:54.021 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:53 smithi153 bash[23852]: cluster 2024-04-06T03:38:53.204355+0000 osd.4 (osd.4) 408 : cluster 0 3.3 scrub ok 2024-04-06T03:38:54.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:54.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:54.165 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:54.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:54.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:54.327 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.162515+0000 osd.4 (osd.4) 405 : cluster 0 6.17 scrub starts 2024-04-06T03:38:54.327 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.303747+0000 osd.4 (osd.4) 406 : cluster 0 6.17 scrub ok 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.933074+0000 mon.a (mon.0) 1490 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: audit 2024-04-06T03:38:52.940076+0000 mon.a (mon.0) 1491 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: cluster 2024-04-06T03:38:52.944527+0000 mon.a (mon.0) 1492 : cluster 0 osdmap e286: 8 total, 8 up, 8 in 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: cluster 2024-04-06T03:38:53.197870+0000 osd.4 (osd.4) 407 : cluster 0 3.3 scrub starts 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[21467]: cluster 2024-04-06T03:38:53.204355+0000 osd.4 (osd.4) 408 : cluster 0 3.3 scrub ok 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.162515+0000 osd.4 (osd.4) 405 : cluster 0 6.17 scrub starts 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.303747+0000 osd.4 (osd.4) 406 : cluster 0 6.17 scrub ok 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.933074+0000 mon.a (mon.0) 1490 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: audit 2024-04-06T03:38:52.940076+0000 mon.a (mon.0) 1491 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: cluster 2024-04-06T03:38:52.944527+0000 mon.a (mon.0) 1492 : cluster 0 osdmap e286: 8 total, 8 up, 8 in 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: cluster 2024-04-06T03:38:53.197870+0000 osd.4 (osd.4) 407 : cluster 0 3.3 scrub starts 2024-04-06T03:38:54.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:53 smithi050 bash[29221]: cluster 2024-04-06T03:38:53.204355+0000 osd.4 (osd.4) 408 : cluster 0 3.3 scrub ok 2024-04-06T03:38:54.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:54.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:54.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:54.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:54.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:54.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:54.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:54.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:54.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:54.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:54.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:55.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:55.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:55.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:55.182 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:54 smithi050 bash[21467]: cluster 2024-04-06T03:38:54.373005+0000 mgr.y (mgr.24368) 661 : cluster 0 pgmap v704: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:55.182 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:54 smithi050 bash[29221]: cluster 2024-04-06T03:38:54.373005+0000 mgr.y (mgr.24368) 661 : cluster 0 pgmap v704: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:55.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:55.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:55.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:54 smithi153 bash[23852]: cluster 2024-04-06T03:38:54.373005+0000 mgr.y (mgr.24368) 661 : cluster 0 pgmap v704: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-06T03:38:55.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:55.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:55.456 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:55.559 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:55.586 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:55.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:55.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:55.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:55.848 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:55.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:55.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:55.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:56.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:56.113 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:56.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:56.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:55 smithi050 bash[21467]: cluster 2024-04-06T03:38:54.794787+0000 osd.0 (osd.0) 418 : cluster 0 3.10 scrub starts 2024-04-06T03:38:56.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:55 smithi050 bash[21467]: cluster 2024-04-06T03:38:54.798634+0000 osd.0 (osd.0) 419 : cluster 0 3.10 scrub ok 2024-04-06T03:38:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:55 smithi050 bash[29221]: cluster 2024-04-06T03:38:54.794787+0000 osd.0 (osd.0) 418 : cluster 0 3.10 scrub starts 2024-04-06T03:38:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:55 smithi050 bash[29221]: cluster 2024-04-06T03:38:54.798634+0000 osd.0 (osd.0) 419 : cluster 0 3.10 scrub ok 2024-04-06T03:38:56.245 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:56.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:55 smithi153 bash[23852]: cluster 2024-04-06T03:38:54.794787+0000 osd.0 (osd.0) 418 : cluster 0 3.10 scrub starts 2024-04-06T03:38:56.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:55 smithi153 bash[23852]: cluster 2024-04-06T03:38:54.798634+0000 osd.0 (osd.0) 419 : cluster 0 3.10 scrub ok 2024-04-06T03:38:56.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:56.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:56.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:56.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:56.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:56.637 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:56.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:56.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:56.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:38:56.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:56.902 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:56.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:57.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:57.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:57.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:57.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:56 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.229411+0000 osd.3 (osd.3) 436 : cluster 0 2.12 scrub starts 2024-04-06T03:38:57.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:56 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.232378+0000 osd.3 (osd.3) 437 : cluster 0 2.12 scrub ok 2024-04-06T03:38:57.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:56 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.373687+0000 mgr.y (mgr.24368) 662 : cluster 0 pgmap v705: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 793 B/s rd, 0 op/s 2024-04-06T03:38:57.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:56 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.229411+0000 osd.3 (osd.3) 436 : cluster 0 2.12 scrub starts 2024-04-06T03:38:57.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:56 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.232378+0000 osd.3 (osd.3) 437 : cluster 0 2.12 scrub ok 2024-04-06T03:38:57.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:56 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.373687+0000 mgr.y (mgr.24368) 662 : cluster 0 pgmap v705: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 793 B/s rd, 0 op/s 2024-04-06T03:38:57.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:57.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:57.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:56 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.229411+0000 osd.3 (osd.3) 436 : cluster 0 2.12 scrub starts 2024-04-06T03:38:57.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:56 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.232378+0000 osd.3 (osd.3) 437 : cluster 0 2.12 scrub ok 2024-04-06T03:38:57.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:56 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.373687+0000 mgr.y (mgr.24368) 662 : cluster 0 pgmap v705: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 793 B/s rd, 0 op/s 2024-04-06T03:38:57.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:38:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:38:57.368 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:57.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:57.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:57.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:57.560 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:57.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:57.691 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:57.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:57.821 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:57.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:57.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:58.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:58.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:58.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:58.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:58.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.121996+0000 osd.2 (osd.2) 353 : cluster 0 2.1 scrub starts 2024-04-06T03:38:58.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.123314+0000 osd.2 (osd.2) 354 : cluster 0 2.1 scrub ok 2024-04-06T03:38:58.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.693053+0000 osd.1 (osd.1) 345 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:38:58.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:56.770301+0000 osd.1 (osd.1) 346 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.055778+0000 osd.4 (osd.4) 409 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.058398+0000 osd.4 (osd.4) 410 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.122727+0000 osd.2 (osd.2) 355 : cluster 0 3.4 scrub starts 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.125237+0000 osd.2 (osd.2) 356 : cluster 0 3.4 scrub ok 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.279113+0000 osd.3 (osd.3) 438 : cluster 0 3.b scrub starts 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.282007+0000 osd.3 (osd.3) 439 : cluster 0 3.b scrub ok 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.121996+0000 osd.2 (osd.2) 353 : cluster 0 2.1 scrub starts 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.123314+0000 osd.2 (osd.2) 354 : cluster 0 2.1 scrub ok 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.693053+0000 osd.1 (osd.1) 345 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:56.770301+0000 osd.1 (osd.1) 346 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:38:58.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.055778+0000 osd.4 (osd.4) 409 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:38:58.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.058398+0000 osd.4 (osd.4) 410 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:38:58.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.122727+0000 osd.2 (osd.2) 355 : cluster 0 3.4 scrub starts 2024-04-06T03:38:58.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.125237+0000 osd.2 (osd.2) 356 : cluster 0 3.4 scrub ok 2024-04-06T03:38:58.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.279113+0000 osd.3 (osd.3) 438 : cluster 0 3.b scrub starts 2024-04-06T03:38:58.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:57 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.282007+0000 osd.3 (osd.3) 439 : cluster 0 3.b scrub ok 2024-04-06T03:38:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.121996+0000 osd.2 (osd.2) 353 : cluster 0 2.1 scrub starts 2024-04-06T03:38:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.123314+0000 osd.2 (osd.2) 354 : cluster 0 2.1 scrub ok 2024-04-06T03:38:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.693053+0000 osd.1 (osd.1) 345 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:38:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:56.770301+0000 osd.1 (osd.1) 346 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:38:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.055778+0000 osd.4 (osd.4) 409 : cluster 0 3.1a deep-scrub starts 2024-04-06T03:38:58.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.058398+0000 osd.4 (osd.4) 410 : cluster 0 3.1a deep-scrub ok 2024-04-06T03:38:58.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.122727+0000 osd.2 (osd.2) 355 : cluster 0 3.4 scrub starts 2024-04-06T03:38:58.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.125237+0000 osd.2 (osd.2) 356 : cluster 0 3.4 scrub ok 2024-04-06T03:38:58.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.279113+0000 osd.3 (osd.3) 438 : cluster 0 3.b scrub starts 2024-04-06T03:38:58.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:57 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.282007+0000 osd.3 (osd.3) 439 : cluster 0 3.b scrub ok 2024-04-06T03:38:58.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:58.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:38:58.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:38:58.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:58.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:58.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:58.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:58.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:58.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:58.865 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:58.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:58.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:58.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:59.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:59.124 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:38:59.227 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:59.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.719316+0000 osd.1 (osd.1) 347 : cluster 0 2.2 scrub starts 2024-04-06T03:38:59.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[21467]: cluster 2024-04-06T03:38:57.721617+0000 osd.1 (osd.1) 348 : cluster 0 2.2 scrub ok 2024-04-06T03:38:59.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[21467]: cluster 2024-04-06T03:38:58.008698+0000 osd.7 (osd.7) 457 : cluster 0 2.8 scrub starts 2024-04-06T03:38:59.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[21467]: cluster 2024-04-06T03:38:58.011067+0000 osd.7 (osd.7) 458 : cluster 0 2.8 scrub ok 2024-04-06T03:38:59.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[21467]: cluster 2024-04-06T03:38:58.374668+0000 mgr.y (mgr.24368) 663 : cluster 0 pgmap v706: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.719316+0000 osd.1 (osd.1) 347 : cluster 0 2.2 scrub starts 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[29221]: cluster 2024-04-06T03:38:57.721617+0000 osd.1 (osd.1) 348 : cluster 0 2.2 scrub ok 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[29221]: cluster 2024-04-06T03:38:58.008698+0000 osd.7 (osd.7) 457 : cluster 0 2.8 scrub starts 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[29221]: cluster 2024-04-06T03:38:58.011067+0000 osd.7 (osd.7) 458 : cluster 0 2.8 scrub ok 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:38:58 smithi050 bash[29221]: cluster 2024-04-06T03:38:58.374668+0000 mgr.y (mgr.24368) 663 : cluster 0 pgmap v706: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:58 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.719316+0000 osd.1 (osd.1) 347 : cluster 0 2.2 scrub starts 2024-04-06T03:38:59.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:58 smithi153 bash[23852]: cluster 2024-04-06T03:38:57.721617+0000 osd.1 (osd.1) 348 : cluster 0 2.2 scrub ok 2024-04-06T03:38:59.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:58 smithi153 bash[23852]: cluster 2024-04-06T03:38:58.008698+0000 osd.7 (osd.7) 457 : cluster 0 2.8 scrub starts 2024-04-06T03:38:59.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:58 smithi153 bash[23852]: cluster 2024-04-06T03:38:58.011067+0000 osd.7 (osd.7) 458 : cluster 0 2.8 scrub ok 2024-04-06T03:38:59.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:58 smithi153 bash[23852]: cluster 2024-04-06T03:38:58.374668+0000 mgr.y (mgr.24368) 663 : cluster 0 pgmap v706: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:38:59.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:38:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:38:59.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:59.387 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:38:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:38:59.416 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:38:59.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:59.514 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:38:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:38:59.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:59.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:38:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:38:59.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:59.775 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:38:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:38:59.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:59.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:38:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:38:59.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:00.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:00.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:00.048 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:39:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:39:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-06T03:39:00.049 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:39:00.109 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:38:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:38:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:00.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:00.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:00.277 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:38:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:39:00.277 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:59 smithi153 bash[23852]: cluster 2024-04-06T03:38:58.873063+0000 osd.0 (osd.0) 420 : cluster 0 6.d deep-scrub starts 2024-04-06T03:39:00.277 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:59 smithi153 bash[23852]: cluster 2024-04-06T03:38:58.987342+0000 osd.0 (osd.0) 421 : cluster 0 6.d deep-scrub ok 2024-04-06T03:39:00.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:59 smithi153 bash[23852]: cluster 2024-04-06T03:38:59.031779+0000 osd.4 (osd.4) 411 : cluster 0 5.c scrub starts 2024-04-06T03:39:00.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:38:59 smithi153 bash[23852]: cluster 2024-04-06T03:38:59.034065+0000 osd.4 (osd.4) 412 : cluster 0 5.c scrub ok 2024-04-06T03:39:00.293 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:00.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:00.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:38:58.873063+0000 osd.0 (osd.0) 420 : cluster 0 6.d deep-scrub starts 2024-04-06T03:39:00.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:38:58.987342+0000 osd.0 (osd.0) 421 : cluster 0 6.d deep-scrub ok 2024-04-06T03:39:00.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:38:59.031779+0000 osd.4 (osd.4) 411 : cluster 0 5.c scrub starts 2024-04-06T03:39:00.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:38:59.034065+0000 osd.4 (osd.4) 412 : cluster 0 5.c scrub ok 2024-04-06T03:39:00.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:38:58.873063+0000 osd.0 (osd.0) 420 : cluster 0 6.d deep-scrub starts 2024-04-06T03:39:00.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:38:58.987342+0000 osd.0 (osd.0) 421 : cluster 0 6.d deep-scrub ok 2024-04-06T03:39:00.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:38:59.031779+0000 osd.4 (osd.4) 411 : cluster 0 5.c scrub starts 2024-04-06T03:39:00.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:38:59.034065+0000 osd.4 (osd.4) 412 : cluster 0 5.c scrub ok 2024-04-06T03:39:00.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:00.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:00.549 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:00.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:00.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:00.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:00.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:00.905 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:00.932 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:00.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:00.996 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-04-06T03:39:01.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:01.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:01.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:01.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:01.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:38:59.835480+0000 osd.0 (osd.0) 422 : cluster 0 6.9 scrub starts 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: audit 2024-04-06T03:38:59.894166+0000 mgr.y (mgr.24368) 664 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:38:59.917281+0000 osd.0 (osd.0) 423 : cluster 0 6.9 scrub ok 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:39:00.030220+0000 osd.4 (osd.4) 413 : cluster 0 5.18 scrub starts 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:39:00.032292+0000 osd.4 (osd.4) 414 : cluster 0 5.18 scrub ok 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:39:00.235871+0000 osd.2 (osd.2) 357 : cluster 0 2.16 scrub starts 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:39:00.237456+0000 osd.2 (osd.2) 358 : cluster 0 2.16 scrub ok 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: cluster 2024-04-06T03:39:00.375350+0000 mgr.y (mgr.24368) 665 : cluster 0 pgmap v707: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:39:01.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: audit 2024-04-06T03:39:00.558038+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.50:0/1825829130' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[29221]: audit 2024-04-06T03:39:00.558401+0000 mon.a (mon.0) 1493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:38:59.835480+0000 osd.0 (osd.0) 422 : cluster 0 6.9 scrub starts 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: audit 2024-04-06T03:38:59.894166+0000 mgr.y (mgr.24368) 664 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:38:59.917281+0000 osd.0 (osd.0) 423 : cluster 0 6.9 scrub ok 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:39:00.030220+0000 osd.4 (osd.4) 413 : cluster 0 5.18 scrub starts 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:39:00.032292+0000 osd.4 (osd.4) 414 : cluster 0 5.18 scrub ok 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:39:00.235871+0000 osd.2 (osd.2) 357 : cluster 0 2.16 scrub starts 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:39:00.237456+0000 osd.2 (osd.2) 358 : cluster 0 2.16 scrub ok 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: cluster 2024-04-06T03:39:00.375350+0000 mgr.y (mgr.24368) 665 : cluster 0 pgmap v707: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: audit 2024-04-06T03:39:00.558038+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.50:0/1825829130' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:39:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:00 smithi050 bash[21467]: audit 2024-04-06T03:39:00.558401+0000 mon.a (mon.0) 1493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:39:01.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:01.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:38:59.835480+0000 osd.0 (osd.0) 422 : cluster 0 6.9 scrub starts 2024-04-06T03:39:01.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: audit 2024-04-06T03:38:59.894166+0000 mgr.y (mgr.24368) 664 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:01.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:38:59.917281+0000 osd.0 (osd.0) 423 : cluster 0 6.9 scrub ok 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:39:00.030220+0000 osd.4 (osd.4) 413 : cluster 0 5.18 scrub starts 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:39:00.032292+0000 osd.4 (osd.4) 414 : cluster 0 5.18 scrub ok 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:39:00.235871+0000 osd.2 (osd.2) 357 : cluster 0 2.16 scrub starts 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:39:00.237456+0000 osd.2 (osd.2) 358 : cluster 0 2.16 scrub ok 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: cluster 2024-04-06T03:39:00.375350+0000 mgr.y (mgr.24368) 665 : cluster 0 pgmap v707: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: audit 2024-04-06T03:39:00.558038+0000 mon.c (mon.2) 88 : audit 1 from='client.? 172.21.15.50:0/1825829130' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:39:01.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:00 smithi153 bash[23852]: audit 2024-04-06T03:39:00.558401+0000 mon.a (mon.0) 1493 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:39:01.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:01.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:01.453 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:01.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:01.583 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:01.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:01.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:01.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:01.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:01.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:01.976 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:01.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:02.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:02.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:02.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:02.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:01 smithi153 bash[23852]: audit 2024-04-06T03:39:00.987042+0000 mon.a (mon.0) 1494 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:39:02.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:01 smithi153 bash[23852]: cluster 2024-04-06T03:39:00.998377+0000 mon.a (mon.0) 1495 : cluster 0 osdmap e287: 8 total, 8 up, 8 in 2024-04-06T03:39:02.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:02.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:01 smithi050 bash[21467]: audit 2024-04-06T03:39:00.987042+0000 mon.a (mon.0) 1494 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:39:02.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:01 smithi050 bash[21467]: cluster 2024-04-06T03:39:00.998377+0000 mon.a (mon.0) 1495 : cluster 0 osdmap e287: 8 total, 8 up, 8 in 2024-04-06T03:39:02.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: audit 2024-04-06T03:39:00.987042+0000 mon.a (mon.0) 1494 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:39:02.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: cluster 2024-04-06T03:39:00.998377+0000 mon.a (mon.0) 1495 : cluster 0 osdmap e287: 8 total, 8 up, 8 in 2024-04-06T03:39:02.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:02.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:02.465 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:02.492 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:02.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:02.624 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:02.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:02.755 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:02.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:02.885 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:02.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:02.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:03.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:03.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:03.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:03.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[21467]: cluster 2024-04-06T03:39:01.757487+0000 osd.1 (osd.1) 349 : cluster 0 6.4 scrub starts 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[21467]: cluster 2024-04-06T03:39:01.759463+0000 osd.1 (osd.1) 350 : cluster 0 6.4 scrub ok 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[21467]: cluster 2024-04-06T03:39:01.845061+0000 osd.0 (osd.0) 424 : cluster 0 3.1f deep-scrub starts 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[21467]: cluster 2024-04-06T03:39:01.848217+0000 osd.0 (osd.0) 425 : cluster 0 3.1f deep-scrub ok 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[21467]: cluster 2024-04-06T03:39:02.376398+0000 mgr.y (mgr.24368) 666 : cluster 0 pgmap v709: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: cluster 2024-04-06T03:39:01.757487+0000 osd.1 (osd.1) 349 : cluster 0 6.4 scrub starts 2024-04-06T03:39:03.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: cluster 2024-04-06T03:39:01.759463+0000 osd.1 (osd.1) 350 : cluster 0 6.4 scrub ok 2024-04-06T03:39:03.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: cluster 2024-04-06T03:39:01.845061+0000 osd.0 (osd.0) 424 : cluster 0 3.1f deep-scrub starts 2024-04-06T03:39:03.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: cluster 2024-04-06T03:39:01.848217+0000 osd.0 (osd.0) 425 : cluster 0 3.1f deep-scrub ok 2024-04-06T03:39:03.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:02 smithi050 bash[29221]: cluster 2024-04-06T03:39:02.376398+0000 mgr.y (mgr.24368) 666 : cluster 0 pgmap v709: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:39:03.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:03.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:02 smithi153 bash[23852]: cluster 2024-04-06T03:39:01.757487+0000 osd.1 (osd.1) 349 : cluster 0 6.4 scrub starts 2024-04-06T03:39:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:02 smithi153 bash[23852]: cluster 2024-04-06T03:39:01.759463+0000 osd.1 (osd.1) 350 : cluster 0 6.4 scrub ok 2024-04-06T03:39:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:02 smithi153 bash[23852]: cluster 2024-04-06T03:39:01.845061+0000 osd.0 (osd.0) 424 : cluster 0 3.1f deep-scrub starts 2024-04-06T03:39:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:02 smithi153 bash[23852]: cluster 2024-04-06T03:39:01.848217+0000 osd.0 (osd.0) 425 : cluster 0 3.1f deep-scrub ok 2024-04-06T03:39:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:02 smithi153 bash[23852]: cluster 2024-04-06T03:39:02.376398+0000 mgr.y (mgr.24368) 666 : cluster 0 pgmap v709: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:39:03.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:03.410 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:03.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:03.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-06T03:39:03.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:03.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:03.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:03.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:03.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:03.795 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:03.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:03.925 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:03.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:04.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:03 smithi153 bash[23852]: cluster 2024-04-06T03:39:02.879812+0000 osd.0 (osd.0) 426 : cluster 0 2.4 scrub starts 2024-04-06T03:39:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:03 smithi153 bash[23852]: cluster 2024-04-06T03:39:02.882701+0000 osd.0 (osd.0) 427 : cluster 0 2.4 scrub ok 2024-04-06T03:39:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:03 smithi153 bash[23852]: cluster 2024-04-06T03:39:02.988829+0000 mon.a (mon.0) 1496 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:39:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:03 smithi153 bash[23852]: cluster 2024-04-06T03:39:02.988859+0000 mon.a (mon.0) 1497 : cluster 1 Cluster is now healthy 2024-04-06T03:39:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:03 smithi153 bash[23852]: cluster 2024-04-06T03:39:03.327533+0000 osd.2 (osd.2) 359 : cluster 0 5.8 scrub starts 2024-04-06T03:39:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:03 smithi153 bash[23852]: cluster 2024-04-06T03:39:03.328956+0000 osd.2 (osd.2) 360 : cluster 0 5.8 scrub ok 2024-04-06T03:39:04.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:04.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:04.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:04.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[21467]: cluster 2024-04-06T03:39:02.879812+0000 osd.0 (osd.0) 426 : cluster 0 2.4 scrub starts 2024-04-06T03:39:04.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[21467]: cluster 2024-04-06T03:39:02.882701+0000 osd.0 (osd.0) 427 : cluster 0 2.4 scrub ok 2024-04-06T03:39:04.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[21467]: cluster 2024-04-06T03:39:02.988829+0000 mon.a (mon.0) 1496 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:39:04.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[21467]: cluster 2024-04-06T03:39:02.988859+0000 mon.a (mon.0) 1497 : cluster 1 Cluster is now healthy 2024-04-06T03:39:04.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[21467]: cluster 2024-04-06T03:39:03.327533+0000 osd.2 (osd.2) 359 : cluster 0 5.8 scrub starts 2024-04-06T03:39:04.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[21467]: cluster 2024-04-06T03:39:03.328956+0000 osd.2 (osd.2) 360 : cluster 0 5.8 scrub ok 2024-04-06T03:39:04.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[29221]: cluster 2024-04-06T03:39:02.879812+0000 osd.0 (osd.0) 426 : cluster 0 2.4 scrub starts 2024-04-06T03:39:04.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[29221]: cluster 2024-04-06T03:39:02.882701+0000 osd.0 (osd.0) 427 : cluster 0 2.4 scrub ok 2024-04-06T03:39:04.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[29221]: cluster 2024-04-06T03:39:02.988829+0000 mon.a (mon.0) 1496 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:39:04.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[29221]: cluster 2024-04-06T03:39:02.988859+0000 mon.a (mon.0) 1497 : cluster 1 Cluster is now healthy 2024-04-06T03:39:04.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[29221]: cluster 2024-04-06T03:39:03.327533+0000 osd.2 (osd.2) 359 : cluster 0 5.8 scrub starts 2024-04-06T03:39:04.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:04 smithi050 bash[29221]: cluster 2024-04-06T03:39:03.328956+0000 osd.2 (osd.2) 360 : cluster 0 5.8 scrub ok 2024-04-06T03:39:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:04.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:04.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:04.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:04.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:04.495 DEBUG:teuthology.orchestra.run.smithi050:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-04-06T03:39:04.498 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-06T03:39:04.498 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-06T03:39:04.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:hints = 1 2024-04-06T03:39:04.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-06T03:39:04.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object prefix: benchmark_data_smithi050_121293 2024-04-06T03:39:04.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:39:04.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 0 0 0 0 0 0 - 0 2024-04-06T03:39:04.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:04.583 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:04.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:04.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:04.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:04.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:04.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:04.974 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:04.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:05.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:05.101 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:05.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:05.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:05.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:05 smithi153 bash[23852]: cluster 2024-04-06T03:39:04.377382+0000 mgr.y (mgr.24368) 667 : cluster 0 pgmap v710: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:39:05.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:05 smithi153 bash[23852]: audit 2024-04-06T03:39:04.854331+0000 mon.a (mon.0) 1498 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:05.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:05 smithi153 bash[23852]: audit 2024-04-06T03:39:04.857405+0000 mon.a (mon.0) 1499 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:05.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:05 smithi050 bash[21467]: cluster 2024-04-06T03:39:04.377382+0000 mgr.y (mgr.24368) 667 : cluster 0 pgmap v710: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:39:05.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:05 smithi050 bash[21467]: audit 2024-04-06T03:39:04.854331+0000 mon.a (mon.0) 1498 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:05.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:05 smithi050 bash[21467]: audit 2024-04-06T03:39:04.857405+0000 mon.a (mon.0) 1499 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:05.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:05 smithi050 bash[29221]: cluster 2024-04-06T03:39:04.377382+0000 mgr.y (mgr.24368) 667 : cluster 0 pgmap v710: 148 pgs: 148 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:39:05.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:05 smithi050 bash[29221]: audit 2024-04-06T03:39:04.854331+0000 mon.a (mon.0) 1498 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:05.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:05 smithi050 bash[29221]: audit 2024-04-06T03:39:04.857405+0000 mon.a (mon.0) 1499 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:05.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:05.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:05.474 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:05.548 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 1 16 1821 1805 112.801 112.812 0.00209738 0.00635871 2024-04-06T03:39:05.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:05.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:05.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:05.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:05.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:05.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:05.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:05.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:06.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:06.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:06.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:06 smithi050 bash[21467]: cluster 2024-04-06T03:39:04.850257+0000 osd.0 (osd.0) 428 : cluster 0 3.12 scrub starts 2024-04-06T03:39:06.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:06 smithi050 bash[21467]: cluster 2024-04-06T03:39:04.854172+0000 osd.0 (osd.0) 429 : cluster 0 3.12 scrub ok 2024-04-06T03:39:06.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:06 smithi050 bash[29221]: cluster 2024-04-06T03:39:04.850257+0000 osd.0 (osd.0) 428 : cluster 0 3.12 scrub starts 2024-04-06T03:39:06.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:06 smithi050 bash[29221]: cluster 2024-04-06T03:39:04.854172+0000 osd.0 (osd.0) 429 : cluster 0 3.12 scrub ok 2024-04-06T03:39:06.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:06.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:06.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:06 smithi153 bash[23852]: cluster 2024-04-06T03:39:04.850257+0000 osd.0 (osd.0) 428 : cluster 0 3.12 scrub starts 2024-04-06T03:39:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:06 smithi153 bash[23852]: cluster 2024-04-06T03:39:04.854172+0000 osd.0 (osd.0) 429 : cluster 0 3.12 scrub ok 2024-04-06T03:39:06.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:06.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:06.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:06.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:06.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 2 16 3201 3185 99.5212 86.25 0.0087746 0.0098781 2024-04-06T03:39:06.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:06.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:06.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:06.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:06.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:06.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:06.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:07.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:07.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:07.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:07.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:07.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.156625+0000 osd.3 (osd.3) 440 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:07.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.156705+0000 osd.3 (osd.3) 441 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:07.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.298572+0000 osd.3 (osd.3) 442 : cluster 0 3.7 scrub starts 2024-04-06T03:39:07.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.307780+0000 osd.3 (osd.3) 443 : cluster 0 3.7 scrub ok 2024-04-06T03:39:07.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.377928+0000 mgr.y (mgr.24368) 668 : cluster 0 pgmap v711: 148 pgs: 148 active+clean; 5.1 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 461 KiB/s wr, 8 op/s 2024-04-06T03:39:07.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.156625+0000 osd.3 (osd.3) 440 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:07.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.156705+0000 osd.3 (osd.3) 441 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:07.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.298572+0000 osd.3 (osd.3) 442 : cluster 0 3.7 scrub starts 2024-04-06T03:39:07.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.307780+0000 osd.3 (osd.3) 443 : cluster 0 3.7 scrub ok 2024-04-06T03:39:07.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:07 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.377928+0000 mgr.y (mgr.24368) 668 : cluster 0 pgmap v711: 148 pgs: 148 active+clean; 5.1 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 461 KiB/s wr, 8 op/s 2024-04-06T03:39:07.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:07.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:07 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.156625+0000 osd.3 (osd.3) 440 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:07 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.156705+0000 osd.3 (osd.3) 441 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:07 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.298572+0000 osd.3 (osd.3) 442 : cluster 0 3.7 scrub starts 2024-04-06T03:39:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:07 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.307780+0000 osd.3 (osd.3) 443 : cluster 0 3.7 scrub ok 2024-04-06T03:39:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:07 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.377928+0000 mgr.y (mgr.24368) 668 : cluster 0 pgmap v711: 148 pgs: 148 active+clean; 5.1 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 461 KiB/s wr, 8 op/s 2024-04-06T03:39:07.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:07.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:07.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:07.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:07.547 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 3 16 5165 5149 107.261 122.75 0.0111564 0.00930458 2024-04-06T03:39:07.568 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:07.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:07.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:07.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:07.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:07.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:07.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:07.945 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:39:07.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:07.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:08.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:08.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:08.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:08.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:08.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.931983+0000 osd.0 (osd.0) 430 : cluster 0 5.1e scrub starts 2024-04-06T03:39:08.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[21467]: cluster 2024-04-06T03:39:06.938943+0000 osd.0 (osd.0) 431 : cluster 0 5.1e scrub ok 2024-04-06T03:39:08.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.030859+0000 mon.a (mon.0) 1500 : cluster 0 osdmap e288: 8 total, 8 up, 8 in 2024-04-06T03:39:08.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.048703+0000 osd.4 (osd.4) 415 : cluster 0 5.11 scrub starts 2024-04-06T03:39:08.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.049870+0000 osd.4 (osd.4) 416 : cluster 0 5.11 scrub ok 2024-04-06T03:39:08.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.931983+0000 osd.0 (osd.0) 430 : cluster 0 5.1e scrub starts 2024-04-06T03:39:08.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[29221]: cluster 2024-04-06T03:39:06.938943+0000 osd.0 (osd.0) 431 : cluster 0 5.1e scrub ok 2024-04-06T03:39:08.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.030859+0000 mon.a (mon.0) 1500 : cluster 0 osdmap e288: 8 total, 8 up, 8 in 2024-04-06T03:39:08.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.048703+0000 osd.4 (osd.4) 415 : cluster 0 5.11 scrub starts 2024-04-06T03:39:08.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:08 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.049870+0000 osd.4 (osd.4) 416 : cluster 0 5.11 scrub ok 2024-04-06T03:39:08.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:08.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:08 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.931983+0000 osd.0 (osd.0) 430 : cluster 0 5.1e scrub starts 2024-04-06T03:39:08.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:08 smithi153 bash[23852]: cluster 2024-04-06T03:39:06.938943+0000 osd.0 (osd.0) 431 : cluster 0 5.1e scrub ok 2024-04-06T03:39:08.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:08 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.030859+0000 mon.a (mon.0) 1500 : cluster 0 osdmap e288: 8 total, 8 up, 8 in 2024-04-06T03:39:08.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:08 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.048703+0000 osd.4 (osd.4) 415 : cluster 0 5.11 scrub starts 2024-04-06T03:39:08.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:08 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.049870+0000 osd.4 (osd.4) 416 : cluster 0 5.11 scrub ok 2024-04-06T03:39:08.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:08.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:08.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:08.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:08.548 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 4 16 7109 7093 110.818 121.5 0.00717835 0.00900777 2024-04-06T03:39:08.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:08.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:08.736 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:08.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:08.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:08.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:08.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:08.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:09.009 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:09.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:09.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:09.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:09.273 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:09.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.944111+0000 osd.0 (osd.0) 432 : cluster 0 5.15 scrub starts 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.946312+0000 osd.0 (osd.0) 433 : cluster 0 5.15 scrub ok 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.978348+0000 osd.7 (osd.7) 459 : cluster 0 5.19 scrub starts 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:07.979614+0000 osd.7 (osd.7) 460 : cluster 0 5.19 scrub ok 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:08.004518+0000 osd.4 (osd.4) 417 : cluster 0 2.1a scrub starts 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:08.007323+0000 osd.4 (osd.4) 418 : cluster 0 2.1a scrub ok 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:08.189775+0000 osd.2 (osd.2) 361 : cluster 0 6.14 scrub starts 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:08.191826+0000 osd.2 (osd.2) 362 : cluster 0 6.14 scrub ok 2024-04-06T03:39:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[23852]: cluster 2024-04-06T03:39:08.378335+0000 mgr.y (mgr.24368) 669 : cluster 0 pgmap v713: 148 pgs: 148 active+clean; 83 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 10 MiB/s wr, 166 op/s 2024-04-06T03:39:09.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:09.402 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:09.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.944111+0000 osd.0 (osd.0) 432 : cluster 0 5.15 scrub starts 2024-04-06T03:39:09.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.946312+0000 osd.0 (osd.0) 433 : cluster 0 5.15 scrub ok 2024-04-06T03:39:09.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.978348+0000 osd.7 (osd.7) 459 : cluster 0 5.19 scrub starts 2024-04-06T03:39:09.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:07.979614+0000 osd.7 (osd.7) 460 : cluster 0 5.19 scrub ok 2024-04-06T03:39:09.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:08.004518+0000 osd.4 (osd.4) 417 : cluster 0 2.1a scrub starts 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:08.007323+0000 osd.4 (osd.4) 418 : cluster 0 2.1a scrub ok 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:08.189775+0000 osd.2 (osd.2) 361 : cluster 0 6.14 scrub starts 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:08.191826+0000 osd.2 (osd.2) 362 : cluster 0 6.14 scrub ok 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[29221]: cluster 2024-04-06T03:39:08.378335+0000 mgr.y (mgr.24368) 669 : cluster 0 pgmap v713: 148 pgs: 148 active+clean; 83 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 10 MiB/s wr, 166 op/s 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.944111+0000 osd.0 (osd.0) 432 : cluster 0 5.15 scrub starts 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.946312+0000 osd.0 (osd.0) 433 : cluster 0 5.15 scrub ok 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.978348+0000 osd.7 (osd.7) 459 : cluster 0 5.19 scrub starts 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:07.979614+0000 osd.7 (osd.7) 460 : cluster 0 5.19 scrub ok 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:08.004518+0000 osd.4 (osd.4) 417 : cluster 0 2.1a scrub starts 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:08.007323+0000 osd.4 (osd.4) 418 : cluster 0 2.1a scrub ok 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:08.189775+0000 osd.2 (osd.2) 361 : cluster 0 6.14 scrub starts 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:08.191826+0000 osd.2 (osd.2) 362 : cluster 0 6.14 scrub ok 2024-04-06T03:39:09.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21467]: cluster 2024-04-06T03:39:08.378335+0000 mgr.y (mgr.24368) 669 : cluster 0 pgmap v713: 148 pgs: 148 active+clean; 83 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 10 MiB/s wr, 166 op/s 2024-04-06T03:39:09.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:09.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:09.548 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 5 16 7927 7911 98.8785 51.125 0.00370068 0.0100958 2024-04-06T03:39:09.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:09.738 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:39:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:39:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:09.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:09.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:09.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:09.919 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:39:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:39:09.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:09.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:10.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:10.050 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[21467]: cluster 2024-04-06T03:39:09.160736+0000 osd.2 (osd.2) 363 : cluster 0 5.b scrub starts 2024-04-06T03:39:10.050 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[21467]: cluster 2024-04-06T03:39:09.163374+0000 osd.2 (osd.2) 364 : cluster 0 5.b scrub ok 2024-04-06T03:39:10.050 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[21467]: cluster 2024-04-06T03:39:09.497042+0000 osd.3 (osd.3) 444 : cluster 0 3.2 scrub starts 2024-04-06T03:39:10.050 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[21467]: cluster 2024-04-06T03:39:09.506450+0000 osd.3 (osd.3) 445 : cluster 0 3.2 scrub ok 2024-04-06T03:39:10.051 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[29221]: cluster 2024-04-06T03:39:09.160736+0000 osd.2 (osd.2) 363 : cluster 0 5.b scrub starts 2024-04-06T03:39:10.051 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[29221]: cluster 2024-04-06T03:39:09.163374+0000 osd.2 (osd.2) 364 : cluster 0 5.b scrub ok 2024-04-06T03:39:10.051 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[29221]: cluster 2024-04-06T03:39:09.497042+0000 osd.3 (osd.3) 444 : cluster 0 3.2 scrub starts 2024-04-06T03:39:10.051 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:10 smithi050 bash[29221]: cluster 2024-04-06T03:39:09.506450+0000 osd.3 (osd.3) 445 : cluster 0 3.2 scrub ok 2024-04-06T03:39:10.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:10.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:10.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:10.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:10 smithi153 bash[23852]: cluster 2024-04-06T03:39:09.160736+0000 osd.2 (osd.2) 363 : cluster 0 5.b scrub starts 2024-04-06T03:39:10.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:10 smithi153 bash[23852]: cluster 2024-04-06T03:39:09.163374+0000 osd.2 (osd.2) 364 : cluster 0 5.b scrub ok 2024-04-06T03:39:10.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:10 smithi153 bash[23852]: cluster 2024-04-06T03:39:09.497042+0000 osd.3 (osd.3) 444 : cluster 0 3.2 scrub starts 2024-04-06T03:39:10.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:10 smithi153 bash[23852]: cluster 2024-04-06T03:39:09.506450+0000 osd.3 (osd.3) 445 : cluster 0 3.2 scrub ok 2024-04-06T03:39:10.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:10.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:10.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:10.526 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:39:10.551 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 6 16 10385 10369 107.938 153.625 0.00436714 0.00925037 2024-04-06T03:39:10.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:10.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:10.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:10.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:10.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:10.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:10.971 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:10.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:11.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:11.102 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:11.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:11.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:09.705143+0000 osd.1 (osd.1) 351 : cluster 0 5.12 scrub starts 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:09.706733+0000 osd.1 (osd.1) 352 : cluster 0 5.12 scrub ok 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: audit 2024-04-06T03:39:09.903566+0000 mgr.y (mgr.24368) 670 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.004739+0000 osd.7 (osd.7) 461 : cluster 0 6.8 scrub starts 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.006338+0000 osd.7 (osd.7) 462 : cluster 0 6.8 scrub ok 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.047156+0000 osd.4 (osd.4) 419 : cluster 0 3.1d scrub starts 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.052203+0000 osd.4 (osd.4) 420 : cluster 0 3.1d scrub ok 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.119701+0000 osd.2 (osd.2) 365 : cluster 0 2.3 scrub starts 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.125587+0000 osd.2 (osd.2) 366 : cluster 0 2.3 scrub ok 2024-04-06T03:39:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:11 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.378983+0000 mgr.y (mgr.24368) 671 : cluster 0 pgmap v714: 148 pgs: 148 active+clean; 381 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 981 B/s rd, 41 MiB/s wr, 649 op/s 2024-04-06T03:39:11.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:11.360 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:11.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:11.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:11.487 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:09.705143+0000 osd.1 (osd.1) 351 : cluster 0 5.12 scrub starts 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:09.706733+0000 osd.1 (osd.1) 352 : cluster 0 5.12 scrub ok 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: audit 2024-04-06T03:39:09.903566+0000 mgr.y (mgr.24368) 670 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.004739+0000 osd.7 (osd.7) 461 : cluster 0 6.8 scrub starts 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.006338+0000 osd.7 (osd.7) 462 : cluster 0 6.8 scrub ok 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.047156+0000 osd.4 (osd.4) 419 : cluster 0 3.1d scrub starts 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.052203+0000 osd.4 (osd.4) 420 : cluster 0 3.1d scrub ok 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.119701+0000 osd.2 (osd.2) 365 : cluster 0 2.3 scrub starts 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.125587+0000 osd.2 (osd.2) 366 : cluster 0 2.3 scrub ok 2024-04-06T03:39:11.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.378983+0000 mgr.y (mgr.24368) 671 : cluster 0 pgmap v714: 148 pgs: 148 active+clean; 381 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 981 B/s rd, 41 MiB/s wr, 649 op/s 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:09.705143+0000 osd.1 (osd.1) 351 : cluster 0 5.12 scrub starts 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:09.706733+0000 osd.1 (osd.1) 352 : cluster 0 5.12 scrub ok 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: audit 2024-04-06T03:39:09.903566+0000 mgr.y (mgr.24368) 670 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.004739+0000 osd.7 (osd.7) 461 : cluster 0 6.8 scrub starts 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.006338+0000 osd.7 (osd.7) 462 : cluster 0 6.8 scrub ok 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.047156+0000 osd.4 (osd.4) 419 : cluster 0 3.1d scrub starts 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.052203+0000 osd.4 (osd.4) 420 : cluster 0 3.1d scrub ok 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.119701+0000 osd.2 (osd.2) 365 : cluster 0 2.3 scrub starts 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.125587+0000 osd.2 (osd.2) 366 : cluster 0 2.3 scrub ok 2024-04-06T03:39:11.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:11 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.378983+0000 mgr.y (mgr.24368) 671 : cluster 0 pgmap v714: 148 pgs: 148 active+clean; 381 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 981 B/s rd, 41 MiB/s wr, 649 op/s 2024-04-06T03:39:11.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 7 16 11389 11373 101.485 62.75 0.0175284 0.00984399 2024-04-06T03:39:11.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:11.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:11.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:11.755 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:11.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:11.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:11.881 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:11.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:11.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:11.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:12.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:12.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:12.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:10.906927+0000 osd.1 (osd.1) 353 : cluster 0 6.19 deep-scrub starts 2024-04-06T03:39:12.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.091673+0000 osd.2 (osd.2) 367 : cluster 0 5.a deep-scrub starts 2024-04-06T03:39:12.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.097303+0000 osd.2 (osd.2) 368 : cluster 0 5.a deep-scrub ok 2024-04-06T03:39:12.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.124942+0000 osd.1 (osd.1) 354 : cluster 0 6.19 deep-scrub ok 2024-04-06T03:39:12.123 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.131526+0000 osd.0 (osd.0) 434 : cluster 0 5.3 scrub starts 2024-04-06T03:39:12.123 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.138138+0000 osd.0 (osd.0) 435 : cluster 0 5.3 scrub ok 2024-04-06T03:39:12.123 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.174353+0000 osd.7 (osd.7) 463 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:12.123 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:12 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.174430+0000 osd.7 (osd.7) 464 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:12.137 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:12.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:12.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:12.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:10.906927+0000 osd.1 (osd.1) 353 : cluster 0 6.19 deep-scrub starts 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.091673+0000 osd.2 (osd.2) 367 : cluster 0 5.a deep-scrub starts 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.097303+0000 osd.2 (osd.2) 368 : cluster 0 5.a deep-scrub ok 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.124942+0000 osd.1 (osd.1) 354 : cluster 0 6.19 deep-scrub ok 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.131526+0000 osd.0 (osd.0) 434 : cluster 0 5.3 scrub starts 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.138138+0000 osd.0 (osd.0) 435 : cluster 0 5.3 scrub ok 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.174353+0000 osd.7 (osd.7) 463 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:12.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.174430+0000 osd.7 (osd.7) 464 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:10.906927+0000 osd.1 (osd.1) 353 : cluster 0 6.19 deep-scrub starts 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.091673+0000 osd.2 (osd.2) 367 : cluster 0 5.a deep-scrub starts 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.097303+0000 osd.2 (osd.2) 368 : cluster 0 5.a deep-scrub ok 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.124942+0000 osd.1 (osd.1) 354 : cluster 0 6.19 deep-scrub ok 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.131526+0000 osd.0 (osd.0) 434 : cluster 0 5.3 scrub starts 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.138138+0000 osd.0 (osd.0) 435 : cluster 0 5.3 scrub ok 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.174353+0000 osd.7 (osd.7) 463 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:12.393 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:12 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.174430+0000 osd.7 (osd.7) 464 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:12.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:12.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:12.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:12.551 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 8 16 13248 13232 103.321 116.188 0.00464162 0.00966893 2024-04-06T03:39:12.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:12.654 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:12.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:12.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:12.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:12.916 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:12.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:13.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:13.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:13.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:13.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:13.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.964782+0000 osd.4 (osd.4) 421 : cluster 0 5.f deep-scrub starts 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:11.966437+0000 osd.4 (osd.4) 422 : cluster 0 5.f deep-scrub ok 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.106364+0000 osd.7 (osd.7) 465 : cluster 0 2.5 scrub starts 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.111691+0000 osd.7 (osd.7) 466 : cluster 0 2.5 scrub ok 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.122357+0000 osd.0 (osd.0) 436 : cluster 0 5.5 scrub starts 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.122918+0000 mon.a (mon.0) 1501 : cluster 0 osdmap e289: 8 total, 8 up, 8 in 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.156569+0000 osd.0 (osd.0) 437 : cluster 0 5.5 scrub ok 2024-04-06T03:39:13.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:13 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.379569+0000 mgr.y (mgr.24368) 672 : cluster 0 pgmap v716: 148 pgs: 148 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-06T03:39:13.302 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:13.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:13.430 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:13.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.964782+0000 osd.4 (osd.4) 421 : cluster 0 5.f deep-scrub starts 2024-04-06T03:39:13.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:11.966437+0000 osd.4 (osd.4) 422 : cluster 0 5.f deep-scrub ok 2024-04-06T03:39:13.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.106364+0000 osd.7 (osd.7) 465 : cluster 0 2.5 scrub starts 2024-04-06T03:39:13.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.111691+0000 osd.7 (osd.7) 466 : cluster 0 2.5 scrub ok 2024-04-06T03:39:13.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.122357+0000 osd.0 (osd.0) 436 : cluster 0 5.5 scrub starts 2024-04-06T03:39:13.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.122918+0000 mon.a (mon.0) 1501 : cluster 0 osdmap e289: 8 total, 8 up, 8 in 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.156569+0000 osd.0 (osd.0) 437 : cluster 0 5.5 scrub ok 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.379569+0000 mgr.y (mgr.24368) 672 : cluster 0 pgmap v716: 148 pgs: 148 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.964782+0000 osd.4 (osd.4) 421 : cluster 0 5.f deep-scrub starts 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:11.966437+0000 osd.4 (osd.4) 422 : cluster 0 5.f deep-scrub ok 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.106364+0000 osd.7 (osd.7) 465 : cluster 0 2.5 scrub starts 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.111691+0000 osd.7 (osd.7) 466 : cluster 0 2.5 scrub ok 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.122357+0000 osd.0 (osd.0) 436 : cluster 0 5.5 scrub starts 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.122918+0000 mon.a (mon.0) 1501 : cluster 0 osdmap e289: 8 total, 8 up, 8 in 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.156569+0000 osd.0 (osd.0) 437 : cluster 0 5.5 scrub ok 2024-04-06T03:39:13.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:13 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.379569+0000 mgr.y (mgr.24368) 672 : cluster 0 pgmap v716: 148 pgs: 148 active+clean; 574 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-04-06T03:39:13.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:13.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 9 16 14985 14969 103.902 108.562 0.0055015 0.0096147 2024-04-06T03:39:13.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:13.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:13.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:13.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:13.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:13.819 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:13.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:13.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:13.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:14.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:14.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:14.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.957856+0000 osd.4 (osd.4) 423 : cluster 0 2.18 scrub starts 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[21467]: cluster 2024-04-06T03:39:12.958841+0000 osd.4 (osd.4) 424 : cluster 0 2.18 scrub ok 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[21467]: cluster 2024-04-06T03:39:13.051109+0000 osd.1 (osd.1) 355 : cluster 0 2.d deep-scrub starts 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[21467]: cluster 2024-04-06T03:39:13.053951+0000 osd.1 (osd.1) 356 : cluster 0 2.d deep-scrub ok 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.957856+0000 osd.4 (osd.4) 423 : cluster 0 2.18 scrub starts 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[29221]: cluster 2024-04-06T03:39:12.958841+0000 osd.4 (osd.4) 424 : cluster 0 2.18 scrub ok 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[29221]: cluster 2024-04-06T03:39:13.051109+0000 osd.1 (osd.1) 355 : cluster 0 2.d deep-scrub starts 2024-04-06T03:39:14.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:14 smithi050 bash[29221]: cluster 2024-04-06T03:39:13.053951+0000 osd.1 (osd.1) 356 : cluster 0 2.d deep-scrub ok 2024-04-06T03:39:14.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:14.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:14.451 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:14.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:14 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.957856+0000 osd.4 (osd.4) 423 : cluster 0 2.18 scrub starts 2024-04-06T03:39:14.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:14 smithi153 bash[23852]: cluster 2024-04-06T03:39:12.958841+0000 osd.4 (osd.4) 424 : cluster 0 2.18 scrub ok 2024-04-06T03:39:14.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:14 smithi153 bash[23852]: cluster 2024-04-06T03:39:13.051109+0000 osd.1 (osd.1) 355 : cluster 0 2.d deep-scrub starts 2024-04-06T03:39:14.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:14 smithi153 bash[23852]: cluster 2024-04-06T03:39:13.053951+0000 osd.1 (osd.1) 356 : cluster 0 2.d deep-scrub ok 2024-04-06T03:39:14.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:14.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 10 16 17186 17170 107.265 137.562 0.014584 0.00931232 2024-04-06T03:39:14.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:14.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:14.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:14.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:14.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:14.853 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:14.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:14.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:14.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:15.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:15.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:15.132 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-06T03:39:15.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:15.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:15.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:15.370 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:15.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:15.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:15 smithi153 bash[23852]: cluster 2024-04-06T03:39:14.102234+0000 osd.0 (osd.0) 438 : cluster 0 2.1f scrub starts 2024-04-06T03:39:15.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:15 smithi153 bash[23852]: cluster 2024-04-06T03:39:14.104788+0000 osd.0 (osd.0) 439 : cluster 0 2.1f scrub ok 2024-04-06T03:39:15.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:15 smithi153 bash[23852]: cluster 2024-04-06T03:39:14.380242+0000 mgr.y (mgr.24368) 673 : cluster 0 pgmap v717: 148 pgs: 148 active+clean; 775 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-06T03:39:15.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:15 smithi153 bash[23852]: cluster 2024-04-06T03:39:14.542550+0000 osd.3 (osd.3) 446 : cluster 0 4.0 scrub starts 2024-04-06T03:39:15.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:15 smithi153 bash[23852]: cluster 2024-04-06T03:39:14.550403+0000 osd.3 (osd.3) 447 : cluster 0 4.0 scrub ok 2024-04-06T03:39:15.485 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:15 smithi153 bash[23852]: audit 2024-04-06T03:39:14.946810+0000 mon.a (mon.0) 1502 : audit 1 from='client.? 172.21.15.50:0/4111251650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:39:15.501 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:15.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 11 16 18453 18437 104.713 79.1875 0.0091641 0.00953893 2024-04-06T03:39:15.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:15.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:15.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[29221]: cluster 2024-04-06T03:39:14.102234+0000 osd.0 (osd.0) 438 : cluster 0 2.1f scrub starts 2024-04-06T03:39:15.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[29221]: cluster 2024-04-06T03:39:14.104788+0000 osd.0 (osd.0) 439 : cluster 0 2.1f scrub ok 2024-04-06T03:39:15.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[29221]: cluster 2024-04-06T03:39:14.380242+0000 mgr.y (mgr.24368) 673 : cluster 0 pgmap v717: 148 pgs: 148 active+clean; 775 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-06T03:39:15.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[29221]: cluster 2024-04-06T03:39:14.542550+0000 osd.3 (osd.3) 446 : cluster 0 4.0 scrub starts 2024-04-06T03:39:15.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[29221]: cluster 2024-04-06T03:39:14.550403+0000 osd.3 (osd.3) 447 : cluster 0 4.0 scrub ok 2024-04-06T03:39:15.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[29221]: audit 2024-04-06T03:39:14.946810+0000 mon.a (mon.0) 1502 : audit 1 from='client.? 172.21.15.50:0/4111251650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:39:15.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[21467]: cluster 2024-04-06T03:39:14.102234+0000 osd.0 (osd.0) 438 : cluster 0 2.1f scrub starts 2024-04-06T03:39:15.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[21467]: cluster 2024-04-06T03:39:14.104788+0000 osd.0 (osd.0) 439 : cluster 0 2.1f scrub ok 2024-04-06T03:39:15.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[21467]: cluster 2024-04-06T03:39:14.380242+0000 mgr.y (mgr.24368) 673 : cluster 0 pgmap v717: 148 pgs: 148 active+clean; 775 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-04-06T03:39:15.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[21467]: cluster 2024-04-06T03:39:14.542550+0000 osd.3 (osd.3) 446 : cluster 0 4.0 scrub starts 2024-04-06T03:39:15.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[21467]: cluster 2024-04-06T03:39:14.550403+0000 osd.3 (osd.3) 447 : cluster 0 4.0 scrub ok 2024-04-06T03:39:15.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:15 smithi050 bash[21467]: audit 2024-04-06T03:39:14.946810+0000 mon.a (mon.0) 1502 : audit 1 from='client.? 172.21.15.50:0/4111251650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:39:15.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:15.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:15.864 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:15.896 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:15.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:16.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:16.032 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:16.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:16.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:16.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:16.292 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:16.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:16.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:16 smithi153 bash[23852]: audit 2024-04-06T03:39:15.129998+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.50:0/4111251650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:39:16.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:16 smithi153 bash[23852]: cluster 2024-04-06T03:39:15.133180+0000 mon.a (mon.0) 1504 : cluster 0 osdmap e290: 8 total, 8 up, 8 in 2024-04-06T03:39:16.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:16 smithi153 bash[23852]: cluster 2024-04-06T03:39:15.515839+0000 osd.3 (osd.3) 448 : cluster 0 3.8 scrub starts 2024-04-06T03:39:16.407 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:16 smithi153 bash[23852]: cluster 2024-04-06T03:39:15.520541+0000 osd.3 (osd.3) 449 : cluster 0 3.8 scrub ok 2024-04-06T03:39:16.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[21467]: audit 2024-04-06T03:39:15.129998+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.50:0/4111251650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[21467]: cluster 2024-04-06T03:39:15.133180+0000 mon.a (mon.0) 1504 : cluster 0 osdmap e290: 8 total, 8 up, 8 in 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[21467]: cluster 2024-04-06T03:39:15.515839+0000 osd.3 (osd.3) 448 : cluster 0 3.8 scrub starts 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[21467]: cluster 2024-04-06T03:39:15.520541+0000 osd.3 (osd.3) 449 : cluster 0 3.8 scrub ok 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[29221]: audit 2024-04-06T03:39:15.129998+0000 mon.a (mon.0) 1503 : audit 1 from='client.? 172.21.15.50:0/4111251650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[29221]: cluster 2024-04-06T03:39:15.133180+0000 mon.a (mon.0) 1504 : cluster 0 osdmap e290: 8 total, 8 up, 8 in 2024-04-06T03:39:16.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[29221]: cluster 2024-04-06T03:39:15.515839+0000 osd.3 (osd.3) 448 : cluster 0 3.8 scrub starts 2024-04-06T03:39:16.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:16 smithi050 bash[29221]: cluster 2024-04-06T03:39:15.520541+0000 osd.3 (osd.3) 449 : cluster 0 3.8 scrub ok 2024-04-06T03:39:16.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:16.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 12 16 20442 20426 106.345 124.312 0.00422551 0.00939578 2024-04-06T03:39:16.554 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:16.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:16.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:16.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:16.814 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:16.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:16.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:17.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:17.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:17.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:17.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:17.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:17.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:17.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:17 smithi153 bash[23852]: cluster 2024-04-06T03:39:15.986479+0000 osd.4 (osd.4) 425 : cluster 0 5.1f scrub starts 2024-04-06T03:39:17.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:17 smithi153 bash[23852]: cluster 2024-04-06T03:39:15.988910+0000 osd.4 (osd.4) 426 : cluster 0 5.1f scrub ok 2024-04-06T03:39:17.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:17 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.380732+0000 mgr.y (mgr.24368) 674 : cluster 0 pgmap v719: 148 pgs: 148 active+clean; 915 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-04-06T03:39:17.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:17 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.487606+0000 osd.3 (osd.3) 450 : cluster 0 6.b scrub starts 2024-04-06T03:39:17.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:17 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.644780+0000 osd.3 (osd.3) 451 : cluster 0 6.b scrub ok 2024-04-06T03:39:17.472 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[21467]: cluster 2024-04-06T03:39:15.986479+0000 osd.4 (osd.4) 425 : cluster 0 5.1f scrub starts 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[21467]: cluster 2024-04-06T03:39:15.988910+0000 osd.4 (osd.4) 426 : cluster 0 5.1f scrub ok 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.380732+0000 mgr.y (mgr.24368) 674 : cluster 0 pgmap v719: 148 pgs: 148 active+clean; 915 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.487606+0000 osd.3 (osd.3) 450 : cluster 0 6.b scrub starts 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.644780+0000 osd.3 (osd.3) 451 : cluster 0 6.b scrub ok 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[29221]: cluster 2024-04-06T03:39:15.986479+0000 osd.4 (osd.4) 425 : cluster 0 5.1f scrub starts 2024-04-06T03:39:17.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[29221]: cluster 2024-04-06T03:39:15.988910+0000 osd.4 (osd.4) 426 : cluster 0 5.1f scrub ok 2024-04-06T03:39:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.380732+0000 mgr.y (mgr.24368) 674 : cluster 0 pgmap v719: 148 pgs: 148 active+clean; 915 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-04-06T03:39:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.487606+0000 osd.3 (osd.3) 450 : cluster 0 6.b scrub starts 2024-04-06T03:39:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:17 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.644780+0000 osd.3 (osd.3) 451 : cluster 0 6.b scrub ok 2024-04-06T03:39:17.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 13 16 22483 22467 107.976 127.562 0.00391362 0.00925351 2024-04-06T03:39:17.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:17.602 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:17.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:17.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:17.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:17.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:17.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:17.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:17.995 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:18.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:18.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:18.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:18.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.951716+0000 osd.4 (osd.4) 427 : cluster 0 2.11 scrub starts 2024-04-06T03:39:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.953504+0000 osd.4 (osd.4) 428 : cluster 0 2.11 scrub ok 2024-04-06T03:39:18.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.988303+0000 osd.7 (osd.7) 467 : cluster 0 5.4 scrub starts 2024-04-06T03:39:18.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[21467]: cluster 2024-04-06T03:39:16.989264+0000 osd.7 (osd.7) 468 : cluster 0 5.4 scrub ok 2024-04-06T03:39:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.951716+0000 osd.4 (osd.4) 427 : cluster 0 2.11 scrub starts 2024-04-06T03:39:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.953504+0000 osd.4 (osd.4) 428 : cluster 0 2.11 scrub ok 2024-04-06T03:39:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.988303+0000 osd.7 (osd.7) 467 : cluster 0 5.4 scrub starts 2024-04-06T03:39:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:18 smithi050 bash[29221]: cluster 2024-04-06T03:39:16.989264+0000 osd.7 (osd.7) 468 : cluster 0 5.4 scrub ok 2024-04-06T03:39:18.397 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:18.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:18 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.951716+0000 osd.4 (osd.4) 427 : cluster 0 2.11 scrub starts 2024-04-06T03:39:18.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:18 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.953504+0000 osd.4 (osd.4) 428 : cluster 0 2.11 scrub ok 2024-04-06T03:39:18.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:18 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.988303+0000 osd.7 (osd.7) 467 : cluster 0 5.4 scrub starts 2024-04-06T03:39:18.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:18 smithi153 bash[23852]: cluster 2024-04-06T03:39:16.989264+0000 osd.7 (osd.7) 468 : cluster 0 5.4 scrub ok 2024-04-06T03:39:18.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:18.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:18.518 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:18.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 14 16 24376 24360 108.713 118.312 0.00293776 0.00915633 2024-04-06T03:39:18.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:18.645 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:18.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:18.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:18.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:18.909 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:18.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:19.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:19.034 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:19.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:19.162 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:19.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:19.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:19.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:19.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:19.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:19.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:19 smithi050 bash[21467]: cluster 2024-04-06T03:39:18.381477+0000 mgr.y (mgr.24368) 675 : cluster 0 pgmap v720: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-04-06T03:39:19.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:39:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:39:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:19.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:19 smithi050 bash[29221]: cluster 2024-04-06T03:39:18.381477+0000 mgr.y (mgr.24368) 675 : cluster 0 pgmap v720: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-04-06T03:39:19.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:19 smithi153 bash[23852]: cluster 2024-04-06T03:39:18.381477+0000 mgr.y (mgr.24368) 675 : cluster 0 pgmap v720: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-04-06T03:39:19.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:19.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 15 15 25317 25302 105.391 58.875 0.00836999 0.00947802 2024-04-06T03:39:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:19.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:19.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:19.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:19.819 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:19.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:19.934 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:39:19.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:19.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:19.987 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:39:20.028 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:39:20.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:20.067 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:39:20.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:20.108 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:39:20.143 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:39:20.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:39:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:39:20.181 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:39:20.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:20.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:20 smithi153 bash[23852]: cluster 2024-04-06T03:39:19.501159+0000 osd.3 (osd.3) 452 : cluster 0 3.5 scrub starts 2024-04-06T03:39:20.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:20 smithi153 bash[23852]: cluster 2024-04-06T03:39:19.509177+0000 osd.3 (osd.3) 453 : cluster 0 3.5 scrub ok 2024-04-06T03:39:20.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:20 smithi153 bash[23852]: audit 2024-04-06T03:39:19.759966+0000 mon.c (mon.2) 89 : audit 0 from='client.? 172.21.15.50:0/1739246505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:39:20.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:20 smithi153 bash[23852]: audit 2024-04-06T03:39:19.880218+0000 mon.a (mon.0) 1505 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:20.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:20.221 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:39:20.226 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:39:20.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:20.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:20.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:20.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:20.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 16 15 26502 26487 103.433 74.0625 0.00556914 0.00965924 2024-04-06T03:39:20.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:20.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:20.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:20.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:20.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[21467]: cluster 2024-04-06T03:39:19.501159+0000 osd.3 (osd.3) 452 : cluster 0 3.5 scrub starts 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[21467]: cluster 2024-04-06T03:39:19.509177+0000 osd.3 (osd.3) 453 : cluster 0 3.5 scrub ok 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[21467]: audit 2024-04-06T03:39:19.759966+0000 mon.c (mon.2) 89 : audit 0 from='client.? 172.21.15.50:0/1739246505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[21467]: audit 2024-04-06T03:39:19.880218+0000 mon.a (mon.0) 1505 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[29221]: cluster 2024-04-06T03:39:19.501159+0000 osd.3 (osd.3) 452 : cluster 0 3.5 scrub starts 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[29221]: cluster 2024-04-06T03:39:19.509177+0000 osd.3 (osd.3) 453 : cluster 0 3.5 scrub ok 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[29221]: audit 2024-04-06T03:39:19.759966+0000 mon.c (mon.2) 89 : audit 0 from='client.? 172.21.15.50:0/1739246505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:39:20.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:20 smithi050 bash[29221]: audit 2024-04-06T03:39:19.880218+0000 mon.a (mon.0) 1505 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:20.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:20.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:20.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:20.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:20.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:21.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:21.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:21.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:21.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:21.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:21.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[21467]: audit 2024-04-06T03:39:19.914400+0000 mgr.y (mgr.24368) 676 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:21.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[21467]: cluster 2024-04-06T03:39:19.989803+0000 osd.4 (osd.4) 429 : cluster 0 6.1a scrub starts 2024-04-06T03:39:21.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[21467]: cluster 2024-04-06T03:39:20.036987+0000 osd.4 (osd.4) 430 : cluster 0 6.1a scrub ok 2024-04-06T03:39:21.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[21467]: cluster 2024-04-06T03:39:20.382283+0000 mgr.y (mgr.24368) 677 : cluster 0 pgmap v721: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 619 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-04-06T03:39:21.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[29221]: audit 2024-04-06T03:39:19.914400+0000 mgr.y (mgr.24368) 676 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:21.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[29221]: cluster 2024-04-06T03:39:19.989803+0000 osd.4 (osd.4) 429 : cluster 0 6.1a scrub starts 2024-04-06T03:39:21.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[29221]: cluster 2024-04-06T03:39:20.036987+0000 osd.4 (osd.4) 430 : cluster 0 6.1a scrub ok 2024-04-06T03:39:21.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:21 smithi050 bash[29221]: cluster 2024-04-06T03:39:20.382283+0000 mgr.y (mgr.24368) 677 : cluster 0 pgmap v721: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 619 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-04-06T03:39:21.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:21.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:21.504 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:21 smithi153 bash[23852]: audit 2024-04-06T03:39:19.914400+0000 mgr.y (mgr.24368) 676 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:21 smithi153 bash[23852]: cluster 2024-04-06T03:39:19.989803+0000 osd.4 (osd.4) 429 : cluster 0 6.1a scrub starts 2024-04-06T03:39:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:21 smithi153 bash[23852]: cluster 2024-04-06T03:39:20.036987+0000 osd.4 (osd.4) 430 : cluster 0 6.1a scrub ok 2024-04-06T03:39:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:21 smithi153 bash[23852]: cluster 2024-04-06T03:39:20.382283+0000 mgr.y (mgr.24368) 677 : cluster 0 pgmap v721: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 619 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-04-06T03:39:21.552 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 17 16 28258 28242 103.8 109.688 0.00557978 0.00962799 2024-04-06T03:39:21.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:21.628 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:21.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:21.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:21.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:21.887 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:21.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:21.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:22.022 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:22.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:22.149 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:22.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[21467]: cluster 2024-04-06T03:39:20.961719+0000 osd.4 (osd.4) 431 : cluster 0 3.a scrub starts 2024-04-06T03:39:22.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[21467]: cluster 2024-04-06T03:39:20.965514+0000 osd.4 (osd.4) 432 : cluster 0 3.a scrub ok 2024-04-06T03:39:22.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[21467]: cluster 2024-04-06T03:39:21.262200+0000 osd.0 (osd.0) 440 : cluster 0 6.a scrub starts 2024-04-06T03:39:22.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[21467]: cluster 2024-04-06T03:39:21.333132+0000 osd.0 (osd.0) 441 : cluster 0 6.a scrub ok 2024-04-06T03:39:22.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[29221]: cluster 2024-04-06T03:39:20.961719+0000 osd.4 (osd.4) 431 : cluster 0 3.a scrub starts 2024-04-06T03:39:22.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[29221]: cluster 2024-04-06T03:39:20.965514+0000 osd.4 (osd.4) 432 : cluster 0 3.a scrub ok 2024-04-06T03:39:22.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[29221]: cluster 2024-04-06T03:39:21.262200+0000 osd.0 (osd.0) 440 : cluster 0 6.a scrub starts 2024-04-06T03:39:22.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:22 smithi050 bash[29221]: cluster 2024-04-06T03:39:21.333132+0000 osd.0 (osd.0) 441 : cluster 0 6.a scrub ok 2024-04-06T03:39:22.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:22.274 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:22.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:22.384 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:22 smithi153 bash[23852]: cluster 2024-04-06T03:39:20.961719+0000 osd.4 (osd.4) 431 : cluster 0 3.a scrub starts 2024-04-06T03:39:22.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:22 smithi153 bash[23852]: cluster 2024-04-06T03:39:20.965514+0000 osd.4 (osd.4) 432 : cluster 0 3.a scrub ok 2024-04-06T03:39:22.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:22 smithi153 bash[23852]: cluster 2024-04-06T03:39:21.262200+0000 osd.0 (osd.0) 440 : cluster 0 6.a scrub starts 2024-04-06T03:39:22.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:22 smithi153 bash[23852]: cluster 2024-04-06T03:39:21.333132+0000 osd.0 (osd.0) 441 : cluster 0 6.a scrub ok 2024-04-06T03:39:22.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:22.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:22.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:22.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 18 15 29007 28992 100.638 46.875 0.00907392 0.00992966 2024-04-06T03:39:22.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:22.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:22.676 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-06T03:39:22.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:39:22.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:22.781 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:22.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:22.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:23.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:23.034 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:23.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:23.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:23.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:23.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:23.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:23 smithi153 bash[23852]: cluster 2024-04-06T03:39:21.972918+0000 osd.4 (osd.4) 433 : cluster 0 6.e scrub starts 2024-04-06T03:39:23.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:23 smithi153 bash[23852]: cluster 2024-04-06T03:39:22.029237+0000 osd.4 (osd.4) 434 : cluster 0 6.e scrub ok 2024-04-06T03:39:23.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:23 smithi153 bash[23852]: cluster 2024-04-06T03:39:22.227693+0000 osd.0 (osd.0) 442 : cluster 0 6.10 scrub starts 2024-04-06T03:39:23.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:23 smithi153 bash[23852]: cluster 2024-04-06T03:39:22.336341+0000 osd.0 (osd.0) 443 : cluster 0 6.10 scrub ok 2024-04-06T03:39:23.428 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:23 smithi153 bash[23852]: cluster 2024-04-06T03:39:22.383225+0000 mgr.y (mgr.24368) 678 : cluster 0 pgmap v722: 148 pgs: 148 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:39:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:23.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[21467]: cluster 2024-04-06T03:39:21.972918+0000 osd.4 (osd.4) 433 : cluster 0 6.e scrub starts 2024-04-06T03:39:23.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[21467]: cluster 2024-04-06T03:39:22.029237+0000 osd.4 (osd.4) 434 : cluster 0 6.e scrub ok 2024-04-06T03:39:23.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[21467]: cluster 2024-04-06T03:39:22.227693+0000 osd.0 (osd.0) 442 : cluster 0 6.10 scrub starts 2024-04-06T03:39:23.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[21467]: cluster 2024-04-06T03:39:22.336341+0000 osd.0 (osd.0) 443 : cluster 0 6.10 scrub ok 2024-04-06T03:39:23.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[21467]: cluster 2024-04-06T03:39:22.383225+0000 mgr.y (mgr.24368) 678 : cluster 0 pgmap v722: 148 pgs: 148 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:39:23.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[29221]: cluster 2024-04-06T03:39:21.972918+0000 osd.4 (osd.4) 433 : cluster 0 6.e scrub starts 2024-04-06T03:39:23.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[29221]: cluster 2024-04-06T03:39:22.029237+0000 osd.4 (osd.4) 434 : cluster 0 6.e scrub ok 2024-04-06T03:39:23.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[29221]: cluster 2024-04-06T03:39:22.227693+0000 osd.0 (osd.0) 442 : cluster 0 6.10 scrub starts 2024-04-06T03:39:23.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[29221]: cluster 2024-04-06T03:39:22.336341+0000 osd.0 (osd.0) 443 : cluster 0 6.10 scrub ok 2024-04-06T03:39:23.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:23 smithi050 bash[29221]: cluster 2024-04-06T03:39:22.383225+0000 mgr.y (mgr.24368) 678 : cluster 0 pgmap v722: 148 pgs: 148 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-04-06T03:39:23.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:23.551 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:23.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 19 16 30099 30083 98.9297 68.1875 0.00276245 0.0100879 2024-04-06T03:39:23.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:23.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:23.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:23.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:23.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:23.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:24.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:24.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:24.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:24.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:24.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:24.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:24.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:24.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:24.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:39:24.557046+0000 min lat: 0.0011709 max lat: 0.652534 avg lat: 0.0101193 2024-04-06T03:39:24.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:39:24.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 20 15 31628 31613 98.7641 95.625 0.0059453 0.0101193 2024-04-06T03:39:24.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:24.607 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:24.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:24.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:24.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:24.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:24.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:25.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:25.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:25.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:25.252 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:25.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:25.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:25.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:25.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:25.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:25 smithi153 bash[23852]: cluster 2024-04-06T03:39:24.383918+0000 mgr.y (mgr.24368) 679 : cluster 0 pgmap v723: 148 pgs: 148 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-06T03:39:25.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:25 smithi153 bash[23852]: cluster 2024-04-06T03:39:24.581203+0000 osd.3 (osd.3) 454 : cluster 0 6.6 deep-scrub starts 2024-04-06T03:39:25.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:25 smithi153 bash[23852]: cluster 2024-04-06T03:39:24.595329+0000 osd.3 (osd.3) 455 : cluster 0 6.6 deep-scrub ok 2024-04-06T03:39:25.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 21 16 32846 32830 97.6827 76.0625 0.00189691 0.0100955 2024-04-06T03:39:25.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:25.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:25.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:25 smithi050 bash[21467]: cluster 2024-04-06T03:39:24.383918+0000 mgr.y (mgr.24368) 679 : cluster 0 pgmap v723: 148 pgs: 148 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-06T03:39:25.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:25 smithi050 bash[21467]: cluster 2024-04-06T03:39:24.581203+0000 osd.3 (osd.3) 454 : cluster 0 6.6 deep-scrub starts 2024-04-06T03:39:25.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:25 smithi050 bash[21467]: cluster 2024-04-06T03:39:24.595329+0000 osd.3 (osd.3) 455 : cluster 0 6.6 deep-scrub ok 2024-04-06T03:39:25.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:25 smithi050 bash[29221]: cluster 2024-04-06T03:39:24.383918+0000 mgr.y (mgr.24368) 679 : cluster 0 pgmap v723: 148 pgs: 148 active+clean; 1.7 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-04-06T03:39:25.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:25 smithi050 bash[29221]: cluster 2024-04-06T03:39:24.581203+0000 osd.3 (osd.3) 454 : cluster 0 6.6 deep-scrub starts 2024-04-06T03:39:25.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:25 smithi050 bash[29221]: cluster 2024-04-06T03:39:24.595329+0000 osd.3 (osd.3) 455 : cluster 0 6.6 deep-scrub ok 2024-04-06T03:39:25.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:25.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:25.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:25.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:26.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:26.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:26.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:26.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:26.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:26.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:26.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:26.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:26.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:26.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:26.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:26 smithi153 bash[23852]: cluster 2024-04-06T03:39:25.011133+0000 osd.4 (osd.4) 435 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:39:26.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:26 smithi153 bash[23852]: cluster 2024-04-06T03:39:25.012438+0000 osd.4 (osd.4) 436 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:39:26.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:26 smithi153 bash[23852]: cluster 2024-04-06T03:39:25.591105+0000 osd.3 (osd.3) 456 : cluster 0 3.c deep-scrub starts 2024-04-06T03:39:26.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:26 smithi153 bash[23852]: cluster 2024-04-06T03:39:25.598836+0000 osd.3 (osd.3) 457 : cluster 0 3.c deep-scrub ok 2024-04-06T03:39:26.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:26 smithi153 bash[23852]: cluster 2024-04-06T03:39:25.665265+0000 osd.6 (osd.6) 99 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:26.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:26 smithi153 bash[23852]: cluster 2024-04-06T03:39:25.665338+0000 osd.6 (osd.6) 100 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:26.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 22 16 34456 34440 97.8161 100.625 0.0118261 0.0102173 2024-04-06T03:39:26.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:26.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:26.696 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:26.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:26.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[21467]: cluster 2024-04-06T03:39:25.011133+0000 osd.4 (osd.4) 435 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[21467]: cluster 2024-04-06T03:39:25.012438+0000 osd.4 (osd.4) 436 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[21467]: cluster 2024-04-06T03:39:25.591105+0000 osd.3 (osd.3) 456 : cluster 0 3.c deep-scrub starts 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[21467]: cluster 2024-04-06T03:39:25.598836+0000 osd.3 (osd.3) 457 : cluster 0 3.c deep-scrub ok 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[21467]: cluster 2024-04-06T03:39:25.665265+0000 osd.6 (osd.6) 99 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[21467]: cluster 2024-04-06T03:39:25.665338+0000 osd.6 (osd.6) 100 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[29221]: cluster 2024-04-06T03:39:25.011133+0000 osd.4 (osd.4) 435 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[29221]: cluster 2024-04-06T03:39:25.012438+0000 osd.4 (osd.4) 436 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[29221]: cluster 2024-04-06T03:39:25.591105+0000 osd.3 (osd.3) 456 : cluster 0 3.c deep-scrub starts 2024-04-06T03:39:26.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[29221]: cluster 2024-04-06T03:39:25.598836+0000 osd.3 (osd.3) 457 : cluster 0 3.c deep-scrub ok 2024-04-06T03:39:26.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[29221]: cluster 2024-04-06T03:39:25.665265+0000 osd.6 (osd.6) 99 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:26.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:26 smithi050 bash[29221]: cluster 2024-04-06T03:39:25.665338+0000 osd.6 (osd.6) 100 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:26.926 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:26.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:26.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:27.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:27.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:27.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:27.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:27.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:27.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:27.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.210972+0000 osd.0 (osd.0) 444 : cluster 0 5.1b scrub starts 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.216713+0000 osd.0 (osd.0) 445 : cluster 0 5.1b scrub ok 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.384528+0000 mgr.y (mgr.24368) 680 : cluster 0 pgmap v724: 148 pgs: 148 active+clean; 1.8 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 909 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.452912+0000 mon.a (mon.0) 1506 : cluster 0 osdmap e291: 8 total, 8 up, 8 in 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.613574+0000 osd.3 (osd.3) 458 : cluster 0 5.14 scrub starts 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.622962+0000 osd.3 (osd.3) 459 : cluster 0 5.14 scrub ok 2024-04-06T03:39:27.463 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.923770+0000 osd.7 (osd.7) 469 : cluster 0 3.f deep-scrub starts 2024-04-06T03:39:27.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:27 smithi153 bash[23852]: cluster 2024-04-06T03:39:26.934747+0000 osd.7 (osd.7) 470 : cluster 0 3.f deep-scrub ok 2024-04-06T03:39:27.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:27.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 23 16 36016 36000 97.8019 97.5 0.00737988 0.0102132 2024-04-06T03:39:27.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:27.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:27.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:27.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.210972+0000 osd.0 (osd.0) 444 : cluster 0 5.1b scrub starts 2024-04-06T03:39:27.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.216713+0000 osd.0 (osd.0) 445 : cluster 0 5.1b scrub ok 2024-04-06T03:39:27.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.384528+0000 mgr.y (mgr.24368) 680 : cluster 0 pgmap v724: 148 pgs: 148 active+clean; 1.8 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 909 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-06T03:39:27.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.452912+0000 mon.a (mon.0) 1506 : cluster 0 osdmap e291: 8 total, 8 up, 8 in 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.613574+0000 osd.3 (osd.3) 458 : cluster 0 5.14 scrub starts 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.622962+0000 osd.3 (osd.3) 459 : cluster 0 5.14 scrub ok 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.923770+0000 osd.7 (osd.7) 469 : cluster 0 3.f deep-scrub starts 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[21467]: cluster 2024-04-06T03:39:26.934747+0000 osd.7 (osd.7) 470 : cluster 0 3.f deep-scrub ok 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.210972+0000 osd.0 (osd.0) 444 : cluster 0 5.1b scrub starts 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.216713+0000 osd.0 (osd.0) 445 : cluster 0 5.1b scrub ok 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.384528+0000 mgr.y (mgr.24368) 680 : cluster 0 pgmap v724: 148 pgs: 148 active+clean; 1.8 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 909 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.452912+0000 mon.a (mon.0) 1506 : cluster 0 osdmap e291: 8 total, 8 up, 8 in 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.613574+0000 osd.3 (osd.3) 458 : cluster 0 5.14 scrub starts 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.622962+0000 osd.3 (osd.3) 459 : cluster 0 5.14 scrub ok 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.923770+0000 osd.7 (osd.7) 469 : cluster 0 3.f deep-scrub starts 2024-04-06T03:39:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:27 smithi050 bash[29221]: cluster 2024-04-06T03:39:26.934747+0000 osd.7 (osd.7) 470 : cluster 0 3.f deep-scrub ok 2024-04-06T03:39:27.740 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:27.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:27.868 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:27.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:27.997 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:27.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:28.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:28.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:28.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:28.266 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:28.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:28.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:28.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:28.501 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:28 smithi153 bash[23852]: cluster 2024-04-06T03:39:27.904581+0000 osd.7 (osd.7) 471 : cluster 0 2.b deep-scrub starts 2024-04-06T03:39:28.501 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:28 smithi153 bash[23852]: cluster 2024-04-06T03:39:27.906527+0000 osd.7 (osd.7) 472 : cluster 0 2.b deep-scrub ok 2024-04-06T03:39:28.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:28.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 24 16 37437 37421 97.4269 88.8125 0.00193219 0.0101863 2024-04-06T03:39:28.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:28.644 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:28.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:39:28.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:28 smithi050 bash[21467]: cluster 2024-04-06T03:39:27.904581+0000 osd.7 (osd.7) 471 : cluster 0 2.b deep-scrub starts 2024-04-06T03:39:28.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:28 smithi050 bash[21467]: cluster 2024-04-06T03:39:27.906527+0000 osd.7 (osd.7) 472 : cluster 0 2.b deep-scrub ok 2024-04-06T03:39:28.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:28 smithi050 bash[29221]: cluster 2024-04-06T03:39:27.904581+0000 osd.7 (osd.7) 471 : cluster 0 2.b deep-scrub starts 2024-04-06T03:39:28.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:28 smithi050 bash[29221]: cluster 2024-04-06T03:39:27.906527+0000 osd.7 (osd.7) 472 : cluster 0 2.b deep-scrub ok 2024-04-06T03:39:28.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:28.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:28.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:28.896 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:28.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:28.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:29.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:29.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:29.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:29.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:29.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:29.396 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:39:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:29.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:29.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:29 smithi153 bash[23852]: cluster 2024-04-06T03:39:28.385146+0000 mgr.y (mgr.24368) 681 : cluster 0 pgmap v726: 148 pgs: 148 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-04-06T03:39:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:29 smithi153 bash[23852]: cluster 2024-04-06T03:39:28.612512+0000 osd.3 (osd.3) 460 : cluster 0 5.13 scrub starts 2024-04-06T03:39:29.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:29 smithi153 bash[23852]: cluster 2024-04-06T03:39:28.618752+0000 osd.3 (osd.3) 461 : cluster 0 5.13 scrub ok 2024-04-06T03:39:29.553 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 25 16 38761 38745 96.8396 82.75 0.00567206 0.0103207 2024-04-06T03:39:29.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:29.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[21467]: cluster 2024-04-06T03:39:28.385146+0000 mgr.y (mgr.24368) 681 : cluster 0 pgmap v726: 148 pgs: 148 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-04-06T03:39:29.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[21467]: cluster 2024-04-06T03:39:28.612512+0000 osd.3 (osd.3) 460 : cluster 0 5.13 scrub starts 2024-04-06T03:39:29.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[21467]: cluster 2024-04-06T03:39:28.618752+0000 osd.3 (osd.3) 461 : cluster 0 5.13 scrub ok 2024-04-06T03:39:29.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[29221]: cluster 2024-04-06T03:39:28.385146+0000 mgr.y (mgr.24368) 681 : cluster 0 pgmap v726: 148 pgs: 148 active+clean; 2.0 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-04-06T03:39:29.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[29221]: cluster 2024-04-06T03:39:28.612512+0000 osd.3 (osd.3) 460 : cluster 0 5.13 scrub starts 2024-04-06T03:39:29.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:29 smithi050 bash[29221]: cluster 2024-04-06T03:39:28.618752+0000 osd.3 (osd.3) 461 : cluster 0 5.13 scrub ok 2024-04-06T03:39:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:29.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.688 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:29.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:29.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:29.817 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:29.912 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:39:29.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:29.944 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:29.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:30.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:30.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:30.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:30.198 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:39:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:39:30.219 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:30.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:30.346 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:30.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:30.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:30.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 26 15 40215 40200 96.6123 90.9375 0.00975857 0.0103454 2024-04-06T03:39:30.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:30.612 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:30.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:30.725 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:30 smithi153 bash[23852]: cluster 2024-04-06T03:39:29.636654+0000 osd.3 (osd.3) 462 : cluster 0 6.c scrub starts 2024-04-06T03:39:30.725 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:30 smithi153 bash[23852]: cluster 2024-04-06T03:39:29.641858+0000 osd.3 (osd.3) 463 : cluster 0 6.c scrub ok 2024-04-06T03:39:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:30.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:30 smithi050 bash[21467]: cluster 2024-04-06T03:39:29.636654+0000 osd.3 (osd.3) 462 : cluster 0 6.c scrub starts 2024-04-06T03:39:30.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:30 smithi050 bash[21467]: cluster 2024-04-06T03:39:29.641858+0000 osd.3 (osd.3) 463 : cluster 0 6.c scrub ok 2024-04-06T03:39:30.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:30 smithi050 bash[29221]: cluster 2024-04-06T03:39:29.636654+0000 osd.3 (osd.3) 462 : cluster 0 6.c scrub starts 2024-04-06T03:39:30.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:30 smithi050 bash[29221]: cluster 2024-04-06T03:39:29.641858+0000 osd.3 (osd.3) 463 : cluster 0 6.c scrub ok 2024-04-06T03:39:30.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:30.857 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:30.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:30.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:30.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:31.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:31.122 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:31.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:31.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:31.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:31.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:31.394 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:31.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[21467]: audit 2024-04-06T03:39:29.922581+0000 mgr.y (mgr.24368) 682 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[21467]: cluster 2024-04-06T03:39:30.233775+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub starts 2024-04-06T03:39:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[21467]: cluster 2024-04-06T03:39:30.292084+0000 osd.0 (osd.0) 447 : cluster 0 6.18 scrub ok 2024-04-06T03:39:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[21467]: cluster 2024-04-06T03:39:30.385990+0000 mgr.y (mgr.24368) 683 : cluster 0 pgmap v727: 148 pgs: 148 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-06T03:39:31.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:31.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:31.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 27 16 41489 41473 95.9806 79.5625 0.0024063 0.0104124 2024-04-06T03:39:31.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:31.656 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:31.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[29221]: audit 2024-04-06T03:39:29.922581+0000 mgr.y (mgr.24368) 682 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:31.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[29221]: cluster 2024-04-06T03:39:30.233775+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub starts 2024-04-06T03:39:31.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[29221]: cluster 2024-04-06T03:39:30.292084+0000 osd.0 (osd.0) 447 : cluster 0 6.18 scrub ok 2024-04-06T03:39:31.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:31 smithi050 bash[29221]: cluster 2024-04-06T03:39:30.385990+0000 mgr.y (mgr.24368) 683 : cluster 0 pgmap v727: 148 pgs: 148 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-06T03:39:31.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:31.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:31 smithi153 bash[23852]: audit 2024-04-06T03:39:29.922581+0000 mgr.y (mgr.24368) 682 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:31 smithi153 bash[23852]: cluster 2024-04-06T03:39:30.233775+0000 osd.0 (osd.0) 446 : cluster 0 6.18 scrub starts 2024-04-06T03:39:31.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:31 smithi153 bash[23852]: cluster 2024-04-06T03:39:30.292084+0000 osd.0 (osd.0) 447 : cluster 0 6.18 scrub ok 2024-04-06T03:39:31.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:31 smithi153 bash[23852]: cluster 2024-04-06T03:39:30.385990+0000 mgr.y (mgr.24368) 683 : cluster 0 pgmap v727: 148 pgs: 148 active+clean; 2.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-04-06T03:39:31.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:31.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:31.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:32.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:32.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:32.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:32.173 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:32.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:32.300 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:32.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:32.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:32.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:32.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 28 16 42686 42670 95.2244 74.8125 0.00537288 0.0104956 2024-04-06T03:39:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:32.673 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:32.709 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:32.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[21467]: cluster 2024-04-06T03:39:31.815689+0000 osd.3 (osd.3) 464 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:39:32.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[21467]: cluster 2024-04-06T03:39:31.921494+0000 osd.3 (osd.3) 465 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:39:32.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[21467]: cluster 2024-04-06T03:39:31.972607+0000 osd.7 (osd.7) 473 : cluster 0 6.12 scrub starts 2024-04-06T03:39:32.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[21467]: cluster 2024-04-06T03:39:31.990683+0000 osd.7 (osd.7) 474 : cluster 0 6.12 scrub ok 2024-04-06T03:39:32.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[29221]: cluster 2024-04-06T03:39:31.815689+0000 osd.3 (osd.3) 464 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:39:32.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[29221]: cluster 2024-04-06T03:39:31.921494+0000 osd.3 (osd.3) 465 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:39:32.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[29221]: cluster 2024-04-06T03:39:31.972607+0000 osd.7 (osd.7) 473 : cluster 0 6.12 scrub starts 2024-04-06T03:39:32.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:32 smithi050 bash[29221]: cluster 2024-04-06T03:39:31.990683+0000 osd.7 (osd.7) 474 : cluster 0 6.12 scrub ok 2024-04-06T03:39:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:32 smithi153 bash[23852]: cluster 2024-04-06T03:39:31.815689+0000 osd.3 (osd.3) 464 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:39:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:32 smithi153 bash[23852]: cluster 2024-04-06T03:39:31.921494+0000 osd.3 (osd.3) 465 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:39:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:32 smithi153 bash[23852]: cluster 2024-04-06T03:39:31.972607+0000 osd.7 (osd.7) 473 : cluster 0 6.12 scrub starts 2024-04-06T03:39:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:32 smithi153 bash[23852]: cluster 2024-04-06T03:39:31.990683+0000 osd.7 (osd.7) 474 : cluster 0 6.12 scrub ok 2024-04-06T03:39:32.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:32.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:32.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:32.956 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:39:32.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:32.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:33.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:33.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:33.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:33.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:33.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:33.359 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:33.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:33.491 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:33.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:33.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 29 16 43779 43763 94.2963 68.3125 0.00219219 0.0105786 2024-04-06T03:39:33.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:33.626 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:33.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:33.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:33 smithi153 bash[23852]: cluster 2024-04-06T03:39:32.309870+0000 osd.0 (osd.0) 448 : cluster 0 5.2 scrub starts 2024-04-06T03:39:33.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:33 smithi153 bash[23852]: cluster 2024-04-06T03:39:32.312894+0000 osd.0 (osd.0) 449 : cluster 0 5.2 scrub ok 2024-04-06T03:39:33.739 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:33 smithi153 bash[23852]: cluster 2024-04-06T03:39:32.386631+0000 mgr.y (mgr.24368) 684 : cluster 0 pgmap v728: 148 pgs: 148 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-06T03:39:33.754 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:33.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:33.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:33.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:33.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:33 smithi050 bash[21467]: cluster 2024-04-06T03:39:32.309870+0000 osd.0 (osd.0) 448 : cluster 0 5.2 scrub starts 2024-04-06T03:39:33.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:33 smithi050 bash[21467]: cluster 2024-04-06T03:39:32.312894+0000 osd.0 (osd.0) 449 : cluster 0 5.2 scrub ok 2024-04-06T03:39:33.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:33 smithi050 bash[21467]: cluster 2024-04-06T03:39:32.386631+0000 mgr.y (mgr.24368) 684 : cluster 0 pgmap v728: 148 pgs: 148 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-06T03:39:33.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:33 smithi050 bash[29221]: cluster 2024-04-06T03:39:32.309870+0000 osd.0 (osd.0) 448 : cluster 0 5.2 scrub starts 2024-04-06T03:39:33.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:33 smithi050 bash[29221]: cluster 2024-04-06T03:39:32.312894+0000 osd.0 (osd.0) 449 : cluster 0 5.2 scrub ok 2024-04-06T03:39:33.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:33 smithi050 bash[29221]: cluster 2024-04-06T03:39:32.386631+0000 mgr.y (mgr.24368) 684 : cluster 0 pgmap v728: 148 pgs: 148 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-06T03:39:33.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:34.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:34.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:34.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:34.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:34.289 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:34.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.420 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:34.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.550 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:34.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 30 16 45160 45144 94.03 86.3125 0.01357 0.0106295 2024-04-06T03:39:34.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:34.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:34.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:34.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:34.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:34 smithi153 bash[23852]: cluster 2024-04-06T03:39:33.008381+0000 osd.4 (osd.4) 437 : cluster 0 6.2 scrub starts 2024-04-06T03:39:34.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:34 smithi153 bash[23852]: cluster 2024-04-06T03:39:33.010621+0000 osd.4 (osd.4) 438 : cluster 0 6.2 scrub ok 2024-04-06T03:39:34.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:34 smithi153 bash[23852]: cluster 2024-04-06T03:39:34.387276+0000 mgr.y (mgr.24368) 685 : cluster 0 pgmap v729: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-06T03:39:34.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:34.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:34.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:34 smithi050 bash[21467]: cluster 2024-04-06T03:39:33.008381+0000 osd.4 (osd.4) 437 : cluster 0 6.2 scrub starts 2024-04-06T03:39:34.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:34 smithi050 bash[21467]: cluster 2024-04-06T03:39:33.010621+0000 osd.4 (osd.4) 438 : cluster 0 6.2 scrub ok 2024-04-06T03:39:34.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:34 smithi050 bash[21467]: cluster 2024-04-06T03:39:34.387276+0000 mgr.y (mgr.24368) 685 : cluster 0 pgmap v729: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-06T03:39:34.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:34 smithi050 bash[29221]: cluster 2024-04-06T03:39:33.008381+0000 osd.4 (osd.4) 437 : cluster 0 6.2 scrub starts 2024-04-06T03:39:34.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:34 smithi050 bash[29221]: cluster 2024-04-06T03:39:33.010621+0000 osd.4 (osd.4) 438 : cluster 0 6.2 scrub ok 2024-04-06T03:39:34.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:34 smithi050 bash[29221]: cluster 2024-04-06T03:39:34.387276+0000 mgr.y (mgr.24368) 685 : cluster 0 pgmap v729: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-06T03:39:35.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:35.055 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:35.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:35.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:35.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:35.295 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:39:35.295 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:39:35.316 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:35.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:35.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:35.548 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:35.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 31 16 45924 45908 92.537 47.75 0.00648118 0.0107946 2024-04-06T03:39:35.574 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:35.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:35.711 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:35.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:35.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:35.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:35.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[29221]: cluster 2024-04-06T03:39:34.000415+0000 osd.4 (osd.4) 439 : cluster 0 5.1c scrub starts 2024-04-06T03:39:35.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[29221]: cluster 2024-04-06T03:39:34.004838+0000 osd.4 (osd.4) 440 : cluster 0 5.1c scrub ok 2024-04-06T03:39:35.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[29221]: audit 2024-04-06T03:39:34.906151+0000 mon.a (mon.0) 1507 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:35.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[29221]: cluster 2024-04-06T03:39:35.031601+0000 osd.7 (osd.7) 475 : cluster 0 6.5 scrub starts 2024-04-06T03:39:35.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[29221]: cluster 2024-04-06T03:39:35.082176+0000 osd.7 (osd.7) 476 : cluster 0 6.5 scrub ok 2024-04-06T03:39:35.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[21467]: cluster 2024-04-06T03:39:34.000415+0000 osd.4 (osd.4) 439 : cluster 0 5.1c scrub starts 2024-04-06T03:39:35.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[21467]: cluster 2024-04-06T03:39:34.004838+0000 osd.4 (osd.4) 440 : cluster 0 5.1c scrub ok 2024-04-06T03:39:35.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[21467]: audit 2024-04-06T03:39:34.906151+0000 mon.a (mon.0) 1507 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:35.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[21467]: cluster 2024-04-06T03:39:35.031601+0000 osd.7 (osd.7) 475 : cluster 0 6.5 scrub starts 2024-04-06T03:39:35.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:35 smithi050 bash[21467]: cluster 2024-04-06T03:39:35.082176+0000 osd.7 (osd.7) 476 : cluster 0 6.5 scrub ok 2024-04-06T03:39:35.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:35.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:36.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:36.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:36.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:36.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:35 smithi153 bash[23852]: cluster 2024-04-06T03:39:34.000415+0000 osd.4 (osd.4) 439 : cluster 0 5.1c scrub starts 2024-04-06T03:39:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:35 smithi153 bash[23852]: cluster 2024-04-06T03:39:34.004838+0000 osd.4 (osd.4) 440 : cluster 0 5.1c scrub ok 2024-04-06T03:39:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:35 smithi153 bash[23852]: audit 2024-04-06T03:39:34.906151+0000 mon.a (mon.0) 1507 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:35 smithi153 bash[23852]: cluster 2024-04-06T03:39:35.031601+0000 osd.7 (osd.7) 475 : cluster 0 6.5 scrub starts 2024-04-06T03:39:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:35 smithi153 bash[23852]: cluster 2024-04-06T03:39:35.082176+0000 osd.7 (osd.7) 476 : cluster 0 6.5 scrub ok 2024-04-06T03:39:36.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:36.371 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:36.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:36.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:36.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:36.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 32 16 46912 46896 91.5747 61.75 0.028715 0.0109075 2024-04-06T03:39:36.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:36.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:36.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:36.781 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:36.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:36.903 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:39:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:36.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:37.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:37.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:37.140 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:37.165 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:37.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:36 smithi153 bash[23852]: cluster 2024-04-06T03:39:35.819959+0000 osd.3 (osd.3) 466 : cluster 0 6.1f scrub starts 2024-04-06T03:39:37.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:36 smithi153 bash[23852]: cluster 2024-04-06T03:39:36.012604+0000 osd.7 (osd.7) 477 : cluster 0 5.10 scrub starts 2024-04-06T03:39:37.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:36 smithi153 bash[23852]: cluster 2024-04-06T03:39:36.013563+0000 osd.7 (osd.7) 478 : cluster 0 5.10 scrub ok 2024-04-06T03:39:37.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:36 smithi153 bash[23852]: cluster 2024-04-06T03:39:36.388002+0000 mgr.y (mgr.24368) 686 : cluster 0 pgmap v730: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-06T03:39:37.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:36 smithi153 bash[23852]: audit 2024-04-06T03:39:36.673317+0000 mon.c (mon.2) 90 : audit 1 from='client.? 172.21.15.50:0/894932475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:39:37.167 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:36 smithi153 bash[23852]: audit 2024-04-06T03:39:36.673898+0000 mon.a (mon.0) 1508 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:39:37.247 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:35.819959+0000 osd.3 (osd.3) 466 : cluster 0 6.1f scrub starts 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:36.012604+0000 osd.7 (osd.7) 477 : cluster 0 5.10 scrub starts 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:36.013563+0000 osd.7 (osd.7) 478 : cluster 0 5.10 scrub ok 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:36.388002+0000 mgr.y (mgr.24368) 686 : cluster 0 pgmap v730: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: audit 2024-04-06T03:39:36.673317+0000 mon.c (mon.2) 90 : audit 1 from='client.? 172.21.15.50:0/894932475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: audit 2024-04-06T03:39:36.673898+0000 mon.a (mon.0) 1508 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:35.819959+0000 osd.3 (osd.3) 466 : cluster 0 6.1f scrub starts 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:36.012604+0000 osd.7 (osd.7) 477 : cluster 0 5.10 scrub starts 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:36.013563+0000 osd.7 (osd.7) 478 : cluster 0 5.10 scrub ok 2024-04-06T03:39:37.248 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:36.388002+0000 mgr.y (mgr.24368) 686 : cluster 0 pgmap v730: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-04-06T03:39:37.249 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: audit 2024-04-06T03:39:36.673317+0000 mon.c (mon.2) 90 : audit 1 from='client.? 172.21.15.50:0/894932475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:39:37.249 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: audit 2024-04-06T03:39:36.673898+0000 mon.a (mon.0) 1508 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:39:37.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:37.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:37.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:37.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:37.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:37.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:37.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 33 16 48153 48137 91.1499 77.5625 0.00493498 0.0109488 2024-04-06T03:39:37.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:37.672 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:37.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:37.801 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:37.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:37.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:37.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:38.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:37 smithi153 bash[23852]: cluster 2024-04-06T03:39:35.942071+0000 osd.4 (osd.4) 441 : cluster 0 5.1 deep-scrub starts 2024-04-06T03:39:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:37 smithi153 bash[23852]: cluster 2024-04-06T03:39:35.943504+0000 osd.4 (osd.4) 442 : cluster 0 5.1 deep-scrub ok 2024-04-06T03:39:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:37 smithi153 bash[23852]: cluster 2024-04-06T03:39:36.076833+0000 osd.3 (osd.3) 467 : cluster 0 6.1f scrub ok 2024-04-06T03:39:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:37 smithi153 bash[23852]: cluster 2024-04-06T03:39:36.899680+0000 mon.a (mon.0) 1509 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:39:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:37 smithi153 bash[23852]: audit 2024-04-06T03:39:36.901111+0000 mon.a (mon.0) 1510 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:39:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:37 smithi153 bash[23852]: cluster 2024-04-06T03:39:36.904796+0000 mon.a (mon.0) 1511 : cluster 0 osdmap e292: 8 total, 8 up, 8 in 2024-04-06T03:39:38.067 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:38.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:38.199 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:38.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:35.942071+0000 osd.4 (osd.4) 441 : cluster 0 5.1 deep-scrub starts 2024-04-06T03:39:38.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:35.943504+0000 osd.4 (osd.4) 442 : cluster 0 5.1 deep-scrub ok 2024-04-06T03:39:38.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:36.076833+0000 osd.3 (osd.3) 467 : cluster 0 6.1f scrub ok 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:36.899680+0000 mon.a (mon.0) 1509 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: audit 2024-04-06T03:39:36.901111+0000 mon.a (mon.0) 1510 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[21467]: cluster 2024-04-06T03:39:36.904796+0000 mon.a (mon.0) 1511 : cluster 0 osdmap e292: 8 total, 8 up, 8 in 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:35.942071+0000 osd.4 (osd.4) 441 : cluster 0 5.1 deep-scrub starts 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:35.943504+0000 osd.4 (osd.4) 442 : cluster 0 5.1 deep-scrub ok 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:36.076833+0000 osd.3 (osd.3) 467 : cluster 0 6.1f scrub ok 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:36.899680+0000 mon.a (mon.0) 1509 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: audit 2024-04-06T03:39:36.901111+0000 mon.a (mon.0) 1510 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:39:38.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:37 smithi050 bash[29221]: cluster 2024-04-06T03:39:36.904796+0000 mon.a (mon.0) 1511 : cluster 0 osdmap e292: 8 total, 8 up, 8 in 2024-04-06T03:39:38.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:38.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:38.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:38.454 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:38.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 34 16 49180 49164 90.3568 64.1875 0.00443898 0.0110621 2024-04-06T03:39:38.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:38.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:38.689 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:38.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:38.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:38.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:38.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:38.956 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:38 smithi153 bash[23852]: cluster 2024-04-06T03:39:38.388652+0000 mgr.y (mgr.24368) 687 : cluster 0 pgmap v732: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-06T03:39:38.969 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:38.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:38.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:39.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:39.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:39.113 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:38 smithi050 bash[29221]: cluster 2024-04-06T03:39:38.388652+0000 mgr.y (mgr.24368) 687 : cluster 0 pgmap v732: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-06T03:39:39.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:38 smithi050 bash[21467]: cluster 2024-04-06T03:39:38.388652+0000 mgr.y (mgr.24368) 687 : cluster 0 pgmap v732: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-04-06T03:39:39.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:39.236 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:39.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:39.361 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:39.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:39.488 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:39:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:39:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:39.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 35 16 49941 49925 89.1341 47.5625 0.00765795 0.0112136 2024-04-06T03:39:39.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:39.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:39.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:39.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:39.861 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:39.892 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:39.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:39.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:40.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:40.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:39:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:39:40.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:39 smithi153 bash[23852]: audit 2024-04-06T03:39:39.713878+0000 mon.a (mon.0) 1512 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:39:40.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-06T03:39:40.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:40.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:40.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:40.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:40.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:40.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:39 smithi050 bash[29221]: audit 2024-04-06T03:39:39.713878+0000 mon.a (mon.0) 1512 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:39:40.354 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:39 smithi050 bash[21467]: audit 2024-04-06T03:39:39.713878+0000 mon.a (mon.0) 1512 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:39:40.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:40.410 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:40.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:40.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:40.554 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 36 16 50867 50851 88.2657 57.875 0.0310284 0.011325 2024-04-06T03:39:40.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:40.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:40.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:40.809 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:40.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:40.936 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:40.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:41.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:41.067 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:41.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:41.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: audit 2024-04-06T03:39:39.930617+0000 mgr.y (mgr.24368) 688 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:41.182 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: cluster 2024-04-06T03:39:40.246700+0000 osd.1 (osd.1) 357 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:41.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: cluster 2024-04-06T03:39:40.246781+0000 osd.1 (osd.1) 358 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:41.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: cluster 2024-04-06T03:39:40.389430+0000 mgr.y (mgr.24368) 689 : cluster 0 pgmap v733: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-04-06T03:39:41.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: audit 2024-04-06T03:39:40.608573+0000 mon.a (mon.0) 1513 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:39:41.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: audit 2024-04-06T03:39:40.613182+0000 mon.a (mon.0) 1514 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:39:41.183 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:40 smithi153 bash[23852]: audit 2024-04-06T03:39:40.623309+0000 mon.a (mon.0) 1515 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:41.194 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: audit 2024-04-06T03:39:39.930617+0000 mgr.y (mgr.24368) 688 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: cluster 2024-04-06T03:39:40.246700+0000 osd.1 (osd.1) 357 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: cluster 2024-04-06T03:39:40.246781+0000 osd.1 (osd.1) 358 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: cluster 2024-04-06T03:39:40.389430+0000 mgr.y (mgr.24368) 689 : cluster 0 pgmap v733: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: audit 2024-04-06T03:39:40.608573+0000 mon.a (mon.0) 1513 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: audit 2024-04-06T03:39:40.613182+0000 mon.a (mon.0) 1514 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[21467]: audit 2024-04-06T03:39:40.623309+0000 mon.a (mon.0) 1515 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: audit 2024-04-06T03:39:39.930617+0000 mgr.y (mgr.24368) 688 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: cluster 2024-04-06T03:39:40.246700+0000 osd.1 (osd.1) 357 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: cluster 2024-04-06T03:39:40.246781+0000 osd.1 (osd.1) 358 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: cluster 2024-04-06T03:39:40.389430+0000 mgr.y (mgr.24368) 689 : cluster 0 pgmap v733: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: audit 2024-04-06T03:39:40.608573+0000 mon.a (mon.0) 1513 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: audit 2024-04-06T03:39:40.613182+0000 mon.a (mon.0) 1514 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:39:41.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:40 smithi050 bash[29221]: audit 2024-04-06T03:39:40.623309+0000 mon.a (mon.0) 1515 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:41.297 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.322 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:41.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:41.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:41.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:41.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 37 16 52109 52093 87.978 77.625 0.00622031 0.011361 2024-04-06T03:39:41.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:41.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:41.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:41.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:41.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:41.858 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:41.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:41.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:41 smithi050 bash[21467]: cluster 2024-04-06T03:39:40.934940+0000 mon.a (mon.0) 1516 : cluster 0 osdmap e293: 8 total, 8 up, 8 in 2024-04-06T03:39:41.988 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:41 smithi050 bash[29221]: cluster 2024-04-06T03:39:40.934940+0000 mon.a (mon.0) 1516 : cluster 0 osdmap e293: 8 total, 8 up, 8 in 2024-04-06T03:39:41.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:41.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:42.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:42.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:42.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:41 smithi153 bash[23852]: cluster 2024-04-06T03:39:40.934940+0000 mon.a (mon.0) 1516 : cluster 0 osdmap e293: 8 total, 8 up, 8 in 2024-04-06T03:39:42.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:42.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:42.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:42.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 38 16 53477 53461 87.9126 85.5 0.0111362 0.0113696 2024-04-06T03:39:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:42.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:42.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:42.916 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:42.943 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:42.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:43.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:39:43.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.070 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:43.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:43.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:42 smithi050 bash[21467]: cluster 2024-04-06T03:39:42.390119+0000 mgr.y (mgr.24368) 690 : cluster 0 pgmap v735: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:39:43.184 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:42 smithi050 bash[29221]: cluster 2024-04-06T03:39:42.390119+0000 mgr.y (mgr.24368) 690 : cluster 0 pgmap v735: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:39:43.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:43.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:43.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:42 smithi153 bash[23852]: cluster 2024-04-06T03:39:42.390119+0000 mgr.y (mgr.24368) 690 : cluster 0 pgmap v735: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:39:43.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.345 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:43.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.475 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:43.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 39 16 54798 54782 87.7752 82.5625 0.00970715 0.011388 2024-04-06T03:39:43.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:43.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:43.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:43.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:43.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:43.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:43.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:43.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:44.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:44.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:44.225 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:44.251 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:44.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:44.392 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:44.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:44.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:44.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:39:44.559048+0000 min lat: 0.0011709 max lat: 0.652534 avg lat: 0.011452 2024-04-06T03:39:44.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:39:44.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 40 16 55884 55868 87.2776 67.875 0.00909012 0.011452 2024-04-06T03:39:44.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:44.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:44.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:44.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:44.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:44.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:44.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:45.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:45.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:44 smithi153 bash[23852]: cluster 2024-04-06T03:39:43.884400+0000 mon.a (mon.0) 1517 : cluster 0 osdmap e294: 8 total, 8 up, 8 in 2024-04-06T03:39:45.032 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:44 smithi153 bash[23852]: cluster 2024-04-06T03:39:44.390754+0000 mgr.y (mgr.24368) 691 : cluster 0 pgmap v737: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:39:45.045 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:45.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:45.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:45.173 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:45.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:44 smithi050 bash[21467]: cluster 2024-04-06T03:39:43.884400+0000 mon.a (mon.0) 1517 : cluster 0 osdmap e294: 8 total, 8 up, 8 in 2024-04-06T03:39:45.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:44 smithi050 bash[21467]: cluster 2024-04-06T03:39:44.390754+0000 mgr.y (mgr.24368) 691 : cluster 0 pgmap v737: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:39:45.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:44 smithi050 bash[29221]: cluster 2024-04-06T03:39:43.884400+0000 mon.a (mon.0) 1517 : cluster 0 osdmap e294: 8 total, 8 up, 8 in 2024-04-06T03:39:45.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:44 smithi050 bash[29221]: cluster 2024-04-06T03:39:44.390754+0000 mgr.y (mgr.24368) 691 : cluster 0 pgmap v737: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:39:45.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:45.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:45.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:45.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:45.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:45.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 41 16 57047 57031 86.9216 72.6875 0.00557815 0.011498 2024-04-06T03:39:45.560 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:45.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:45.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:45.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:45.815 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:45.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:45.940 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:45.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:46.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:46.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:46.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:46.202 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:46.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:46.335 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:46.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.470 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:46.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:46.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 42 16 57940 57924 86.1809 55.8125 0.00432811 0.0115975 2024-04-06T03:39:46.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:46.601 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:46.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:46.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:46.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:46.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:46.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:46.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:46.959 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:39:46.960 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:39:46.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:47.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:47.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:47.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:47.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:47.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:47.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:47.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:47.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:47.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:47.401 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:47.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:47.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:47.530 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:47.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:47.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 43 16 59026 59010 85.7551 67.875 0.0352666 0.0116555 2024-04-06T03:39:47.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:47.699 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:39:47.710 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:39:47.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:47 smithi050 bash[21467]: cluster 2024-04-06T03:39:46.391384+0000 mgr.y (mgr.24368) 692 : cluster 0 pgmap v738: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:39:47.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:47 smithi050 bash[29221]: cluster 2024-04-06T03:39:46.391384+0000 mgr.y (mgr.24368) 692 : cluster 0 pgmap v738: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:39:47.800 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:47 smithi153 bash[23852]: cluster 2024-04-06T03:39:46.391384+0000 mgr.y (mgr.24368) 692 : cluster 0 pgmap v738: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-04-06T03:39:47.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:47.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:48.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:48.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:48.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:48.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:48.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:48.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:48.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:48.555 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 44 16 60044 60028 85.252 63.625 0.0112945 0.0117249 2024-04-06T03:39:48.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:48.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:48.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:48.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[21467]: cluster 2024-04-06T03:39:47.009555+0000 osd.4 (osd.4) 443 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:48.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[21467]: cluster 2024-04-06T03:39:47.009628+0000 osd.4 (osd.4) 444 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:48.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[21467]: cluster 2024-04-06T03:39:47.451554+0000 mon.a (mon.0) 1518 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-04-06T03:39:48.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[21467]: audit 2024-04-06T03:39:47.680667+0000 mgr.y (mgr.24368) 693 : audit 0 from='client.15330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:39:48.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[29221]: cluster 2024-04-06T03:39:47.009555+0000 osd.4 (osd.4) 443 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:48.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[29221]: cluster 2024-04-06T03:39:47.009628+0000 osd.4 (osd.4) 444 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:48.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[29221]: cluster 2024-04-06T03:39:47.451554+0000 mon.a (mon.0) 1518 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-04-06T03:39:48.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:48 smithi050 bash[29221]: audit 2024-04-06T03:39:47.680667+0000 mgr.y (mgr.24368) 693 : audit 0 from='client.15330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:39:48.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:48 smithi153 bash[23852]: cluster 2024-04-06T03:39:47.009555+0000 osd.4 (osd.4) 443 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:48 smithi153 bash[23852]: cluster 2024-04-06T03:39:47.009628+0000 osd.4 (osd.4) 444 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:48 smithi153 bash[23852]: cluster 2024-04-06T03:39:47.451554+0000 mon.a (mon.0) 1518 : cluster 0 osdmap e295: 8 total, 8 up, 8 in 2024-04-06T03:39:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:48 smithi153 bash[23852]: audit 2024-04-06T03:39:47.680667+0000 mgr.y (mgr.24368) 693 : audit 0 from='client.15330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:39:48.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:48.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:48.907 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:48.934 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:48.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:49.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:49.062 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:49.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:49.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:49.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 45 16 60937 60921 84.5978 55.8125 0.0131317 0.0118154 2024-04-06T03:39:49.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:49 smithi050 bash[21467]: cluster 2024-04-06T03:39:48.392017+0000 mgr.y (mgr.24368) 694 : cluster 0 pgmap v740: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-06T03:39:49.716 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:49 smithi050 bash[21467]: cluster 2024-04-06T03:39:48.888772+0000 mon.a (mon.0) 1519 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-04-06T03:39:49.717 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:39:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:39:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:49.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:49 smithi050 bash[29221]: cluster 2024-04-06T03:39:48.392017+0000 mgr.y (mgr.24368) 694 : cluster 0 pgmap v740: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-06T03:39:49.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:49 smithi050 bash[29221]: cluster 2024-04-06T03:39:48.888772+0000 mon.a (mon.0) 1519 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-04-06T03:39:49.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:49.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:49.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:49 smithi153 bash[23852]: cluster 2024-04-06T03:39:48.392017+0000 mgr.y (mgr.24368) 694 : cluster 0 pgmap v740: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 65 MiB/s wr, 1.05k op/s 2024-04-06T03:39:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:49 smithi153 bash[23852]: cluster 2024-04-06T03:39:48.888772+0000 mon.a (mon.0) 1519 : cluster 0 osdmap e296: 8 total, 8 up, 8 in 2024-04-06T03:39:49.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:49.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:49.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:49.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:50.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:50.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:50.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:50.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:39:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:39:50.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:50.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:50.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 46 16 61934 61918 84.1132 62.3125 0.0159673 0.0118832 2024-04-06T03:39:50.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:50.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:50.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:50.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.737 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:50.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:50.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.880 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:50.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:50.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:51.004 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:51.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:51.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:51.134 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:51.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:51.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:51.247 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:50 smithi153 bash[23852]: audit 2024-04-06T03:39:49.942601+0000 mgr.y (mgr.24368) 695 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:51.247 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:50 smithi153 bash[23852]: audit 2024-04-06T03:39:49.953233+0000 mon.a (mon.0) 1520 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:51.247 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:50 smithi153 bash[23852]: audit 2024-04-06T03:39:49.957160+0000 mon.a (mon.0) 1521 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:51.247 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:50 smithi153 bash[23852]: cluster 2024-04-06T03:39:50.392820+0000 mgr.y (mgr.24368) 696 : cluster 0 pgmap v742: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 786 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[21467]: audit 2024-04-06T03:39:49.942601+0000 mgr.y (mgr.24368) 695 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[21467]: audit 2024-04-06T03:39:49.953233+0000 mon.a (mon.0) 1520 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[21467]: audit 2024-04-06T03:39:49.957160+0000 mon.a (mon.0) 1521 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[21467]: cluster 2024-04-06T03:39:50.392820+0000 mgr.y (mgr.24368) 696 : cluster 0 pgmap v742: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 786 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[29221]: audit 2024-04-06T03:39:49.942601+0000 mgr.y (mgr.24368) 695 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[29221]: audit 2024-04-06T03:39:49.953233+0000 mon.a (mon.0) 1520 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[29221]: audit 2024-04-06T03:39:49.957160+0000 mon.a (mon.0) 1521 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:39:51.249 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:51 smithi050 bash[29221]: cluster 2024-04-06T03:39:50.392820+0000 mgr.y (mgr.24368) 696 : cluster 0 pgmap v742: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 786 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-04-06T03:39:51.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:51.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:51.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:51.386 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:51.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:51.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:51.512 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:51.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:51.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 47 16 63053 63037 83.8115 69.9375 0.00273897 0.0119125 2024-04-06T03:39:51.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:51.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:51.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:51.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:52.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:52.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:52.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:52.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:52.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:52.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:52.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 48 16 63817 63801 83.0602 47.75 0.0135953 0.0120343 2024-04-06T03:39:52.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:52.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:52.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:52.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:52.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:52.810 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:52.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:52.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:52.933 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:39:52.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:52.952 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:39:53.269 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2024-04-06T03:39:47.448604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3705936222,"num_objects":56769,"num_object_clones":0,"num_object_copies":170307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56769,"num_whiteouts":0,"num_read":2769,"num_read_kb":2583,"num_write":57632,"num_write_kb":3620957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57789,"ondisk_log_size":57789,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13602316,"kb_used_data":10702648,"kb_used_omap":12,"kb_used_meta":2899571,"kb_avail":736358900,"statfs":{"total":767960285184,"available":754031513600,"internally_reserved":0,"allocated":10959511552,"data_stored":10956589462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2969161265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":510853120,"num_objects":7795,"num_object_clones":0,"num_object_copies":23385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7795,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7795,"num_write_kb":498880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.059790"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":502,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016468+0000","last_change":"2024-04-06T03:38:33.397343+0000","last_active":"2024-04-06T03:39:44.016468+0000","last_peered":"2024-04-06T03:39:44.016468+0000","last_clean":"2024-04-06T03:39:44.016468+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:39:44.016468+0000","last_undegraded":"2024-04-06T03:39:44.016468+0000","last_fullsized":"2024-04-06T03:39:44.016468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:33.397236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:25.064683+0000","last_clean_scrub_stamp":"2024-04-06T03:38:33.397236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:53.334671+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883126+0000","last_change":"2024-04-06T03:39:00.032521+0000","last_active":"2024-04-06T03:39:43.883126+0000","last_peered":"2024-04-06T03:39:43.883126+0000","last_clean":"2024-04-06T03:39:43.883126+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:39:43.883126+0000","last_undegraded":"2024-04-06T03:39:43.883126+0000","last_fullsized":"2024-04-06T03:39:43.883126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:00.032302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:39:00.032302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:28.589523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":534,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027490+0000","last_change":"2024-04-06T03:39:14.104873+0000","last_active":"2024-04-06T03:39:44.027490+0000","last_peered":"2024-04-06T03:39:44.027490+0000","last_clean":"2024-04-06T03:39:44.027490+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:39:44.027490+0000","last_undegraded":"2024-04-06T03:39:44.027490+0000","last_fullsized":"2024-04-06T03:39:44.027490+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:14.104792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:14.104792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.385240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"199'32","reported_seq":560,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017981+0000","last_change":"2024-04-06T03:37:40.241138+0000","last_active":"2024-04-06T03:39:44.017981+0000","last_peered":"2024-04-06T03:39:44.017981+0000","last_clean":"2024-04-06T03:39:44.017981+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:39:44.017981+0000","last_undegraded":"2024-04-06T03:39:44.017981+0000","last_fullsized":"2024-04-06T03:39:44.017981+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'32","last_scrub_stamp":"2024-04-06T03:37:40.241007+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:37:40.241007+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.014466+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":510,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883191+0000","last_change":"2024-04-06T03:39:20.037086+0000","last_active":"2024-04-06T03:39:43.883191+0000","last_peered":"2024-04-06T03:39:43.883191+0000","last_clean":"2024-04-06T03:39:43.883191+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:39:43.883191+0000","last_undegraded":"2024-04-06T03:39:43.883191+0000","last_fullsized":"2024-04-06T03:39:43.883191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:20.036992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:20.036992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:45.175057+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":393,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026317+0000","last_change":"2024-04-06T03:39:07.979712+0000","last_active":"2024-04-06T03:39:44.026317+0000","last_peered":"2024-04-06T03:39:44.026317+0000","last_clean":"2024-04-06T03:39:44.026317+0000","last_became_active":"2024-04-06T03:25:54.480600+0000","last_became_peered":"2024-04-06T03:25:54.480600+0000","last_unstale":"2024-04-06T03:39:44.026317+0000","last_undegraded":"2024-04-06T03:39:44.026317+0000","last_fullsized":"2024-04-06T03:39:44.026317+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:07.979620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:07.979620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:09.243944+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":516,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016688+0000","last_change":"2024-04-06T03:38:52.265617+0000","last_active":"2024-04-06T03:39:44.016688+0000","last_peered":"2024-04-06T03:39:44.016688+0000","last_clean":"2024-04-06T03:39:44.016688+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:39:44.016688+0000","last_undegraded":"2024-04-06T03:39:44.016688+0000","last_fullsized":"2024-04-06T03:39:44.016688+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:52.265537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:38:52.265537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:53.808001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"199'31","reported_seq":567,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027317+0000","last_change":"2024-04-06T03:39:01.848367+0000","last_active":"2024-04-06T03:39:44.027317+0000","last_peered":"2024-04-06T03:39:44.027317+0000","last_clean":"2024-04-06T03:39:44.027317+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:39:44.027317+0000","last_undegraded":"2024-04-06T03:39:44.027317+0000","last_fullsized":"2024-04-06T03:39:44.027317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:01.848222+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:14.574150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":397,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024142+0000","last_change":"2024-04-06T03:39:11.125137+0000","last_active":"2024-04-06T03:39:44.024142+0000","last_peered":"2024-04-06T03:39:44.024142+0000","last_clean":"2024-04-06T03:39:44.024142+0000","last_became_active":"2024-04-06T03:25:54.484120+0000","last_became_peered":"2024-04-06T03:25:54.484120+0000","last_unstale":"2024-04-06T03:39:44.024142+0000","last_undegraded":"2024-04-06T03:39:44.024142+0000","last_fullsized":"2024-04-06T03:39:44.024142+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.124952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:29.954996+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":504,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026745+0000","last_change":"2024-04-06T03:38:39.136548+0000","last_active":"2024-04-06T03:39:44.026745+0000","last_peered":"2024-04-06T03:39:44.026745+0000","last_clean":"2024-04-06T03:39:44.026745+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:39:44.026745+0000","last_undegraded":"2024-04-06T03:39:44.026745+0000","last_fullsized":"2024-04-06T03:39:44.026745+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.136452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.136452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:53.411380+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":528,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.028150+0000","last_change":"2024-04-06T03:38:37.173203+0000","last_active":"2024-04-06T03:39:44.028150+0000","last_peered":"2024-04-06T03:39:44.028150+0000","last_clean":"2024-04-06T03:39:44.028150+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:39:44.028150+0000","last_undegraded":"2024-04-06T03:39:44.028150+0000","last_fullsized":"2024-04-06T03:39:44.028150+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:38:37.173088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.173088+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"199'34","reported_seq":510,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883695+0000","last_change":"2024-04-06T03:38:30.770446+0000","last_active":"2024-04-06T03:39:43.883695+0000","last_peered":"2024-04-06T03:39:43.883695+0000","last_clean":"2024-04-06T03:39:43.883695+0000","last_became_active":"2024-04-06T03:25:54.489462+0000","last_became_peered":"2024-04-06T03:25:54.489462+0000","last_unstale":"2024-04-06T03:39:43.883695+0000","last_undegraded":"2024-04-06T03:39:43.883695+0000","last_fullsized":"2024-04-06T03:39:43.883695+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'34","last_scrub_stamp":"2024-04-06T03:38:30.770300+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:38:30.770300+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:57.560892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":508,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.023592+0000","last_change":"2024-04-06T03:39:30.292163+0000","last_active":"2024-04-06T03:39:44.023592+0000","last_peered":"2024-04-06T03:39:44.023592+0000","last_clean":"2024-04-06T03:39:44.023592+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:39:44.023592+0000","last_undegraded":"2024-04-06T03:39:44.023592+0000","last_fullsized":"2024-04-06T03:39:44.023592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:30.292088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:30.292088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:55.827484+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":389,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025679+0000","last_change":"2024-04-06T03:39:26.216855+0000","last_active":"2024-04-06T03:39:44.025679+0000","last_peered":"2024-04-06T03:39:44.025679+0000","last_clean":"2024-04-06T03:39:44.025679+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:39:44.025679+0000","last_undegraded":"2024-04-06T03:39:44.025679+0000","last_fullsized":"2024-04-06T03:39:44.025679+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:26.216722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:26.216722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:36.642647+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":512,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026887+0000","last_change":"2024-04-06T03:37:28.610037+0000","last_active":"2024-04-06T03:39:44.026887+0000","last_peered":"2024-04-06T03:39:44.026887+0000","last_clean":"2024-04-06T03:39:44.026887+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:39:44.026887+0000","last_undegraded":"2024-04-06T03:39:44.026887+0000","last_fullsized":"2024-04-06T03:39:44.026887+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:37:28.609910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"199'30","reported_seq":488,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883973+0000","last_change":"2024-04-06T03:39:10.052379+0000","last_active":"2024-04-06T03:39:43.883973+0000","last_peered":"2024-04-06T03:39:43.883973+0000","last_clean":"2024-04-06T03:39:43.883973+0000","last_became_active":"2024-04-06T03:25:54.483976+0000","last_became_peered":"2024-04-06T03:25:54.483976+0000","last_unstale":"2024-04-06T03:39:43.883973+0000","last_undegraded":"2024-04-06T03:39:43.883973+0000","last_fullsized":"2024-04-06T03:39:43.883973+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'30","last_scrub_stamp":"2024-04-06T03:39:10.052211+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:39:10.052211+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.899653+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":512,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017717+0000","last_change":"2024-04-06T03:39:36.076966+0000","last_active":"2024-04-06T03:39:44.017717+0000","last_peered":"2024-04-06T03:39:44.017717+0000","last_clean":"2024-04-06T03:39:44.017717+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:39:44.017717+0000","last_undegraded":"2024-04-06T03:39:44.017717+0000","last_fullsized":"2024-04-06T03:39:44.017717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:36.076841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:36.076841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:49.718545+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":522,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.884039+0000","last_change":"2024-04-06T03:39:34.004957+0000","last_active":"2024-04-06T03:39:43.884039+0000","last_peered":"2024-04-06T03:39:43.884039+0000","last_clean":"2024-04-06T03:39:43.884039+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:39:43.884039+0000","last_undegraded":"2024-04-06T03:39:43.884039+0000","last_fullsized":"2024-04-06T03:39:43.884039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:34.004843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:39:34.004843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:58.509671+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016637+0000","last_change":"2024-04-06T03:38:32.280669+0000","last_active":"2024-04-06T03:39:44.016637+0000","last_peered":"2024-04-06T03:39:44.016637+0000","last_clean":"2024-04-06T03:39:44.016637+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:39:44.016637+0000","last_undegraded":"2024-04-06T03:39:44.016637+0000","last_fullsized":"2024-04-06T03:39:44.016637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:32.280559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:32.280559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:46.949757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"199'33","reported_seq":573,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883318+0000","last_change":"2024-04-06T03:38:57.058614+0000","last_active":"2024-04-06T03:39:43.883318+0000","last_peered":"2024-04-06T03:39:43.883318+0000","last_clean":"2024-04-06T03:39:43.883318+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:39:43.883318+0000","last_undegraded":"2024-04-06T03:39:43.883318+0000","last_fullsized":"2024-04-06T03:39:43.883318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.058408+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.110170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":504,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883964+0000","last_change":"2024-04-06T03:38:44.104984+0000","last_active":"2024-04-06T03:39:43.883964+0000","last_peered":"2024-04-06T03:39:43.883964+0000","last_clean":"2024-04-06T03:39:43.883964+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:39:43.883964+0000","last_undegraded":"2024-04-06T03:39:43.883964+0000","last_fullsized":"2024-04-06T03:39:43.883964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:44.104755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:44.104755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:13.537650+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":520,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024834+0000","last_change":"2024-04-06T03:38:43.800795+0000","last_active":"2024-04-06T03:39:44.024834+0000","last_peered":"2024-04-06T03:39:44.024834+0000","last_clean":"2024-04-06T03:39:44.024834+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:39:44.024834+0000","last_undegraded":"2024-04-06T03:39:44.024834+0000","last_fullsized":"2024-04-06T03:39:44.024834+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:43.800561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:38:43.800561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:59.835149+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":375,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883394+0000","last_change":"2024-04-06T03:39:08.007471+0000","last_active":"2024-04-06T03:39:43.883394+0000","last_peered":"2024-04-06T03:39:43.883394+0000","last_clean":"2024-04-06T03:39:43.883394+0000","last_became_active":"2024-04-06T03:26:18.617121+0000","last_became_peered":"2024-04-06T03:26:18.617121+0000","last_unstale":"2024-04-06T03:39:43.883394+0000","last_undegraded":"2024-04-06T03:39:43.883394+0000","last_fullsized":"2024-04-06T03:39:43.883394+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:08.007331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:08.007331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:23.215346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"224'16","reported_seq":544,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025916+0000","last_change":"2024-04-06T03:37:38.582652+0000","last_active":"2024-04-06T03:39:44.025916+0000","last_peered":"2024-04-06T03:39:44.025916+0000","last_clean":"2024-04-06T03:39:44.025916+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:39:44.025916+0000","last_undegraded":"2024-04-06T03:39:44.025916+0000","last_fullsized":"2024-04-06T03:39:44.025916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16","last_scrub_stamp":"2024-04-06T03:37:38.582497+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:37:38.582497+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:44.487799+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":384,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883600+0000","last_change":"2024-04-06T03:38:51.251747+0000","last_active":"2024-04-06T03:39:43.883600+0000","last_peered":"2024-04-06T03:39:43.883600+0000","last_clean":"2024-04-06T03:39:43.883600+0000","last_became_active":"2024-04-06T03:25:54.483894+0000","last_became_peered":"2024-04-06T03:25:54.483894+0000","last_unstale":"2024-04-06T03:39:43.883600+0000","last_undegraded":"2024-04-06T03:39:43.883600+0000","last_fullsized":"2024-04-06T03:39:43.883600+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:51.251490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:51.251490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:18.791411+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":514,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027272+0000","last_change":"2024-04-06T03:39:06.939070+0000","last_active":"2024-04-06T03:39:44.027272+0000","last_peered":"2024-04-06T03:39:44.027272+0000","last_clean":"2024-04-06T03:39:44.027272+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:39:44.027272+0000","last_undegraded":"2024-04-06T03:39:44.027272+0000","last_fullsized":"2024-04-06T03:39:44.027272+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:06.938951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:39:06.938951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:08.530385+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":507,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016827+0000","last_change":"2024-04-06T03:37:30.256605+0000","last_active":"2024-04-06T03:39:44.016827+0000","last_peered":"2024-04-06T03:39:44.016827+0000","last_clean":"2024-04-06T03:39:44.016827+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:39:44.016827+0000","last_undegraded":"2024-04-06T03:39:44.016827+0000","last_fullsized":"2024-04-06T03:39:44.016827+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:37:30.256067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:37:30.256067+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.014394+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"199'26","reported_seq":544,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016498+0000","last_change":"2024-04-06T03:38:47.204903+0000","last_active":"2024-04-06T03:39:44.016498+0000","last_peered":"2024-04-06T03:39:44.016498+0000","last_clean":"2024-04-06T03:39:44.016498+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:39:44.016498+0000","last_undegraded":"2024-04-06T03:39:44.016498+0000","last_fullsized":"2024-04-06T03:39:44.016498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'26","last_scrub_stamp":"2024-04-06T03:38:47.204658+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:38:47.204658+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:06.390894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":503,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027326+0000","last_change":"2024-04-06T03:38:40.108037+0000","last_active":"2024-04-06T03:39:44.027326+0000","last_peered":"2024-04-06T03:39:44.027326+0000","last_clean":"2024-04-06T03:39:44.027326+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:39:44.027326+0000","last_undegraded":"2024-04-06T03:39:44.027326+0000","last_fullsized":"2024-04-06T03:39:44.027326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:38:40.107808+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:31:51.905353+0000","last_clean_scrub_stamp":"2024-04-06T03:38:40.107808+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:51.997382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"139'13","reported_seq":620,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883771+0000","last_change":"2024-04-06T03:39:15.989066+0000","last_active":"2024-04-06T03:39:43.883771+0000","last_peered":"2024-04-06T03:39:43.883771+0000","last_clean":"2024-04-06T03:39:43.883771+0000","last_became_active":"2024-04-06T03:26:18.615209+0000","last_became_peered":"2024-04-06T03:26:18.615209+0000","last_unstale":"2024-04-06T03:39:43.883771+0000","last_undegraded":"2024-04-06T03:39:43.883771+0000","last_fullsized":"2024-04-06T03:39:43.883771+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'13","last_scrub_stamp":"2024-04-06T03:39:15.988918+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:39:15.988918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.121161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":389,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883092+0000","last_change":"2024-04-06T03:39:12.958930+0000","last_active":"2024-04-06T03:39:43.883092+0000","last_peered":"2024-04-06T03:39:43.883092+0000","last_clean":"2024-04-06T03:39:43.883092+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:39:43.883092+0000","last_undegraded":"2024-04-06T03:39:43.883092+0000","last_fullsized":"2024-04-06T03:39:43.883092+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:12.958845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:39:12.958845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.107423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'62","reported_seq":626,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.023951+0000","last_change":"2024-04-06T03:38:39.710296+0000","last_active":"2024-04-06T03:39:44.023951+0000","last_peered":"2024-04-06T03:39:44.023951+0000","last_clean":"2024-04-06T03:39:44.023951+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:39:44.023951+0000","last_undegraded":"2024-04-06T03:39:44.023951+0000","last_fullsized":"2024-04-06T03:39:44.023951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'62","last_scrub_stamp":"2024-04-06T03:38:39.710124+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.710124+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:07.839617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":495,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017314+0000","last_change":"2024-04-06T03:37:28.166141+0000","last_active":"2024-04-06T03:39:44.017314+0000","last_peered":"2024-04-06T03:39:44.017314+0000","last_clean":"2024-04-06T03:39:44.017314+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:39:44.017314+0000","last_undegraded":"2024-04-06T03:39:44.017314+0000","last_fullsized":"2024-04-06T03:39:44.017314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:28.165908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:37:28.165908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.014430+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":513,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025930+0000","last_change":"2024-04-06T03:39:36.013647+0000","last_active":"2024-04-06T03:39:44.025930+0000","last_peered":"2024-04-06T03:39:44.025930+0000","last_clean":"2024-04-06T03:39:44.025930+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:39:44.025930+0000","last_undegraded":"2024-04-06T03:39:44.025930+0000","last_fullsized":"2024-04-06T03:39:44.025930+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:36.013567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:39:36.013567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.319086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":391,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.018663+0000","last_change":"2024-04-06T03:38:29.927678+0000","last_active":"2024-04-06T03:39:44.018663+0000","last_peered":"2024-04-06T03:39:44.018663+0000","last_clean":"2024-04-06T03:39:44.018663+0000","last_became_active":"2024-04-06T03:26:18.635349+0000","last_became_peered":"2024-04-06T03:26:18.635349+0000","last_unstale":"2024-04-06T03:39:44.018663+0000","last_undegraded":"2024-04-06T03:39:44.018663+0000","last_fullsized":"2024-04-06T03:39:44.018663+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:29.927479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:38:29.927479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:57.169833+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"199'27","reported_seq":449,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.028030+0000","last_change":"2024-04-06T03:38:32.273967+0000","last_active":"2024-04-06T03:39:44.028030+0000","last_peered":"2024-04-06T03:39:44.028030+0000","last_clean":"2024-04-06T03:39:44.028030+0000","last_became_active":"2024-04-06T03:25:54.491804+0000","last_became_peered":"2024-04-06T03:25:54.491804+0000","last_unstale":"2024-04-06T03:39:44.028030+0000","last_undegraded":"2024-04-06T03:39:44.028030+0000","last_fullsized":"2024-04-06T03:39:44.028030+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:38:32.273593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:32.273593+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:45.071205+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":509,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025110+0000","last_change":"2024-04-06T03:39:31.990811+0000","last_active":"2024-04-06T03:39:44.025110+0000","last_peered":"2024-04-06T03:39:44.025110+0000","last_clean":"2024-04-06T03:39:44.025110+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:39:44.025110+0000","last_undegraded":"2024-04-06T03:39:44.025110+0000","last_fullsized":"2024-04-06T03:39:44.025110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:31.990689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:39:31.990689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:57.086019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":376,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883085+0000","last_change":"2024-04-06T03:39:07.049989+0000","last_active":"2024-04-06T03:39:43.883085+0000","last_peered":"2024-04-06T03:39:43.883085+0000","last_clean":"2024-04-06T03:39:43.883085+0000","last_became_active":"2024-04-06T03:26:18.618151+0000","last_became_peered":"2024-04-06T03:26:18.618151+0000","last_unstale":"2024-04-06T03:39:43.883085+0000","last_undegraded":"2024-04-06T03:39:43.883085+0000","last_fullsized":"2024-04-06T03:39:43.883085+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:07.049875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:07.049875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:18.063806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":394,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.021173+0000","last_change":"2024-04-06T03:39:00.237609+0000","last_active":"2024-04-06T03:39:44.021173+0000","last_peered":"2024-04-06T03:39:44.021173+0000","last_clean":"2024-04-06T03:39:44.021173+0000","last_became_active":"2024-04-06T03:26:18.637825+0000","last_became_peered":"2024-04-06T03:26:18.637825+0000","last_unstale":"2024-04-06T03:39:44.021173+0000","last_undegraded":"2024-04-06T03:39:44.021173+0000","last_fullsized":"2024-04-06T03:39:44.021173+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:00.237461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:00.237461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:24.578366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"199'22","reported_seq":558,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027210+0000","last_change":"2024-04-06T03:38:36.725277+0000","last_active":"2024-04-06T03:39:44.027210+0000","last_peered":"2024-04-06T03:39:44.027210+0000","last_clean":"2024-04-06T03:39:44.027210+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:39:44.027210+0000","last_undegraded":"2024-04-06T03:39:44.027210+0000","last_fullsized":"2024-04-06T03:39:44.027210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'22","last_scrub_stamp":"2024-04-06T03:38:36.725153+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.725153+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:55.114254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":495,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016612+0000","last_change":"2024-04-06T03:38:34.251308+0000","last_active":"2024-04-06T03:39:44.016612+0000","last_peered":"2024-04-06T03:39:44.016612+0000","last_clean":"2024-04-06T03:39:44.016612+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:39:44.016612+0000","last_undegraded":"2024-04-06T03:39:44.016612+0000","last_fullsized":"2024-04-06T03:39:44.016612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:34.251170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:34.251170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.014775+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":521,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025566+0000","last_change":"2024-04-06T03:39:09.707299+0000","last_active":"2024-04-06T03:39:44.025566+0000","last_peered":"2024-04-06T03:39:44.025566+0000","last_clean":"2024-04-06T03:39:44.025566+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:39:44.025566+0000","last_undegraded":"2024-04-06T03:39:44.025566+0000","last_fullsized":"2024-04-06T03:39:44.025566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:09.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:09.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:31.639900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":536,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024565+0000","last_change":"2024-04-06T03:38:44.792267+0000","last_active":"2024-04-06T03:39:44.024565+0000","last_peered":"2024-04-06T03:39:44.024565+0000","last_clean":"2024-04-06T03:39:44.024565+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:39:44.024565+0000","last_undegraded":"2024-04-06T03:39:44.024565+0000","last_fullsized":"2024-04-06T03:39:44.024565+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:44.792080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:38:44.792080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:12.046132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"199'33","reported_seq":574,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883926+0000","last_change":"2024-04-06T03:38:31.820449+0000","last_active":"2024-04-06T03:39:43.883926+0000","last_peered":"2024-04-06T03:39:43.883926+0000","last_clean":"2024-04-06T03:39:43.883926+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:39:43.883926+0000","last_undegraded":"2024-04-06T03:39:43.883926+0000","last_fullsized":"2024-04-06T03:39:43.883926+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:38:31.820183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:31.820183+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:45.918838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":509,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.023474+0000","last_change":"2024-04-06T03:39:22.337187+0000","last_active":"2024-04-06T03:39:44.023474+0000","last_peered":"2024-04-06T03:39:44.023474+0000","last_clean":"2024-04-06T03:39:44.023474+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:39:44.023474+0000","last_undegraded":"2024-04-06T03:39:44.023474+0000","last_fullsized":"2024-04-06T03:39:44.023474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:22.336350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:22.336350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:49.007992+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":507,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017081+0000","last_change":"2024-04-06T03:39:28.619043+0000","last_active":"2024-04-06T03:39:44.017081+0000","last_peered":"2024-04-06T03:39:44.017081+0000","last_clean":"2024-04-06T03:39:44.017081+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:39:44.017081+0000","last_undegraded":"2024-04-06T03:39:44.017081+0000","last_fullsized":"2024-04-06T03:39:44.017081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:28.618765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_clean_scrub_stamp":"2024-04-06T03:39:28.618765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:42.794174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":367,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.018152+0000","last_change":"2024-04-06T03:37:43.208701+0000","last_active":"2024-04-06T03:39:44.018152+0000","last_peered":"2024-04-06T03:39:44.018152+0000","last_clean":"2024-04-06T03:39:44.018152+0000","last_became_active":"2024-04-06T03:26:18.643806+0000","last_became_peered":"2024-04-06T03:26:18.643806+0000","last_unstale":"2024-04-06T03:39:44.018152+0000","last_undegraded":"2024-04-06T03:39:44.018152+0000","last_fullsized":"2024-04-06T03:39:44.018152+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:43.208163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:37:43.208163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.014742+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"199'21","reported_seq":545,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027796+0000","last_change":"2024-04-06T03:37:39.557911+0000","last_active":"2024-04-06T03:39:44.027796+0000","last_peered":"2024-04-06T03:39:44.027796+0000","last_clean":"2024-04-06T03:39:44.027796+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:39:44.027796+0000","last_undegraded":"2024-04-06T03:39:44.027796+0000","last_fullsized":"2024-04-06T03:39:44.027796+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:37:39.557640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:37:39.557640+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":510,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883889+0000","last_change":"2024-04-06T03:38:52.303957+0000","last_active":"2024-04-06T03:39:43.883889+0000","last_peered":"2024-04-06T03:39:43.883889+0000","last_clean":"2024-04-06T03:39:43.883889+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:39:43.883889+0000","last_undegraded":"2024-04-06T03:39:43.883889+0000","last_fullsized":"2024-04-06T03:39:43.883889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:52.303759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:52.303759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:11.407085+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":745,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.018246+0000","last_change":"2024-04-06T03:39:26.623101+0000","last_active":"2024-04-06T03:39:44.018246+0000","last_peered":"2024-04-06T03:39:44.018246+0000","last_clean":"2024-04-06T03:39:44.018246+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:39:44.018246+0000","last_undegraded":"2024-04-06T03:39:44.018246+0000","last_fullsized":"2024-04-06T03:39:44.018246+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:39:26.622968+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:39:26.622968+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.086767+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":521,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026019+0000","last_change":"2024-04-06T03:38:44.739027+0000","last_active":"2024-04-06T03:39:44.026019+0000","last_peered":"2024-04-06T03:39:44.026019+0000","last_clean":"2024-04-06T03:39:44.026019+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:39:44.026019+0000","last_undegraded":"2024-04-06T03:39:44.026019+0000","last_fullsized":"2024-04-06T03:39:44.026019+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:44.738912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:44.738912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:10.631546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"199'15","reported_seq":542,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027693+0000","last_change":"2024-04-06T03:39:04.854322+0000","last_active":"2024-04-06T03:39:44.027693+0000","last_peered":"2024-04-06T03:39:44.027693+0000","last_clean":"2024-04-06T03:39:44.027693+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:39:44.027693+0000","last_undegraded":"2024-04-06T03:39:44.027693+0000","last_fullsized":"2024-04-06T03:39:44.027693+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'15","last_scrub_stamp":"2024-04-06T03:39:04.854179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:04.854179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.515388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":494,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027514+0000","last_change":"2024-04-06T03:38:39.773890+0000","last_active":"2024-04-06T03:39:44.027514+0000","last_peered":"2024-04-06T03:39:44.027514+0000","last_clean":"2024-04-06T03:39:44.027514+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:39:44.027514+0000","last_undegraded":"2024-04-06T03:39:44.027514+0000","last_fullsized":"2024-04-06T03:39:44.027514+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.773789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.773789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:08.923157+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"154'13","reported_seq":523,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.032589+0000","last_change":"2024-04-06T03:39:07.946464+0000","last_active":"2024-04-06T03:39:44.032589+0000","last_peered":"2024-04-06T03:39:44.032589+0000","last_clean":"2024-04-06T03:39:44.032589+0000","last_became_active":"2024-04-06T03:27:10.498254+0000","last_became_peered":"2024-04-06T03:27:10.498254+0000","last_unstale":"2024-04-06T03:39:44.032589+0000","last_undegraded":"2024-04-06T03:39:44.032589+0000","last_fullsized":"2024-04-06T03:39:44.032589+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'13","last_scrub_stamp":"2024-04-06T03:39:07.946321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:07.946321+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.741276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":379,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017235+0000","last_change":"2024-04-06T03:38:56.232797+0000","last_active":"2024-04-06T03:39:44.017235+0000","last_peered":"2024-04-06T03:39:44.017235+0000","last_clean":"2024-04-06T03:39:44.017235+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:39:44.017235+0000","last_undegraded":"2024-04-06T03:39:44.017235+0000","last_fullsized":"2024-04-06T03:39:44.017235+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:56.232398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_clean_scrub_stamp":"2024-04-06T03:38:56.232398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:18.749154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"199'24","reported_seq":545,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026068+0000","last_change":"2024-04-06T03:37:30.552999+0000","last_active":"2024-04-06T03:39:44.026068+0000","last_peered":"2024-04-06T03:39:44.026068+0000","last_clean":"2024-04-06T03:39:44.026068+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:39:44.026068+0000","last_undegraded":"2024-04-06T03:39:44.026068+0000","last_fullsized":"2024-04-06T03:39:44.026068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:37:30.552697+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":508,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026281+0000","last_change":"2024-04-06T03:38:36.191151+0000","last_active":"2024-04-06T03:39:44.026281+0000","last_peered":"2024-04-06T03:39:44.026281+0000","last_clean":"2024-04-06T03:39:44.026281+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:39:44.026281+0000","last_undegraded":"2024-04-06T03:39:44.026281+0000","last_fullsized":"2024-04-06T03:39:44.026281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:36.191036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.191036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:44.259985+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":373,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016563+0000","last_change":"2024-04-06T03:38:37.301179+0000","last_active":"2024-04-06T03:39:44.016563+0000","last_peered":"2024-04-06T03:39:44.016563+0000","last_clean":"2024-04-06T03:39:44.016563+0000","last_became_active":"2024-04-06T03:25:54.485733+0000","last_became_peered":"2024-04-06T03:25:54.485733+0000","last_unstale":"2024-04-06T03:39:44.016563+0000","last_undegraded":"2024-04-06T03:39:44.016563+0000","last_fullsized":"2024-04-06T03:39:44.016563+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:37.300978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.300978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:57.895790+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":375,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883736+0000","last_change":"2024-04-06T03:39:16.953596+0000","last_active":"2024-04-06T03:39:43.883736+0000","last_peered":"2024-04-06T03:39:43.883736+0000","last_clean":"2024-04-06T03:39:43.883736+0000","last_became_active":"2024-04-06T03:26:18.615468+0000","last_became_peered":"2024-04-06T03:26:18.615468+0000","last_unstale":"2024-04-06T03:39:43.883736+0000","last_undegraded":"2024-04-06T03:39:43.883736+0000","last_fullsized":"2024-04-06T03:39:43.883736+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:16.953510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:16.953510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:32.122677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"199'10","reported_seq":470,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025085+0000","last_change":"2024-04-06T03:38:54.798995+0000","last_active":"2024-04-06T03:39:44.025085+0000","last_peered":"2024-04-06T03:39:44.025085+0000","last_clean":"2024-04-06T03:39:44.025085+0000","last_became_active":"2024-04-06T03:26:18.639720+0000","last_became_peered":"2024-04-06T03:26:18.639720+0000","last_unstale":"2024-04-06T03:39:44.025085+0000","last_undegraded":"2024-04-06T03:39:44.025085+0000","last_fullsized":"2024-04-06T03:39:44.025085+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10","last_scrub_stamp":"2024-04-06T03:38:54.798646+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:38:54.798646+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:14.947433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":529,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017121+0000","last_change":"2024-04-06T03:39:08.191957+0000","last_active":"2024-04-06T03:39:44.017121+0000","last_peered":"2024-04-06T03:39:44.017121+0000","last_clean":"2024-04-06T03:39:44.017121+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:39:44.017121+0000","last_undegraded":"2024-04-06T03:39:44.017121+0000","last_fullsized":"2024-04-06T03:39:44.017121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:39:08.191830+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:39:08.191830+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:11.183367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017886+0000","last_change":"2024-04-06T03:38:31.250220+0000","last_active":"2024-04-06T03:39:44.017886+0000","last_peered":"2024-04-06T03:39:44.017886+0000","last_clean":"2024-04-06T03:39:44.017886+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:39:44.017886+0000","last_undegraded":"2024-04-06T03:39:44.017886+0000","last_fullsized":"2024-04-06T03:39:44.017886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:31.250106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:31.250106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.014807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":539,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.015942+0000","last_change":"2024-04-06T03:38:30.958769+0000","last_active":"2024-04-06T03:39:44.015942+0000","last_peered":"2024-04-06T03:39:44.015942+0000","last_clean":"2024-04-06T03:39:44.015942+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:39:44.015942+0000","last_undegraded":"2024-04-06T03:39:44.015942+0000","last_fullsized":"2024-04-06T03:39:44.015942+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:30.958503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:30.958503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:58.741593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"199'31","reported_seq":566,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026181+0000","last_change":"2024-04-06T03:38:35.196050+0000","last_active":"2024-04-06T03:39:44.026181+0000","last_peered":"2024-04-06T03:39:44.026181+0000","last_clean":"2024-04-06T03:39:44.026181+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:39:44.026181+0000","last_undegraded":"2024-04-06T03:39:44.026181+0000","last_fullsized":"2024-04-06T03:39:44.026181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:38:35.195635+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:38:35.195635+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"294'3655","reported_seq":3693,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:46.949260+0000","last_change":"2024-04-06T03:38:41.243073+0000","last_active":"2024-04-06T03:39:46.949260+0000","last_peered":"2024-04-06T03:39:46.949260+0000","last_clean":"2024-04-06T03:39:46.949260+0000","last_became_active":"2024-04-06T03:38:41.242239+0000","last_became_peered":"2024-04-06T03:38:41.242239+0000","last_unstale":"2024-04-06T03:39:46.949260+0000","last_undegraded":"2024-04-06T03:39:46.949260+0000","last_fullsized":"2024-04-06T03:39:46.949260+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:47.607643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239534080,"num_objects":3655,"num_object_clones":0,"num_object_copies":10965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3655,"num_write_kb":233920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":504,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017170+0000","last_change":"2024-04-06T03:39:16.644919+0000","last_active":"2024-04-06T03:39:44.017170+0000","last_peered":"2024-04-06T03:39:44.017170+0000","last_clean":"2024-04-06T03:39:44.017170+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:39:44.017170+0000","last_undegraded":"2024-04-06T03:39:44.017170+0000","last_fullsized":"2024-04-06T03:39:44.017170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:16.644788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:16.644788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:17.121912+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":541,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883572+0000","last_change":"2024-04-06T03:38:40.956357+0000","last_active":"2024-04-06T03:39:43.883572+0000","last_peered":"2024-04-06T03:39:43.883572+0000","last_clean":"2024-04-06T03:39:43.883572+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:39:43.883572+0000","last_undegraded":"2024-04-06T03:39:43.883572+0000","last_fullsized":"2024-04-06T03:39:43.883572+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:38:40.956237+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:38:40.956237+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:04.896291+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'31","reported_seq":566,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027759+0000","last_change":"2024-04-06T03:37:37.543171+0000","last_active":"2024-04-06T03:39:44.027759+0000","last_peered":"2024-04-06T03:39:44.027759+0000","last_clean":"2024-04-06T03:39:44.027759+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:39:44.027759+0000","last_undegraded":"2024-04-06T03:39:44.027759+0000","last_fullsized":"2024-04-06T03:39:44.027759+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'31","last_scrub_stamp":"2024-04-06T03:37:37.543013+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:37:37.543013+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":531,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.015686+0000","last_change":"2024-04-06T03:39:03.329103+0000","last_active":"2024-04-06T03:39:44.015686+0000","last_peered":"2024-04-06T03:39:44.015686+0000","last_clean":"2024-04-06T03:39:44.015686+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:39:44.015686+0000","last_undegraded":"2024-04-06T03:39:44.015686+0000","last_fullsized":"2024-04-06T03:39:44.015686+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:03.328962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:03.328962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:20.620710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"294'3570","reported_seq":3608,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.371767+0000","last_change":"2024-04-06T03:38:40.968041+0000","last_active":"2024-04-06T03:39:44.371767+0000","last_peered":"2024-04-06T03:39:44.371767+0000","last_clean":"2024-04-06T03:39:44.371767+0000","last_became_active":"2024-04-06T03:38:40.967708+0000","last_became_peered":"2024-04-06T03:38:40.967708+0000","last_unstale":"2024-04-06T03:39:44.371767+0000","last_undegraded":"2024-04-06T03:39:44.371767+0000","last_fullsized":"2024-04-06T03:39:44.371767+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:46.949281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233963520,"num_objects":3570,"num_object_clones":0,"num_object_copies":10710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":228480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":510,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.023587+0000","last_change":"2024-04-06T03:38:37.764505+0000","last_active":"2024-04-06T03:39:44.023587+0000","last_peered":"2024-04-06T03:39:44.023587+0000","last_clean":"2024-04-06T03:39:44.023587+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:39:44.023587+0000","last_undegraded":"2024-04-06T03:39:44.023587+0000","last_fullsized":"2024-04-06T03:39:44.023587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:37.764177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.764177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:04.226958+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":470,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027632+0000","last_change":"2024-04-06T03:39:02.882952+0000","last_active":"2024-04-06T03:39:44.027632+0000","last_peered":"2024-04-06T03:39:44.027632+0000","last_clean":"2024-04-06T03:39:44.027632+0000","last_became_active":"2024-04-06T03:27:10.500422+0000","last_became_peered":"2024-04-06T03:27:10.500422+0000","last_unstale":"2024-04-06T03:39:44.027632+0000","last_undegraded":"2024-04-06T03:39:44.027632+0000","last_fullsized":"2024-04-06T03:39:44.027632+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:39:02.882714+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:39:02.882714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:25.341566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"199'37","reported_seq":617,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017818+0000","last_change":"2024-04-06T03:39:19.509800+0000","last_active":"2024-04-06T03:39:44.017818+0000","last_peered":"2024-04-06T03:39:44.017818+0000","last_clean":"2024-04-06T03:39:44.017818+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:39:44.017818+0000","last_undegraded":"2024-04-06T03:39:44.017818+0000","last_fullsized":"2024-04-06T03:39:44.017818+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:39:19.509187+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:39:19.509187+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:46.311872+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":534,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025649+0000","last_change":"2024-04-06T03:38:37.780281+0000","last_active":"2024-04-06T03:39:44.025649+0000","last_peered":"2024-04-06T03:39:44.025649+0000","last_clean":"2024-04-06T03:39:44.025649+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:39:44.025649+0000","last_undegraded":"2024-04-06T03:39:44.025649+0000","last_fullsized":"2024-04-06T03:39:44.025649+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:38:37.780066+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.780066+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:59.992585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":751,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025838+0000","last_change":"2024-04-06T03:39:11.138297+0000","last_active":"2024-04-06T03:39:44.025838+0000","last_peered":"2024-04-06T03:39:44.025838+0000","last_clean":"2024-04-06T03:39:44.025838+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:39:44.025838+0000","last_undegraded":"2024-04-06T03:39:44.025838+0000","last_fullsized":"2024-04-06T03:39:44.025838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:39:11.138146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.138146+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:21.825735+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"294'3609","reported_seq":3647,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:46.389863+0000","last_change":"2024-04-06T03:38:41.244498+0000","last_active":"2024-04-06T03:39:46.389863+0000","last_peered":"2024-04-06T03:39:46.389863+0000","last_clean":"2024-04-06T03:39:46.389863+0000","last_became_active":"2024-04-06T03:38:41.244000+0000","last_became_peered":"2024-04-06T03:38:41.244000+0000","last_unstale":"2024-04-06T03:39:46.389863+0000","last_undegraded":"2024-04-06T03:39:46.389863+0000","last_fullsized":"2024-04-06T03:39:46.389863+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:02.354005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236519424,"num_objects":3609,"num_object_clones":0,"num_object_copies":10827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3609,"num_write_kb":230976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":503,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016696+0000","last_change":"2024-04-06T03:39:24.595475+0000","last_active":"2024-04-06T03:39:44.016696+0000","last_peered":"2024-04-06T03:39:44.016696+0000","last_clean":"2024-04-06T03:39:44.016696+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:39:44.016696+0000","last_undegraded":"2024-04-06T03:39:44.016696+0000","last_fullsized":"2024-04-06T03:39:44.016696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:39:24.595337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:28.464510+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":397,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025111+0000","last_change":"2024-04-06T03:38:57.721802+0000","last_active":"2024-04-06T03:39:44.025111+0000","last_peered":"2024-04-06T03:39:44.025111+0000","last_clean":"2024-04-06T03:39:44.025111+0000","last_became_active":"2024-04-06T03:25:54.484235+0000","last_became_peered":"2024-04-06T03:25:54.484235+0000","last_unstale":"2024-04-06T03:39:44.025111+0000","last_undegraded":"2024-04-06T03:39:44.025111+0000","last_fullsized":"2024-04-06T03:39:44.025111+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:57.721624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.721624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:24.378648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"199'44","reported_seq":584,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883518+0000","last_change":"2024-04-06T03:38:53.204522+0000","last_active":"2024-04-06T03:39:43.883518+0000","last_peered":"2024-04-06T03:39:43.883518+0000","last_clean":"2024-04-06T03:39:43.883518+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:39:43.883518+0000","last_undegraded":"2024-04-06T03:39:43.883518+0000","last_fullsized":"2024-04-06T03:39:43.883518+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'44","last_scrub_stamp":"2024-04-06T03:38:53.204362+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:38:53.204362+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:12.386087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":514,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.029713+0000","last_change":"2024-04-06T03:39:12.156813+0000","last_active":"2024-04-06T03:39:44.029713+0000","last_peered":"2024-04-06T03:39:44.029713+0000","last_clean":"2024-04-06T03:39:44.029713+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:39:44.029713+0000","last_undegraded":"2024-04-06T03:39:44.029713+0000","last_fullsized":"2024-04-06T03:39:44.029713+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:12.156581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:39:12.156581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:38.634867+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"294'3503","reported_seq":3541,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.737416+0000","last_change":"2024-04-06T03:38:40.975902+0000","last_active":"2024-04-06T03:39:44.737416+0000","last_peered":"2024-04-06T03:39:44.737416+0000","last_clean":"2024-04-06T03:39:44.737416+0000","last_became_active":"2024-04-06T03:38:40.974686+0000","last_became_peered":"2024-04-06T03:38:40.974686+0000","last_unstale":"2024-04-06T03:39:44.737416+0000","last_undegraded":"2024-04-06T03:39:44.737416+0000","last_fullsized":"2024-04-06T03:39:44.737416+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:55.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229572608,"num_objects":3503,"num_object_clones":0,"num_object_copies":10509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":224192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":509,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026413+0000","last_change":"2024-04-06T03:39:35.082316+0000","last_active":"2024-04-06T03:39:44.026413+0000","last_peered":"2024-04-06T03:39:44.026413+0000","last_clean":"2024-04-06T03:39:44.026413+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:39:44.026413+0000","last_undegraded":"2024-04-06T03:39:44.026413+0000","last_fullsized":"2024-04-06T03:39:44.026413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:35.082184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:39:35.082184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:35.161516+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"199'42","reported_seq":447,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.015998+0000","last_change":"2024-04-06T03:38:42.981579+0000","last_active":"2024-04-06T03:39:44.015998+0000","last_peered":"2024-04-06T03:39:44.015998+0000","last_clean":"2024-04-06T03:39:44.015998+0000","last_became_active":"2024-04-06T03:27:10.495663+0000","last_became_peered":"2024-04-06T03:27:10.495663+0000","last_unstale":"2024-04-06T03:39:44.015998+0000","last_undegraded":"2024-04-06T03:39:44.015998+0000","last_fullsized":"2024-04-06T03:39:44.015998+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'42","last_scrub_stamp":"2024-04-06T03:38:42.981382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:42.981382+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:05.900192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":540,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017466+0000","last_change":"2024-04-06T03:38:56.123425+0000","last_active":"2024-04-06T03:39:44.017466+0000","last_peered":"2024-04-06T03:39:44.017466+0000","last_clean":"2024-04-06T03:39:44.017466+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:39:44.017466+0000","last_undegraded":"2024-04-06T03:39:44.017466+0000","last_fullsized":"2024-04-06T03:39:44.017466+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:56.123319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:38:56.123319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:20.667797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":531,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.021183+0000","last_change":"2024-04-06T03:38:49.030047+0000","last_active":"2024-04-06T03:39:44.021183+0000","last_peered":"2024-04-06T03:39:44.021183+0000","last_clean":"2024-04-06T03:39:44.021183+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:39:44.021183+0000","last_undegraded":"2024-04-06T03:39:44.021183+0000","last_fullsized":"2024-04-06T03:39:44.021183+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:49.029797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:49.029797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:54.997649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"294'3553","reported_seq":3591,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024432+0000","last_change":"2024-04-06T03:38:41.244345+0000","last_active":"2024-04-06T03:39:44.024432+0000","last_peered":"2024-04-06T03:39:44.024432+0000","last_clean":"2024-04-06T03:39:44.024432+0000","last_became_active":"2024-04-06T03:38:41.243355+0000","last_became_peered":"2024-04-06T03:38:41.243355+0000","last_unstale":"2024-04-06T03:39:44.024432+0000","last_undegraded":"2024-04-06T03:39:44.024432+0000","last_fullsized":"2024-04-06T03:39:44.024432+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:48.459883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3553,"num_write_kb":227392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":375,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017981+0000","last_change":"2024-04-06T03:39:31.921644+0000","last_active":"2024-04-06T03:39:44.017981+0000","last_peered":"2024-04-06T03:39:44.017981+0000","last_clean":"2024-04-06T03:39:44.017981+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:39:44.017981+0000","last_undegraded":"2024-04-06T03:39:44.017981+0000","last_fullsized":"2024-04-06T03:39:44.017981+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:31.921501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:31.921501+0000","last_clean_scrub_stamp":"2024-04-06T03:39:31.921501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:55.722354+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"224'32","reported_seq":560,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017059+0000","last_change":"2024-04-06T03:39:09.506926+0000","last_active":"2024-04-06T03:39:44.017059+0000","last_peered":"2024-04-06T03:39:44.017059+0000","last_clean":"2024-04-06T03:39:44.017059+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:39:44.017059+0000","last_undegraded":"2024-04-06T03:39:44.017059+0000","last_fullsized":"2024-04-06T03:39:44.017059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'32","last_scrub_stamp":"2024-04-06T03:39:09.506469+0000","last_deep_scrub":"224'32","last_deep_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_clean_scrub_stamp":"2024-04-06T03:39:09.506469+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.441786+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016358+0000","last_change":"2024-04-06T03:39:10.125890+0000","last_active":"2024-04-06T03:39:44.016358+0000","last_peered":"2024-04-06T03:39:44.016358+0000","last_clean":"2024-04-06T03:39:44.016358+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:39:44.016358+0000","last_undegraded":"2024-04-06T03:39:44.016358+0000","last_fullsized":"2024-04-06T03:39:44.016358+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:10.125595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:10.125595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.887097+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":773,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025514+0000","last_change":"2024-04-06T03:37:43.550112+0000","last_active":"2024-04-06T03:39:44.025514+0000","last_peered":"2024-04-06T03:39:44.025514+0000","last_clean":"2024-04-06T03:39:44.025514+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:39:44.025514+0000","last_undegraded":"2024-04-06T03:39:44.025514+0000","last_fullsized":"2024-04-06T03:39:44.025514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:37:43.550007+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:37:43.550007+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":514,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027287+0000","last_change":"2024-04-06T03:39:16.989351+0000","last_active":"2024-04-06T03:39:44.027287+0000","last_peered":"2024-04-06T03:39:44.027287+0000","last_clean":"2024-04-06T03:39:44.027287+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:39:44.027287+0000","last_undegraded":"2024-04-06T03:39:44.027287+0000","last_fullsized":"2024-04-06T03:39:44.027287+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:16.989269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:39:16.989269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:40.769757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"294'3578","reported_seq":3616,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:46.396059+0000","last_change":"2024-04-06T03:38:40.985375+0000","last_active":"2024-04-06T03:39:46.396059+0000","last_peered":"2024-04-06T03:39:46.396059+0000","last_clean":"2024-04-06T03:39:46.396059+0000","last_became_active":"2024-04-06T03:38:40.984476+0000","last_became_peered":"2024-04-06T03:38:40.984476+0000","last_unstale":"2024-04-06T03:39:46.396059+0000","last_undegraded":"2024-04-06T03:39:46.396059+0000","last_fullsized":"2024-04-06T03:39:46.396059+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3578,"log_dups_size":0,"ondisk_log_size":3578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:53.683760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234487808,"num_objects":3578,"num_object_clones":0,"num_object_copies":10734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":228992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":517,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024434+0000","last_change":"2024-04-06T03:39:01.759596+0000","last_active":"2024-04-06T03:39:44.024434+0000","last_peered":"2024-04-06T03:39:44.024434+0000","last_clean":"2024-04-06T03:39:44.024434+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:39:44.024434+0000","last_undegraded":"2024-04-06T03:39:44.024434+0000","last_fullsized":"2024-04-06T03:39:44.024434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:01.759468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:37.513299+0000","last_clean_scrub_stamp":"2024-04-06T03:39:01.759468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:03.078805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"199'35","reported_seq":573,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027378+0000","last_change":"2024-04-06T03:38:30.725252+0000","last_active":"2024-04-06T03:39:44.027378+0000","last_peered":"2024-04-06T03:39:44.027378+0000","last_clean":"2024-04-06T03:39:44.027378+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:39:44.027378+0000","last_undegraded":"2024-04-06T03:39:44.027378+0000","last_fullsized":"2024-04-06T03:39:44.027378+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'35","last_scrub_stamp":"2024-04-06T03:38:30.724996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:30.724996+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:47.225132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":514,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025387+0000","last_change":"2024-04-06T03:38:41.076829+0000","last_active":"2024-04-06T03:39:44.025387+0000","last_peered":"2024-04-06T03:39:44.025387+0000","last_clean":"2024-04-06T03:39:44.025387+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:39:44.025387+0000","last_undegraded":"2024-04-06T03:39:44.025387+0000","last_fullsized":"2024-04-06T03:39:44.025387+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:41.076607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:41.076607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:45.452751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":389,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024517+0000","last_change":"2024-04-06T03:38:35.753231+0000","last_active":"2024-04-06T03:39:44.024517+0000","last_peered":"2024-04-06T03:39:44.024517+0000","last_clean":"2024-04-06T03:39:44.024517+0000","last_became_active":"2024-04-06T03:25:54.492757+0000","last_became_peered":"2024-04-06T03:25:54.492757+0000","last_unstale":"2024-04-06T03:39:44.024517+0000","last_undegraded":"2024-04-06T03:39:44.024517+0000","last_fullsized":"2024-04-06T03:39:44.024517+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:35.752989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_clean_scrub_stamp":"2024-04-06T03:38:35.752989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:50.130112+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"294'3326","reported_seq":3363,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.201261+0000","last_change":"2024-04-06T03:38:41.239398+0000","last_active":"2024-04-06T03:39:44.201261+0000","last_peered":"2024-04-06T03:39:44.201261+0000","last_clean":"2024-04-06T03:39:44.201261+0000","last_became_active":"2024-04-06T03:38:41.238215+0000","last_became_peered":"2024-04-06T03:38:41.238215+0000","last_unstale":"2024-04-06T03:39:44.201261+0000","last_undegraded":"2024-04-06T03:39:44.201261+0000","last_fullsized":"2024-04-06T03:39:44.201261+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:45.918866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217972736,"num_objects":3326,"num_object_clones":0,"num_object_copies":9978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":212864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":518,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024182+0000","last_change":"2024-04-06T03:38:56.770480+0000","last_active":"2024-04-06T03:39:44.024182+0000","last_peered":"2024-04-06T03:39:44.024182+0000","last_clean":"2024-04-06T03:39:44.024182+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:39:44.024182+0000","last_undegraded":"2024-04-06T03:39:44.024182+0000","last_fullsized":"2024-04-06T03:39:44.024182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_clean_scrub_stamp":"2024-04-06T03:38:56.770310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.899370+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":522,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026591+0000","last_change":"2024-04-06T03:39:12.124333+0000","last_active":"2024-04-06T03:39:44.026591+0000","last_peered":"2024-04-06T03:39:44.026591+0000","last_clean":"2024-04-06T03:39:44.026591+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:39:44.026591+0000","last_undegraded":"2024-04-06T03:39:44.026591+0000","last_fullsized":"2024-04-06T03:39:44.026591+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:12.111707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:39:12.111707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.440876+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"251'51","reported_seq":722,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017019+0000","last_change":"2024-04-06T03:38:57.125352+0000","last_active":"2024-04-06T03:39:44.017019+0000","last_peered":"2024-04-06T03:39:44.017019+0000","last_clean":"2024-04-06T03:39:44.017019+0000","last_became_active":"2024-04-06T03:35:56.084984+0000","last_became_peered":"2024-04-06T03:35:56.084984+0000","last_unstale":"2024-04-06T03:39:44.017019+0000","last_undegraded":"2024-04-06T03:39:44.017019+0000","last_fullsized":"2024-04-06T03:39:44.017019+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'51","last_scrub_stamp":"2024-04-06T03:38:57.125241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.125241+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:15.087256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":385,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027802+0000","last_change":"2024-04-06T03:39:32.313014+0000","last_active":"2024-04-06T03:39:44.027802+0000","last_peered":"2024-04-06T03:39:44.027802+0000","last_clean":"2024-04-06T03:39:44.027802+0000","last_became_active":"2024-04-06T03:26:18.643845+0000","last_became_peered":"2024-04-06T03:26:18.643845+0000","last_unstale":"2024-04-06T03:39:44.027802+0000","last_undegraded":"2024-04-06T03:39:44.027802+0000","last_fullsized":"2024-04-06T03:39:44.027802+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:32.312900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:32.312900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:43.961384+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"294'3404","reported_seq":3442,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.204347+0000","last_change":"2024-04-06T03:38:41.240014+0000","last_active":"2024-04-06T03:39:44.204347+0000","last_peered":"2024-04-06T03:39:44.204347+0000","last_clean":"2024-04-06T03:39:44.204347+0000","last_became_active":"2024-04-06T03:38:41.239246+0000","last_became_peered":"2024-04-06T03:38:41.239246+0000","last_unstale":"2024-04-06T03:39:44.204347+0000","last_undegraded":"2024-04-06T03:39:44.204347+0000","last_fullsized":"2024-04-06T03:39:44.204347+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3404,"log_dups_size":0,"ondisk_log_size":3404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:03.574043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223084544,"num_objects":3404,"num_object_clones":0,"num_object_copies":10212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3404,"num_write_kb":217856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":512,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883258+0000","last_change":"2024-04-06T03:39:33.010891+0000","last_active":"2024-04-06T03:39:43.883258+0000","last_peered":"2024-04-06T03:39:43.883258+0000","last_clean":"2024-04-06T03:39:43.883258+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:39:43.883258+0000","last_undegraded":"2024-04-06T03:39:43.883258+0000","last_fullsized":"2024-04-06T03:39:43.883258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:33.010635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:39:33.010635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.375211+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":530,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024656+0000","last_change":"2024-04-06T03:38:38.749440+0000","last_active":"2024-04-06T03:39:44.024656+0000","last_peered":"2024-04-06T03:39:44.024656+0000","last_clean":"2024-04-06T03:39:44.024656+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:39:44.024656+0000","last_undegraded":"2024-04-06T03:39:44.024656+0000","last_fullsized":"2024-04-06T03:39:44.024656+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:38:38.749329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:38.749329+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:05.260691+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"199'23","reported_seq":556,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017608+0000","last_change":"2024-04-06T03:39:06.307894+0000","last_active":"2024-04-06T03:39:44.017608+0000","last_peered":"2024-04-06T03:39:44.017608+0000","last_clean":"2024-04-06T03:39:44.017608+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:39:44.017608+0000","last_undegraded":"2024-04-06T03:39:44.017608+0000","last_fullsized":"2024-04-06T03:39:44.017608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'23","last_scrub_stamp":"2024-04-06T03:39:06.307786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:06.307786+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.806563+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1626,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:45.125650+0000","last_change":"2024-04-06T03:39:14.550550+0000","last_active":"2024-04-06T03:39:45.125650+0000","last_peered":"2024-04-06T03:39:45.125650+0000","last_clean":"2024-04-06T03:39:45.125650+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:39:45.125650+0000","last_undegraded":"2024-04-06T03:39:45.125650+0000","last_fullsized":"2024-04-06T03:39:45.125650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:39:14.550411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:39:14.550411+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.362228+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1106,"num_read_kb":1101,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":516,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883253+0000","last_change":"2024-04-06T03:39:35.943774+0000","last_active":"2024-04-06T03:39:43.883253+0000","last_peered":"2024-04-06T03:39:43.883253+0000","last_clean":"2024-04-06T03:39:43.883253+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:39:43.883253+0000","last_undegraded":"2024-04-06T03:39:43.883253+0000","last_fullsized":"2024-04-06T03:39:43.883253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_clean_scrub_stamp":"2024-04-06T03:39:35.943511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:44.828686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"294'3602","reported_seq":3640,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:46.939531+0000","last_change":"2024-04-06T03:38:41.242596+0000","last_active":"2024-04-06T03:39:46.939531+0000","last_peered":"2024-04-06T03:39:46.939531+0000","last_clean":"2024-04-06T03:39:46.939531+0000","last_became_active":"2024-04-06T03:38:41.242148+0000","last_became_peered":"2024-04-06T03:38:41.242148+0000","last_unstale":"2024-04-06T03:39:46.939531+0000","last_undegraded":"2024-04-06T03:39:46.939531+0000","last_fullsized":"2024-04-06T03:39:46.939531+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:06.294118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236060672,"num_objects":3602,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3602,"num_write_kb":230528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":502,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025914+0000","last_change":"2024-04-06T03:37:31.563861+0000","last_active":"2024-04-06T03:39:44.025914+0000","last_peered":"2024-04-06T03:39:44.025914+0000","last_clean":"2024-04-06T03:39:44.025914+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:39:44.025914+0000","last_undegraded":"2024-04-06T03:39:44.025914+0000","last_fullsized":"2024-04-06T03:39:44.025914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:37:31.563716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":376,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.028169+0000","last_change":"2024-04-06T03:37:34.532858+0000","last_active":"2024-04-06T03:39:44.028169+0000","last_peered":"2024-04-06T03:39:44.028169+0000","last_clean":"2024-04-06T03:39:44.028169+0000","last_became_active":"2024-04-06T03:26:18.621727+0000","last_became_peered":"2024-04-06T03:26:18.621727+0000","last_unstale":"2024-04-06T03:39:44.028169+0000","last_undegraded":"2024-04-06T03:39:44.028169+0000","last_fullsized":"2024-04-06T03:39:44.028169+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:34.532778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:37:34.532778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"199'24","reported_seq":553,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025557+0000","last_change":"2024-04-06T03:38:31.679871+0000","last_active":"2024-04-06T03:39:44.025557+0000","last_peered":"2024-04-06T03:39:44.025557+0000","last_clean":"2024-04-06T03:39:44.025557+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:39:44.025557+0000","last_undegraded":"2024-04-06T03:39:44.025557+0000","last_fullsized":"2024-04-06T03:39:44.025557+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:38:31.679737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:31.679737+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:49.326094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":525,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883370+0000","last_change":"2024-04-06T03:39:25.012594+0000","last_active":"2024-04-06T03:39:43.883370+0000","last_peered":"2024-04-06T03:39:43.883370+0000","last_clean":"2024-04-06T03:39:43.883370+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:39:43.883370+0000","last_undegraded":"2024-04-06T03:39:43.883370+0000","last_fullsized":"2024-04-06T03:39:43.883370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:25.012444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:25.012444+0000","last_clean_scrub_stamp":"2024-04-06T03:39:25.012444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:43.525977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":746,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.017405+0000","last_change":"2024-04-06T03:38:42.258790+0000","last_active":"2024-04-06T03:39:44.017405+0000","last_peered":"2024-04-06T03:39:44.017405+0000","last_clean":"2024-04-06T03:39:44.017405+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:39:44.017405+0000","last_undegraded":"2024-04-06T03:39:44.017405+0000","last_fullsized":"2024-04-06T03:39:44.017405+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:38:42.258672+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:38:42.258672+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:07.507957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"294'3454","reported_seq":3492,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.214209+0000","last_change":"2024-04-06T03:38:41.241641+0000","last_active":"2024-04-06T03:39:44.214209+0000","last_peered":"2024-04-06T03:39:44.214209+0000","last_clean":"2024-04-06T03:39:44.214209+0000","last_became_active":"2024-04-06T03:38:41.240235+0000","last_became_peered":"2024-04-06T03:38:41.240235+0000","last_unstale":"2024-04-06T03:39:44.214209+0000","last_undegraded":"2024-04-06T03:39:44.214209+0000","last_fullsized":"2024-04-06T03:39:44.214209+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3454,"log_dups_size":0,"ondisk_log_size":3454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:00.584498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226361344,"num_objects":3454,"num_object_clones":0,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3454,"num_write_kb":221056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":503,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.019261+0000","last_change":"2024-04-06T03:39:29.642108+0000","last_active":"2024-04-06T03:39:44.019261+0000","last_peered":"2024-04-06T03:39:44.019261+0000","last_clean":"2024-04-06T03:39:44.019261+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:39:44.019261+0000","last_undegraded":"2024-04-06T03:39:44.019261+0000","last_fullsized":"2024-04-06T03:39:44.019261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:29.641865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:39:29.641865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:55.761400+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":382,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883775+0000","last_change":"2024-04-06T03:39:11.966576+0000","last_active":"2024-04-06T03:39:43.883775+0000","last_peered":"2024-04-06T03:39:43.883775+0000","last_clean":"2024-04-06T03:39:43.883775+0000","last_became_active":"2024-04-06T03:25:54.491253+0000","last_became_peered":"2024-04-06T03:25:54.491253+0000","last_unstale":"2024-04-06T03:39:43.883775+0000","last_undegraded":"2024-04-06T03:39:43.883775+0000","last_fullsized":"2024-04-06T03:39:43.883775+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.966446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:38.797317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":513,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025649+0000","last_change":"2024-04-06T03:38:58.011298+0000","last_active":"2024-04-06T03:39:44.025649+0000","last_peered":"2024-04-06T03:39:44.025649+0000","last_clean":"2024-04-06T03:39:44.025649+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:39:44.025649+0000","last_undegraded":"2024-04-06T03:39:44.025649+0000","last_fullsized":"2024-04-06T03:39:44.025649+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:58.011077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:58.011077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:24.074597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"199'37","reported_seq":574,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883276+0000","last_change":"2024-04-06T03:38:37.957427+0000","last_active":"2024-04-06T03:39:43.883276+0000","last_peered":"2024-04-06T03:39:43.883276+0000","last_clean":"2024-04-06T03:39:43.883276+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:39:43.883276+0000","last_undegraded":"2024-04-06T03:39:43.883276+0000","last_fullsized":"2024-04-06T03:39:43.883276+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:38:37.957275+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.957275+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:43.925715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"294'3550","reported_seq":3588,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.194017+0000","last_change":"2024-04-06T03:38:40.976607+0000","last_active":"2024-04-06T03:39:44.194017+0000","last_peered":"2024-04-06T03:39:44.194017+0000","last_clean":"2024-04-06T03:39:44.194017+0000","last_became_active":"2024-04-06T03:38:40.976213+0000","last_became_peered":"2024-04-06T03:38:40.976213+0000","last_unstale":"2024-04-06T03:39:44.194017+0000","last_undegraded":"2024-04-06T03:39:44.194017+0000","last_fullsized":"2024-04-06T03:39:44.194017+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3550,"log_dups_size":0,"ondisk_log_size":3550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:48.955302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232652800,"num_objects":3550,"num_object_clones":0,"num_object_copies":10650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3550,"num_write_kb":227200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":341,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.029580+0000","last_change":"2024-04-06T03:38:58.987637+0000","last_active":"2024-04-06T03:39:44.029580+0000","last_peered":"2024-04-06T03:39:44.029580+0000","last_clean":"2024-04-06T03:39:44.029580+0000","last_became_active":"2024-04-06T03:27:10.497525+0000","last_became_peered":"2024-04-06T03:27:10.497525+0000","last_unstale":"2024-04-06T03:39:44.029580+0000","last_undegraded":"2024-04-06T03:39:44.029580+0000","last_fullsized":"2024-04-06T03:39:44.029580+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_clean_scrub_stamp":"2024-04-06T03:38:58.987355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:25.095281+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":753,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883785+0000","last_change":"2024-04-06T03:37:42.672606+0000","last_active":"2024-04-06T03:39:43.883785+0000","last_peered":"2024-04-06T03:39:43.883785+0000","last_clean":"2024-04-06T03:39:43.883785+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:39:43.883785+0000","last_undegraded":"2024-04-06T03:39:43.883785+0000","last_fullsized":"2024-04-06T03:39:43.883785+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:37:42.672471+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:35:33.269156+0000","last_clean_scrub_stamp":"2024-04-06T03:37:42.672471+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":529,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024044+0000","last_change":"2024-04-06T03:38:50.753603+0000","last_active":"2024-04-06T03:39:44.024044+0000","last_peered":"2024-04-06T03:39:44.024044+0000","last_clean":"2024-04-06T03:39:44.024044+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:39:44.024044+0000","last_undegraded":"2024-04-06T03:39:44.024044+0000","last_fullsized":"2024-04-06T03:39:44.024044+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:50.753393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:50.753393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:51.859862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"199'39","reported_seq":578,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016673+0000","last_change":"2024-04-06T03:39:15.520741+0000","last_active":"2024-04-06T03:39:44.016673+0000","last_peered":"2024-04-06T03:39:44.016673+0000","last_clean":"2024-04-06T03:39:44.016673+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:39:44.016673+0000","last_undegraded":"2024-04-06T03:39:44.016673+0000","last_fullsized":"2024-04-06T03:39:44.016673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:39:15.520549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:15.520549+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.831610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"294'3446","reported_seq":3484,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025962+0000","last_change":"2024-04-06T03:38:40.984725+0000","last_active":"2024-04-06T03:39:44.025962+0000","last_peered":"2024-04-06T03:39:44.025962+0000","last_clean":"2024-04-06T03:39:44.025962+0000","last_became_active":"2024-04-06T03:38:40.984168+0000","last_became_peered":"2024-04-06T03:38:40.984168+0000","last_unstale":"2024-04-06T03:39:44.025962+0000","last_undegraded":"2024-04-06T03:39:44.025962+0000","last_fullsized":"2024-04-06T03:39:44.025962+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3446,"log_dups_size":0,"ondisk_log_size":3446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225837056,"num_objects":3446,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3446,"num_write_kb":220544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":511,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883910+0000","last_change":"2024-04-06T03:39:22.029484+0000","last_active":"2024-04-06T03:39:43.883910+0000","last_peered":"2024-04-06T03:39:43.883910+0000","last_clean":"2024-04-06T03:39:43.883910+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:39:43.883910+0000","last_undegraded":"2024-04-06T03:39:43.883910+0000","last_fullsized":"2024-04-06T03:39:43.883910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:22.029278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:22.029278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:44.736300+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":769,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016088+0000","last_change":"2024-04-06T03:38:40.981295+0000","last_active":"2024-04-06T03:39:44.016088+0000","last_peered":"2024-04-06T03:39:44.016088+0000","last_clean":"2024-04-06T03:39:44.016088+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:39:44.016088+0000","last_undegraded":"2024-04-06T03:39:44.016088+0000","last_fullsized":"2024-04-06T03:39:44.016088+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:38:40.981122+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:38:40.981122+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:53.999742+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":392,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016944+0000","last_change":"2024-04-06T03:38:36.268944+0000","last_active":"2024-04-06T03:39:44.016944+0000","last_peered":"2024-04-06T03:39:44.016944+0000","last_clean":"2024-04-06T03:39:44.016944+0000","last_became_active":"2024-04-06T03:25:19.549690+0000","last_became_peered":"2024-04-06T03:25:19.549690+0000","last_unstale":"2024-04-06T03:39:44.016944+0000","last_undegraded":"2024-04-06T03:39:44.016944+0000","last_fullsized":"2024-04-06T03:39:44.016944+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:36.268788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:23.908290+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.268788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:48.982681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"199'21","reported_seq":548,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016972+0000","last_change":"2024-04-06T03:38:57.282224+0000","last_active":"2024-04-06T03:39:44.016972+0000","last_peered":"2024-04-06T03:39:44.016972+0000","last_clean":"2024-04-06T03:39:44.016972+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:39:44.016972+0000","last_undegraded":"2024-04-06T03:39:44.016972+0000","last_fullsized":"2024-04-06T03:39:44.016972+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:38:57.282016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.282016+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:05.356143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"294'3488","reported_seq":3526,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.737066+0000","last_change":"2024-04-06T03:38:40.975757+0000","last_active":"2024-04-06T03:39:44.737066+0000","last_peered":"2024-04-06T03:39:44.737066+0000","last_clean":"2024-04-06T03:39:44.737066+0000","last_became_active":"2024-04-06T03:38:40.974964+0000","last_became_peered":"2024-04-06T03:38:40.974964+0000","last_unstale":"2024-04-06T03:39:44.737066+0000","last_undegraded":"2024-04-06T03:39:44.737066+0000","last_fullsized":"2024-04-06T03:39:44.737066+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:59.438135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228589568,"num_objects":3488,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3488,"num_write_kb":223232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":527,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.018935+0000","last_change":"2024-04-06T03:38:35.085842+0000","last_active":"2024-04-06T03:39:44.018935+0000","last_peered":"2024-04-06T03:39:44.018935+0000","last_clean":"2024-04-06T03:39:44.018935+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:39:44.018935+0000","last_undegraded":"2024-04-06T03:39:44.018935+0000","last_fullsized":"2024-04-06T03:39:44.018935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:35.085716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:38:35.085716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:51.398525+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":338,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883962+0000","last_change":"2024-04-06T03:38:59.034375+0000","last_active":"2024-04-06T03:39:43.883962+0000","last_peered":"2024-04-06T03:39:43.883962+0000","last_clean":"2024-04-06T03:39:43.883962+0000","last_became_active":"2024-04-06T03:27:10.501454+0000","last_became_peered":"2024-04-06T03:27:10.501454+0000","last_unstale":"2024-04-06T03:39:43.883962+0000","last_undegraded":"2024-04-06T03:39:43.883962+0000","last_fullsized":"2024-04-06T03:39:43.883962+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:59.034078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:59.034078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.898507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":521,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026045+0000","last_change":"2024-04-06T03:39:27.906651+0000","last_active":"2024-04-06T03:39:44.026045+0000","last_peered":"2024-04-06T03:39:44.026045+0000","last_clean":"2024-04-06T03:39:44.026045+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:39:44.026045+0000","last_undegraded":"2024-04-06T03:39:44.026045+0000","last_fullsized":"2024-04-06T03:39:44.026045+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:39:27.906531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:36.128418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"199'41","reported_seq":470,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:43.883538+0000","last_change":"2024-04-06T03:39:20.965683+0000","last_active":"2024-04-06T03:39:43.883538+0000","last_peered":"2024-04-06T03:39:43.883538+0000","last_clean":"2024-04-06T03:39:43.883538+0000","last_became_active":"2024-04-06T03:26:18.616944+0000","last_became_peered":"2024-04-06T03:26:18.616944+0000","last_unstale":"2024-04-06T03:39:43.883538+0000","last_undegraded":"2024-04-06T03:39:43.883538+0000","last_fullsized":"2024-04-06T03:39:43.883538+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'41","last_scrub_stamp":"2024-04-06T03:39:20.965523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:20.965523+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.036992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"294'3613","reported_seq":3651,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:46.408805+0000","last_change":"2024-04-06T03:38:40.977014+0000","last_active":"2024-04-06T03:39:46.408805+0000","last_peered":"2024-04-06T03:39:46.408805+0000","last_clean":"2024-04-06T03:39:46.408805+0000","last_became_active":"2024-04-06T03:38:40.976105+0000","last_became_peered":"2024-04-06T03:38:40.976105+0000","last_unstale":"2024-04-06T03:39:46.408805+0000","last_undegraded":"2024-04-06T03:39:46.408805+0000","last_fullsized":"2024-04-06T03:39:46.408805+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236781568,"num_objects":3613,"num_object_clones":0,"num_object_copies":10839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3613,"num_write_kb":231232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":501,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.025102+0000","last_change":"2024-04-06T03:39:10.006464+0000","last_active":"2024-04-06T03:39:44.025102+0000","last_peered":"2024-04-06T03:39:44.025102+0000","last_clean":"2024-04-06T03:39:44.025102+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:39:44.025102+0000","last_undegraded":"2024-04-06T03:39:44.025102+0000","last_fullsized":"2024-04-06T03:39:44.025102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:10.006346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:10.006346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:15.094954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":531,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016197+0000","last_change":"2024-04-06T03:39:09.163498+0000","last_active":"2024-04-06T03:39:44.016197+0000","last_peered":"2024-04-06T03:39:44.016197+0000","last_clean":"2024-04-06T03:39:44.016197+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:39:44.016197+0000","last_undegraded":"2024-04-06T03:39:44.016197+0000","last_fullsized":"2024-04-06T03:39:44.016197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:09.163381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:39:09.163381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:28.061368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":531,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.016919+0000","last_change":"2024-04-06T03:38:36.976257+0000","last_active":"2024-04-06T03:39:44.016919+0000","last_peered":"2024-04-06T03:39:44.016919+0000","last_clean":"2024-04-06T03:39:44.016919+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:39:44.016919+0000","last_undegraded":"2024-04-06T03:39:44.016919+0000","last_fullsized":"2024-04-06T03:39:44.016919+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:36.976013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.976013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:52.675803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"199'39","reported_seq":566,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027647+0000","last_change":"2024-04-06T03:38:45.009952+0000","last_active":"2024-04-06T03:39:44.027647+0000","last_peered":"2024-04-06T03:39:44.027647+0000","last_clean":"2024-04-06T03:39:44.027647+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:39:44.027647+0000","last_undegraded":"2024-04-06T03:39:44.027647+0000","last_fullsized":"2024-04-06T03:39:44.027647+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:38:45.009705+0000","last_deep_scrub":"199'39","last_deep_scrub_stamp":"2024-04-06T03:35:49.576918+0000","last_clean_scrub_stamp":"2024-04-06T03:38:45.009705+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:51.339059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"294'3499","reported_seq":3537,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.716936+0000","last_change":"2024-04-06T03:38:40.982609+0000","last_active":"2024-04-06T03:39:44.716936+0000","last_peered":"2024-04-06T03:39:44.716936+0000","last_clean":"2024-04-06T03:39:44.716936+0000","last_became_active":"2024-04-06T03:38:40.981962+0000","last_became_peered":"2024-04-06T03:38:40.981962+0000","last_unstale":"2024-04-06T03:39:44.716936+0000","last_undegraded":"2024-04-06T03:39:44.716936+0000","last_fullsized":"2024-04-06T03:39:44.716936+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3499,"log_dups_size":0,"ondisk_log_size":3499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:50.823731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229310464,"num_objects":3499,"num_object_clones":0,"num_object_copies":10497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3499,"num_write_kb":223936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":509,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027493+0000","last_change":"2024-04-06T03:38:59.917478+0000","last_active":"2024-04-06T03:39:44.027493+0000","last_peered":"2024-04-06T03:39:44.027493+0000","last_clean":"2024-04-06T03:39:44.027493+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:39:44.027493+0000","last_undegraded":"2024-04-06T03:39:44.027493+0000","last_fullsized":"2024-04-06T03:39:44.027493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:59.917290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:59.917290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:17.591600+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":531,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.015609+0000","last_change":"2024-04-06T03:39:11.097645+0000","last_active":"2024-04-06T03:39:44.015609+0000","last_peered":"2024-04-06T03:39:44.015609+0000","last_clean":"2024-04-06T03:39:44.015609+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:39:44.015609+0000","last_undegraded":"2024-04-06T03:39:44.015609+0000","last_fullsized":"2024-04-06T03:39:44.015609+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.097314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:17.929540+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":537,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.024737+0000","last_change":"2024-04-06T03:39:13.054289+0000","last_active":"2024-04-06T03:39:44.024737+0000","last_peered":"2024-04-06T03:39:44.024737+0000","last_clean":"2024-04-06T03:39:44.024737+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:39:44.024737+0000","last_undegraded":"2024-04-06T03:39:44.024737+0000","last_fullsized":"2024-04-06T03:39:44.024737+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:39:13.053965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:30.746261+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"199'25","reported_seq":524,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.018044+0000","last_change":"2024-04-06T03:39:25.599044+0000","last_active":"2024-04-06T03:39:44.018044+0000","last_peered":"2024-04-06T03:39:44.018044+0000","last_clean":"2024-04-06T03:39:44.018044+0000","last_became_active":"2024-04-06T03:25:54.495696+0000","last_became_peered":"2024-04-06T03:25:54.495696+0000","last_unstale":"2024-04-06T03:39:44.018044+0000","last_undegraded":"2024-04-06T03:39:44.018044+0000","last_fullsized":"2024-04-06T03:39:44.018044+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'25","last_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:39:25.598852+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:35.014424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"294'3689","reported_seq":3727,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:46.400025+0000","last_change":"2024-04-06T03:38:40.986276+0000","last_active":"2024-04-06T03:39:46.400025+0000","last_peered":"2024-04-06T03:39:46.400025+0000","last_clean":"2024-04-06T03:39:46.400025+0000","last_became_active":"2024-04-06T03:38:40.985850+0000","last_became_peered":"2024-04-06T03:38:40.985850+0000","last_unstale":"2024-04-06T03:39:46.400025+0000","last_undegraded":"2024-04-06T03:39:46.400025+0000","last_fullsized":"2024-04-06T03:39:46.400025+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:39:53.289278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241762304,"num_objects":3689,"num_object_clones":0,"num_object_copies":11067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":236096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":380,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.027627+0000","last_change":"2024-04-06T03:39:21.333266+0000","last_active":"2024-04-06T03:39:44.027627+0000","last_peered":"2024-04-06T03:39:44.027627+0000","last_clean":"2024-04-06T03:39:44.027627+0000","last_became_active":"2024-04-06T03:26:18.641975+0000","last_became_peered":"2024-04-06T03:26:18.641975+0000","last_unstale":"2024-04-06T03:39:44.027627+0000","last_undegraded":"2024-04-06T03:39:44.027627+0000","last_fullsized":"2024-04-06T03:39:44.027627+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:21.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:39:21.333139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:39.938998+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":539,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.020839+0000","last_change":"2024-04-06T03:38:41.932988+0000","last_active":"2024-04-06T03:39:44.020839+0000","last_peered":"2024-04-06T03:39:44.020839+0000","last_clean":"2024-04-06T03:39:44.020839+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:39:44.020839+0000","last_undegraded":"2024-04-06T03:39:44.020839+0000","last_fullsized":"2024-04-06T03:39:44.020839+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:41.932886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:38:41.932886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:04.861631+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"291'34","reported_seq":567,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.028516+0000","last_change":"2024-04-06T03:39:26.934856+0000","last_active":"2024-04-06T03:39:44.028516+0000","last_peered":"2024-04-06T03:39:44.028516+0000","last_clean":"2024-04-06T03:39:44.028516+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:39:44.028516+0000","last_undegraded":"2024-04-06T03:39:44.028516+0000","last_fullsized":"2024-04-06T03:39:44.028516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'34","last_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_deep_scrub":"291'34","last_deep_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_clean_scrub_stamp":"2024-04-06T03:39:26.934752+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.324584+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'12","reported_seq":751,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-04-06T03:39:44.026461+0000","last_change":"2024-04-06T03:38:38.151300+0000","last_active":"2024-04-06T03:39:44.026461+0000","last_peered":"2024-04-06T03:39:44.026461+0000","last_clean":"2024-04-06T03:39:44.026461+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:39:44.026461+0000","last_undegraded":"2024-04-06T03:39:44.026461+0000","last_fullsized":"2024-04-06T03:39:44.026461+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'12","last_scrub_stamp":"2024-04-06T03:38:38.151186+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:38:38.151186+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3705339904,"num_objects":56539,"num_object_clones":0,"num_object_copies":169617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56539,"num_write_kb":3618496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11078270976,"data_stored":11078270976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56539,"ondisk_log_size":56539,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1414,"num_read_kb":1139,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1002,"ondisk_log_size":1002,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1106,"num_read_kb":1101,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743661,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1247280,"kb_used_data":895396,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92497872,"statfs":{"total":95995035648,"available":94717820928,"internally_reserved":0,"allocated":916885504,"data_stored":916479445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136119,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1861548,"kb_used_data":1529124,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":91883604,"statfs":{"total":95995035648,"available":94088810496,"internally_reserved":0,"allocated":1565822976,"data_stored":1565472299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340392390},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575148,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052280,"kb_used_data":660144,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92692872,"statfs":{"total":95995035648,"available":94917500928,"internally_reserved":0,"allocated":675987456,"data_stored":675623801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738678,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2690192,"kb_used_data":2289732,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91054960,"statfs":{"total":95995035648,"available":93240279040,"internally_reserved":0,"allocated":2344685568,"data_stored":2344311252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378914,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1643132,"kb_used_data":1314864,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92102020,"statfs":{"total":95995035648,"available":94312468480,"internally_reserved":0,"allocated":1346420736,"data_stored":1346064901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084566,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1554164,"kb_used_data":1143976,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92190988,"statfs":{"total":95995035648,"available":94403571712,"internally_reserved":0,"allocated":1171431424,"data_stored":1171083695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855629,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2052640,"kb_used_data":1768788,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91692512,"statfs":{"total":95995035648,"available":93893132288,"internally_reserved":0,"allocated":1811238912,"data_stored":1810846698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495875,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501080,"kb_used_data":1100624,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92244072,"statfs":{"total":95995035648,"available":94457929728,"internally_reserved":0,"allocated":1127038976,"data_stored":1126707371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2375090176,"data_stored":2375090176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":683868160,"data_stored":683868160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1368391680,"data_stored":1368391680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1192099840,"data_stored":1192099840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1818886144,"data_stored":1818886144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1137049600,"data_stored":1137049600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1580466176,"data_stored":1580466176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":922419200,"data_stored":922419200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:39:53.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:39:53.329 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:39:53.370 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:39:53.409 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:53.417 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:39:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:53.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:53.458 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:39:53.490 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:39:53.514 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:39:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:39:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:39:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:39:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:39:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:39:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:39:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:39:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:39:53.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:53.543 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:53.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:53.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 49 16 64381 64365 82.0844 35.25 0.00816702 0.0121582 2024-04-06T03:39:53.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:53.672 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:53.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:53.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[21467]: cluster 2024-04-06T03:39:52.393467+0000 mgr.y (mgr.24368) 697 : cluster 0 pgmap v743: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[21467]: cluster 2024-04-06T03:39:52.850631+0000 osd.2 (osd.2) 369 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[21467]: cluster 2024-04-06T03:39:52.850703+0000 osd.2 (osd.2) 370 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[21467]: audit 2024-04-06T03:39:52.916015+0000 mon.c (mon.2) 91 : audit 0 from='client.? 172.21.15.50:0/1047648682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[29221]: cluster 2024-04-06T03:39:52.393467+0000 mgr.y (mgr.24368) 697 : cluster 0 pgmap v743: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[29221]: cluster 2024-04-06T03:39:52.850631+0000 osd.2 (osd.2) 369 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[29221]: cluster 2024-04-06T03:39:52.850703+0000 osd.2 (osd.2) 370 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:53.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:53 smithi050 bash[29221]: audit 2024-04-06T03:39:52.916015+0000 mon.c (mon.2) 91 : audit 0 from='client.? 172.21.15.50:0/1047648682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:39:53.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:53.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:53 smithi153 bash[23852]: cluster 2024-04-06T03:39:52.393467+0000 mgr.y (mgr.24368) 697 : cluster 0 pgmap v743: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s 2024-04-06T03:39:53.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:53 smithi153 bash[23852]: cluster 2024-04-06T03:39:52.850631+0000 osd.2 (osd.2) 369 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:53.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:53 smithi153 bash[23852]: cluster 2024-04-06T03:39:52.850703+0000 osd.2 (osd.2) 370 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:53.785 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:53 smithi153 bash[23852]: audit 2024-04-06T03:39:52.916015+0000 mon.c (mon.2) 91 : audit 0 from='client.? 172.21.15.50:0/1047648682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:39:53.810 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:53.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:53.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:53.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:53.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:54.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:54.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:54.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:54.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:54.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:54.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:54.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:54.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:54.556 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 50 16 65416 65400 81.7364 64.6875 0.0133288 0.0122285 2024-04-06T03:39:54.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:54.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:54 smithi153 bash[23852]: cluster 2024-04-06T03:39:53.467759+0000 mon.a (mon.0) 1522 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-04-06T03:39:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:54 smithi153 bash[23852]: cluster 2024-04-06T03:39:53.691482+0000 osd.0 (osd.0) 450 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:54 smithi153 bash[23852]: cluster 2024-04-06T03:39:53.691549+0000 osd.0 (osd.0) 451 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:54 smithi153 bash[23852]: cluster 2024-04-06T03:39:53.888589+0000 mon.a (mon.0) 1523 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-04-06T03:39:54.606 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:54.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:54.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:54.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:54.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:54.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:54.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[21467]: cluster 2024-04-06T03:39:53.467759+0000 mon.a (mon.0) 1522 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-04-06T03:39:54.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[21467]: cluster 2024-04-06T03:39:53.691482+0000 osd.0 (osd.0) 450 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:54.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[21467]: cluster 2024-04-06T03:39:53.691549+0000 osd.0 (osd.0) 451 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:54.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[21467]: cluster 2024-04-06T03:39:53.888589+0000 mon.a (mon.0) 1523 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-04-06T03:39:54.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[29221]: cluster 2024-04-06T03:39:53.467759+0000 mon.a (mon.0) 1522 : cluster 0 osdmap e297: 8 total, 8 up, 8 in 2024-04-06T03:39:54.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[29221]: cluster 2024-04-06T03:39:53.691482+0000 osd.0 (osd.0) 450 : cluster 0 purged_snaps scrub starts 2024-04-06T03:39:54.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[29221]: cluster 2024-04-06T03:39:53.691549+0000 osd.0 (osd.0) 451 : cluster 0 purged_snaps scrub ok 2024-04-06T03:39:54.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:54 smithi050 bash[29221]: cluster 2024-04-06T03:39:53.888589+0000 mon.a (mon.0) 1523 : cluster 0 osdmap e298: 8 total, 8 up, 8 in 2024-04-06T03:39:54.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:55.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:55.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:55.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:55.145 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:55.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:55.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:55.274 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:55.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:55.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:55.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:55.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:55.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:55.612 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 51 16 66583 66567 81.5 72.9375 0.00472636 0.0122431 2024-04-06T03:39:55.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:55 smithi050 bash[21467]: cluster 2024-04-06T03:39:54.394042+0000 mgr.y (mgr.24368) 698 : cluster 0 pgmap v746: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-04-06T03:39:55.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:55 smithi050 bash[29221]: cluster 2024-04-06T03:39:54.394042+0000 mgr.y (mgr.24368) 698 : cluster 0 pgmap v746: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-04-06T03:39:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:55 smithi153 bash[23852]: cluster 2024-04-06T03:39:54.394042+0000 mgr.y (mgr.24368) 698 : cluster 0 pgmap v746: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-04-06T03:39:55.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:55.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:55.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:56.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:56.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:56.135 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:56.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:56.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:56.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:56.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:39:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:39:56.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:56.535 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:56.612 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 52 16 67218 67202 80.672 39.6875 0.0053442 0.0123885 2024-04-06T03:39:56.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:56.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:56.803 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:56.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:56.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:56.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:57.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:57.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:57.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:57.166 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:57.189 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:57.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:57.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:57.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:57.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:57.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:57.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:39:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:57.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:39:57.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:57.586 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:57.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:57.612 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 53 16 68229 68213 80.3424 63.1875 0.0168355 0.0124412 2024-04-06T03:39:57.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:57.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:57 smithi050 bash[21467]: cluster 2024-04-06T03:39:56.394704+0000 mgr.y (mgr.24368) 699 : cluster 0 pgmap v747: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-06T03:39:57.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:57 smithi050 bash[29221]: cluster 2024-04-06T03:39:56.394704+0000 mgr.y (mgr.24368) 699 : cluster 0 pgmap v747: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-06T03:39:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:57 smithi153 bash[23852]: cluster 2024-04-06T03:39:56.394704+0000 mgr.y (mgr.24368) 699 : cluster 0 pgmap v747: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-04-06T03:39:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:57.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:58.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:58.091 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:58.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:58.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:58.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:58.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:58.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:58.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:58.498 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:58.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:58.612 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 54 15 69334 69319 80.1348 69.125 0.0100402 0.0124737 2024-04-06T03:39:58.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:58.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:58.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:58.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:58.908 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:39:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:58.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:39:59.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:59.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:39:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:59.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:39:59.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:39:59.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:39:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:59.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:39:59.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:39:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:59.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:39:59.520 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:39:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:59.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:39:59.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:59.747 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 55 16 70329 70313 79.8076 62.125 0.00857339 0.012525 2024-04-06T03:39:59.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:59 smithi050 bash[21467]: cluster 2024-04-06T03:39:58.395312+0000 mgr.y (mgr.24368) 700 : cluster 0 pgmap v748: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s 2024-04-06T03:39:59.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:39:59 smithi050 bash[21467]: cluster 2024-04-06T03:39:58.892318+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-04-06T03:39:59.748 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:39:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:39:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:39:59.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:59 smithi050 bash[29221]: cluster 2024-04-06T03:39:58.395312+0000 mgr.y (mgr.24368) 700 : cluster 0 pgmap v748: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s 2024-04-06T03:39:59.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:39:59 smithi050 bash[29221]: cluster 2024-04-06T03:39:58.892318+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-04-06T03:39:59.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:59 smithi153 bash[23852]: cluster 2024-04-06T03:39:58.395312+0000 mgr.y (mgr.24368) 700 : cluster 0 pgmap v748: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s 2024-04-06T03:39:59.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:39:59 smithi153 bash[23852]: cluster 2024-04-06T03:39:58.892318+0000 mon.a (mon.0) 1524 : cluster 0 osdmap e299: 8 total, 8 up, 8 in 2024-04-06T03:39:59.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:59.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:39:59.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:59.984 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:39:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:39:59.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:00.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:00.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:00.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:00.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:00.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:39:59 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:40:00.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:00.360 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:00.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:00.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:00.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:00.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 56 16 71018 71002 79.152 43.0625 0.00905572 0.0126291 2024-04-06T03:40:00.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:00 smithi153 bash[23852]: cluster 2024-04-06T03:40:00.000166+0000 mon.a (mon.0) 1525 : cluster 3 Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-06T03:40:00.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:00 smithi153 bash[23852]: cluster 2024-04-06T03:40:00.000191+0000 mon.a (mon.0) 1526 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-06T03:40:00.637 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:00.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:00 smithi050 bash[21467]: cluster 2024-04-06T03:40:00.000166+0000 mon.a (mon.0) 1525 : cluster 3 Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-06T03:40:00.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:00 smithi050 bash[21467]: cluster 2024-04-06T03:40:00.000191+0000 mon.a (mon.0) 1526 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-06T03:40:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:00 smithi050 bash[29221]: cluster 2024-04-06T03:40:00.000166+0000 mon.a (mon.0) 1525 : cluster 3 Health detail: HEALTH_WARN noscrub flag(s) set 2024-04-06T03:40:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:00 smithi050 bash[29221]: cluster 2024-04-06T03:40:00.000191+0000 mon.a (mon.0) 1526 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-04-06T03:40:00.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:00.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:00.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:00.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:00.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:01.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:01.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:01.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:01.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:01.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:01.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:01.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:01.377 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:01.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:01.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:01.498 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:01.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:01.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:01.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 57 16 71967 71951 78.8043 59.3125 0.0167895 0.0126848 2024-04-06T03:40:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:01 smithi153 bash[23852]: audit 2024-04-06T03:39:59.953751+0000 mgr.y (mgr.24368) 701 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:01.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:01 smithi153 bash[23852]: cluster 2024-04-06T03:40:00.396047+0000 mgr.y (mgr.24368) 702 : cluster 0 pgmap v750: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 738 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:40:01.926 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:01.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:01 smithi050 bash[21467]: audit 2024-04-06T03:39:59.953751+0000 mgr.y (mgr.24368) 701 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:01.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:01 smithi050 bash[21467]: cluster 2024-04-06T03:40:00.396047+0000 mgr.y (mgr.24368) 702 : cluster 0 pgmap v750: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 738 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:40:01.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:01 smithi050 bash[29221]: audit 2024-04-06T03:39:59.953751+0000 mgr.y (mgr.24368) 701 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:01.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:01 smithi050 bash[29221]: cluster 2024-04-06T03:40:00.396047+0000 mgr.y (mgr.24368) 702 : cluster 0 pgmap v750: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 738 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:40:01.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:02.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:02.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:02.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:02.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:02.356 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:02.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 58 16 72655 72639 78.1875 43 0.0132214 0.0127852 2024-04-06T03:40:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:03.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:03.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:03.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:03.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:03.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:03.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:03.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:03.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:03.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 59 16 73484 73468 77.7408 51.8125 0.00704334 0.0128491 2024-04-06T03:40:03.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:03 smithi050 bash[21467]: cluster 2024-04-06T03:40:02.399390+0000 mgr.y (mgr.24368) 703 : cluster 0 pgmap v751: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-06T03:40:03.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:03 smithi050 bash[29221]: cluster 2024-04-06T03:40:02.399390+0000 mgr.y (mgr.24368) 703 : cluster 0 pgmap v751: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-06T03:40:03.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:03.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:03.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:03 smithi153 bash[23852]: cluster 2024-04-06T03:40:02.399390+0000 mgr.y (mgr.24368) 703 : cluster 0 pgmap v751: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-06T03:40:03.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:03.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:03.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:03.986 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:03.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:04.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:04.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:04.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:04.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:04.324 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:04.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:04.383 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:04.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:04.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:40:04.616607+0000 min lat: 0.0011709 max lat: 0.652534 avg lat: 0.012894 2024-04-06T03:40:04.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:40:04.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 60 16 74342 74326 77.3392 53.625 0.0137317 0.012894 2024-04-06T03:40:04.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:04.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:04.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:04.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:04.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:04.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:04.885 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:04.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:04.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:05.002 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:40:05.015 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:05.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:05.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:05.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:05.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:04 smithi050 bash[21467]: cluster 2024-04-06T03:40:04.400074+0000 mgr.y (mgr.24368) 704 : cluster 0 pgmap v752: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 916 op/s 2024-04-06T03:40:05.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:04 smithi050 bash[21467]: audit 2024-04-06T03:40:04.747254+0000 mon.c (mon.2) 92 : audit 1 from='client.? 172.21.15.50:0/1070155923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:40:05.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:04 smithi050 bash[21467]: audit 2024-04-06T03:40:04.747604+0000 mon.a (mon.0) 1527 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:40:05.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:04 smithi050 bash[29221]: cluster 2024-04-06T03:40:04.400074+0000 mgr.y (mgr.24368) 704 : cluster 0 pgmap v752: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 916 op/s 2024-04-06T03:40:05.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:04 smithi050 bash[29221]: audit 2024-04-06T03:40:04.747254+0000 mon.c (mon.2) 92 : audit 1 from='client.? 172.21.15.50:0/1070155923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:40:05.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:04 smithi050 bash[29221]: audit 2024-04-06T03:40:04.747604+0000 mon.a (mon.0) 1527 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:40:05.254 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:05.263 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:04 smithi153 bash[23852]: cluster 2024-04-06T03:40:04.400074+0000 mgr.y (mgr.24368) 704 : cluster 0 pgmap v752: 148 pgs: 148 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 57 MiB/s wr, 916 op/s 2024-04-06T03:40:05.263 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:04 smithi153 bash[23852]: audit 2024-04-06T03:40:04.747254+0000 mon.c (mon.2) 92 : audit 1 from='client.? 172.21.15.50:0/1070155923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:40:05.263 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:04 smithi153 bash[23852]: audit 2024-04-06T03:40:04.747604+0000 mon.a (mon.0) 1527 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:40:05.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:05.383 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:05.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:05.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:05.539 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:05.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 61 16 75154 75138 76.9036 50.75 0.0113748 0.0129984 2024-04-06T03:40:05.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:05.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:05.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:05.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:05.933 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:05.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:05.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:06.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:06.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:06.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:06.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:06.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[21467]: cluster 2024-04-06T03:40:04.924015+0000 mon.a (mon.0) 1528 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:40:06.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[21467]: audit 2024-04-06T03:40:04.978510+0000 mon.a (mon.0) 1529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:40:06.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[21467]: cluster 2024-04-06T03:40:04.988915+0000 mon.a (mon.0) 1530 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-04-06T03:40:06.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[21467]: audit 2024-04-06T03:40:04.996520+0000 mon.a (mon.0) 1531 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:06.268 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[29221]: cluster 2024-04-06T03:40:04.924015+0000 mon.a (mon.0) 1528 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:40:06.268 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[29221]: audit 2024-04-06T03:40:04.978510+0000 mon.a (mon.0) 1529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:40:06.268 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[29221]: cluster 2024-04-06T03:40:04.988915+0000 mon.a (mon.0) 1530 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-04-06T03:40:06.268 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:05 smithi050 bash[29221]: audit 2024-04-06T03:40:04.996520+0000 mon.a (mon.0) 1531 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:06.288 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:06.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:05 smithi153 bash[23852]: cluster 2024-04-06T03:40:04.924015+0000 mon.a (mon.0) 1528 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:40:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:05 smithi153 bash[23852]: audit 2024-04-06T03:40:04.978510+0000 mon.a (mon.0) 1529 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:40:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:05 smithi153 bash[23852]: cluster 2024-04-06T03:40:04.988915+0000 mon.a (mon.0) 1530 : cluster 0 osdmap e300: 8 total, 8 up, 8 in 2024-04-06T03:40:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:05 smithi153 bash[23852]: audit 2024-04-06T03:40:04.996520+0000 mon.a (mon.0) 1531 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:06.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:06.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:06.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:06.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:06.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:06.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:06.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:06.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 62 16 75744 75728 76.2585 36.875 0.00618253 0.0130251 2024-04-06T03:40:06.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:06.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:06.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:06.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:06.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:06.886 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:40:06.886 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:40:06.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:40:06.886 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:40:06.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74150 8.e 8:75d6fd5a:::benchmark_data_smithi050_86143_object74149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.034619+0000", 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 553.84908541899995, 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42533740599999997, 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74150 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.887 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034619+0000", 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034619+0000", 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034620+0000", 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3039999999999999e-06 2024-04-06T03:40:06.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034647+0000", 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6968e-05 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034655+0000", 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9929999999999995e-06 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034660+0000", 2024-04-06T03:40:06.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3630000000000002e-06 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034737+0000", 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6669999999999996e-05 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034823+0000", 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6012000000000003e-05 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:40:06.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.034903+0000", 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0284999999999994e-05 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.042936+0000", 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080326440000000002 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.045702+0000", 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027660229999999998 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.459849+0000", 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.41414750299999997 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.459916+0000", 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7033999999999995e-05 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.459956+0000", 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9607000000000002e-05 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.892 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74166 8.9 8:9a57aa0f:::benchmark_data_smithi050_86143_object74165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.051856+0000", 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 553.83184771900005, 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43935837799999999, 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74166 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.893 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051856+0000", 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051856+0000", 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051857+0000", 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0589999999999999e-06 2024-04-06T03:40:06.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051881+0000", 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3932000000000001e-05 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051892+0000", 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0609e-05 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051900+0000", 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.242e-06 2024-04-06T03:40:06.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.051967+0000", 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7236999999999993e-05 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.052101+0000", 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013316900000000001 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.052257+0000", 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015600299999999999 2024-04-06T03:40:06.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.054653+0000", 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023967429999999998 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.057456+0000", 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028024019999999998 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491145+0000", 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43368966799999997 2024-04-06T03:40:06.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491184+0000", 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8547999999999998e-05 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491215+0000", 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0766000000000003e-05 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:74172 8.9 8:9f0adc0c:::benchmark_data_smithi050_86143_object74171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:30:53.060899+0000", 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 553.82280478600001, 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43049049900000003, 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74172 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060899+0000", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060899+0000", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060900+0000", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2109999999999999e-06 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060931+0000", 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0071999999999999e-05 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060945+0000", 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4661000000000001e-05 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.060956+0000", 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1314999999999999e-05 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.061030+0000", 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3783999999999997e-05 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.061205+0000", 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017448400000000001 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.061380+0000", 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017525 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.064573+0000", 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031926400000000001 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.064685+0000", 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011252300000000001 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491276+0000", 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.426590786 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491343+0000", 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6563000000000006e-05 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:30:53.491390+0000", 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7209999999999997e-05 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:134289 8.e 8:7bc21049:::benchmark_data_smithi050_86143_object27978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:03.585004+0000", 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 483.29869992699997, 2024-04-06T03:40:06.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42752563999999998, 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 134289 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585004+0000", 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585004+0000", 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585005+0000", 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.33e-06 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585011+0000", 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6500000000000001e-06 2024-04-06T03:40:06.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585022+0000", 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0664e-05 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.585030+0000", 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8800000000000008e-06 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.982776+0000", 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39774652999999999 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.982892+0000", 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116047 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.983016+0000", 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123681 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.991834+0000", 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088182160000000002 2024-04-06T03:40:06.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:03.997793+0000", 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059593930000000003 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:04.012230+0000", 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014436818000000001 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:04.012475+0000", 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024445599999999999 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:04.012530+0000", 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4975000000000001e-05 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160886 8.e 8:7b9e2e5b:::benchmark_data_smithi050_86143_object54575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.766409+0000", 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 452.11729467200001, 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50982000999999999, 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.910 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160886 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766409+0000", 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766409+0000", 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766411+0000", 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.358e-06 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766415+0000", 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2640000000000002e-06 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766429+0000", 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3719e-05 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.766435+0000", 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3230000000000003e-06 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.772525+0000", 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060896880000000002 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.772763+0000", 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023817599999999999 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.772875+0000", 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011236299999999999 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.775963+0000", 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030874169999999999 2024-04-06T03:40:06.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.776480+0000", 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051693400000000003 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.275924+0000", 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49944450899999998 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.276105+0000", 2024-04-06T03:40:06.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018112900000000001 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.276229+0000", 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012412999999999999 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14928.0:160906 8.e 8:7f9037e1:::benchmark_data_smithi050_86143_object54595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:32:34.786493+0000", 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 452.09721098400001, 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49019417900000001, 2024-04-06T03:40:06.916 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14928", 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2914994031", 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 160906 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.786493+0000", 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.786493+0000", 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.786494+0000", 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.282e-06 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.786500+0000", 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6200000000000004e-06 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.786513+0000", 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.346e-05 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.786521+0000", 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6580000000000002e-06 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.788457+0000", 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019358439999999999 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.788580+0000", 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123108 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.788659+0000", 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8532e-05 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.793948+0000", 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0052899510000000002 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:34.793959+0000", 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0180000000000001e-05 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.276333+0000", 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48237435899999997 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.276549+0000", 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021563100000000001 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:32:35.276687+0000", 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013855399999999999 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:114962 9.5 9:a6224ab0:::benchmark_data_smithi050_105098_object18052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:36:37.866340+0000", 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.01736409899999, 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.44513242400000003, 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.922 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 114962 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866340+0000", 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866340+0000", 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866342+0000", 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.761e-06 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866349+0000", 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.905e-06 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866363+0000", 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3957999999999999e-05 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866372+0000", 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6830000000000003e-06 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866448+0000", 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6119999999999996e-05 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866570+0000", 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012140499999999999 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,7", 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.866674+0000", 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104483 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872430+0000", 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057554640000000001 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311063+0000", 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43863292799999998 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311206+0000", 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014369099999999999 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311341+0000", 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013497699999999999 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311472+0000", 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131049 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.927 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15135.0:114957 9.8 e259/231 9:114144da:::benchmark_data_smithi050_105098_object18047:head v 259'7045, mlcod=259'7041)", 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:36:37.872584+0000", 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.01111952900001, 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43891751200000001, 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872584+0000", 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872584+0000", 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872585+0000", 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6000000000000003e-07 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872592+0000", 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3160000000000002e-06 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872602+0000", 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0616e-05 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872610+0000", 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.481e-06 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872859+0000", 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024916400000000002 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.872879+0000", 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.969e-05 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311401+0000", 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.438522036 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311502+0000", 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101449 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:114969 9.9 9:9513d4a6:::benchmark_data_smithi050_105098_object18059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:36:37.875312+0000", 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.00839186799999, 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45110768699999998, 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 114969 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.932 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875312+0000", 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875312+0000", 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875314+0000", 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6840000000000001e-06 2024-04-06T03:40:06.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875321+0000", 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4889999999999998e-06 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875336+0000", 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4987000000000001e-05 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875346+0000", 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8819999999999996e-06 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875416+0000", 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0086000000000006e-05 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875554+0000", 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013808399999999999 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,7", 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875674+0000", 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011964300000000001 2024-04-06T03:40:06.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311229+0000", 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.435554791 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311487+0000", 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025854200000000001 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.326193+0000", 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014705505000000001 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.326321+0000", 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012807500000000001 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.326420+0000", 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8919000000000006e-05 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.937 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15135.0:114962 9.5 e259/231 ondisk, result = 0)", 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:36:37.875482+0000", 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.008222292, 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43600176600000001, 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875482+0000", 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875482+0000", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875483+0000", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4600000000000003e-07 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875488+0000", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8030000000000002e-06 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875500+0000", 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1204e-05 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.875507+0000", 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1199999999999996e-06 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311000+0000", 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43549299299999999 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311042+0000", 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1921000000000001e-05 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311483+0000", 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000441779 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15135.0:114969 9.9 e259/231 ondisk, result = 0)", 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:36:37.878527+0000", 2024-04-06T03:40:06.941 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.005176656, 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43283599099999998, 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878527+0000", 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878527+0000", 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878528+0000", 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1399999999999995e-07 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878540+0000", 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2071e-05 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311016+0000", 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.432476044 2024-04-06T03:40:06.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311046+0000", 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9909000000000001e-05 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311185+0000", 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138477 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311211+0000", 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5775999999999999e-05 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311363+0000", 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0001528 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:114974 9.5 9:a884ca09:::benchmark_data_smithi050_105098_object18064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:36:37.878663+0000", 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 209.005040732, 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.46832014999999999, 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.945 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 114974 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878663+0000", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878663+0000", 2024-04-06T03:40:06.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878664+0000", 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1939999999999999e-06 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:37.878670+0000", 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2299999999999999e-06 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311241+0000", 2024-04-06T03:40:06.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.43257147200000001 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311272+0000", 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0805000000000001e-05 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311590+0000", 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031760700000000002 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.311815+0000", 2024-04-06T03:40:06.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000225625 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,7", 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.312086+0000", 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027098500000000001 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.330038+0000", 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017951363000000001 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.330135+0000", 2024-04-06T03:40:06.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7221999999999999e-05 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.346789+0000", 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016654262999999999 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.346896+0000", 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000106982 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:36:38.346983+0000", 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7402000000000001e-05 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15135.0:158007 9.b 9:d021809f:::benchmark_data_smithi050_105098_object61097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:37:26.274017+0000", 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 160.60968724200001, 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39655604300000002, 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15135", 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2755370750", 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 158007 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274017+0000", 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274017+0000", 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274018+0000", 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.449e-06 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274025+0000", 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8369999999999999e-06 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274037+0000", 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1579999999999999e-05 2024-04-06T03:40:06.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274045+0000", 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4090000000000003e-06 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274112+0000", 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6742000000000006e-05 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274227+0000", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115581 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,6", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.274342+0000", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011469899999999999 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.277426+0000", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030837389999999998 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.281787+0000", 2024-04-06T03:40:06.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043613790000000003 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.670071+0000", 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.38828404300000002 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.670440+0000", 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000368527 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:37:26.670573+0000", 2024-04-06T03:40:06.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000133058 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:1809 10.4 10:275bf1b0:::benchmark_data_smithi050_121293_object1808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:05.309885+0000", 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 61.573818911000004, 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53563720400000003, 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1809 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309885+0000", 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309885+0000", 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309886+0000", 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5199999999999995e-07 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309914+0000", 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7656000000000001e-05 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309931+0000", 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7091999999999998e-05 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309943+0000", 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2605999999999999e-05 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309996+0000", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2771e-05 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.310095+0000", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8741999999999998e-05 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,4", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.310177+0000", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2316000000000003e-05 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.310657+0000", 2024-04-06T03:40:06.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047996099999999999 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.312229+0000", 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015715219999999999 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.845162+0000", 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53293317699999998 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.845357+0000", 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019494400000000001 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.845522+0000", 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000165465 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:1816 10.4 10:274890ab:::benchmark_data_smithi050_121293_object1815:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:05.317668+0000", 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 61.566035585000002, 2024-04-06T03:40:06.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52821830400000003, 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1816 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317668+0000", 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317668+0000", 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317670+0000", 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.297e-06 2024-04-06T03:40:06.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317702+0000", 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1856000000000001e-05 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317720+0000", 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7985000000000001e-05 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317733+0000", 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3557000000000001e-05 2024-04-06T03:40:06.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317804+0000", 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1171e-05 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.317949+0000", 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014448099999999999 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,4", 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.318115+0000", 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166293 2024-04-06T03:40:06.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.318664+0000", 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054853699999999998 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.319633+0000", 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00096907199999999997 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.845635+0000", 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52600193299999998 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.845782+0000", 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000147582 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.845887+0000", 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010454 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:70566 10.6 10:60a8064e:::benchmark_data_smithi050_121293_object70565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-04-06T03:40:06.969 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:59.804161+0000", 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.0795432360000001, 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.438681141, 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 70566 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.804161+0000", 2024-04-06T03:40:06.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.804161+0000", 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.804161+0000", 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4499999999999998e-07 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.804178+0000", 2024-04-06T03:40:06.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6616000000000002e-05 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.804183+0000", 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0139999999999998e-06 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.804187+0000", 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.89e-06 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.856061+0000", 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.051874284 2024-04-06T03:40:06.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.856174+0000", 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113083 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,5", 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.856298+0000", 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000123535 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859566+0000", 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032682240000000001 2024-04-06T03:40:06.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.862819+0000", 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032531510000000001 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.242747+0000", 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.37992799500000002 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.242802+0000", 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5562000000000001e-05 2024-04-06T03:40:06.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.242842+0000", 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9441999999999999e-05 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:70571 10.4 10:22b0aaa5:::benchmark_data_smithi050_121293_object70570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:59.810507+0000", 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.0731969250000004, 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.42988573699999999, 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.975 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 70571 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.810507+0000", 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.810507+0000", 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.810508+0000", 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.02e-06 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.810554+0000", 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5945000000000001e-05 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.852910+0000", 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042356421999999998 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.852920+0000", 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5219999999999996e-06 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.853202+0000", 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028221000000000002 2024-04-06T03:40:06.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.853316+0000", 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011333 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,4", 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.853445+0000", 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012991899999999999 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.861182+0000", 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0077364579999999999 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.861193+0000", 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1311999999999999e-05 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.863202+0000", 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002009187 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.240337+0000", 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.37713454499999999 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.240393+0000", 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5866999999999999e-05 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:70580 10.6 10:63f49bbf:::benchmark_data_smithi050_121293_object70579:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:59.859058+0000", 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.024645585, 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.39674334900000002, 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 70580 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859058+0000", 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859058+0000", 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859059+0000", 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.51e-07 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859084+0000", 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5276999999999999e-05 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859094+0000", 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2499999999999995e-06 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.859101+0000", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1130000000000003e-06 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.862842+0000", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003741229 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.862961+0000", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118925 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,5", 2024-04-06T03:40:06.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:59.863078+0000", 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011721400000000001 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.242917+0000", 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.379839121 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.252846+0000", 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099286410000000002 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.255686+0000", 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028403130000000001 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.255748+0000", 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2349999999999998e-05 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:00.255802+0000", 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3365000000000002e-05 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:74439 10.4 10:22e78327:::benchmark_data_smithi050_121293_object74438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:04.772530+0000", 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.1111742360000001, 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.225550363, 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74439 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772530+0000", 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772530+0000", 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772530+0000", 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8100000000000003e-07 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772550+0000", 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9559e-05 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772557+0000", 2024-04-06T03:40:06.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5220000000000001e-06 2024-04-06T03:40:06.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:06.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772563+0000", 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7150000000000003e-06 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772623+0000", 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0217000000000003e-05 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772771+0000", 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000147257 2024-04-06T03:40:06.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,4", 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772927+0000", 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000156911 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.776711+0000", 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003783561 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.782335+0000", 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005623762 2024-04-06T03:40:06.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.997968+0000", 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.21563326899999999 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.998031+0000", 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3088000000000003e-05 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.998080+0000", 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9020999999999997e-05 2024-04-06T03:40:06.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:74440 10.4 10:2063c33c:::benchmark_data_smithi050_121293_object74439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e299)", 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:04.772575+0000", 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.1111293560000002, 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.22882371500000001, 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:40:06.994 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74440 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772575+0000", 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772575+0000", 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772575+0000", 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4099999999999999e-07 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772595+0000", 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9729999999999999e-05 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772604+0000", 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000008e-06 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.772610+0000", 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9100000000000002e-06 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.774854+0000", 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022437360000000001 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.774956+0000", 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101826 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,4", 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.775041+0000", 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5675999999999998e-05 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.776837+0000", 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017954729999999999 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:04.782387+0000", 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0055498819999999999 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:05.001337+0000", 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.218950644 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:05.001371+0000", 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3353999999999999e-05 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:05.001398+0000", 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7823000000000001e-05 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:06.999 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:07.000 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:07.000 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:40:07.000 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:40:07.000 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:40:07.015 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:07.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:07.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:06 smithi153 bash[23852]: cluster 2024-04-06T03:40:06.400631+0000 mgr.y (mgr.24368) 705 : cluster 0 pgmap v754: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 925 op/s 2024-04-06T03:40:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:06 smithi050 bash[21467]: cluster 2024-04-06T03:40:06.400631+0000 mgr.y (mgr.24368) 705 : cluster 0 pgmap v754: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 925 op/s 2024-04-06T03:40:07.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:06 smithi050 bash[29221]: cluster 2024-04-06T03:40:06.400631+0000 mgr.y (mgr.24368) 705 : cluster 0 pgmap v754: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 925 op/s 2024-04-06T03:40:07.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:07.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:07.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:07.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:07.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:07.613 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 63 15 76372 76357 75.6726 39.3125 0.0283465 0.0132098 2024-04-06T03:40:07.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:07.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:07.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:07.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:07.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:07.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:07.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:07.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:07.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:07.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:08.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:08.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:08.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:08.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:08.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:08.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:08.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:08.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:08.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 64 16 77343 77327 75.4376 60.625 0.00864591 0.0132506 2024-04-06T03:40:08.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:08.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:08.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:08.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:08.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:08.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:09.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:09.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:09.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:09.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:09.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:09.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:09.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:09.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:09.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 65 16 78076 78060 74.9821 45.8125 0.00395763 0.0133174 2024-04-06T03:40:09.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:09.658 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:09 smithi050 bash[21467]: cluster 2024-04-06T03:40:08.401297+0000 mgr.y (mgr.24368) 706 : cluster 0 pgmap v755: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 932 op/s 2024-04-06T03:40:09.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:09 smithi050 bash[29221]: cluster 2024-04-06T03:40:08.401297+0000 mgr.y (mgr.24368) 706 : cluster 0 pgmap v755: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 932 op/s 2024-04-06T03:40:09.659 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:40:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:40:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:40:09.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:09.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:09.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:09 smithi153 bash[23852]: cluster 2024-04-06T03:40:08.401297+0000 mgr.y (mgr.24368) 706 : cluster 0 pgmap v755: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 932 op/s 2024-04-06T03:40:09.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:09.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:09.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:09.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:10.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:10.210 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:10.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:10.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:40:09 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:40:10.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:10.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:10.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:10.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:10.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:10.591 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:10.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 66 16 78909 78893 74.6351 52.0625 0.00732668 0.0133933 2024-04-06T03:40:10.618 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:10.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:10.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:10.749 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:10.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:10.855 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:10.876 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:10.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:10.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:11.009 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:11.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:11.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:11.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:11.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:11.285 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:11.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:11.416 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:11.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:11.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:11.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 67 16 79778 79762 74.332 54.3125 0.0112584 0.0134481 2024-04-06T03:40:11.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:11.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:11 smithi153 bash[23852]: audit 2024-04-06T03:40:09.963943+0000 mgr.y (mgr.24368) 707 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:11.667 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:11 smithi153 bash[23852]: cluster 2024-04-06T03:40:10.402109+0000 mgr.y (mgr.24368) 708 : cluster 0 pgmap v756: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-06T03:40:11.699 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:11.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:11 smithi050 bash[29221]: audit 2024-04-06T03:40:09.963943+0000 mgr.y (mgr.24368) 707 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:11.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:11 smithi050 bash[29221]: cluster 2024-04-06T03:40:10.402109+0000 mgr.y (mgr.24368) 708 : cluster 0 pgmap v756: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-06T03:40:11.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:11 smithi050 bash[21467]: audit 2024-04-06T03:40:09.963943+0000 mgr.y (mgr.24368) 707 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:11.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:11 smithi050 bash[21467]: cluster 2024-04-06T03:40:10.402109+0000 mgr.y (mgr.24368) 708 : cluster 0 pgmap v756: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 814 op/s 2024-04-06T03:40:11.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:11.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:11.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:11.949 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:11.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:12.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:12.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:12.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:12.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:12.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:12.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:12.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-06T03:40:12.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:12.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:12.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:12.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:12.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 68 16 80475 80459 73.8798 43.5625 0.00334299 0.0135181 2024-04-06T03:40:12.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:12.709 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:12.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:40:12.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:12.837 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:12.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:12.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:12.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:13.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:13.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:13.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:13.249 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:13.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:13.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:13.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:13.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:13.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 69 16 81374 81358 73.6236 56.1875 0.0117705 0.0135778 2024-04-06T03:40:13.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:13.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:13.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:13.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:13 smithi153 bash[23852]: cluster 2024-04-06T03:40:12.101049+0000 osd.5 (osd.5) 143 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:13.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:13 smithi153 bash[23852]: cluster 2024-04-06T03:40:12.101127+0000 osd.5 (osd.5) 144 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:13.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:13 smithi153 bash[23852]: cluster 2024-04-06T03:40:12.402760+0000 mgr.y (mgr.24368) 709 : cluster 0 pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 782 op/s 2024-04-06T03:40:13.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:13 smithi153 bash[23852]: cluster 2024-04-06T03:40:12.492837+0000 mon.a (mon.0) 1532 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-04-06T03:40:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:13.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:13.894 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:13.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:13.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[21467]: cluster 2024-04-06T03:40:12.101049+0000 osd.5 (osd.5) 143 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[21467]: cluster 2024-04-06T03:40:12.101127+0000 osd.5 (osd.5) 144 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[21467]: cluster 2024-04-06T03:40:12.402760+0000 mgr.y (mgr.24368) 709 : cluster 0 pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 782 op/s 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[21467]: cluster 2024-04-06T03:40:12.492837+0000 mon.a (mon.0) 1532 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[29221]: cluster 2024-04-06T03:40:12.101049+0000 osd.5 (osd.5) 143 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[29221]: cluster 2024-04-06T03:40:12.101127+0000 osd.5 (osd.5) 144 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:13.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[29221]: cluster 2024-04-06T03:40:12.402760+0000 mgr.y (mgr.24368) 709 : cluster 0 pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 49 MiB/s wr, 782 op/s 2024-04-06T03:40:13.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:13 smithi050 bash[29221]: cluster 2024-04-06T03:40:12.492837+0000 mon.a (mon.0) 1532 : cluster 0 osdmap e301: 8 total, 8 up, 8 in 2024-04-06T03:40:14.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:14.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:14.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:14.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:14.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:14.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:14.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:14.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:14.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:14.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:14.578 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:14.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 70 16 82399 82383 73.487 64.0625 0.00803992 0.0136022 2024-04-06T03:40:14.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:14.708 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:14.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:14.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:14.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:14.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:14.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:15.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:14 smithi153 bash[23852]: cluster 2024-04-06T03:40:14.403394+0000 mgr.y (mgr.24368) 710 : cluster 0 pgmap v759: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 720 op/s 2024-04-06T03:40:15.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:15.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:15.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:14 smithi050 bash[21467]: cluster 2024-04-06T03:40:14.403394+0000 mgr.y (mgr.24368) 710 : cluster 0 pgmap v759: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 720 op/s 2024-04-06T03:40:15.144 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:14 smithi050 bash[29221]: cluster 2024-04-06T03:40:14.403394+0000 mgr.y (mgr.24368) 710 : cluster 0 pgmap v759: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 720 op/s 2024-04-06T03:40:15.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:15.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:15.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:15.354 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:15.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:15.488 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:15.591 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:15.614 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 71 16 83119 83103 73.086 45 0.0034228 0.0136564 2024-04-06T03:40:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:15.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:15.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:15.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:15.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:15.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:15.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:16.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:16.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:16.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:16.293 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:16.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:16.420 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:16.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:16.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:16.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 72 16 83738 83722 72.6086 38.6875 0.004593 0.0137665 2024-04-06T03:40:16.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:16.706 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:16.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:16.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:16.937 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:16.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:16 smithi153 bash[23852]: cluster 2024-04-06T03:40:15.354457+0000 osd.7 (osd.7) 479 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:16.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:16 smithi153 bash[23852]: cluster 2024-04-06T03:40:15.354511+0000 osd.7 (osd.7) 480 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:16.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:16 smithi153 bash[23852]: cluster 2024-04-06T03:40:15.902504+0000 mon.a (mon.0) 1533 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-04-06T03:40:16.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:16 smithi153 bash[23852]: cluster 2024-04-06T03:40:16.403779+0000 mgr.y (mgr.24368) 711 : cluster 0 pgmap v761: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 803 op/s 2024-04-06T03:40:16.960 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:16.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:17.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:17.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:17.137 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:40:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[21467]: cluster 2024-04-06T03:40:15.354457+0000 osd.7 (osd.7) 479 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[21467]: cluster 2024-04-06T03:40:15.354511+0000 osd.7 (osd.7) 480 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[21467]: cluster 2024-04-06T03:40:15.902504+0000 mon.a (mon.0) 1533 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[21467]: cluster 2024-04-06T03:40:16.403779+0000 mgr.y (mgr.24368) 711 : cluster 0 pgmap v761: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 803 op/s 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[29221]: cluster 2024-04-06T03:40:15.354457+0000 osd.7 (osd.7) 479 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[29221]: cluster 2024-04-06T03:40:15.354511+0000 osd.7 (osd.7) 480 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[29221]: cluster 2024-04-06T03:40:15.902504+0000 mon.a (mon.0) 1533 : cluster 0 osdmap e302: 8 total, 8 up, 8 in 2024-04-06T03:40:17.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:16 smithi050 bash[29221]: cluster 2024-04-06T03:40:16.403779+0000 mgr.y (mgr.24368) 711 : cluster 0 pgmap v761: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 803 op/s 2024-04-06T03:40:17.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:17.228 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:17.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:17.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:17.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:17.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:17.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:17.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:17.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 73 16 84328 84312 72.1195 36.875 0.00580905 0.0137977 2024-04-06T03:40:17.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:17.741 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:17.845 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:17.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:17.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:17.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:17.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:17 smithi153 bash[23852]: audit 2024-04-06T03:40:17.111142+0000 mgr.y (mgr.24368) 712 : audit 0 from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:40:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.002 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:18.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:18.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:18.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:18.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:18.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:18.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:17 smithi050 bash[21467]: audit 2024-04-06T03:40:17.111142+0000 mgr.y (mgr.24368) 712 : audit 0 from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:40:18.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:17 smithi050 bash[29221]: audit 2024-04-06T03:40:17.111142+0000 mgr.y (mgr.24368) 712 : audit 0 from='client.15348 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:40:18.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:18.399 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:18.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:18.532 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:18.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 74 16 84989 84973 71.7034 41.3125 0.00324867 0.0139288 2024-04-06T03:40:18.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:18.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:18.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:18.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:18.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:18.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:19.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:19.028 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:18 smithi153 bash[23852]: cluster 2024-04-06T03:40:18.404389+0000 mgr.y (mgr.24368) 713 : cluster 0 pgmap v762: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 700 op/s 2024-04-06T03:40:19.046 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:19.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:19.152 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:19.175 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:19.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:18 smithi050 bash[21467]: cluster 2024-04-06T03:40:18.404389+0000 mgr.y (mgr.24368) 713 : cluster 0 pgmap v762: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 700 op/s 2024-04-06T03:40:19.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:18 smithi050 bash[29221]: cluster 2024-04-06T03:40:18.404389+0000 mgr.y (mgr.24368) 713 : cluster 0 pgmap v762: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 700 op/s 2024-04-06T03:40:19.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:19.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:19.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.439 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:19.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:19.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:19.566 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:19.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 75 16 85958 85942 71.5549 60.5625 0.0125355 0.0139701 2024-04-06T03:40:19.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:19.699 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:19.739 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:40:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:40:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:40:19.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:19.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:19.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:19.981 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":761,"stamp":"2024-04-06T03:40:16.403565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5322709342,"num_objects":81439,"num_object_clones":0,"num_object_copies":244317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81439,"num_whiteouts":0,"num_read":2794,"num_read_kb":2608,"num_write":82302,"num_write_kb":5199837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80559,"ondisk_log_size":80559,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18647264,"kb_used_data":15553108,"kb_used_omap":12,"kb_used_meta":3093939,"kb_avail":731313952,"statfs":{"total":767960285184,"available":748865486848,"internally_reserved":0,"allocated":15926382592,"data_stored":15923880026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3168194097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,3,5,1,0,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":420872192,"num_objects":6422,"num_object_clones":0,"num_object_copies":19266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6422,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6422,"num_write_kb":411008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002747"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":514,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994118+0000","last_change":"2024-04-06T03:38:33.397343+0000","last_active":"2024-04-06T03:40:04.994118+0000","last_peered":"2024-04-06T03:40:04.994118+0000","last_clean":"2024-04-06T03:40:04.994118+0000","last_became_active":"2024-04-06T03:20:58.140524+0000","last_became_peered":"2024-04-06T03:20:58.140524+0000","last_unstale":"2024-04-06T03:40:04.994118+0000","last_undegraded":"2024-04-06T03:40:04.994118+0000","last_fullsized":"2024-04-06T03:40:04.994118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:33.397236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:25.064683+0000","last_clean_scrub_stamp":"2024-04-06T03:38:33.397236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":520,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483710+0000","last_change":"2024-04-06T03:39:00.032521+0000","last_active":"2024-04-06T03:40:12.483710+0000","last_peered":"2024-04-06T03:40:12.483710+0000","last_clean":"2024-04-06T03:40:12.483710+0000","last_became_active":"2024-04-06T03:20:55.770877+0000","last_became_peered":"2024-04-06T03:20:55.770877+0000","last_unstale":"2024-04-06T03:40:12.483710+0000","last_undegraded":"2024-04-06T03:40:12.483710+0000","last_fullsized":"2024-04-06T03:40:12.483710+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:00.032302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:39:00.032302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:28.589523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":546,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.999171+0000","last_change":"2024-04-06T03:39:14.104873+0000","last_active":"2024-04-06T03:40:04.999171+0000","last_peered":"2024-04-06T03:40:04.999171+0000","last_clean":"2024-04-06T03:40:04.999171+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:40:04.999171+0000","last_undegraded":"2024-04-06T03:40:04.999171+0000","last_fullsized":"2024-04-06T03:40:04.999171+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:14.104792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:14.104792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.385240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"199'32","reported_seq":572,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.992300+0000","last_change":"2024-04-06T03:37:40.241138+0000","last_active":"2024-04-06T03:40:04.992300+0000","last_peered":"2024-04-06T03:40:04.992300+0000","last_clean":"2024-04-06T03:40:04.992300+0000","last_became_active":"2024-04-06T03:20:53.341462+0000","last_became_peered":"2024-04-06T03:20:53.341462+0000","last_unstale":"2024-04-06T03:40:04.992300+0000","last_undegraded":"2024-04-06T03:40:04.992300+0000","last_fullsized":"2024-04-06T03:40:04.992300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'32","last_scrub_stamp":"2024-04-06T03:37:40.241007+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:37:40.241007+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483872+0000","last_change":"2024-04-06T03:39:20.037086+0000","last_active":"2024-04-06T03:40:12.483872+0000","last_peered":"2024-04-06T03:40:12.483872+0000","last_clean":"2024-04-06T03:40:12.483872+0000","last_became_active":"2024-04-06T03:20:57.825690+0000","last_became_peered":"2024-04-06T03:20:57.825690+0000","last_unstale":"2024-04-06T03:40:12.483872+0000","last_undegraded":"2024-04-06T03:40:12.483872+0000","last_fullsized":"2024-04-06T03:40:12.483872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:20.036992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:20.036992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:45.175057+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":407,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501385+0000","last_change":"2024-04-06T03:39:07.979712+0000","last_active":"2024-04-06T03:40:12.501385+0000","last_peered":"2024-04-06T03:40:12.501385+0000","last_clean":"2024-04-06T03:40:12.501385+0000","last_became_active":"2024-04-06T03:25:54.480600+0000","last_became_peered":"2024-04-06T03:25:54.480600+0000","last_unstale":"2024-04-06T03:40:12.501385+0000","last_undegraded":"2024-04-06T03:40:12.501385+0000","last_fullsized":"2024-04-06T03:40:12.501385+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:07.979620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:07.979620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":528,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.992014+0000","last_change":"2024-04-06T03:38:52.265617+0000","last_active":"2024-04-06T03:40:04.992014+0000","last_peered":"2024-04-06T03:40:04.992014+0000","last_clean":"2024-04-06T03:40:04.992014+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:40:04.992014+0000","last_undegraded":"2024-04-06T03:40:04.992014+0000","last_fullsized":"2024-04-06T03:40:04.992014+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:52.265537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:38:52.265537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"199'31","reported_seq":579,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998552+0000","last_change":"2024-04-06T03:39:01.848367+0000","last_active":"2024-04-06T03:40:04.998552+0000","last_peered":"2024-04-06T03:40:04.998552+0000","last_clean":"2024-04-06T03:40:04.998552+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:40:04.998552+0000","last_undegraded":"2024-04-06T03:40:04.998552+0000","last_fullsized":"2024-04-06T03:40:04.998552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:01.848222+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:14.574150+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":411,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.490547+0000","last_change":"2024-04-06T03:39:11.125137+0000","last_active":"2024-04-06T03:40:12.490547+0000","last_peered":"2024-04-06T03:40:12.490547+0000","last_clean":"2024-04-06T03:40:12.490547+0000","last_became_active":"2024-04-06T03:25:54.484120+0000","last_became_peered":"2024-04-06T03:25:54.484120+0000","last_unstale":"2024-04-06T03:40:12.490547+0000","last_undegraded":"2024-04-06T03:40:12.490547+0000","last_fullsized":"2024-04-06T03:40:12.490547+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.124952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:29.954996+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501381+0000","last_change":"2024-04-06T03:38:39.136548+0000","last_active":"2024-04-06T03:40:12.501381+0000","last_peered":"2024-04-06T03:40:12.501381+0000","last_clean":"2024-04-06T03:40:12.501381+0000","last_became_active":"2024-04-06T03:20:55.753736+0000","last_became_peered":"2024-04-06T03:20:55.753736+0000","last_unstale":"2024-04-06T03:40:12.501381+0000","last_undegraded":"2024-04-06T03:40:12.501381+0000","last_fullsized":"2024-04-06T03:40:12.501381+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.136452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:31.422613+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.136452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":542,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501799+0000","last_change":"2024-04-06T03:38:37.173203+0000","last_active":"2024-04-06T03:40:12.501799+0000","last_peered":"2024-04-06T03:40:12.501799+0000","last_clean":"2024-04-06T03:40:12.501799+0000","last_became_active":"2024-04-06T03:20:51.141672+0000","last_became_peered":"2024-04-06T03:20:51.141672+0000","last_unstale":"2024-04-06T03:40:12.501799+0000","last_undegraded":"2024-04-06T03:40:12.501799+0000","last_fullsized":"2024-04-06T03:40:12.501799+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:38:37.173088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.173088+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"199'34","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484179+0000","last_change":"2024-04-06T03:38:30.770446+0000","last_active":"2024-04-06T03:40:12.484179+0000","last_peered":"2024-04-06T03:40:12.484179+0000","last_clean":"2024-04-06T03:40:12.484179+0000","last_became_active":"2024-04-06T03:25:54.489462+0000","last_became_peered":"2024-04-06T03:25:54.489462+0000","last_unstale":"2024-04-06T03:40:12.484179+0000","last_undegraded":"2024-04-06T03:40:12.484179+0000","last_fullsized":"2024-04-06T03:40:12.484179+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'34","last_scrub_stamp":"2024-04-06T03:38:30.770300+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:38:30.770300+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:14.050758+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":520,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.993416+0000","last_change":"2024-04-06T03:39:30.292163+0000","last_active":"2024-04-06T03:40:04.993416+0000","last_peered":"2024-04-06T03:40:04.993416+0000","last_clean":"2024-04-06T03:40:04.993416+0000","last_became_active":"2024-04-06T03:20:57.831714+0000","last_became_peered":"2024-04-06T03:20:57.831714+0000","last_unstale":"2024-04-06T03:40:04.993416+0000","last_undegraded":"2024-04-06T03:40:04.993416+0000","last_fullsized":"2024-04-06T03:40:04.993416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:30.292088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:30.292088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:55.827484+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":401,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998107+0000","last_change":"2024-04-06T03:39:26.216855+0000","last_active":"2024-04-06T03:40:04.998107+0000","last_peered":"2024-04-06T03:40:04.998107+0000","last_clean":"2024-04-06T03:40:04.998107+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:40:04.998107+0000","last_undegraded":"2024-04-06T03:40:04.998107+0000","last_fullsized":"2024-04-06T03:40:04.998107+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:26.216722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:26.216722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:36.642647+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":526,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501626+0000","last_change":"2024-04-06T03:37:28.610037+0000","last_active":"2024-04-06T03:40:12.501626+0000","last_peered":"2024-04-06T03:40:12.501626+0000","last_clean":"2024-04-06T03:40:12.501626+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:40:12.501626+0000","last_undegraded":"2024-04-06T03:40:12.501626+0000","last_fullsized":"2024-04-06T03:40:12.501626+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:37:28.609910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"199'30","reported_seq":502,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484464+0000","last_change":"2024-04-06T03:39:10.052379+0000","last_active":"2024-04-06T03:40:12.484464+0000","last_peered":"2024-04-06T03:40:12.484464+0000","last_clean":"2024-04-06T03:40:12.484464+0000","last_became_active":"2024-04-06T03:25:54.483976+0000","last_became_peered":"2024-04-06T03:25:54.483976+0000","last_unstale":"2024-04-06T03:40:12.484464+0000","last_undegraded":"2024-04-06T03:40:12.484464+0000","last_fullsized":"2024-04-06T03:40:12.484464+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'30","last_scrub_stamp":"2024-04-06T03:39:10.052211+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:39:10.052211+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.899653+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":524,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:05.002317+0000","last_change":"2024-04-06T03:39:36.076966+0000","last_active":"2024-04-06T03:40:05.002317+0000","last_peered":"2024-04-06T03:40:05.002317+0000","last_clean":"2024-04-06T03:40:05.002317+0000","last_became_active":"2024-04-06T03:20:58.131735+0000","last_became_peered":"2024-04-06T03:20:58.131735+0000","last_unstale":"2024-04-06T03:40:05.002317+0000","last_undegraded":"2024-04-06T03:40:05.002317+0000","last_fullsized":"2024-04-06T03:40:05.002317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:36.076841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:36.076841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:49.718545+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":536,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484591+0000","last_change":"2024-04-06T03:39:34.004957+0000","last_active":"2024-04-06T03:40:12.484591+0000","last_peered":"2024-04-06T03:40:12.484591+0000","last_clean":"2024-04-06T03:40:12.484591+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:40:12.484591+0000","last_undegraded":"2024-04-06T03:40:12.484591+0000","last_fullsized":"2024-04-06T03:40:12.484591+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:34.004843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:39:34.004843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:58.509671+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":518,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994367+0000","last_change":"2024-04-06T03:38:32.280669+0000","last_active":"2024-04-06T03:40:04.994367+0000","last_peered":"2024-04-06T03:40:04.994367+0000","last_clean":"2024-04-06T03:40:04.994367+0000","last_became_active":"2024-04-06T03:20:51.141917+0000","last_became_peered":"2024-04-06T03:20:51.141917+0000","last_unstale":"2024-04-06T03:40:04.994367+0000","last_undegraded":"2024-04-06T03:40:04.994367+0000","last_fullsized":"2024-04-06T03:40:04.994367+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:32.280559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:32.280559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"199'33","reported_seq":587,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484129+0000","last_change":"2024-04-06T03:38:57.058614+0000","last_active":"2024-04-06T03:40:12.484129+0000","last_peered":"2024-04-06T03:40:12.484129+0000","last_clean":"2024-04-06T03:40:12.484129+0000","last_became_active":"2024-04-06T03:20:53.356220+0000","last_became_peered":"2024-04-06T03:20:53.356220+0000","last_unstale":"2024-04-06T03:40:12.484129+0000","last_undegraded":"2024-04-06T03:40:12.484129+0000","last_fullsized":"2024-04-06T03:40:12.484129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.058408+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.110170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":518,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.485154+0000","last_change":"2024-04-06T03:38:44.104984+0000","last_active":"2024-04-06T03:40:12.485154+0000","last_peered":"2024-04-06T03:40:12.485154+0000","last_clean":"2024-04-06T03:40:12.485154+0000","last_became_active":"2024-04-06T03:20:58.131621+0000","last_became_peered":"2024-04-06T03:20:58.131621+0000","last_unstale":"2024-04-06T03:40:12.485154+0000","last_undegraded":"2024-04-06T03:40:12.485154+0000","last_fullsized":"2024-04-06T03:40:12.485154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:44.104755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:44.104755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:13.537650+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":534,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.491657+0000","last_change":"2024-04-06T03:38:43.800795+0000","last_active":"2024-04-06T03:40:12.491657+0000","last_peered":"2024-04-06T03:40:12.491657+0000","last_clean":"2024-04-06T03:40:12.491657+0000","last_became_active":"2024-04-06T03:20:55.773513+0000","last_became_peered":"2024-04-06T03:20:55.773513+0000","last_unstale":"2024-04-06T03:40:12.491657+0000","last_undegraded":"2024-04-06T03:40:12.491657+0000","last_fullsized":"2024-04-06T03:40:12.491657+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:43.800561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:38:43.800561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":389,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484251+0000","last_change":"2024-04-06T03:39:08.007471+0000","last_active":"2024-04-06T03:40:12.484251+0000","last_peered":"2024-04-06T03:40:12.484251+0000","last_clean":"2024-04-06T03:40:12.484251+0000","last_became_active":"2024-04-06T03:26:18.617121+0000","last_became_peered":"2024-04-06T03:26:18.617121+0000","last_unstale":"2024-04-06T03:40:12.484251+0000","last_undegraded":"2024-04-06T03:40:12.484251+0000","last_fullsized":"2024-04-06T03:40:12.484251+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:08.007331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:08.007331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:23.215346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"224'16","reported_seq":556,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998299+0000","last_change":"2024-04-06T03:37:38.582652+0000","last_active":"2024-04-06T03:40:04.998299+0000","last_peered":"2024-04-06T03:40:04.998299+0000","last_clean":"2024-04-06T03:40:04.998299+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:40:04.998299+0000","last_undegraded":"2024-04-06T03:40:04.998299+0000","last_fullsized":"2024-04-06T03:40:04.998299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'16","last_scrub_stamp":"2024-04-06T03:37:38.582497+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:37:38.582497+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":398,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484175+0000","last_change":"2024-04-06T03:38:51.251747+0000","last_active":"2024-04-06T03:40:12.484175+0000","last_peered":"2024-04-06T03:40:12.484175+0000","last_clean":"2024-04-06T03:40:12.484175+0000","last_became_active":"2024-04-06T03:25:54.483894+0000","last_became_peered":"2024-04-06T03:25:54.483894+0000","last_unstale":"2024-04-06T03:40:12.484175+0000","last_undegraded":"2024-04-06T03:40:12.484175+0000","last_fullsized":"2024-04-06T03:40:12.484175+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:51.251490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:51.251490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:18.791411+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":526,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.999107+0000","last_change":"2024-04-06T03:39:06.939070+0000","last_active":"2024-04-06T03:40:04.999107+0000","last_peered":"2024-04-06T03:40:04.999107+0000","last_clean":"2024-04-06T03:40:04.999107+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:40:04.999107+0000","last_undegraded":"2024-04-06T03:40:04.999107+0000","last_fullsized":"2024-04-06T03:40:04.999107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:06.938951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:34.009837+0000","last_clean_scrub_stamp":"2024-04-06T03:39:06.938951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:08.530385+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":519,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991408+0000","last_change":"2024-04-06T03:37:30.256605+0000","last_active":"2024-04-06T03:40:04.991408+0000","last_peered":"2024-04-06T03:40:04.991408+0000","last_clean":"2024-04-06T03:40:04.991408+0000","last_became_active":"2024-04-06T03:20:51.142015+0000","last_became_peered":"2024-04-06T03:20:51.142015+0000","last_unstale":"2024-04-06T03:40:04.991408+0000","last_undegraded":"2024-04-06T03:40:04.991408+0000","last_fullsized":"2024-04-06T03:40:04.991408+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:37:30.256067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:37:30.256067+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"199'26","reported_seq":556,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.990966+0000","last_change":"2024-04-06T03:38:47.204903+0000","last_active":"2024-04-06T03:40:04.990966+0000","last_peered":"2024-04-06T03:40:04.990966+0000","last_clean":"2024-04-06T03:40:04.990966+0000","last_became_active":"2024-04-06T03:20:53.351009+0000","last_became_peered":"2024-04-06T03:20:53.351009+0000","last_unstale":"2024-04-06T03:40:04.990966+0000","last_undegraded":"2024-04-06T03:40:04.990966+0000","last_fullsized":"2024-04-06T03:40:04.990966+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'26","last_scrub_stamp":"2024-04-06T03:38:47.204658+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:38:47.204658+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:06.390894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":517,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502174+0000","last_change":"2024-04-06T03:38:40.108037+0000","last_active":"2024-04-06T03:40:12.502174+0000","last_peered":"2024-04-06T03:40:12.502174+0000","last_clean":"2024-04-06T03:40:12.502174+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:40:12.502174+0000","last_undegraded":"2024-04-06T03:40:12.502174+0000","last_fullsized":"2024-04-06T03:40:12.502174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:38:40.107808+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:31:51.905353+0000","last_clean_scrub_stamp":"2024-04-06T03:38:40.107808+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"139'13","reported_seq":640,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484523+0000","last_change":"2024-04-06T03:39:15.989066+0000","last_active":"2024-04-06T03:40:12.484523+0000","last_peered":"2024-04-06T03:40:12.484523+0000","last_clean":"2024-04-06T03:40:12.484523+0000","last_became_active":"2024-04-06T03:26:18.615209+0000","last_became_peered":"2024-04-06T03:26:18.615209+0000","last_unstale":"2024-04-06T03:40:12.484523+0000","last_undegraded":"2024-04-06T03:40:12.484523+0000","last_fullsized":"2024-04-06T03:40:12.484523+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'13","last_scrub_stamp":"2024-04-06T03:39:15.988918+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:39:15.988918+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.121161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":403,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483653+0000","last_change":"2024-04-06T03:39:12.958930+0000","last_active":"2024-04-06T03:40:12.483653+0000","last_peered":"2024-04-06T03:40:12.483653+0000","last_clean":"2024-04-06T03:40:12.483653+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:40:12.483653+0000","last_undegraded":"2024-04-06T03:40:12.483653+0000","last_fullsized":"2024-04-06T03:40:12.483653+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:12.958845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:39:12.958845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.107423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"199'62","reported_seq":640,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.489346+0000","last_change":"2024-04-06T03:38:39.710296+0000","last_active":"2024-04-06T03:40:12.489346+0000","last_peered":"2024-04-06T03:40:12.489346+0000","last_clean":"2024-04-06T03:40:12.489346+0000","last_became_active":"2024-04-06T03:20:53.362149+0000","last_became_peered":"2024-04-06T03:20:53.362149+0000","last_unstale":"2024-04-06T03:40:12.489346+0000","last_undegraded":"2024-04-06T03:40:12.489346+0000","last_fullsized":"2024-04-06T03:40:12.489346+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'62","last_scrub_stamp":"2024-04-06T03:38:39.710124+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.710124+0000","objects_scrubbed":11,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:13.408510+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":507,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994850+0000","last_change":"2024-04-06T03:37:28.166141+0000","last_active":"2024-04-06T03:40:04.994850+0000","last_peered":"2024-04-06T03:40:04.994850+0000","last_clean":"2024-04-06T03:40:04.994850+0000","last_became_active":"2024-04-06T03:20:58.140433+0000","last_became_peered":"2024-04-06T03:20:58.140433+0000","last_unstale":"2024-04-06T03:40:04.994850+0000","last_undegraded":"2024-04-06T03:40:04.994850+0000","last_fullsized":"2024-04-06T03:40:04.994850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:28.165908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:37:28.165908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":527,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.503048+0000","last_change":"2024-04-06T03:39:36.013647+0000","last_active":"2024-04-06T03:40:12.503048+0000","last_peered":"2024-04-06T03:40:12.503048+0000","last_clean":"2024-04-06T03:40:12.503048+0000","last_became_active":"2024-04-06T03:20:55.760865+0000","last_became_peered":"2024-04-06T03:20:55.760865+0000","last_unstale":"2024-04-06T03:40:12.503048+0000","last_undegraded":"2024-04-06T03:40:12.503048+0000","last_fullsized":"2024-04-06T03:40:12.503048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:36.013567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:39:36.013567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.319086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":405,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502110+0000","last_change":"2024-04-06T03:38:29.927678+0000","last_active":"2024-04-06T03:40:12.502110+0000","last_peered":"2024-04-06T03:40:12.502110+0000","last_clean":"2024-04-06T03:40:12.502110+0000","last_became_active":"2024-04-06T03:26:18.635349+0000","last_became_peered":"2024-04-06T03:26:18.635349+0000","last_unstale":"2024-04-06T03:40:12.502110+0000","last_undegraded":"2024-04-06T03:40:12.502110+0000","last_fullsized":"2024-04-06T03:40:12.502110+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:29.927479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:38:29.927479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"199'27","reported_seq":463,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502454+0000","last_change":"2024-04-06T03:38:32.273967+0000","last_active":"2024-04-06T03:40:12.502454+0000","last_peered":"2024-04-06T03:40:12.502454+0000","last_clean":"2024-04-06T03:40:12.502454+0000","last_became_active":"2024-04-06T03:25:54.491804+0000","last_became_peered":"2024-04-06T03:25:54.491804+0000","last_unstale":"2024-04-06T03:40:12.502454+0000","last_undegraded":"2024-04-06T03:40:12.502454+0000","last_fullsized":"2024-04-06T03:40:12.502454+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'27","last_scrub_stamp":"2024-04-06T03:38:32.273593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:32.273593+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":523,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501265+0000","last_change":"2024-04-06T03:39:31.990811+0000","last_active":"2024-04-06T03:40:12.501265+0000","last_peered":"2024-04-06T03:40:12.501265+0000","last_clean":"2024-04-06T03:40:12.501265+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:40:12.501265+0000","last_undegraded":"2024-04-06T03:40:12.501265+0000","last_fullsized":"2024-04-06T03:40:12.501265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:31.990689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:39:31.990689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:57.086019+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":390,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483664+0000","last_change":"2024-04-06T03:39:07.049989+0000","last_active":"2024-04-06T03:40:12.483664+0000","last_peered":"2024-04-06T03:40:12.483664+0000","last_clean":"2024-04-06T03:40:12.483664+0000","last_became_active":"2024-04-06T03:26:18.618151+0000","last_became_peered":"2024-04-06T03:26:18.618151+0000","last_unstale":"2024-04-06T03:40:12.483664+0000","last_undegraded":"2024-04-06T03:40:12.483664+0000","last_fullsized":"2024-04-06T03:40:12.483664+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:07.049875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:07.049875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:18.063806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":408,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502152+0000","last_change":"2024-04-06T03:39:00.237609+0000","last_active":"2024-04-06T03:40:12.502152+0000","last_peered":"2024-04-06T03:40:12.502152+0000","last_clean":"2024-04-06T03:40:12.502152+0000","last_became_active":"2024-04-06T03:26:18.637825+0000","last_became_peered":"2024-04-06T03:26:18.637825+0000","last_unstale":"2024-04-06T03:40:12.502152+0000","last_undegraded":"2024-04-06T03:40:12.502152+0000","last_fullsized":"2024-04-06T03:40:12.502152+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:00.237461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:00.237461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:24.578366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"199'22","reported_seq":570,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.996011+0000","last_change":"2024-04-06T03:38:36.725277+0000","last_active":"2024-04-06T03:40:04.996011+0000","last_peered":"2024-04-06T03:40:04.996011+0000","last_clean":"2024-04-06T03:40:04.996011+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:40:04.996011+0000","last_undegraded":"2024-04-06T03:40:04.996011+0000","last_fullsized":"2024-04-06T03:40:04.996011+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'22","last_scrub_stamp":"2024-04-06T03:38:36.725153+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.725153+0000","objects_scrubbed":4,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":507,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991373+0000","last_change":"2024-04-06T03:38:34.251308+0000","last_active":"2024-04-06T03:40:04.991373+0000","last_peered":"2024-04-06T03:40:04.991373+0000","last_clean":"2024-04-06T03:40:04.991373+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:40:04.991373+0000","last_undegraded":"2024-04-06T03:40:04.991373+0000","last_fullsized":"2024-04-06T03:40:04.991373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:34.251170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:34.251170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":535,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.490889+0000","last_change":"2024-04-06T03:39:09.707299+0000","last_active":"2024-04-06T03:40:12.490889+0000","last_peered":"2024-04-06T03:40:12.490889+0000","last_clean":"2024-04-06T03:40:12.490889+0000","last_became_active":"2024-04-06T03:20:55.765045+0000","last_became_peered":"2024-04-06T03:20:55.765045+0000","last_unstale":"2024-04-06T03:40:12.490889+0000","last_undegraded":"2024-04-06T03:40:12.490889+0000","last_fullsized":"2024-04-06T03:40:12.490889+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:09.706740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:09.706740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:31.639900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":551,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.491448+0000","last_change":"2024-04-06T03:38:44.792267+0000","last_active":"2024-04-06T03:40:12.491448+0000","last_peered":"2024-04-06T03:40:12.491448+0000","last_clean":"2024-04-06T03:40:12.491448+0000","last_became_active":"2024-04-06T03:20:51.149235+0000","last_became_peered":"2024-04-06T03:20:51.149235+0000","last_unstale":"2024-04-06T03:40:12.491448+0000","last_undegraded":"2024-04-06T03:40:12.491448+0000","last_fullsized":"2024-04-06T03:40:12.491448+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:44.792080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:38:44.792080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"199'33","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484479+0000","last_change":"2024-04-06T03:38:31.820449+0000","last_active":"2024-04-06T03:40:12.484479+0000","last_peered":"2024-04-06T03:40:12.484479+0000","last_clean":"2024-04-06T03:40:12.484479+0000","last_became_active":"2024-04-06T03:20:53.337972+0000","last_became_peered":"2024-04-06T03:20:53.337972+0000","last_unstale":"2024-04-06T03:40:12.484479+0000","last_undegraded":"2024-04-06T03:40:12.484479+0000","last_fullsized":"2024-04-06T03:40:12.484479+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'33","last_scrub_stamp":"2024-04-06T03:38:31.820183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:31.820183+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:13.094749+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":41,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":521,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.993251+0000","last_change":"2024-04-06T03:39:22.337187+0000","last_active":"2024-04-06T03:40:04.993251+0000","last_peered":"2024-04-06T03:40:04.993251+0000","last_clean":"2024-04-06T03:40:04.993251+0000","last_became_active":"2024-04-06T03:20:57.828457+0000","last_became_peered":"2024-04-06T03:20:57.828457+0000","last_unstale":"2024-04-06T03:40:04.993251+0000","last_undegraded":"2024-04-06T03:40:04.993251+0000","last_fullsized":"2024-04-06T03:40:04.993251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:22.336350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:22.336350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:49.007992+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":519,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994598+0000","last_change":"2024-04-06T03:39:28.619043+0000","last_active":"2024-04-06T03:40:04.994598+0000","last_peered":"2024-04-06T03:40:04.994598+0000","last_clean":"2024-04-06T03:40:04.994598+0000","last_became_active":"2024-04-06T03:20:55.759685+0000","last_became_peered":"2024-04-06T03:20:55.759685+0000","last_unstale":"2024-04-06T03:40:04.994598+0000","last_undegraded":"2024-04-06T03:40:04.994598+0000","last_fullsized":"2024-04-06T03:40:04.994598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:28.618765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:59.155815+0000","last_clean_scrub_stamp":"2024-04-06T03:39:28.618765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:42.794174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":379,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.995169+0000","last_change":"2024-04-06T03:37:43.208701+0000","last_active":"2024-04-06T03:40:04.995169+0000","last_peered":"2024-04-06T03:40:04.995169+0000","last_clean":"2024-04-06T03:40:04.995169+0000","last_became_active":"2024-04-06T03:26:18.643806+0000","last_became_peered":"2024-04-06T03:26:18.643806+0000","last_unstale":"2024-04-06T03:40:04.995169+0000","last_undegraded":"2024-04-06T03:40:04.995169+0000","last_fullsized":"2024-04-06T03:40:04.995169+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:43.208163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:37:43.208163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"199'21","reported_seq":560,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502049+0000","last_change":"2024-04-06T03:37:39.557911+0000","last_active":"2024-04-06T03:40:12.502049+0000","last_peered":"2024-04-06T03:40:12.502049+0000","last_clean":"2024-04-06T03:40:12.502049+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:40:12.502049+0000","last_undegraded":"2024-04-06T03:40:12.502049+0000","last_fullsized":"2024-04-06T03:40:12.502049+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:37:39.557640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:37:39.557640+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484643+0000","last_change":"2024-04-06T03:38:52.303957+0000","last_active":"2024-04-06T03:40:12.484643+0000","last_peered":"2024-04-06T03:40:12.484643+0000","last_clean":"2024-04-06T03:40:12.484643+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:40:12.484643+0000","last_undegraded":"2024-04-06T03:40:12.484643+0000","last_fullsized":"2024-04-06T03:40:12.484643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:52.303759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:52.303759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":762,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.995296+0000","last_change":"2024-04-06T03:39:26.623101+0000","last_active":"2024-04-06T03:40:04.995296+0000","last_peered":"2024-04-06T03:40:04.995296+0000","last_clean":"2024-04-06T03:40:04.995296+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:40:04.995296+0000","last_undegraded":"2024-04-06T03:40:04.995296+0000","last_fullsized":"2024-04-06T03:40:04.995296+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:39:26.622968+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:39:26.622968+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.086767+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":533,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998397+0000","last_change":"2024-04-06T03:38:44.739027+0000","last_active":"2024-04-06T03:40:04.998397+0000","last_peered":"2024-04-06T03:40:04.998397+0000","last_clean":"2024-04-06T03:40:04.998397+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:40:04.998397+0000","last_undegraded":"2024-04-06T03:40:04.998397+0000","last_fullsized":"2024-04-06T03:40:04.998397+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:44.738912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:44.738912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:10.631546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"199'15","reported_seq":554,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:05.000541+0000","last_change":"2024-04-06T03:39:04.854322+0000","last_active":"2024-04-06T03:40:05.000541+0000","last_peered":"2024-04-06T03:40:05.000541+0000","last_clean":"2024-04-06T03:40:05.000541+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:40:05.000541+0000","last_undegraded":"2024-04-06T03:40:05.000541+0000","last_fullsized":"2024-04-06T03:40:05.000541+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'15","last_scrub_stamp":"2024-04-06T03:39:04.854179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:04.854179+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.515388+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":506,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.999321+0000","last_change":"2024-04-06T03:38:39.773890+0000","last_active":"2024-04-06T03:40:04.999321+0000","last_peered":"2024-04-06T03:40:04.999321+0000","last_clean":"2024-04-06T03:40:04.999321+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:40:04.999321+0000","last_undegraded":"2024-04-06T03:40:04.999321+0000","last_fullsized":"2024-04-06T03:40:04.999321+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.773789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.773789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:08.923157+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"154'13","reported_seq":540,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998938+0000","last_change":"2024-04-06T03:39:07.946464+0000","last_active":"2024-04-06T03:40:04.998938+0000","last_peered":"2024-04-06T03:40:04.998938+0000","last_clean":"2024-04-06T03:40:04.998938+0000","last_became_active":"2024-04-06T03:27:10.498254+0000","last_became_peered":"2024-04-06T03:27:10.498254+0000","last_unstale":"2024-04-06T03:40:04.998938+0000","last_undegraded":"2024-04-06T03:40:04.998938+0000","last_fullsized":"2024-04-06T03:40:04.998938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'13","last_scrub_stamp":"2024-04-06T03:39:07.946321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:07.946321+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.741276+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":392,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.992004+0000","last_change":"2024-04-06T03:38:56.232797+0000","last_active":"2024-04-06T03:40:04.992004+0000","last_peered":"2024-04-06T03:40:04.992004+0000","last_clean":"2024-04-06T03:40:04.992004+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:40:04.992004+0000","last_undegraded":"2024-04-06T03:40:04.992004+0000","last_fullsized":"2024-04-06T03:40:04.992004+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:56.232398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:42.516373+0000","last_clean_scrub_stamp":"2024-04-06T03:38:56.232398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:18.749154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"199'24","reported_seq":559,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501770+0000","last_change":"2024-04-06T03:37:30.552999+0000","last_active":"2024-04-06T03:40:12.501770+0000","last_peered":"2024-04-06T03:40:12.501770+0000","last_clean":"2024-04-06T03:40:12.501770+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:40:12.501770+0000","last_undegraded":"2024-04-06T03:40:12.501770+0000","last_fullsized":"2024-04-06T03:40:12.501770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:37:30.552697+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":522,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501529+0000","last_change":"2024-04-06T03:38:36.191151+0000","last_active":"2024-04-06T03:40:12.501529+0000","last_peered":"2024-04-06T03:40:12.501529+0000","last_clean":"2024-04-06T03:40:12.501529+0000","last_became_active":"2024-04-06T03:20:58.131656+0000","last_became_peered":"2024-04-06T03:20:58.131656+0000","last_unstale":"2024-04-06T03:40:12.501529+0000","last_undegraded":"2024-04-06T03:40:12.501529+0000","last_fullsized":"2024-04-06T03:40:12.501529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:36.191036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.191036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":385,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991874+0000","last_change":"2024-04-06T03:38:37.301179+0000","last_active":"2024-04-06T03:40:04.991874+0000","last_peered":"2024-04-06T03:40:04.991874+0000","last_clean":"2024-04-06T03:40:04.991874+0000","last_became_active":"2024-04-06T03:25:54.485733+0000","last_became_peered":"2024-04-06T03:25:54.485733+0000","last_unstale":"2024-04-06T03:40:04.991874+0000","last_undegraded":"2024-04-06T03:40:04.991874+0000","last_fullsized":"2024-04-06T03:40:04.991874+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:37.300978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.300978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":389,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483970+0000","last_change":"2024-04-06T03:39:16.953596+0000","last_active":"2024-04-06T03:40:12.483970+0000","last_peered":"2024-04-06T03:40:12.483970+0000","last_clean":"2024-04-06T03:40:12.483970+0000","last_became_active":"2024-04-06T03:26:18.615468+0000","last_became_peered":"2024-04-06T03:26:18.615468+0000","last_unstale":"2024-04-06T03:40:12.483970+0000","last_undegraded":"2024-04-06T03:40:12.483970+0000","last_fullsized":"2024-04-06T03:40:12.483970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:16.953510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:16.953510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:32.122677+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"199'10","reported_seq":482,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.993815+0000","last_change":"2024-04-06T03:38:54.798995+0000","last_active":"2024-04-06T03:40:04.993815+0000","last_peered":"2024-04-06T03:40:04.993815+0000","last_clean":"2024-04-06T03:40:04.993815+0000","last_became_active":"2024-04-06T03:26:18.639720+0000","last_became_peered":"2024-04-06T03:26:18.639720+0000","last_unstale":"2024-04-06T03:40:04.993815+0000","last_undegraded":"2024-04-06T03:40:04.993815+0000","last_fullsized":"2024-04-06T03:40:04.993815+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'10","last_scrub_stamp":"2024-04-06T03:38:54.798646+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:38:54.798646+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:14.947433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":543,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501250+0000","last_change":"2024-04-06T03:39:08.191957+0000","last_active":"2024-04-06T03:40:12.501250+0000","last_peered":"2024-04-06T03:40:12.501250+0000","last_clean":"2024-04-06T03:40:12.501250+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:40:12.501250+0000","last_undegraded":"2024-04-06T03:40:12.501250+0000","last_fullsized":"2024-04-06T03:40:12.501250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:39:08.191830+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:39:08.191830+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":518,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:05.002416+0000","last_change":"2024-04-06T03:38:31.250220+0000","last_active":"2024-04-06T03:40:05.002416+0000","last_peered":"2024-04-06T03:40:05.002416+0000","last_clean":"2024-04-06T03:40:05.002416+0000","last_became_active":"2024-04-06T03:20:55.757134+0000","last_became_peered":"2024-04-06T03:20:55.757134+0000","last_unstale":"2024-04-06T03:40:05.002416+0000","last_undegraded":"2024-04-06T03:40:05.002416+0000","last_fullsized":"2024-04-06T03:40:05.002416+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:31.250106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:31.250106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":553,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.496609+0000","last_change":"2024-04-06T03:38:30.958769+0000","last_active":"2024-04-06T03:40:12.496609+0000","last_peered":"2024-04-06T03:40:12.496609+0000","last_clean":"2024-04-06T03:40:12.496609+0000","last_became_active":"2024-04-06T03:20:51.151787+0000","last_became_peered":"2024-04-06T03:20:51.151787+0000","last_unstale":"2024-04-06T03:40:12.496609+0000","last_undegraded":"2024-04-06T03:40:12.496609+0000","last_fullsized":"2024-04-06T03:40:12.496609+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:30.958503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:30.958503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"199'31","reported_seq":580,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501317+0000","last_change":"2024-04-06T03:38:35.196050+0000","last_active":"2024-04-06T03:40:12.501317+0000","last_peered":"2024-04-06T03:40:12.501317+0000","last_clean":"2024-04-06T03:40:12.501317+0000","last_became_active":"2024-04-06T03:20:53.350126+0000","last_became_peered":"2024-04-06T03:20:53.350126+0000","last_unstale":"2024-04-06T03:40:12.501317+0000","last_undegraded":"2024-04-06T03:40:12.501317+0000","last_fullsized":"2024-04-06T03:40:12.501317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'31","last_scrub_stamp":"2024-04-06T03:38:35.195635+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:38:35.195635+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"300'5018","reported_seq":5068,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:11.788517+0000","last_change":"2024-04-06T03:38:41.243073+0000","last_active":"2024-04-06T03:40:11.788517+0000","last_peered":"2024-04-06T03:40:11.788517+0000","last_clean":"2024-04-06T03:40:11.788517+0000","last_became_active":"2024-04-06T03:38:41.242239+0000","last_became_peered":"2024-04-06T03:38:41.242239+0000","last_unstale":"2024-04-06T03:40:11.788517+0000","last_undegraded":"2024-04-06T03:40:11.788517+0000","last_fullsized":"2024-04-06T03:40:11.788517+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:12.680936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328859648,"num_objects":5018,"num_object_clones":0,"num_object_copies":15054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5018,"num_write_kb":321152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":516,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994723+0000","last_change":"2024-04-06T03:39:16.644919+0000","last_active":"2024-04-06T03:40:04.994723+0000","last_peered":"2024-04-06T03:40:04.994723+0000","last_clean":"2024-04-06T03:40:04.994723+0000","last_became_active":"2024-04-06T03:20:57.837846+0000","last_became_peered":"2024-04-06T03:20:57.837846+0000","last_unstale":"2024-04-06T03:40:04.994723+0000","last_undegraded":"2024-04-06T03:40:04.994723+0000","last_fullsized":"2024-04-06T03:40:04.994723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:16.644788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:16.644788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:17.121912+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":555,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484250+0000","last_change":"2024-04-06T03:38:40.956357+0000","last_active":"2024-04-06T03:40:12.484250+0000","last_peered":"2024-04-06T03:40:12.484250+0000","last_clean":"2024-04-06T03:40:12.484250+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:40:12.484250+0000","last_undegraded":"2024-04-06T03:40:12.484250+0000","last_fullsized":"2024-04-06T03:40:12.484250+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:38:40.956237+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:38:40.956237+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:16.043394+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"207'31","reported_seq":580,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502274+0000","last_change":"2024-04-06T03:37:37.543171+0000","last_active":"2024-04-06T03:40:12.502274+0000","last_peered":"2024-04-06T03:40:12.502274+0000","last_clean":"2024-04-06T03:40:12.502274+0000","last_became_active":"2024-04-06T03:20:53.353010+0000","last_became_peered":"2024-04-06T03:20:53.353010+0000","last_unstale":"2024-04-06T03:40:12.502274+0000","last_undegraded":"2024-04-06T03:40:12.502274+0000","last_fullsized":"2024-04-06T03:40:12.502274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'31","last_scrub_stamp":"2024-04-06T03:37:37.543013+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:37:37.543013+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.496546+0000","last_change":"2024-04-06T03:39:03.329103+0000","last_active":"2024-04-06T03:40:12.496546+0000","last_peered":"2024-04-06T03:40:12.496546+0000","last_clean":"2024-04-06T03:40:12.496546+0000","last_became_active":"2024-04-06T03:20:55.761060+0000","last_became_peered":"2024-04-06T03:20:55.761060+0000","last_unstale":"2024-04-06T03:40:12.496546+0000","last_undegraded":"2024-04-06T03:40:12.496546+0000","last_fullsized":"2024-04-06T03:40:12.496546+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:03.328962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:03.328962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:20.620710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"301'5213","reported_seq":5264,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.415300+0000","last_change":"2024-04-06T03:38:40.968041+0000","last_active":"2024-04-06T03:40:14.415300+0000","last_peered":"2024-04-06T03:40:14.415300+0000","last_clean":"2024-04-06T03:40:14.415300+0000","last_became_active":"2024-04-06T03:38:40.967708+0000","last_became_peered":"2024-04-06T03:38:40.967708+0000","last_unstale":"2024-04-06T03:40:14.415300+0000","last_undegraded":"2024-04-06T03:40:14.415300+0000","last_fullsized":"2024-04-06T03:40:14.415300+0000","mapping_epoch":280,"log_start":"288'500","ondisk_log_start":"288'500","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4713,"log_dups_size":0,"ondisk_log_size":4713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341639168,"num_objects":5213,"num_object_clones":0,"num_object_copies":15639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5213,"num_write_kb":333632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":522,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.993305+0000","last_change":"2024-04-06T03:38:37.764505+0000","last_active":"2024-04-06T03:40:04.993305+0000","last_peered":"2024-04-06T03:40:04.993305+0000","last_clean":"2024-04-06T03:40:04.993305+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:40:04.993305+0000","last_undegraded":"2024-04-06T03:40:04.993305+0000","last_fullsized":"2024-04-06T03:40:04.993305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:37.764177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.764177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":482,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998407+0000","last_change":"2024-04-06T03:39:02.882952+0000","last_active":"2024-04-06T03:40:04.998407+0000","last_peered":"2024-04-06T03:40:04.998407+0000","last_clean":"2024-04-06T03:40:04.998407+0000","last_became_active":"2024-04-06T03:27:10.500422+0000","last_became_peered":"2024-04-06T03:27:10.500422+0000","last_unstale":"2024-04-06T03:40:04.998407+0000","last_undegraded":"2024-04-06T03:40:04.998407+0000","last_fullsized":"2024-04-06T03:40:04.998407+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:39:02.882714+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:39:02.882714+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:25.341566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"199'37","reported_seq":634,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991915+0000","last_change":"2024-04-06T03:39:19.509800+0000","last_active":"2024-04-06T03:40:04.991915+0000","last_peered":"2024-04-06T03:40:04.991915+0000","last_clean":"2024-04-06T03:40:04.991915+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:40:04.991915+0000","last_undegraded":"2024-04-06T03:40:04.991915+0000","last_fullsized":"2024-04-06T03:40:04.991915+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:39:19.509187+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:39:19.509187+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:46.311872+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":548,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.490969+0000","last_change":"2024-04-06T03:38:37.780281+0000","last_active":"2024-04-06T03:40:12.490969+0000","last_peered":"2024-04-06T03:40:12.490969+0000","last_clean":"2024-04-06T03:40:12.490969+0000","last_became_active":"2024-04-06T03:20:54.752187+0000","last_became_peered":"2024-04-06T03:20:54.752187+0000","last_unstale":"2024-04-06T03:40:12.490969+0000","last_undegraded":"2024-04-06T03:40:12.490969+0000","last_fullsized":"2024-04-06T03:40:12.490969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:38:37.780066+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.780066+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"85'11","reported_seq":768,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998247+0000","last_change":"2024-04-06T03:39:11.138297+0000","last_active":"2024-04-06T03:40:04.998247+0000","last_peered":"2024-04-06T03:40:04.998247+0000","last_clean":"2024-04-06T03:40:04.998247+0000","last_became_active":"2024-04-06T03:20:55.774021+0000","last_became_peered":"2024-04-06T03:20:55.774021+0000","last_unstale":"2024-04-06T03:40:04.998247+0000","last_undegraded":"2024-04-06T03:40:04.998247+0000","last_fullsized":"2024-04-06T03:40:04.998247+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'11","last_scrub_stamp":"2024-04-06T03:39:11.138146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.138146+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:21.825735+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"300'4979","reported_seq":5029,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:11.409710+0000","last_change":"2024-04-06T03:38:41.244498+0000","last_active":"2024-04-06T03:40:11.409710+0000","last_peered":"2024-04-06T03:40:11.409710+0000","last_clean":"2024-04-06T03:40:11.409710+0000","last_became_active":"2024-04-06T03:38:41.244000+0000","last_became_peered":"2024-04-06T03:38:41.244000+0000","last_unstale":"2024-04-06T03:40:11.409710+0000","last_undegraded":"2024-04-06T03:40:11.409710+0000","last_fullsized":"2024-04-06T03:40:11.409710+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4979,"log_dups_size":0,"ondisk_log_size":4979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:11.660317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326303744,"num_objects":4979,"num_object_clones":0,"num_object_copies":14937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":318656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":515,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.992150+0000","last_change":"2024-04-06T03:39:24.595475+0000","last_active":"2024-04-06T03:40:04.992150+0000","last_peered":"2024-04-06T03:40:04.992150+0000","last_clean":"2024-04-06T03:40:04.992150+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:40:04.992150+0000","last_undegraded":"2024-04-06T03:40:04.992150+0000","last_fullsized":"2024-04-06T03:40:04.992150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:39:24.595337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:28.464510+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":411,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.490499+0000","last_change":"2024-04-06T03:38:57.721802+0000","last_active":"2024-04-06T03:40:12.490499+0000","last_peered":"2024-04-06T03:40:12.490499+0000","last_clean":"2024-04-06T03:40:12.490499+0000","last_became_active":"2024-04-06T03:25:54.484235+0000","last_became_peered":"2024-04-06T03:25:54.484235+0000","last_unstale":"2024-04-06T03:40:12.490499+0000","last_undegraded":"2024-04-06T03:40:12.490499+0000","last_fullsized":"2024-04-06T03:40:12.490499+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:57.721624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.721624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:24.378648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"199'44","reported_seq":598,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484236+0000","last_change":"2024-04-06T03:38:53.204522+0000","last_active":"2024-04-06T03:40:12.484236+0000","last_peered":"2024-04-06T03:40:12.484236+0000","last_clean":"2024-04-06T03:40:12.484236+0000","last_became_active":"2024-04-06T03:20:53.339593+0000","last_became_peered":"2024-04-06T03:20:53.339593+0000","last_unstale":"2024-04-06T03:40:12.484236+0000","last_undegraded":"2024-04-06T03:40:12.484236+0000","last_fullsized":"2024-04-06T03:40:12.484236+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'44","last_scrub_stamp":"2024-04-06T03:38:53.204362+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:38:53.204362+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":526,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998508+0000","last_change":"2024-04-06T03:39:12.156813+0000","last_active":"2024-04-06T03:40:04.998508+0000","last_peered":"2024-04-06T03:40:04.998508+0000","last_clean":"2024-04-06T03:40:04.998508+0000","last_became_active":"2024-04-06T03:20:55.775079+0000","last_became_peered":"2024-04-06T03:20:55.775079+0000","last_unstale":"2024-04-06T03:40:04.998508+0000","last_undegraded":"2024-04-06T03:40:04.998508+0000","last_fullsized":"2024-04-06T03:40:04.998508+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:12.156581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:39:12.156581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:38.634867+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"301'5187","reported_seq":5239,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.785290+0000","last_change":"2024-04-06T03:38:40.975902+0000","last_active":"2024-04-06T03:40:14.785290+0000","last_peered":"2024-04-06T03:40:14.785290+0000","last_clean":"2024-04-06T03:40:14.785290+0000","last_became_active":"2024-04-06T03:38:40.974686+0000","last_became_peered":"2024-04-06T03:38:40.974686+0000","last_unstale":"2024-04-06T03:40:14.785290+0000","last_undegraded":"2024-04-06T03:40:14.785290+0000","last_fullsized":"2024-04-06T03:40:14.785290+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.444859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339935232,"num_objects":5187,"num_object_clones":0,"num_object_copies":15561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5187,"num_write_kb":331968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":524,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501609+0000","last_change":"2024-04-06T03:39:35.082316+0000","last_active":"2024-04-06T03:40:12.501609+0000","last_peered":"2024-04-06T03:40:12.501609+0000","last_clean":"2024-04-06T03:40:12.501609+0000","last_became_active":"2024-04-06T03:20:58.128440+0000","last_became_peered":"2024-04-06T03:20:58.128440+0000","last_unstale":"2024-04-06T03:40:12.501609+0000","last_undegraded":"2024-04-06T03:40:12.501609+0000","last_fullsized":"2024-04-06T03:40:12.501609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:35.082184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:39:35.082184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:35.161516+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"199'42","reported_seq":461,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.496696+0000","last_change":"2024-04-06T03:38:42.981579+0000","last_active":"2024-04-06T03:40:12.496696+0000","last_peered":"2024-04-06T03:40:12.496696+0000","last_clean":"2024-04-06T03:40:12.496696+0000","last_became_active":"2024-04-06T03:27:10.495663+0000","last_became_peered":"2024-04-06T03:27:10.495663+0000","last_unstale":"2024-04-06T03:40:12.496696+0000","last_undegraded":"2024-04-06T03:40:12.496696+0000","last_fullsized":"2024-04-06T03:40:12.496696+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'42","last_scrub_stamp":"2024-04-06T03:38:42.981382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:42.981382+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":554,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.500983+0000","last_change":"2024-04-06T03:38:56.123425+0000","last_active":"2024-04-06T03:40:12.500983+0000","last_peered":"2024-04-06T03:40:12.500983+0000","last_clean":"2024-04-06T03:40:12.500983+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:40:12.500983+0000","last_undegraded":"2024-04-06T03:40:12.500983+0000","last_fullsized":"2024-04-06T03:40:12.500983+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:56.123319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:38:56.123319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:20.667797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502249+0000","last_change":"2024-04-06T03:38:49.030047+0000","last_active":"2024-04-06T03:40:12.502249+0000","last_peered":"2024-04-06T03:40:12.502249+0000","last_clean":"2024-04-06T03:40:12.502249+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:40:12.502249+0000","last_undegraded":"2024-04-06T03:40:12.502249+0000","last_fullsized":"2024-04-06T03:40:12.502249+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:49.029797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:49.029797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"301'5177","reported_seq":5229,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.055774+0000","last_change":"2024-04-06T03:38:41.244345+0000","last_active":"2024-04-06T03:40:14.055774+0000","last_peered":"2024-04-06T03:40:14.055774+0000","last_clean":"2024-04-06T03:40:14.055774+0000","last_became_active":"2024-04-06T03:38:41.243355+0000","last_became_peered":"2024-04-06T03:38:41.243355+0000","last_unstale":"2024-04-06T03:40:14.055774+0000","last_undegraded":"2024-04-06T03:40:14.055774+0000","last_fullsized":"2024-04-06T03:40:14.055774+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339279872,"num_objects":5177,"num_object_clones":0,"num_object_copies":15531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5177,"num_write_kb":331328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":387,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:05.002510+0000","last_change":"2024-04-06T03:39:31.921644+0000","last_active":"2024-04-06T03:40:05.002510+0000","last_peered":"2024-04-06T03:40:05.002510+0000","last_clean":"2024-04-06T03:40:05.002510+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:40:05.002510+0000","last_undegraded":"2024-04-06T03:40:05.002510+0000","last_fullsized":"2024-04-06T03:40:05.002510+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:31.921501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:31.921501+0000","last_clean_scrub_stamp":"2024-04-06T03:39:31.921501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:55.722354+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"224'32","reported_seq":572,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991828+0000","last_change":"2024-04-06T03:39:09.506926+0000","last_active":"2024-04-06T03:40:04.991828+0000","last_peered":"2024-04-06T03:40:04.991828+0000","last_clean":"2024-04-06T03:40:04.991828+0000","last_became_active":"2024-04-06T03:20:53.346261+0000","last_became_peered":"2024-04-06T03:20:53.346261+0000","last_unstale":"2024-04-06T03:40:04.991828+0000","last_undegraded":"2024-04-06T03:40:04.991828+0000","last_fullsized":"2024-04-06T03:40:04.991828+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'32","last_scrub_stamp":"2024-04-06T03:39:09.506469+0000","last_deep_scrub":"224'32","last_deep_scrub_stamp":"2024-04-06T03:33:50.491205+0000","last_clean_scrub_stamp":"2024-04-06T03:39:09.506469+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.441786+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":412,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501311+0000","last_change":"2024-04-06T03:39:10.125890+0000","last_active":"2024-04-06T03:40:12.501311+0000","last_peered":"2024-04-06T03:40:12.501311+0000","last_clean":"2024-04-06T03:40:12.501311+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:40:12.501311+0000","last_undegraded":"2024-04-06T03:40:12.501311+0000","last_fullsized":"2024-04-06T03:40:12.501311+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:10.125595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:39:10.125595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.887097+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":787,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502854+0000","last_change":"2024-04-06T03:37:43.550112+0000","last_active":"2024-04-06T03:40:12.502854+0000","last_peered":"2024-04-06T03:40:12.502854+0000","last_clean":"2024-04-06T03:40:12.502854+0000","last_became_active":"2024-04-06T03:20:14.056883+0000","last_became_peered":"2024-04-06T03:20:14.056883+0000","last_unstale":"2024-04-06T03:40:12.502854+0000","last_undegraded":"2024-04-06T03:40:12.502854+0000","last_fullsized":"2024-04-06T03:40:12.502854+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:37:43.550007+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:37:43.550007+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":528,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502024+0000","last_change":"2024-04-06T03:39:16.989351+0000","last_active":"2024-04-06T03:40:12.502024+0000","last_peered":"2024-04-06T03:40:12.502024+0000","last_clean":"2024-04-06T03:40:12.502024+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:40:12.502024+0000","last_undegraded":"2024-04-06T03:40:12.502024+0000","last_fullsized":"2024-04-06T03:40:12.502024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:16.989269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:39:16.989269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:40.769757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"300'4958","reported_seq":5008,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:11.424378+0000","last_change":"2024-04-06T03:38:40.985375+0000","last_active":"2024-04-06T03:40:11.424378+0000","last_peered":"2024-04-06T03:40:11.424378+0000","last_clean":"2024-04-06T03:40:11.424378+0000","last_became_active":"2024-04-06T03:38:40.984476+0000","last_became_peered":"2024-04-06T03:38:40.984476+0000","last_unstale":"2024-04-06T03:40:11.424378+0000","last_undegraded":"2024-04-06T03:40:11.424378+0000","last_fullsized":"2024-04-06T03:40:11.424378+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4958,"log_dups_size":0,"ondisk_log_size":4958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:11.660280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324927488,"num_objects":4958,"num_object_clones":0,"num_object_copies":14874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4958,"num_write_kb":317312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":532,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.489278+0000","last_change":"2024-04-06T03:39:01.759596+0000","last_active":"2024-04-06T03:40:12.489278+0000","last_peered":"2024-04-06T03:40:12.489278+0000","last_clean":"2024-04-06T03:40:12.489278+0000","last_became_active":"2024-04-06T03:20:57.843542+0000","last_became_peered":"2024-04-06T03:20:57.843542+0000","last_unstale":"2024-04-06T03:40:12.489278+0000","last_undegraded":"2024-04-06T03:40:12.489278+0000","last_fullsized":"2024-04-06T03:40:12.489278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:01.759468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:37.513299+0000","last_clean_scrub_stamp":"2024-04-06T03:39:01.759468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"199'35","reported_seq":585,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.993754+0000","last_change":"2024-04-06T03:38:30.725252+0000","last_active":"2024-04-06T03:40:04.993754+0000","last_peered":"2024-04-06T03:40:04.993754+0000","last_clean":"2024-04-06T03:40:04.993754+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:40:04.993754+0000","last_undegraded":"2024-04-06T03:40:04.993754+0000","last_fullsized":"2024-04-06T03:40:04.993754+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'35","last_scrub_stamp":"2024-04-06T03:38:30.724996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:30.724996+0000","objects_scrubbed":8,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":528,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502796+0000","last_change":"2024-04-06T03:38:41.076829+0000","last_active":"2024-04-06T03:40:12.502796+0000","last_peered":"2024-04-06T03:40:12.502796+0000","last_clean":"2024-04-06T03:40:12.502796+0000","last_became_active":"2024-04-06T03:20:51.151045+0000","last_became_peered":"2024-04-06T03:20:51.151045+0000","last_unstale":"2024-04-06T03:40:12.502796+0000","last_undegraded":"2024-04-06T03:40:12.502796+0000","last_fullsized":"2024-04-06T03:40:12.502796+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:41.076607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:41.076607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":403,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.491313+0000","last_change":"2024-04-06T03:38:35.753231+0000","last_active":"2024-04-06T03:40:12.491313+0000","last_peered":"2024-04-06T03:40:12.491313+0000","last_clean":"2024-04-06T03:40:12.491313+0000","last_became_active":"2024-04-06T03:25:54.492757+0000","last_became_peered":"2024-04-06T03:25:54.492757+0000","last_unstale":"2024-04-06T03:40:12.491313+0000","last_undegraded":"2024-04-06T03:40:12.491313+0000","last_fullsized":"2024-04-06T03:40:12.491313+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:35.752989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:33:41.412744+0000","last_clean_scrub_stamp":"2024-04-06T03:38:35.752989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"301'4978","reported_seq":5030,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.228042+0000","last_change":"2024-04-06T03:38:41.239398+0000","last_active":"2024-04-06T03:40:14.228042+0000","last_peered":"2024-04-06T03:40:14.228042+0000","last_clean":"2024-04-06T03:40:14.228042+0000","last_became_active":"2024-04-06T03:38:41.238215+0000","last_became_peered":"2024-04-06T03:38:41.238215+0000","last_unstale":"2024-04-06T03:40:14.228042+0000","last_undegraded":"2024-04-06T03:40:14.228042+0000","last_fullsized":"2024-04-06T03:40:14.228042+0000","mapping_epoch":280,"log_start":"287'200","ondisk_log_start":"287'200","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326238208,"num_objects":4978,"num_object_clones":0,"num_object_copies":14934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4978,"num_write_kb":318592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":532,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.490436+0000","last_change":"2024-04-06T03:38:56.770480+0000","last_active":"2024-04-06T03:40:12.490436+0000","last_peered":"2024-04-06T03:40:12.490436+0000","last_clean":"2024-04-06T03:40:12.490436+0000","last_became_active":"2024-04-06T03:20:58.131608+0000","last_became_peered":"2024-04-06T03:20:58.131608+0000","last_unstale":"2024-04-06T03:40:12.490436+0000","last_undegraded":"2024-04-06T03:40:12.490436+0000","last_fullsized":"2024-04-06T03:40:12.490436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_clean_scrub_stamp":"2024-04-06T03:38:56.770310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.899370+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":536,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501707+0000","last_change":"2024-04-06T03:39:12.124333+0000","last_active":"2024-04-06T03:40:12.501707+0000","last_peered":"2024-04-06T03:40:12.501707+0000","last_clean":"2024-04-06T03:40:12.501707+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:40:12.501707+0000","last_undegraded":"2024-04-06T03:40:12.501707+0000","last_fullsized":"2024-04-06T03:40:12.501707+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:12.111707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:39:12.111707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:27.440876+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"251'51","reported_seq":743,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501133+0000","last_change":"2024-04-06T03:38:57.125352+0000","last_active":"2024-04-06T03:40:12.501133+0000","last_peered":"2024-04-06T03:40:12.501133+0000","last_clean":"2024-04-06T03:40:12.501133+0000","last_became_active":"2024-04-06T03:35:56.084984+0000","last_became_peered":"2024-04-06T03:35:56.084984+0000","last_unstale":"2024-04-06T03:40:12.501133+0000","last_undegraded":"2024-04-06T03:40:12.501133+0000","last_fullsized":"2024-04-06T03:40:12.501133+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"251'51","last_scrub_stamp":"2024-04-06T03:38:57.125241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.125241+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:15.087256+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":397,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:05.000826+0000","last_change":"2024-04-06T03:39:32.313014+0000","last_active":"2024-04-06T03:40:05.000826+0000","last_peered":"2024-04-06T03:40:05.000826+0000","last_clean":"2024-04-06T03:40:05.000826+0000","last_became_active":"2024-04-06T03:26:18.643845+0000","last_became_peered":"2024-04-06T03:26:18.643845+0000","last_unstale":"2024-04-06T03:40:05.000826+0000","last_undegraded":"2024-04-06T03:40:05.000826+0000","last_fullsized":"2024-04-06T03:40:05.000826+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:32.312900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:39:32.312900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:43.961384+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"301'4991","reported_seq":5043,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.205474+0000","last_change":"2024-04-06T03:38:41.240014+0000","last_active":"2024-04-06T03:40:14.205474+0000","last_peered":"2024-04-06T03:40:14.205474+0000","last_clean":"2024-04-06T03:40:14.205474+0000","last_became_active":"2024-04-06T03:38:41.239246+0000","last_became_peered":"2024-04-06T03:38:41.239246+0000","last_unstale":"2024-04-06T03:40:14.205474+0000","last_undegraded":"2024-04-06T03:40:14.205474+0000","last_fullsized":"2024-04-06T03:40:14.205474+0000","mapping_epoch":280,"log_start":"288'300","ondisk_log_start":"288'300","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327090176,"num_objects":4991,"num_object_clones":0,"num_object_copies":14973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4991,"num_write_kb":319424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":526,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483974+0000","last_change":"2024-04-06T03:39:33.010891+0000","last_active":"2024-04-06T03:40:12.483974+0000","last_peered":"2024-04-06T03:40:12.483974+0000","last_clean":"2024-04-06T03:40:12.483974+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:40:12.483974+0000","last_undegraded":"2024-04-06T03:40:12.483974+0000","last_fullsized":"2024-04-06T03:40:12.483974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:33.010635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:39:33.010635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.375211+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":544,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.492009+0000","last_change":"2024-04-06T03:38:38.749440+0000","last_active":"2024-04-06T03:40:12.492009+0000","last_peered":"2024-04-06T03:40:12.492009+0000","last_clean":"2024-04-06T03:40:12.492009+0000","last_became_active":"2024-04-06T03:20:51.153239+0000","last_became_peered":"2024-04-06T03:20:51.153239+0000","last_unstale":"2024-04-06T03:40:12.492009+0000","last_undegraded":"2024-04-06T03:40:12.492009+0000","last_fullsized":"2024-04-06T03:40:12.492009+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:38:38.749329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:38.749329+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"199'23","reported_seq":568,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:05.002221+0000","last_change":"2024-04-06T03:39:06.307894+0000","last_active":"2024-04-06T03:40:05.002221+0000","last_peered":"2024-04-06T03:40:05.002221+0000","last_clean":"2024-04-06T03:40:05.002221+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:40:05.002221+0000","last_undegraded":"2024-04-06T03:40:05.002221+0000","last_fullsized":"2024-04-06T03:40:05.002221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'23","last_scrub_stamp":"2024-04-06T03:39:06.307786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:06.307786+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.806563+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":1663,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:10.342276+0000","last_change":"2024-04-06T03:39:14.550550+0000","last_active":"2024-04-06T03:40:10.342276+0000","last_peered":"2024-04-06T03:40:10.342276+0000","last_clean":"2024-04-06T03:40:10.342276+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:40:10.342276+0000","last_undegraded":"2024-04-06T03:40:10.342276+0000","last_fullsized":"2024-04-06T03:40:10.342276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:39:14.550411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:39:14.550411+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.362228+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1131,"num_read_kb":1126,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":530,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483779+0000","last_change":"2024-04-06T03:39:35.943774+0000","last_active":"2024-04-06T03:40:12.483779+0000","last_peered":"2024-04-06T03:40:12.483779+0000","last_clean":"2024-04-06T03:40:12.483779+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:40:12.483779+0000","last_undegraded":"2024-04-06T03:40:12.483779+0000","last_fullsized":"2024-04-06T03:40:12.483779+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_clean_scrub_stamp":"2024-04-06T03:39:35.943511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:44.828686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"300'4944","reported_seq":4994,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:11.709018+0000","last_change":"2024-04-06T03:38:41.242596+0000","last_active":"2024-04-06T03:40:11.709018+0000","last_peered":"2024-04-06T03:40:11.709018+0000","last_clean":"2024-04-06T03:40:11.709018+0000","last_became_active":"2024-04-06T03:38:41.242148+0000","last_became_peered":"2024-04-06T03:38:41.242148+0000","last_unstale":"2024-04-06T03:40:11.709018+0000","last_undegraded":"2024-04-06T03:40:11.709018+0000","last_fullsized":"2024-04-06T03:40:11.709018+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4944,"log_dups_size":0,"ondisk_log_size":4944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:12.681095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324009984,"num_objects":4944,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4944,"num_write_kb":316416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":516,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501609+0000","last_change":"2024-04-06T03:37:31.563861+0000","last_active":"2024-04-06T03:40:12.501609+0000","last_peered":"2024-04-06T03:40:12.501609+0000","last_clean":"2024-04-06T03:40:12.501609+0000","last_became_active":"2024-04-06T03:20:58.132354+0000","last_became_peered":"2024-04-06T03:20:58.132354+0000","last_unstale":"2024-04-06T03:40:12.501609+0000","last_undegraded":"2024-04-06T03:40:12.501609+0000","last_fullsized":"2024-04-06T03:40:12.501609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:37:31.563716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":390,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501916+0000","last_change":"2024-04-06T03:37:34.532858+0000","last_active":"2024-04-06T03:40:12.501916+0000","last_peered":"2024-04-06T03:40:12.501916+0000","last_clean":"2024-04-06T03:40:12.501916+0000","last_became_active":"2024-04-06T03:26:18.621727+0000","last_became_peered":"2024-04-06T03:26:18.621727+0000","last_unstale":"2024-04-06T03:40:12.501916+0000","last_undegraded":"2024-04-06T03:40:12.501916+0000","last_fullsized":"2024-04-06T03:40:12.501916+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:37:34.532778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:37:34.532778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"199'24","reported_seq":565,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.999008+0000","last_change":"2024-04-06T03:38:31.679871+0000","last_active":"2024-04-06T03:40:04.999008+0000","last_peered":"2024-04-06T03:40:04.999008+0000","last_clean":"2024-04-06T03:40:04.999008+0000","last_became_active":"2024-04-06T03:20:53.351381+0000","last_became_peered":"2024-04-06T03:20:53.351381+0000","last_unstale":"2024-04-06T03:40:04.999008+0000","last_undegraded":"2024-04-06T03:40:04.999008+0000","last_fullsized":"2024-04-06T03:40:04.999008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'24","last_scrub_stamp":"2024-04-06T03:38:31.679737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:31.679737+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":539,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483864+0000","last_change":"2024-04-06T03:39:25.012594+0000","last_active":"2024-04-06T03:40:12.483864+0000","last_peered":"2024-04-06T03:40:12.483864+0000","last_clean":"2024-04-06T03:40:12.483864+0000","last_became_active":"2024-04-06T03:20:54.745646+0000","last_became_peered":"2024-04-06T03:20:54.745646+0000","last_unstale":"2024-04-06T03:40:12.483864+0000","last_undegraded":"2024-04-06T03:40:12.483864+0000","last_fullsized":"2024-04-06T03:40:12.483864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:25.012444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:25.012444+0000","last_clean_scrub_stamp":"2024-04-06T03:39:25.012444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:43.525977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"109'12","reported_seq":763,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991530+0000","last_change":"2024-04-06T03:38:42.258790+0000","last_active":"2024-04-06T03:40:04.991530+0000","last_peered":"2024-04-06T03:40:04.991530+0000","last_clean":"2024-04-06T03:40:04.991530+0000","last_became_active":"2024-04-06T03:20:55.768496+0000","last_became_peered":"2024-04-06T03:20:55.768496+0000","last_unstale":"2024-04-06T03:40:04.991530+0000","last_undegraded":"2024-04-06T03:40:04.991530+0000","last_fullsized":"2024-04-06T03:40:04.991530+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:38:42.258672+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:58.657247+0000","last_clean_scrub_stamp":"2024-04-06T03:38:42.258672+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:07.507957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"301'5096","reported_seq":5148,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.226644+0000","last_change":"2024-04-06T03:38:41.241641+0000","last_active":"2024-04-06T03:40:14.226644+0000","last_peered":"2024-04-06T03:40:14.226644+0000","last_clean":"2024-04-06T03:40:14.226644+0000","last_became_active":"2024-04-06T03:38:41.240235+0000","last_became_peered":"2024-04-06T03:38:41.240235+0000","last_unstale":"2024-04-06T03:40:14.226644+0000","last_undegraded":"2024-04-06T03:40:14.226644+0000","last_fullsized":"2024-04-06T03:40:14.226644+0000","mapping_epoch":280,"log_start":"288'400","ondisk_log_start":"288'400","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333971456,"num_objects":5096,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":326144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":515,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.995361+0000","last_change":"2024-04-06T03:39:29.642108+0000","last_active":"2024-04-06T03:40:04.995361+0000","last_peered":"2024-04-06T03:40:04.995361+0000","last_clean":"2024-04-06T03:40:04.995361+0000","last_became_active":"2024-04-06T03:20:58.131753+0000","last_became_peered":"2024-04-06T03:20:58.131753+0000","last_unstale":"2024-04-06T03:40:04.995361+0000","last_undegraded":"2024-04-06T03:40:04.995361+0000","last_fullsized":"2024-04-06T03:40:04.995361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:29.641865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:39:29.641865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:55.761400+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":396,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484524+0000","last_change":"2024-04-06T03:39:11.966576+0000","last_active":"2024-04-06T03:40:12.484524+0000","last_peered":"2024-04-06T03:40:12.484524+0000","last_clean":"2024-04-06T03:40:12.484524+0000","last_became_active":"2024-04-06T03:25:54.491253+0000","last_became_peered":"2024-04-06T03:25:54.491253+0000","last_unstale":"2024-04-06T03:40:12.484524+0000","last_undegraded":"2024-04-06T03:40:12.484524+0000","last_fullsized":"2024-04-06T03:40:12.484524+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.966446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:38.797317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":527,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502891+0000","last_change":"2024-04-06T03:38:58.011298+0000","last_active":"2024-04-06T03:40:12.502891+0000","last_peered":"2024-04-06T03:40:12.502891+0000","last_clean":"2024-04-06T03:40:12.502891+0000","last_became_active":"2024-04-06T03:20:51.151036+0000","last_became_peered":"2024-04-06T03:20:51.151036+0000","last_unstale":"2024-04-06T03:40:12.502891+0000","last_undegraded":"2024-04-06T03:40:12.502891+0000","last_fullsized":"2024-04-06T03:40:12.502891+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:58.011077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:58.011077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:24.074597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"199'37","reported_seq":588,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.483839+0000","last_change":"2024-04-06T03:38:37.957427+0000","last_active":"2024-04-06T03:40:12.483839+0000","last_peered":"2024-04-06T03:40:12.483839+0000","last_clean":"2024-04-06T03:40:12.483839+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:40:12.483839+0000","last_undegraded":"2024-04-06T03:40:12.483839+0000","last_fullsized":"2024-04-06T03:40:12.483839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'37","last_scrub_stamp":"2024-04-06T03:38:37.957275+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:38:37.957275+0000","objects_scrubbed":7,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:13.094730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"301'5193","reported_seq":5245,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.238258+0000","last_change":"2024-04-06T03:38:40.976607+0000","last_active":"2024-04-06T03:40:14.238258+0000","last_peered":"2024-04-06T03:40:14.238258+0000","last_clean":"2024-04-06T03:40:14.238258+0000","last_became_active":"2024-04-06T03:38:40.976213+0000","last_became_peered":"2024-04-06T03:38:40.976213+0000","last_unstale":"2024-04-06T03:40:14.238258+0000","last_undegraded":"2024-04-06T03:40:14.238258+0000","last_fullsized":"2024-04-06T03:40:14.238258+0000","mapping_epoch":280,"log_start":"288'500","ondisk_log_start":"288'500","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340328448,"num_objects":5193,"num_object_clones":0,"num_object_copies":15579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":332352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":353,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.995547+0000","last_change":"2024-04-06T03:38:58.987637+0000","last_active":"2024-04-06T03:40:04.995547+0000","last_peered":"2024-04-06T03:40:04.995547+0000","last_clean":"2024-04-06T03:40:04.995547+0000","last_became_active":"2024-04-06T03:27:10.497525+0000","last_became_peered":"2024-04-06T03:27:10.497525+0000","last_unstale":"2024-04-06T03:40:04.995547+0000","last_undegraded":"2024-04-06T03:40:04.995547+0000","last_fullsized":"2024-04-06T03:40:04.995547+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_clean_scrub_stamp":"2024-04-06T03:38:58.987355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:25.095281+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":773,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484887+0000","last_change":"2024-04-06T03:37:42.672606+0000","last_active":"2024-04-06T03:40:12.484887+0000","last_peered":"2024-04-06T03:40:12.484887+0000","last_clean":"2024-04-06T03:40:12.484887+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:40:12.484887+0000","last_undegraded":"2024-04-06T03:40:12.484887+0000","last_fullsized":"2024-04-06T03:40:12.484887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:37:42.672471+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:35:33.269156+0000","last_clean_scrub_stamp":"2024-04-06T03:37:42.672471+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:13.094709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":543,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.489452+0000","last_change":"2024-04-06T03:38:50.753603+0000","last_active":"2024-04-06T03:40:12.489452+0000","last_peered":"2024-04-06T03:40:12.489452+0000","last_clean":"2024-04-06T03:40:12.489452+0000","last_became_active":"2024-04-06T03:20:51.167088+0000","last_became_peered":"2024-04-06T03:20:51.167088+0000","last_unstale":"2024-04-06T03:40:12.489452+0000","last_undegraded":"2024-04-06T03:40:12.489452+0000","last_fullsized":"2024-04-06T03:40:12.489452+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:50.753393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:38:50.753393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"199'39","reported_seq":590,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991136+0000","last_change":"2024-04-06T03:39:15.520741+0000","last_active":"2024-04-06T03:40:04.991136+0000","last_peered":"2024-04-06T03:40:04.991136+0000","last_clean":"2024-04-06T03:40:04.991136+0000","last_became_active":"2024-04-06T03:20:53.349962+0000","last_became_peered":"2024-04-06T03:20:53.349962+0000","last_unstale":"2024-04-06T03:40:04.991136+0000","last_undegraded":"2024-04-06T03:40:04.991136+0000","last_fullsized":"2024-04-06T03:40:04.991136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:39:15.520549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:15.520549+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:33.831610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"301'5175","reported_seq":5227,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.040220+0000","last_change":"2024-04-06T03:38:40.984725+0000","last_active":"2024-04-06T03:40:14.040220+0000","last_peered":"2024-04-06T03:40:14.040220+0000","last_clean":"2024-04-06T03:40:14.040220+0000","last_became_active":"2024-04-06T03:38:40.984168+0000","last_became_peered":"2024-04-06T03:38:40.984168+0000","last_unstale":"2024-04-06T03:40:14.040220+0000","last_undegraded":"2024-04-06T03:40:14.040220+0000","last_fullsized":"2024-04-06T03:40:14.040220+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":331200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":525,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.485075+0000","last_change":"2024-04-06T03:39:22.029484+0000","last_active":"2024-04-06T03:40:12.485075+0000","last_peered":"2024-04-06T03:40:12.485075+0000","last_clean":"2024-04-06T03:40:12.485075+0000","last_became_active":"2024-04-06T03:20:57.825561+0000","last_became_peered":"2024-04-06T03:20:57.825561+0000","last_unstale":"2024-04-06T03:40:12.485075+0000","last_undegraded":"2024-04-06T03:40:12.485075+0000","last_fullsized":"2024-04-06T03:40:12.485075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:22.029278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:22.029278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:44.736300+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":789,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502186+0000","last_change":"2024-04-06T03:38:40.981295+0000","last_active":"2024-04-06T03:40:12.502186+0000","last_peered":"2024-04-06T03:40:12.502186+0000","last_clean":"2024-04-06T03:40:12.502186+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:40:12.502186+0000","last_undegraded":"2024-04-06T03:40:12.502186+0000","last_fullsized":"2024-04-06T03:40:12.502186+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:38:40.981122+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:38:40.981122+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":404,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.991437+0000","last_change":"2024-04-06T03:38:36.268944+0000","last_active":"2024-04-06T03:40:04.991437+0000","last_peered":"2024-04-06T03:40:04.991437+0000","last_clean":"2024-04-06T03:40:04.991437+0000","last_became_active":"2024-04-06T03:25:19.549690+0000","last_became_peered":"2024-04-06T03:25:19.549690+0000","last_unstale":"2024-04-06T03:40:04.991437+0000","last_undegraded":"2024-04-06T03:40:04.991437+0000","last_fullsized":"2024-04-06T03:40:04.991437+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:36.268788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:23.908290+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.268788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"199'21","reported_seq":560,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994472+0000","last_change":"2024-04-06T03:38:57.282224+0000","last_active":"2024-04-06T03:40:04.994472+0000","last_peered":"2024-04-06T03:40:04.994472+0000","last_clean":"2024-04-06T03:40:04.994472+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:40:04.994472+0000","last_undegraded":"2024-04-06T03:40:04.994472+0000","last_fullsized":"2024-04-06T03:40:04.994472+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'21","last_scrub_stamp":"2024-04-06T03:38:57.282016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:38:57.282016+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:05.356143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"301'5192","reported_seq":5244,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.747681+0000","last_change":"2024-04-06T03:38:40.975757+0000","last_active":"2024-04-06T03:40:14.747681+0000","last_peered":"2024-04-06T03:40:14.747681+0000","last_clean":"2024-04-06T03:40:14.747681+0000","last_became_active":"2024-04-06T03:38:40.974964+0000","last_became_peered":"2024-04-06T03:38:40.974964+0000","last_unstale":"2024-04-06T03:40:14.747681+0000","last_undegraded":"2024-04-06T03:40:14.747681+0000","last_fullsized":"2024-04-06T03:40:14.747681+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.444903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340262912,"num_objects":5192,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5192,"num_write_kb":332288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":541,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501978+0000","last_change":"2024-04-06T03:38:35.085842+0000","last_active":"2024-04-06T03:40:12.501978+0000","last_peered":"2024-04-06T03:40:12.501978+0000","last_clean":"2024-04-06T03:40:12.501978+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:40:12.501978+0000","last_undegraded":"2024-04-06T03:40:12.501978+0000","last_fullsized":"2024-04-06T03:40:12.501978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:35.085716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:38:35.085716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":352,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484637+0000","last_change":"2024-04-06T03:38:59.034375+0000","last_active":"2024-04-06T03:40:12.484637+0000","last_peered":"2024-04-06T03:40:12.484637+0000","last_clean":"2024-04-06T03:40:12.484637+0000","last_became_active":"2024-04-06T03:27:10.501454+0000","last_became_peered":"2024-04-06T03:27:10.501454+0000","last_unstale":"2024-04-06T03:40:12.484637+0000","last_undegraded":"2024-04-06T03:40:12.484637+0000","last_fullsized":"2024-04-06T03:40:12.484637+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:59.034078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:38:59.034078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.898507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":535,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501714+0000","last_change":"2024-04-06T03:39:27.906651+0000","last_active":"2024-04-06T03:40:12.501714+0000","last_peered":"2024-04-06T03:40:12.501714+0000","last_clean":"2024-04-06T03:40:12.501714+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:40:12.501714+0000","last_undegraded":"2024-04-06T03:40:12.501714+0000","last_fullsized":"2024-04-06T03:40:12.501714+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:39:27.906531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:36.128418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"199'41","reported_seq":484,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.484433+0000","last_change":"2024-04-06T03:39:20.965683+0000","last_active":"2024-04-06T03:40:12.484433+0000","last_peered":"2024-04-06T03:40:12.484433+0000","last_clean":"2024-04-06T03:40:12.484433+0000","last_became_active":"2024-04-06T03:26:18.616944+0000","last_became_peered":"2024-04-06T03:26:18.616944+0000","last_unstale":"2024-04-06T03:40:12.484433+0000","last_undegraded":"2024-04-06T03:40:12.484433+0000","last_fullsized":"2024-04-06T03:40:12.484433+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'41","last_scrub_stamp":"2024-04-06T03:39:20.965523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:39:20.965523+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:22.036992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"300'4906","reported_seq":4956,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:11.364756+0000","last_change":"2024-04-06T03:38:40.977014+0000","last_active":"2024-04-06T03:40:11.364756+0000","last_peered":"2024-04-06T03:40:11.364756+0000","last_clean":"2024-04-06T03:40:11.364756+0000","last_became_active":"2024-04-06T03:38:40.976105+0000","last_became_peered":"2024-04-06T03:38:40.976105+0000","last_unstale":"2024-04-06T03:40:11.364756+0000","last_undegraded":"2024-04-06T03:40:11.364756+0000","last_fullsized":"2024-04-06T03:40:11.364756+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":4906,"log_dups_size":0,"ondisk_log_size":4906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:11.660206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321519616,"num_objects":4906,"num_object_clones":0,"num_object_copies":14718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4906,"num_write_kb":313984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":515,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501208+0000","last_change":"2024-04-06T03:39:10.006464+0000","last_active":"2024-04-06T03:40:12.501208+0000","last_peered":"2024-04-06T03:40:12.501208+0000","last_clean":"2024-04-06T03:40:12.501208+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:40:12.501208+0000","last_undegraded":"2024-04-06T03:40:12.501208+0000","last_fullsized":"2024-04-06T03:40:12.501208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:10.006346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:39:10.006346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:15.094954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501141+0000","last_change":"2024-04-06T03:39:09.163498+0000","last_active":"2024-04-06T03:40:12.501141+0000","last_peered":"2024-04-06T03:40:12.501141+0000","last_clean":"2024-04-06T03:40:12.501141+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:40:12.501141+0000","last_undegraded":"2024-04-06T03:40:12.501141+0000","last_fullsized":"2024-04-06T03:40:12.501141+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:09.163381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:43.565930+0000","last_clean_scrub_stamp":"2024-04-06T03:39:09.163381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:28.061368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501025+0000","last_change":"2024-04-06T03:38:36.976257+0000","last_active":"2024-04-06T03:40:12.501025+0000","last_peered":"2024-04-06T03:40:12.501025+0000","last_clean":"2024-04-06T03:40:12.501025+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:40:12.501025+0000","last_undegraded":"2024-04-06T03:40:12.501025+0000","last_fullsized":"2024-04-06T03:40:12.501025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:36.976013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:38:36.976013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"199'39","reported_seq":580,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501907+0000","last_change":"2024-04-06T03:38:45.009952+0000","last_active":"2024-04-06T03:40:12.501907+0000","last_peered":"2024-04-06T03:40:12.501907+0000","last_clean":"2024-04-06T03:40:12.501907+0000","last_became_active":"2024-04-06T03:20:53.354866+0000","last_became_peered":"2024-04-06T03:20:53.354866+0000","last_unstale":"2024-04-06T03:40:12.501907+0000","last_undegraded":"2024-04-06T03:40:12.501907+0000","last_fullsized":"2024-04-06T03:40:12.501907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'39","last_scrub_stamp":"2024-04-06T03:38:45.009705+0000","last_deep_scrub":"199'39","last_deep_scrub_stamp":"2024-04-06T03:35:49.576918+0000","last_clean_scrub_stamp":"2024-04-06T03:38:45.009705+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"301'5165","reported_seq":5217,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:14.786894+0000","last_change":"2024-04-06T03:38:40.982609+0000","last_active":"2024-04-06T03:40:14.786894+0000","last_peered":"2024-04-06T03:40:14.786894+0000","last_clean":"2024-04-06T03:40:14.786894+0000","last_became_active":"2024-04-06T03:38:40.981962+0000","last_became_peered":"2024-04-06T03:38:40.981962+0000","last_unstale":"2024-04-06T03:40:14.786894+0000","last_undegraded":"2024-04-06T03:40:14.786894+0000","last_fullsized":"2024-04-06T03:40:14.786894+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:19.444736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338493440,"num_objects":5165,"num_object_clones":0,"num_object_copies":15495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5165,"num_write_kb":330560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":521,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.994012+0000","last_change":"2024-04-06T03:38:59.917478+0000","last_active":"2024-04-06T03:40:04.994012+0000","last_peered":"2024-04-06T03:40:04.994012+0000","last_clean":"2024-04-06T03:40:04.994012+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:40:04.994012+0000","last_undegraded":"2024-04-06T03:40:04.994012+0000","last_fullsized":"2024-04-06T03:40:04.994012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:59.917290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:38:59.917290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:17.591600+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":545,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501072+0000","last_change":"2024-04-06T03:39:11.097645+0000","last_active":"2024-04-06T03:40:12.501072+0000","last_peered":"2024-04-06T03:40:12.501072+0000","last_clean":"2024-04-06T03:40:12.501072+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:40:12.501072+0000","last_undegraded":"2024-04-06T03:40:12.501072+0000","last_fullsized":"2024-04-06T03:40:12.501072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_clean_scrub_stamp":"2024-04-06T03:39:11.097314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:17.929540+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":551,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.491563+0000","last_change":"2024-04-06T03:39:13.054289+0000","last_active":"2024-04-06T03:40:12.491563+0000","last_peered":"2024-04-06T03:40:12.491563+0000","last_clean":"2024-04-06T03:40:12.491563+0000","last_became_active":"2024-04-06T03:20:51.161299+0000","last_became_peered":"2024-04-06T03:20:51.161299+0000","last_unstale":"2024-04-06T03:40:12.491563+0000","last_undegraded":"2024-04-06T03:40:12.491563+0000","last_fullsized":"2024-04-06T03:40:12.491563+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:39:13.053965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:30.746261+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"199'25","reported_seq":536,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.998747+0000","last_change":"2024-04-06T03:39:25.599044+0000","last_active":"2024-04-06T03:40:04.998747+0000","last_peered":"2024-04-06T03:40:04.998747+0000","last_clean":"2024-04-06T03:40:04.998747+0000","last_became_active":"2024-04-06T03:25:54.495696+0000","last_became_peered":"2024-04-06T03:25:54.495696+0000","last_unstale":"2024-04-06T03:40:04.998747+0000","last_undegraded":"2024-04-06T03:40:04.998747+0000","last_fullsized":"2024-04-06T03:40:04.998747+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'25","last_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:39:25.598852+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:35.014424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"300'5037","reported_seq":5087,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:11.380234+0000","last_change":"2024-04-06T03:38:40.986276+0000","last_active":"2024-04-06T03:40:11.380234+0000","last_peered":"2024-04-06T03:40:11.380234+0000","last_clean":"2024-04-06T03:40:11.380234+0000","last_became_active":"2024-04-06T03:38:40.985850+0000","last_became_peered":"2024-04-06T03:38:40.985850+0000","last_unstale":"2024-04-06T03:40:11.380234+0000","last_undegraded":"2024-04-06T03:40:11.380234+0000","last_fullsized":"2024-04-06T03:40:11.380234+0000","mapping_epoch":280,"log_start":"0'0","ondisk_log_start":"0'0","created":280,"last_epoch_clean":281,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:39.929477+0000","last_clean_scrub_stamp":"2024-04-06T03:38:39.929477+0000","objects_scrubbed":0,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:11.660261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330104832,"num_objects":5037,"num_object_clones":0,"num_object_copies":15111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5037,"num_write_kb":322368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":392,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-04-06T03:40:04.999410+0000","last_change":"2024-04-06T03:39:21.333266+0000","last_active":"2024-04-06T03:40:04.999410+0000","last_peered":"2024-04-06T03:40:04.999410+0000","last_clean":"2024-04-06T03:40:04.999410+0000","last_became_active":"2024-04-06T03:26:18.641975+0000","last_became_peered":"2024-04-06T03:26:18.641975+0000","last_unstale":"2024-04-06T03:40:04.999410+0000","last_undegraded":"2024-04-06T03:40:04.999410+0000","last_fullsized":"2024-04-06T03:40:04.999410+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:39:21.333139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:39:21.333139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:39.938998+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":553,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501929+0000","last_change":"2024-04-06T03:38:41.932988+0000","last_active":"2024-04-06T03:40:12.501929+0000","last_peered":"2024-04-06T03:40:12.501929+0000","last_clean":"2024-04-06T03:40:12.501929+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:40:12.501929+0000","last_undegraded":"2024-04-06T03:40:12.501929+0000","last_fullsized":"2024-04-06T03:40:12.501929+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:38:41.932886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:38:41.932886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"291'34","reported_seq":581,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.502068+0000","last_change":"2024-04-06T03:39:26.934856+0000","last_active":"2024-04-06T03:40:12.502068+0000","last_peered":"2024-04-06T03:40:12.502068+0000","last_clean":"2024-04-06T03:40:12.502068+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:40:12.502068+0000","last_undegraded":"2024-04-06T03:40:12.502068+0000","last_fullsized":"2024-04-06T03:40:12.502068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'34","last_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_deep_scrub":"291'34","last_deep_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_clean_scrub_stamp":"2024-04-06T03:39:26.934752+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:40:41.324584+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"209'12","reported_seq":771,"reported_epoch":301,"state":"active+clean","last_fresh":"2024-04-06T03:40:12.501490+0000","last_change":"2024-04-06T03:38:38.151300+0000","last_active":"2024-04-06T03:40:12.501490+0000","last_peered":"2024-04-06T03:40:12.501490+0000","last_clean":"2024-04-06T03:40:12.501490+0000","last_became_active":"2024-04-06T03:20:55.765257+0000","last_became_peered":"2024-04-06T03:20:55.765257+0000","last_unstale":"2024-04-06T03:40:12.501490+0000","last_undegraded":"2024-04-06T03:40:12.501490+0000","last_fullsized":"2024-04-06T03:40:12.501490+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'12","last_scrub_stamp":"2024-04-06T03:38:38.151186+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:38:38.151186+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5322113024,"num_objects":81209,"num_object_clones":0,"num_object_copies":243627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81209,"num_write_kb":5197376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16000090112,"data_stored":16000090112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79309,"ondisk_log_size":79309,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1414,"num_read_kb":1139,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1002,"ondisk_log_size":1002,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1131,"num_read_kb":1126,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743667,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1702404,"kb_used_data":1324156,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92042748,"statfs":{"total":95995035648,"available":94251773952,"internally_reserved":0,"allocated":1355935744,"data_stored":1355588976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":387316170},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136125,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2587920,"kb_used_data":2263752,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":91157232,"statfs":{"total":95995035648,"available":93345005568,"internally_reserved":0,"allocated":2318082048,"data_stored":2317786450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575154,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1394972,"kb_used_data":980628,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92350180,"statfs":{"total":95995035648,"available":94566584320,"internally_reserved":0,"allocated":1004163072,"data_stored":1003854496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":424278470},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738683,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3644384,"kb_used_data":3171736,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":90100768,"statfs":{"total":95995035648,"available":92263186432,"internally_reserved":0,"allocated":3247857664,"data_stored":3247521555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378920,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2301200,"kb_used_data":1967368,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91443952,"statfs":{"total":95995035648,"available":93638606848,"internally_reserved":0,"allocated":2014584832,"data_stored":2014288288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084571,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2065732,"kb_used_data":1593084,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91679420,"statfs":{"total":95995035648,"available":93879726080,"internally_reserved":0,"allocated":1631318016,"data_stored":1631008494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":483981767},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855635,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2909696,"kb_used_data":2611884,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":90835456,"statfs":{"total":95995035648,"available":93015506944,"internally_reserved":0,"allocated":2674569216,"data_stored":2674236293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495881,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2040956,"kb_used_data":1640500,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91704196,"statfs":{"total":95995035648,"available":93905096704,"internally_reserved":0,"allocated":1679872000,"data_stored":1679595474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3267624960,"data_stored":3267624960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1012137984,"data_stored":1012137984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2021785600,"data_stored":2021785600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1632894976,"data_stored":1632894976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2681077760,"data_stored":2681077760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1688797184,"data_stored":1688797184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2335375360,"data_stored":2335375360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1360396288,"data_stored":1360396288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:40:19.988 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-06T03:40:19.989 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:40:19.989 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-04-06T03:40:19.989 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-06T03:40:19.989 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-06T03:40:19.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-04-06T03:40:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:19.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:20.083 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:40:19 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:40:20.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:20.127 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:20.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:20.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:20 smithi050 bash[21467]: audit 2024-04-06T03:40:20.029666+0000 mon.a (mon.0) 1534 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:20 smithi050 bash[29221]: audit 2024-04-06T03:40:20.029666+0000 mon.a (mon.0) 1534 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:20.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:20 smithi153 bash[23852]: audit 2024-04-06T03:40:20.029666+0000 mon.a (mon.0) 1534 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:20.266 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:20.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:20.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:20.401 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:20.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:20.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:20.548 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:40:20.594 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:40:20.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 76 16 86999 86983 71.4694 65.0625 0.00643013 0.0139869 2024-04-06T03:40:20.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:20.653 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:40:20.657 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:20.689 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:40:20.726 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:40:20.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:20.765 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:40:20.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:20.809 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:40:20.844 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:40:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:40:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:40:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:40:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:40:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:40:20.850 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:40:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:40:20.851 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:40:20.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:20.926 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:20.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:21.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:21.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:21.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:21.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:21.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:21 smithi050 bash[29221]: audit 2024-04-06T03:40:19.973924+0000 mgr.y (mgr.24368) 714 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:21.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:21 smithi050 bash[29221]: cluster 2024-04-06T03:40:20.405214+0000 mgr.y (mgr.24368) 715 : cluster 0 pgmap v763: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 797 op/s 2024-04-06T03:40:21.166 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:21 smithi050 bash[29221]: audit 2024-04-06T03:40:20.482308+0000 mon.c (mon.2) 93 : audit 0 from='client.? 172.21.15.50:0/248701211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:40:21.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:21 smithi050 bash[21467]: audit 2024-04-06T03:40:19.973924+0000 mgr.y (mgr.24368) 714 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:21.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:21 smithi050 bash[21467]: cluster 2024-04-06T03:40:20.405214+0000 mgr.y (mgr.24368) 715 : cluster 0 pgmap v763: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 797 op/s 2024-04-06T03:40:21.168 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:21 smithi050 bash[21467]: audit 2024-04-06T03:40:20.482308+0000 mon.c (mon.2) 93 : audit 0 from='client.? 172.21.15.50:0/248701211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:40:21.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:21.284 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:21.294 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:21 smithi153 bash[23852]: audit 2024-04-06T03:40:19.973924+0000 mgr.y (mgr.24368) 714 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:21.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:21 smithi153 bash[23852]: cluster 2024-04-06T03:40:20.405214+0000 mgr.y (mgr.24368) 715 : cluster 0 pgmap v763: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 797 op/s 2024-04-06T03:40:21.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:21 smithi153 bash[23852]: audit 2024-04-06T03:40:20.482308+0000 mon.c (mon.2) 93 : audit 0 from='client.? 172.21.15.50:0/248701211' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:40:21.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:21.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:21.436 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:21.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:21.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:21.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 77 16 87787 87771 71.181 49.25 0.0222729 0.0140024 2024-04-06T03:40:21.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:21.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:21.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:21.833 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:21.937 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:21.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:21.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:22.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:22.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:22.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:22.216 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:22 smithi153 bash[23852]: cluster 2024-04-06T03:40:20.241849+0000 osd.3 (osd.3) 468 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:22.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:22 smithi153 bash[23852]: cluster 2024-04-06T03:40:20.241922+0000 osd.3 (osd.3) 469 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:22.217 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:22 smithi153 bash[23852]: cluster 2024-04-06T03:40:21.100946+0000 mon.a (mon.0) 1535 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-04-06T03:40:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:22.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:22.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:22 smithi050 bash[21467]: cluster 2024-04-06T03:40:20.241849+0000 osd.3 (osd.3) 468 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:22.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:22 smithi050 bash[21467]: cluster 2024-04-06T03:40:20.241922+0000 osd.3 (osd.3) 469 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:22.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:22 smithi050 bash[21467]: cluster 2024-04-06T03:40:21.100946+0000 mon.a (mon.0) 1535 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-04-06T03:40:22.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:22 smithi050 bash[29221]: cluster 2024-04-06T03:40:20.241849+0000 osd.3 (osd.3) 468 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:22.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:22 smithi050 bash[29221]: cluster 2024-04-06T03:40:20.241922+0000 osd.3 (osd.3) 469 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:22.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:22 smithi050 bash[29221]: cluster 2024-04-06T03:40:21.100946+0000 mon.a (mon.0) 1535 : cluster 0 osdmap e303: 8 total, 8 up, 8 in 2024-04-06T03:40:22.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:22.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:22.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 78 16 88393 88377 70.7542 37.875 0.0168931 0.0141286 2024-04-06T03:40:22.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:22.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:22.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:22.813 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:22.919 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:22.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:23.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:23.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:23.154 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:23.155 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:23.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:23.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:23.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:23.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:23.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:23.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:23 smithi153 bash[23852]: cluster 2024-04-06T03:40:22.405961+0000 mgr.y (mgr.24368) 716 : cluster 0 pgmap v765: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 799 op/s 2024-04-06T03:40:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:23.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:23.472 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:23 smithi050 bash[21467]: cluster 2024-04-06T03:40:22.405961+0000 mgr.y (mgr.24368) 716 : cluster 0 pgmap v765: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 799 op/s 2024-04-06T03:40:23.473 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:23 smithi050 bash[29221]: cluster 2024-04-06T03:40:22.405961+0000 mgr.y (mgr.24368) 716 : cluster 0 pgmap v765: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 MiB/s wr, 799 op/s 2024-04-06T03:40:23.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:23.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:23.615 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 79 16 89405 89389 70.6593 63.25 0.00872053 0.0141468 2024-04-06T03:40:23.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:23.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:23.763 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:23.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:23.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:23.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:24.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:24.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:24.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:24.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:24.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:24.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:24.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:24.436 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:24.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:24.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:24.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:24.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:40:24.618719+0000 min lat: 0.0011709 max lat: 0.652534 avg lat: 0.0141599 2024-04-06T03:40:24.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:40:24.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 80 16 90456 90440 70.5971 65.6875 0.0122672 0.0141599 2024-04-06T03:40:24.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:24.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:40:24.708 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:24.808 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:40:24.814 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:24.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:24.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:24.979 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:24.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:25.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:25.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:25.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:25.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:24 smithi050 bash[21467]: cluster 2024-04-06T03:40:24.406600+0000 mgr.y (mgr.24368) 717 : cluster 0 pgmap v766: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 822 op/s 2024-04-06T03:40:25.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:24 smithi050 bash[29221]: cluster 2024-04-06T03:40:24.406600+0000 mgr.y (mgr.24368) 717 : cluster 0 pgmap v766: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 822 op/s 2024-04-06T03:40:25.257 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:24 smithi153 bash[23852]: cluster 2024-04-06T03:40:24.406600+0000 mgr.y (mgr.24368) 717 : cluster 0 pgmap v766: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 822 op/s 2024-04-06T03:40:25.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:25.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:25.518 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:25.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 81 15 91432 91417 70.4794 61.0625 0.0212278 0.0141832 2024-04-06T03:40:25.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:25.647 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:25.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:25.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:25.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:25.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:25.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:26.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:26.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:26.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:26.171 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:26.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:26.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:26.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:26.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:26.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:26.569 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:26.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 82 15 92497 92482 70.4316 66.5625 0.0102735 0.0141931 2024-04-06T03:40:26.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:26.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:26.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:26.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:26.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:26.963 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:26.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:27.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:27.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:27.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:27.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:27.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:27.313 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:27.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:27.460 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:27.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:27.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:27.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 83 16 93125 93109 70.0554 39.1875 0.0122691 0.0142691 2024-04-06T03:40:27.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:27.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:27 smithi153 bash[23852]: cluster 2024-04-06T03:40:26.409242+0000 mgr.y (mgr.24368) 718 : cluster 0 pgmap v767: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 812 op/s 2024-04-06T03:40:27.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:27.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:27 smithi050 bash[21467]: cluster 2024-04-06T03:40:26.409242+0000 mgr.y (mgr.24368) 718 : cluster 0 pgmap v767: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 812 op/s 2024-04-06T03:40:27.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:27 smithi050 bash[29221]: cluster 2024-04-06T03:40:26.409242+0000 mgr.y (mgr.24368) 718 : cluster 0 pgmap v767: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 51 MiB/s wr, 812 op/s 2024-04-06T03:40:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:27.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:27.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:27.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:27.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:28.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:28.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:28.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:28.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:28.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:28.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:28.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:28.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 84 16 94034 94018 69.8977 56.8125 0.00255145 0.0142878 2024-04-06T03:40:28.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:28.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:28.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:28.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:28.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:28.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:28.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:29.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:29.067 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:29.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:29.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:29.332 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:29.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:29.447 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:40:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:40:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:40:29.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:29.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:29.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:29.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 85 15 94771 94756 69.6182 46.125 0.0100294 0.0143592 2024-04-06T03:40:29.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:29.737 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:29 smithi153 bash[23852]: cluster 2024-04-06T03:40:28.409795+0000 mgr.y (mgr.24368) 719 : cluster 0 pgmap v768: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 53 MiB/s wr, 841 op/s 2024-04-06T03:40:29.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:29 smithi050 bash[21467]: cluster 2024-04-06T03:40:28.409795+0000 mgr.y (mgr.24368) 719 : cluster 0 pgmap v768: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 53 MiB/s wr, 841 op/s 2024-04-06T03:40:29.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:29 smithi050 bash[29221]: cluster 2024-04-06T03:40:28.409795+0000 mgr.y (mgr.24368) 719 : cluster 0 pgmap v768: 148 pgs: 148 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 53 MiB/s wr, 841 op/s 2024-04-06T03:40:29.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:29.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:29.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:29.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:30.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:30.083 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:40:30.083 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-06T03:40:30.083 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:40:30.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:40:29 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:40:30.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:30.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:30.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:30.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:30.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:30.616 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 86 16 95413 95397 69.2747 40.0625 0.00798871 0.0144305 2024-04-06T03:40:30.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.705 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:30.810 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:30.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:30.963 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:30.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:31.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:31.095 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:31.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:31.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:31.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:31.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:31.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:31.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:31.533 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:31.568 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:40:31.569 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:40:31.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 87 16 96341 96325 69.1447 58 0.00835409 0.0144568 2024-04-06T03:40:31.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:31.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:31.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:31.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:31 smithi153 bash[23852]: audit 2024-04-06T03:40:29.981756+0000 mgr.y (mgr.24368) 720 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:31.761 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:31 smithi153 bash[23852]: cluster 2024-04-06T03:40:30.410803+0000 mgr.y (mgr.24368) 721 : cluster 0 pgmap v769: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 889 op/s 2024-04-06T03:40:31.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:31.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:31.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:31 smithi050 bash[21467]: audit 2024-04-06T03:40:29.981756+0000 mgr.y (mgr.24368) 720 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:31.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:31 smithi050 bash[21467]: cluster 2024-04-06T03:40:30.410803+0000 mgr.y (mgr.24368) 721 : cluster 0 pgmap v769: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 889 op/s 2024-04-06T03:40:31.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:31 smithi050 bash[29221]: audit 2024-04-06T03:40:29.981756+0000 mgr.y (mgr.24368) 720 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:31.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:31 smithi050 bash[29221]: cluster 2024-04-06T03:40:30.410803+0000 mgr.y (mgr.24368) 721 : cluster 0 pgmap v769: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 889 op/s 2024-04-06T03:40:31.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:31.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:32.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:32.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:32.139 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:32.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:32.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:32.302 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:32.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:32.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:32.495 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:40:32.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:32.543 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:32 smithi153 bash[23852]: audit 2024-04-06T03:40:31.903752+0000 mon.a (mon.0) 1536 : audit 1 from='client.? 172.21.15.50:0/1187963080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:40:32.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:32.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 88 15 97297 97282 69.0386 59.8125 0.0235403 0.0144804 2024-04-06T03:40:32.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:32.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:32.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:32.819 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:32.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:32.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:32 smithi050 bash[21467]: audit 2024-04-06T03:40:31.903752+0000 mon.a (mon.0) 1536 : audit 1 from='client.? 172.21.15.50:0/1187963080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:40:32.938 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:32 smithi050 bash[29221]: audit 2024-04-06T03:40:31.903752+0000 mon.a (mon.0) 1536 : audit 1 from='client.? 172.21.15.50:0/1187963080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:40:32.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:32.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:33.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:33.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:33.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:33.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:33.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:33.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[21467]: cluster 2024-04-06T03:40:32.411389+0000 mgr.y (mgr.24368) 722 : cluster 0 pgmap v770: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 57 MiB/s wr, 914 op/s 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[21467]: cluster 2024-04-06T03:40:32.482609+0000 mon.a (mon.0) 1537 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[21467]: audit 2024-04-06T03:40:32.489857+0000 mon.a (mon.0) 1538 : audit 1 from='client.? 172.21.15.50:0/1187963080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[21467]: cluster 2024-04-06T03:40:32.497552+0000 mon.a (mon.0) 1539 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[21467]: cluster 2024-04-06T03:40:32.575079+0000 osd.2 (osd.2) 371 : cluster 0 2.c scrub starts 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[21467]: cluster 2024-04-06T03:40:32.580808+0000 osd.2 (osd.2) 372 : cluster 0 2.c scrub ok 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[29221]: cluster 2024-04-06T03:40:32.411389+0000 mgr.y (mgr.24368) 722 : cluster 0 pgmap v770: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 57 MiB/s wr, 914 op/s 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[29221]: cluster 2024-04-06T03:40:32.482609+0000 mon.a (mon.0) 1537 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:40:33.616 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[29221]: audit 2024-04-06T03:40:32.489857+0000 mon.a (mon.0) 1538 : audit 1 from='client.? 172.21.15.50:0/1187963080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:40:33.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[29221]: cluster 2024-04-06T03:40:32.497552+0000 mon.a (mon.0) 1539 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-04-06T03:40:33.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[29221]: cluster 2024-04-06T03:40:32.575079+0000 osd.2 (osd.2) 371 : cluster 0 2.c scrub starts 2024-04-06T03:40:33.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:33 smithi050 bash[29221]: cluster 2024-04-06T03:40:32.580808+0000 osd.2 (osd.2) 372 : cluster 0 2.c scrub ok 2024-04-06T03:40:33.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 89 16 98029 98013 68.7764 45.6875 0.00370638 0.0145225 2024-04-06T03:40:33.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:33.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:33 smithi153 bash[23852]: cluster 2024-04-06T03:40:32.411389+0000 mgr.y (mgr.24368) 722 : cluster 0 pgmap v770: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 57 MiB/s wr, 914 op/s 2024-04-06T03:40:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:33 smithi153 bash[23852]: cluster 2024-04-06T03:40:32.482609+0000 mon.a (mon.0) 1537 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:40:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:33 smithi153 bash[23852]: audit 2024-04-06T03:40:32.489857+0000 mon.a (mon.0) 1538 : audit 1 from='client.? 172.21.15.50:0/1187963080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:40:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:33 smithi153 bash[23852]: cluster 2024-04-06T03:40:32.497552+0000 mon.a (mon.0) 1539 : cluster 0 osdmap e304: 8 total, 8 up, 8 in 2024-04-06T03:40:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:33 smithi153 bash[23852]: cluster 2024-04-06T03:40:32.575079+0000 osd.2 (osd.2) 371 : cluster 0 2.c scrub starts 2024-04-06T03:40:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:33 smithi153 bash[23852]: cluster 2024-04-06T03:40:32.580808+0000 osd.2 (osd.2) 372 : cluster 0 2.c scrub ok 2024-04-06T03:40:33.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:33.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:34.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:34.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:34.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:34.209 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:34.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:34.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:34.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:34.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:34.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:34.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:34 smithi050 bash[21467]: cluster 2024-04-06T03:40:33.344832+0000 osd.0 (osd.0) 452 : cluster 0 2.1f scrub starts 2024-04-06T03:40:34.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:34 smithi050 bash[21467]: cluster 2024-04-06T03:40:33.349479+0000 osd.0 (osd.0) 453 : cluster 0 2.1f scrub ok 2024-04-06T03:40:34.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:34 smithi050 bash[29221]: cluster 2024-04-06T03:40:33.344832+0000 osd.0 (osd.0) 452 : cluster 0 2.1f scrub starts 2024-04-06T03:40:34.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:34 smithi050 bash[29221]: cluster 2024-04-06T03:40:33.349479+0000 osd.0 (osd.0) 453 : cluster 0 2.1f scrub ok 2024-04-06T03:40:34.607 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:34.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total time run: 90.011 2024-04-06T03:40:34.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total writes made: 98814 2024-04-06T03:40:34.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Write size: 65536 2024-04-06T03:40:34.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object size: 65536 2024-04-06T03:40:34.617 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Bandwidth (MB/sec): 68.6124 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Bandwidth: 25.7024 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max bandwidth (MB/sec): 153.625 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min bandwidth (MB/sec): 35.25 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average IOPS: 1097 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev IOPS: 411.238 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max IOPS: 2458 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min IOPS: 564 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average Latency(s): 0.0145702 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Latency(s): 0.0288017 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max latency(s): 0.652534 2024-04-06T03:40:34.618 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min latency(s): 0.0011709 2024-04-06T03:40:34.625 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Cleaning up (deleting benchmark objects) 2024-04-06T03:40:34.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:34.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:34 smithi153 bash[23852]: cluster 2024-04-06T03:40:33.344832+0000 osd.0 (osd.0) 452 : cluster 0 2.1f scrub starts 2024-04-06T03:40:34.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:34 smithi153 bash[23852]: cluster 2024-04-06T03:40:33.349479+0000 osd.0 (osd.0) 453 : cluster 0 2.1f scrub ok 2024-04-06T03:40:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:34.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:34.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:34.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:34.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:34.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:34.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:35.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:35.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:35.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:35.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:35.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:35.401 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:35.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:35.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:35.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:35.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:35.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[21467]: cluster 2024-04-06T03:40:34.328166+0000 osd.0 (osd.0) 454 : cluster 0 3.1b scrub starts 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[21467]: cluster 2024-04-06T03:40:34.334328+0000 osd.0 (osd.0) 455 : cluster 0 3.1b scrub ok 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[21467]: cluster 2024-04-06T03:40:34.411995+0000 mgr.y (mgr.24368) 723 : cluster 0 pgmap v772: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 866 op/s 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[21467]: audit 2024-04-06T03:40:35.049689+0000 mon.a (mon.0) 1540 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.173410+0000 osd.7 (osd.7) 481 : cluster 0 1.0 scrub starts 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.179383+0000 osd.7 (osd.7) 482 : cluster 0 1.0 scrub ok 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[29221]: cluster 2024-04-06T03:40:34.328166+0000 osd.0 (osd.0) 454 : cluster 0 3.1b scrub starts 2024-04-06T03:40:35.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[29221]: cluster 2024-04-06T03:40:34.334328+0000 osd.0 (osd.0) 455 : cluster 0 3.1b scrub ok 2024-04-06T03:40:35.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[29221]: cluster 2024-04-06T03:40:34.411995+0000 mgr.y (mgr.24368) 723 : cluster 0 pgmap v772: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 866 op/s 2024-04-06T03:40:35.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[29221]: audit 2024-04-06T03:40:35.049689+0000 mon.a (mon.0) 1540 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:35.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.173410+0000 osd.7 (osd.7) 481 : cluster 0 1.0 scrub starts 2024-04-06T03:40:35.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:35 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.179383+0000 osd.7 (osd.7) 482 : cluster 0 1.0 scrub ok 2024-04-06T03:40:35.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:35 smithi153 bash[23852]: cluster 2024-04-06T03:40:34.328166+0000 osd.0 (osd.0) 454 : cluster 0 3.1b scrub starts 2024-04-06T03:40:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:35 smithi153 bash[23852]: cluster 2024-04-06T03:40:34.334328+0000 osd.0 (osd.0) 455 : cluster 0 3.1b scrub ok 2024-04-06T03:40:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:35 smithi153 bash[23852]: cluster 2024-04-06T03:40:34.411995+0000 mgr.y (mgr.24368) 723 : cluster 0 pgmap v772: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 54 MiB/s wr, 866 op/s 2024-04-06T03:40:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:35 smithi153 bash[23852]: audit 2024-04-06T03:40:35.049689+0000 mon.a (mon.0) 1540 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:35 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.173410+0000 osd.7 (osd.7) 481 : cluster 0 1.0 scrub starts 2024-04-06T03:40:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:35 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.179383+0000 osd.7 (osd.7) 482 : cluster 0 1.0 scrub ok 2024-04-06T03:40:35.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:35.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:35.933 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:35.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:36.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:36.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:36.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-06T03:40:36.112 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:40:36.112 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:40:36.113 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-06T03:40:36.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump -f json-pretty 2024-04-06T03:40:36.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:36.186 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:36.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:36.313 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:36.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:36.445 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:36.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[21467]: cluster 2024-04-06T03:40:34.647813+0000 osd.1 (osd.1) 359 : cluster 0 10.2 scrub starts 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.147359+0000 osd.3 (osd.3) 470 : cluster 0 2.19 scrub starts 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.149226+0000 osd.3 (osd.3) 471 : cluster 0 2.19 scrub ok 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.367606+0000 osd.0 (osd.0) 456 : cluster 0 10.4 scrub starts 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.515020+0000 osd.2 (osd.2) 373 : cluster 0 5.d scrub starts 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[21467]: cluster 2024-04-06T03:40:35.520109+0000 osd.2 (osd.2) 374 : cluster 0 5.d scrub ok 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[29221]: cluster 2024-04-06T03:40:34.647813+0000 osd.1 (osd.1) 359 : cluster 0 10.2 scrub starts 2024-04-06T03:40:36.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.147359+0000 osd.3 (osd.3) 470 : cluster 0 2.19 scrub starts 2024-04-06T03:40:36.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.149226+0000 osd.3 (osd.3) 471 : cluster 0 2.19 scrub ok 2024-04-06T03:40:36.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.367606+0000 osd.0 (osd.0) 456 : cluster 0 10.4 scrub starts 2024-04-06T03:40:36.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.515020+0000 osd.2 (osd.2) 373 : cluster 0 5.d scrub starts 2024-04-06T03:40:36.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:36 smithi050 bash[29221]: cluster 2024-04-06T03:40:35.520109+0000 osd.2 (osd.2) 374 : cluster 0 5.d scrub ok 2024-04-06T03:40:36.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:36.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:36.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:36 smithi153 bash[23852]: cluster 2024-04-06T03:40:34.647813+0000 osd.1 (osd.1) 359 : cluster 0 10.2 scrub starts 2024-04-06T03:40:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:36 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.147359+0000 osd.3 (osd.3) 470 : cluster 0 2.19 scrub starts 2024-04-06T03:40:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:36 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.149226+0000 osd.3 (osd.3) 471 : cluster 0 2.19 scrub ok 2024-04-06T03:40:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:36 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.367606+0000 osd.0 (osd.0) 456 : cluster 0 10.4 scrub starts 2024-04-06T03:40:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:36 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.515020+0000 osd.2 (osd.2) 373 : cluster 0 5.d scrub starts 2024-04-06T03:40:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:36 smithi153 bash[23852]: cluster 2024-04-06T03:40:35.520109+0000 osd.2 (osd.2) 374 : cluster 0 5.d scrub ok 2024-04-06T03:40:36.705 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:36.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:36.835 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:36.941 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:36.964 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:36.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:37.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:37.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:37.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:37.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:37.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:37.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:37.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:37.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:37.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:37 smithi153 bash[23852]: cluster 2024-04-06T03:40:36.187587+0000 osd.3 (osd.3) 472 : cluster 0 6.13 scrub starts 2024-04-06T03:40:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:37 smithi153 bash[23852]: cluster 2024-04-06T03:40:36.312591+0000 osd.3 (osd.3) 473 : cluster 0 6.13 scrub ok 2024-04-06T03:40:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:37 smithi153 bash[23852]: cluster 2024-04-06T03:40:36.412747+0000 mgr.y (mgr.24368) 724 : cluster 0 pgmap v773: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 53 MiB/s wr, 852 op/s 2024-04-06T03:40:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:37 smithi153 bash[23852]: cluster 2024-04-06T03:40:36.429350+0000 osd.0 (osd.0) 457 : cluster 0 3.1 scrub starts 2024-04-06T03:40:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:37 smithi153 bash[23852]: cluster 2024-04-06T03:40:36.436285+0000 osd.0 (osd.0) 458 : cluster 0 3.1 scrub ok 2024-04-06T03:40:37.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:37.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:37.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:37.824 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:37.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:37.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[21467]: cluster 2024-04-06T03:40:36.187587+0000 osd.3 (osd.3) 472 : cluster 0 6.13 scrub starts 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[21467]: cluster 2024-04-06T03:40:36.312591+0000 osd.3 (osd.3) 473 : cluster 0 6.13 scrub ok 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[21467]: cluster 2024-04-06T03:40:36.412747+0000 mgr.y (mgr.24368) 724 : cluster 0 pgmap v773: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 53 MiB/s wr, 852 op/s 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[21467]: cluster 2024-04-06T03:40:36.429350+0000 osd.0 (osd.0) 457 : cluster 0 3.1 scrub starts 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[21467]: cluster 2024-04-06T03:40:36.436285+0000 osd.0 (osd.0) 458 : cluster 0 3.1 scrub ok 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[29221]: cluster 2024-04-06T03:40:36.187587+0000 osd.3 (osd.3) 472 : cluster 0 6.13 scrub starts 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[29221]: cluster 2024-04-06T03:40:36.312591+0000 osd.3 (osd.3) 473 : cluster 0 6.13 scrub ok 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[29221]: cluster 2024-04-06T03:40:36.412747+0000 mgr.y (mgr.24368) 724 : cluster 0 pgmap v773: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 53 MiB/s wr, 852 op/s 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[29221]: cluster 2024-04-06T03:40:36.429350+0000 osd.0 (osd.0) 457 : cluster 0 3.1 scrub starts 2024-04-06T03:40:37.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:37 smithi050 bash[29221]: cluster 2024-04-06T03:40:36.436285+0000 osd.0 (osd.0) 458 : cluster 0 3.1 scrub ok 2024-04-06T03:40:37.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:37.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:38.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:38.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:38.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:40:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:38.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:38.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:38.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:38.352 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:38.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:38.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:38.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:38.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:38.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:38.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:38.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[21467]: cluster 2024-04-06T03:40:36.666079+0000 osd.1 (osd.1) 360 : cluster 0 10.b scrub starts 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[21467]: cluster 2024-04-06T03:40:37.222402+0000 osd.3 (osd.3) 474 : cluster 0 3.1e scrub starts 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[21467]: cluster 2024-04-06T03:40:37.230321+0000 osd.3 (osd.3) 475 : cluster 0 3.1e scrub ok 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[21467]: cluster 2024-04-06T03:40:37.318358+0000 osd.1 (osd.1) 361 : cluster 0 10.2 scrub ok 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.173232+0000 osd.7 (osd.7) 483 : cluster 0 3.13 scrub starts 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.178561+0000 osd.7 (osd.7) 484 : cluster 0 3.13 scrub ok 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[29221]: cluster 2024-04-06T03:40:36.666079+0000 osd.1 (osd.1) 360 : cluster 0 10.b scrub starts 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[29221]: cluster 2024-04-06T03:40:37.222402+0000 osd.3 (osd.3) 474 : cluster 0 3.1e scrub starts 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[29221]: cluster 2024-04-06T03:40:37.230321+0000 osd.3 (osd.3) 475 : cluster 0 3.1e scrub ok 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[29221]: cluster 2024-04-06T03:40:37.318358+0000 osd.1 (osd.1) 361 : cluster 0 10.2 scrub ok 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.173232+0000 osd.7 (osd.7) 483 : cluster 0 3.13 scrub starts 2024-04-06T03:40:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:38 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.178561+0000 osd.7 (osd.7) 484 : cluster 0 3.13 scrub ok 2024-04-06T03:40:38.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:38 smithi153 bash[23852]: cluster 2024-04-06T03:40:36.666079+0000 osd.1 (osd.1) 360 : cluster 0 10.b scrub starts 2024-04-06T03:40:38.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:38 smithi153 bash[23852]: cluster 2024-04-06T03:40:37.222402+0000 osd.3 (osd.3) 474 : cluster 0 3.1e scrub starts 2024-04-06T03:40:38.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:38 smithi153 bash[23852]: cluster 2024-04-06T03:40:37.230321+0000 osd.3 (osd.3) 475 : cluster 0 3.1e scrub ok 2024-04-06T03:40:38.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:38 smithi153 bash[23852]: cluster 2024-04-06T03:40:37.318358+0000 osd.1 (osd.1) 361 : cluster 0 10.2 scrub ok 2024-04-06T03:40:38.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:38 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.173232+0000 osd.7 (osd.7) 483 : cluster 0 3.13 scrub starts 2024-04-06T03:40:38.743 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:38 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.178561+0000 osd.7 (osd.7) 484 : cluster 0 3.13 scrub ok 2024-04-06T03:40:38.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:38.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:38.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:38.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:38.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:39.033 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:39.136 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:39.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:39.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:39.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:39.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:39.434 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:39.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:39.579 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:39.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:39.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.367192+0000 osd.0 (osd.0) 459 : cluster 0 3.6 scrub starts 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.384904+0000 osd.0 (osd.0) 460 : cluster 0 3.6 scrub ok 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.413408+0000 mgr.y (mgr.24368) 725 : cluster 0 pgmap v774: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 829 op/s 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.420530+0000 osd.0 (osd.0) 461 : cluster 0 10.4 scrub ok 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.539910+0000 osd.2 (osd.2) 375 : cluster 0 10.5 scrub starts 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.367192+0000 osd.0 (osd.0) 459 : cluster 0 3.6 scrub starts 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.384904+0000 osd.0 (osd.0) 460 : cluster 0 3.6 scrub ok 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.413408+0000 mgr.y (mgr.24368) 725 : cluster 0 pgmap v774: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 829 op/s 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.420530+0000 osd.0 (osd.0) 461 : cluster 0 10.4 scrub ok 2024-04-06T03:40:39.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.539910+0000 osd.2 (osd.2) 375 : cluster 0 10.5 scrub starts 2024-04-06T03:40:39.694 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:40:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:40:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:40:39.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:39 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.367192+0000 osd.0 (osd.0) 459 : cluster 0 3.6 scrub starts 2024-04-06T03:40:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:39 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.384904+0000 osd.0 (osd.0) 460 : cluster 0 3.6 scrub ok 2024-04-06T03:40:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:39 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.413408+0000 mgr.y (mgr.24368) 725 : cluster 0 pgmap v774: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 47 MiB/s wr, 829 op/s 2024-04-06T03:40:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:39 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.420530+0000 osd.0 (osd.0) 461 : cluster 0 10.4 scrub ok 2024-04-06T03:40:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:39 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.539910+0000 osd.2 (osd.2) 375 : cluster 0 10.5 scrub starts 2024-04-06T03:40:39.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:39.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:39.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:39.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:40.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:40.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:40.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:40.297 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:40:39 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:40:40.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:40.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:40.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:40.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:40.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:40.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:40.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.723477+0000 osd.1 (osd.1) 362 : cluster 0 5.7 scrub starts 2024-04-06T03:40:40.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.758193+0000 osd.1 (osd.1) 363 : cluster 0 5.7 scrub ok 2024-04-06T03:40:40.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[21467]: cluster 2024-04-06T03:40:38.761251+0000 osd.1 (osd.1) 364 : cluster 0 10.b scrub ok 2024-04-06T03:40:40.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[21467]: cluster 2024-04-06T03:40:39.108059+0000 osd.4 (osd.4) 445 : cluster 0 5.e scrub starts 2024-04-06T03:40:40.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[21467]: cluster 2024-04-06T03:40:39.110866+0000 osd.4 (osd.4) 446 : cluster 0 5.e scrub ok 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[21467]: cluster 2024-04-06T03:40:39.380893+0000 osd.0 (osd.0) 462 : cluster 0 10.6 scrub starts 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.723477+0000 osd.1 (osd.1) 362 : cluster 0 5.7 scrub starts 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.758193+0000 osd.1 (osd.1) 363 : cluster 0 5.7 scrub ok 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[29221]: cluster 2024-04-06T03:40:38.761251+0000 osd.1 (osd.1) 364 : cluster 0 10.b scrub ok 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[29221]: cluster 2024-04-06T03:40:39.108059+0000 osd.4 (osd.4) 445 : cluster 0 5.e scrub starts 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[29221]: cluster 2024-04-06T03:40:39.110866+0000 osd.4 (osd.4) 446 : cluster 0 5.e scrub ok 2024-04-06T03:40:40.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:40 smithi050 bash[29221]: cluster 2024-04-06T03:40:39.380893+0000 osd.0 (osd.0) 462 : cluster 0 10.6 scrub starts 2024-04-06T03:40:40.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:40.757 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:40 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.723477+0000 osd.1 (osd.1) 362 : cluster 0 5.7 scrub starts 2024-04-06T03:40:40.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:40 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.758193+0000 osd.1 (osd.1) 363 : cluster 0 5.7 scrub ok 2024-04-06T03:40:40.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:40 smithi153 bash[23852]: cluster 2024-04-06T03:40:38.761251+0000 osd.1 (osd.1) 364 : cluster 0 10.b scrub ok 2024-04-06T03:40:40.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:40 smithi153 bash[23852]: cluster 2024-04-06T03:40:39.108059+0000 osd.4 (osd.4) 445 : cluster 0 5.e scrub starts 2024-04-06T03:40:40.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:40 smithi153 bash[23852]: cluster 2024-04-06T03:40:39.110866+0000 osd.4 (osd.4) 446 : cluster 0 5.e scrub ok 2024-04-06T03:40:40.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:40 smithi153 bash[23852]: cluster 2024-04-06T03:40:39.380893+0000 osd.0 (osd.0) 462 : cluster 0 10.6 scrub starts 2024-04-06T03:40:40.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:40.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:40.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:40.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:41.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:41.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:41.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:41.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:41.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:41.307 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:41.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:41.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:41.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:41.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: audit 2024-04-06T03:40:39.989455+0000 mgr.y (mgr.24368) 726 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:41.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.293126+0000 osd.3 (osd.3) 476 : cluster 0 2.14 scrub starts 2024-04-06T03:40:41.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.294691+0000 osd.3 (osd.3) 477 : cluster 0 2.14 scrub ok 2024-04-06T03:40:41.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.333981+0000 osd.0 (osd.0) 463 : cluster 0 10.8 scrub starts 2024-04-06T03:40:41.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.414039+0000 mgr.y (mgr.24368) 727 : cluster 0 pgmap v775: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 31 MiB/s wr, 834 op/s 2024-04-06T03:40:41.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: audit 2024-04-06T03:40:40.704893+0000 mon.a (mon.0) 1541 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:40:41.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:41 smithi153 bash[23852]: cluster 2024-04-06T03:40:41.121863+0000 osd.2 (osd.2) 376 : cluster 0 10.5 scrub ok 2024-04-06T03:40:41.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:41.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:41.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:41.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: audit 2024-04-06T03:40:39.989455+0000 mgr.y (mgr.24368) 726 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.293126+0000 osd.3 (osd.3) 476 : cluster 0 2.14 scrub starts 2024-04-06T03:40:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.294691+0000 osd.3 (osd.3) 477 : cluster 0 2.14 scrub ok 2024-04-06T03:40:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.333981+0000 osd.0 (osd.0) 463 : cluster 0 10.8 scrub starts 2024-04-06T03:40:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.414039+0000 mgr.y (mgr.24368) 727 : cluster 0 pgmap v775: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 31 MiB/s wr, 834 op/s 2024-04-06T03:40:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: audit 2024-04-06T03:40:40.704893+0000 mon.a (mon.0) 1541 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[21467]: cluster 2024-04-06T03:40:41.121863+0000 osd.2 (osd.2) 376 : cluster 0 10.5 scrub ok 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: audit 2024-04-06T03:40:39.989455+0000 mgr.y (mgr.24368) 726 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.293126+0000 osd.3 (osd.3) 476 : cluster 0 2.14 scrub starts 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.294691+0000 osd.3 (osd.3) 477 : cluster 0 2.14 scrub ok 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.333981+0000 osd.0 (osd.0) 463 : cluster 0 10.8 scrub starts 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.414039+0000 mgr.y (mgr.24368) 727 : cluster 0 pgmap v775: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 31 MiB/s wr, 834 op/s 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: audit 2024-04-06T03:40:40.704893+0000 mon.a (mon.0) 1541 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:40:41.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:41 smithi050 bash[29221]: cluster 2024-04-06T03:40:41.121863+0000 osd.2 (osd.2) 376 : cluster 0 10.5 scrub ok 2024-04-06T03:40:41.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:41.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:41.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:42.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:42.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:42.161 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:42.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:42.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:42.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:42.429 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:42.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.569 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:42.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:42.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:42.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.662222+0000 osd.1 (osd.1) 365 : cluster 0 2.9 scrub starts 2024-04-06T03:40:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.663498+0000 osd.1 (osd.1) 366 : cluster 0 2.9 scrub ok 2024-04-06T03:40:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.927024+0000 osd.6 (osd.6) 101 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:40.927097+0000 osd.6 (osd.6) 102 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:41.105821+0000 osd.4 (osd.4) 447 : cluster 0 3.9 scrub starts 2024-04-06T03:40:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:41.110545+0000 osd.4 (osd.4) 448 : cluster 0 3.9 scrub ok 2024-04-06T03:40:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:41.335838+0000 osd.3 (osd.3) 478 : cluster 0 6.11 scrub starts 2024-04-06T03:40:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:41.399465+0000 osd.3 (osd.3) 479 : cluster 0 6.11 scrub ok 2024-04-06T03:40:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: cluster 2024-04-06T03:40:41.556336+0000 mon.a (mon.0) 1542 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-04-06T03:40:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: audit 2024-04-06T03:40:41.801652+0000 mon.a (mon.0) 1543 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:40:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: audit 2024-04-06T03:40:41.802772+0000 mon.a (mon.0) 1544 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:40:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:42 smithi153 bash[23852]: audit 2024-04-06T03:40:41.809608+0000 mon.a (mon.0) 1545 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:40:42.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:42.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:42.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:42.957 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:42.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:42.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.662222+0000 osd.1 (osd.1) 365 : cluster 0 2.9 scrub starts 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.663498+0000 osd.1 (osd.1) 366 : cluster 0 2.9 scrub ok 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.927024+0000 osd.6 (osd.6) 101 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:40.927097+0000 osd.6 (osd.6) 102 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:41.105821+0000 osd.4 (osd.4) 447 : cluster 0 3.9 scrub starts 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:41.110545+0000 osd.4 (osd.4) 448 : cluster 0 3.9 scrub ok 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:41.335838+0000 osd.3 (osd.3) 478 : cluster 0 6.11 scrub starts 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:41.399465+0000 osd.3 (osd.3) 479 : cluster 0 6.11 scrub ok 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: cluster 2024-04-06T03:40:41.556336+0000 mon.a (mon.0) 1542 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: audit 2024-04-06T03:40:41.801652+0000 mon.a (mon.0) 1543 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: audit 2024-04-06T03:40:41.802772+0000 mon.a (mon.0) 1544 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[21467]: audit 2024-04-06T03:40:41.809608+0000 mon.a (mon.0) 1545 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:40:42.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.662222+0000 osd.1 (osd.1) 365 : cluster 0 2.9 scrub starts 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.663498+0000 osd.1 (osd.1) 366 : cluster 0 2.9 scrub ok 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.927024+0000 osd.6 (osd.6) 101 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:40.927097+0000 osd.6 (osd.6) 102 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:41.105821+0000 osd.4 (osd.4) 447 : cluster 0 3.9 scrub starts 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:41.110545+0000 osd.4 (osd.4) 448 : cluster 0 3.9 scrub ok 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:41.335838+0000 osd.3 (osd.3) 478 : cluster 0 6.11 scrub starts 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:41.399465+0000 osd.3 (osd.3) 479 : cluster 0 6.11 scrub ok 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: cluster 2024-04-06T03:40:41.556336+0000 mon.a (mon.0) 1542 : cluster 0 osdmap e305: 8 total, 8 up, 8 in 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: audit 2024-04-06T03:40:41.801652+0000 mon.a (mon.0) 1543 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: audit 2024-04-06T03:40:41.802772+0000 mon.a (mon.0) 1544 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:40:42.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:42 smithi050 bash[29221]: audit 2024-04-06T03:40:41.809608+0000 mon.a (mon.0) 1545 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:40:43.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:43.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:43.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:43.232 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:43.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:43.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:43.427 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:43.428 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:43.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:43.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:43.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.637 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:43.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:43.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:43.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:43.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:43.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:43.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:43.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.067469+0000 osd.0 (osd.0) 464 : cluster 0 10.6 scrub ok 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.353948+0000 osd.3 (osd.3) 480 : cluster 0 2.1b scrub starts 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.355567+0000 osd.3 (osd.3) 481 : cluster 0 2.1b scrub ok 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.388493+0000 osd.0 (osd.0) 465 : cluster 0 3.17 scrub starts 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.390783+0000 osd.0 (osd.0) 466 : cluster 0 3.17 scrub ok 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.414701+0000 mgr.y (mgr.24368) 728 : cluster 0 pgmap v777: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 871 op/s 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.516298+0000 osd.2 (osd.2) 377 : cluster 0 6.f scrub starts 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.679856+0000 osd.2 (osd.2) 378 : cluster 0 6.f scrub ok 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:43.132542+0000 osd.7 (osd.7) 485 : cluster 0 2.1c scrub starts 2024-04-06T03:40:44.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:43 smithi153 bash[23852]: cluster 2024-04-06T03:40:43.133465+0000 osd.7 (osd.7) 486 : cluster 0 2.1c scrub ok 2024-04-06T03:40:44.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:44.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:44.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:44.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:44.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:44.442 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:44.444 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:40:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:44.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.067469+0000 osd.0 (osd.0) 464 : cluster 0 10.6 scrub ok 2024-04-06T03:40:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.353948+0000 osd.3 (osd.3) 480 : cluster 0 2.1b scrub starts 2024-04-06T03:40:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.355567+0000 osd.3 (osd.3) 481 : cluster 0 2.1b scrub ok 2024-04-06T03:40:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.388493+0000 osd.0 (osd.0) 465 : cluster 0 3.17 scrub starts 2024-04-06T03:40:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.390783+0000 osd.0 (osd.0) 466 : cluster 0 3.17 scrub ok 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.414701+0000 mgr.y (mgr.24368) 728 : cluster 0 pgmap v777: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 871 op/s 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.516298+0000 osd.2 (osd.2) 377 : cluster 0 6.f scrub starts 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.679856+0000 osd.2 (osd.2) 378 : cluster 0 6.f scrub ok 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:43.132542+0000 osd.7 (osd.7) 485 : cluster 0 2.1c scrub starts 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[21467]: cluster 2024-04-06T03:40:43.133465+0000 osd.7 (osd.7) 486 : cluster 0 2.1c scrub ok 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.067469+0000 osd.0 (osd.0) 464 : cluster 0 10.6 scrub ok 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.353948+0000 osd.3 (osd.3) 480 : cluster 0 2.1b scrub starts 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.355567+0000 osd.3 (osd.3) 481 : cluster 0 2.1b scrub ok 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.388493+0000 osd.0 (osd.0) 465 : cluster 0 3.17 scrub starts 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.390783+0000 osd.0 (osd.0) 466 : cluster 0 3.17 scrub ok 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.414701+0000 mgr.y (mgr.24368) 728 : cluster 0 pgmap v777: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 871 op/s 2024-04-06T03:40:44.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.516298+0000 osd.2 (osd.2) 377 : cluster 0 6.f scrub starts 2024-04-06T03:40:44.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.679856+0000 osd.2 (osd.2) 378 : cluster 0 6.f scrub ok 2024-04-06T03:40:44.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:43.132542+0000 osd.7 (osd.7) 485 : cluster 0 2.1c scrub starts 2024-04-06T03:40:44.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:43 smithi050 bash[29221]: cluster 2024-04-06T03:40:43.133465+0000 osd.7 (osd.7) 486 : cluster 0 2.1c scrub ok 2024-04-06T03:40:44.575 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:44.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:44.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:44.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:44.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:44.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:44.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:44 smithi050 bash[21467]: cluster 2024-04-06T03:40:42.963813+0000 osd.0 (osd.0) 467 : cluster 0 10.8 scrub ok 2024-04-06T03:40:44.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:44 smithi050 bash[21467]: cluster 2024-04-06T03:40:43.408296+0000 osd.0 (osd.0) 468 : cluster 0 10.a scrub starts 2024-04-06T03:40:44.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:44 smithi050 bash[29221]: cluster 2024-04-06T03:40:42.963813+0000 osd.0 (osd.0) 467 : cluster 0 10.8 scrub ok 2024-04-06T03:40:44.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:44 smithi050 bash[29221]: cluster 2024-04-06T03:40:43.408296+0000 osd.0 (osd.0) 468 : cluster 0 10.a scrub starts 2024-04-06T03:40:44.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:44.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:45.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:45.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:45.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:45.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:44 smithi153 bash[23852]: cluster 2024-04-06T03:40:42.963813+0000 osd.0 (osd.0) 467 : cluster 0 10.8 scrub ok 2024-04-06T03:40:45.241 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:44 smithi153 bash[23852]: cluster 2024-04-06T03:40:43.408296+0000 osd.0 (osd.0) 468 : cluster 0 10.a scrub starts 2024-04-06T03:40:45.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:45.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:45.409 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:45.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:45.532 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:45.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:45.657 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:45.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:45.780 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:45.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:45.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:46.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:46.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:46.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:45 smithi153 bash[23852]: cluster 2024-04-06T03:40:44.415320+0000 mgr.y (mgr.24368) 729 : cluster 0 pgmap v778: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 12 MiB/s wr, 879 op/s 2024-04-06T03:40:46.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:46.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:45 smithi050 bash[21467]: cluster 2024-04-06T03:40:44.415320+0000 mgr.y (mgr.24368) 729 : cluster 0 pgmap v778: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 12 MiB/s wr, 879 op/s 2024-04-06T03:40:46.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:45 smithi050 bash[29221]: cluster 2024-04-06T03:40:44.415320+0000 mgr.y (mgr.24368) 729 : cluster 0 pgmap v778: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 12 MiB/s wr, 879 op/s 2024-04-06T03:40:46.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:46.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:46.307 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:46.412 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:46.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:46.558 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:46.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:46.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:46.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:46.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:46.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:46.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:46.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:45.164869+0000 osd.4 (osd.4) 449 : cluster 0 3.14 scrub starts 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:45.168904+0000 osd.4 (osd.4) 450 : cluster 0 3.14 scrub ok 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:45.432348+0000 osd.0 (osd.0) 469 : cluster 0 10.a scrub ok 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:45.434981+0000 osd.0 (osd.0) 470 : cluster 0 6.0 scrub starts 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:45.681696+0000 osd.0 (osd.0) 471 : cluster 0 6.0 scrub ok 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:46.157334+0000 osd.7 (osd.7) 487 : cluster 0 3.e scrub starts 2024-04-06T03:40:47.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:46 smithi153 bash[23852]: cluster 2024-04-06T03:40:46.165683+0000 osd.7 (osd.7) 488 : cluster 0 3.e scrub ok 2024-04-06T03:40:47.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:47.075 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:45.164869+0000 osd.4 (osd.4) 449 : cluster 0 3.14 scrub starts 2024-04-06T03:40:47.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:45.168904+0000 osd.4 (osd.4) 450 : cluster 0 3.14 scrub ok 2024-04-06T03:40:47.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:45.432348+0000 osd.0 (osd.0) 469 : cluster 0 10.a scrub ok 2024-04-06T03:40:47.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:45.434981+0000 osd.0 (osd.0) 470 : cluster 0 6.0 scrub starts 2024-04-06T03:40:47.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:45.681696+0000 osd.0 (osd.0) 471 : cluster 0 6.0 scrub ok 2024-04-06T03:40:47.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:46.157334+0000 osd.7 (osd.7) 487 : cluster 0 3.e scrub starts 2024-04-06T03:40:47.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[21467]: cluster 2024-04-06T03:40:46.165683+0000 osd.7 (osd.7) 488 : cluster 0 3.e scrub ok 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:45.164869+0000 osd.4 (osd.4) 449 : cluster 0 3.14 scrub starts 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:45.168904+0000 osd.4 (osd.4) 450 : cluster 0 3.14 scrub ok 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:45.432348+0000 osd.0 (osd.0) 469 : cluster 0 10.a scrub ok 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:45.434981+0000 osd.0 (osd.0) 470 : cluster 0 6.0 scrub starts 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:45.681696+0000 osd.0 (osd.0) 471 : cluster 0 6.0 scrub ok 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:46.157334+0000 osd.7 (osd.7) 487 : cluster 0 3.e scrub starts 2024-04-06T03:40:47.082 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:46 smithi050 bash[29221]: cluster 2024-04-06T03:40:46.165683+0000 osd.7 (osd.7) 488 : cluster 0 3.e scrub ok 2024-04-06T03:40:47.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:47.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:47.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:47.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:47.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:47.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:47.525 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:47.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:47.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:47.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:47.796 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:47.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:47.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:48.038 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:48.070 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:48.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:48.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[21467]: cluster 2024-04-06T03:40:46.415853+0000 mgr.y (mgr.24368) 730 : cluster 0 pgmap v779: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.3 MiB/s wr, 896 op/s 2024-04-06T03:40:48.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[21467]: cluster 2024-04-06T03:40:46.619299+0000 osd.2 (osd.2) 379 : cluster 0 5.6 scrub starts 2024-04-06T03:40:48.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[21467]: cluster 2024-04-06T03:40:46.620419+0000 osd.2 (osd.2) 380 : cluster 0 5.6 scrub ok 2024-04-06T03:40:48.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[21467]: cluster 2024-04-06T03:40:47.190839+0000 osd.7 (osd.7) 489 : cluster 0 6.3 scrub starts 2024-04-06T03:40:48.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[29221]: cluster 2024-04-06T03:40:46.415853+0000 mgr.y (mgr.24368) 730 : cluster 0 pgmap v779: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.3 MiB/s wr, 896 op/s 2024-04-06T03:40:48.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[29221]: cluster 2024-04-06T03:40:46.619299+0000 osd.2 (osd.2) 379 : cluster 0 5.6 scrub starts 2024-04-06T03:40:48.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[29221]: cluster 2024-04-06T03:40:46.620419+0000 osd.2 (osd.2) 380 : cluster 0 5.6 scrub ok 2024-04-06T03:40:48.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:47 smithi050 bash[29221]: cluster 2024-04-06T03:40:47.190839+0000 osd.7 (osd.7) 489 : cluster 0 6.3 scrub starts 2024-04-06T03:40:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:48.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:47 smithi153 bash[23852]: cluster 2024-04-06T03:40:46.415853+0000 mgr.y (mgr.24368) 730 : cluster 0 pgmap v779: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.3 MiB/s wr, 896 op/s 2024-04-06T03:40:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:47 smithi153 bash[23852]: cluster 2024-04-06T03:40:46.619299+0000 osd.2 (osd.2) 379 : cluster 0 5.6 scrub starts 2024-04-06T03:40:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:47 smithi153 bash[23852]: cluster 2024-04-06T03:40:46.620419+0000 osd.2 (osd.2) 380 : cluster 0 5.6 scrub ok 2024-04-06T03:40:48.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:47 smithi153 bash[23852]: cluster 2024-04-06T03:40:47.190839+0000 osd.7 (osd.7) 489 : cluster 0 6.3 scrub starts 2024-04-06T03:40:48.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:48.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:48.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:48.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:48.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.619 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:48.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:48.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:48.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.884 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:48.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:48.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:48.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:48 smithi050 bash[21467]: cluster 2024-04-06T03:40:47.410286+0000 osd.7 (osd.7) 490 : cluster 0 6.3 scrub ok 2024-04-06T03:40:48.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:48 smithi050 bash[21467]: cluster 2024-04-06T03:40:47.411517+0000 osd.3 (osd.3) 482 : cluster 0 5.17 scrub starts 2024-04-06T03:40:48.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:48 smithi050 bash[21467]: cluster 2024-04-06T03:40:47.419389+0000 osd.3 (osd.3) 483 : cluster 0 5.17 scrub ok 2024-04-06T03:40:48.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:48 smithi050 bash[29221]: cluster 2024-04-06T03:40:47.410286+0000 osd.7 (osd.7) 490 : cluster 0 6.3 scrub ok 2024-04-06T03:40:48.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:48 smithi050 bash[29221]: cluster 2024-04-06T03:40:47.411517+0000 osd.3 (osd.3) 482 : cluster 0 5.17 scrub starts 2024-04-06T03:40:48.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:48 smithi050 bash[29221]: cluster 2024-04-06T03:40:47.419389+0000 osd.3 (osd.3) 483 : cluster 0 5.17 scrub ok 2024-04-06T03:40:49.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:49.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:49.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:49.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:49.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:48 smithi153 bash[23852]: cluster 2024-04-06T03:40:47.410286+0000 osd.7 (osd.7) 490 : cluster 0 6.3 scrub ok 2024-04-06T03:40:49.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:48 smithi153 bash[23852]: cluster 2024-04-06T03:40:47.411517+0000 osd.3 (osd.3) 482 : cluster 0 5.17 scrub starts 2024-04-06T03:40:49.272 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:48 smithi153 bash[23852]: cluster 2024-04-06T03:40:47.419389+0000 osd.3 (osd.3) 483 : cluster 0 5.17 scrub ok 2024-04-06T03:40:49.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:49.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:49.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:49.489 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:40:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:40:49.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:49.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:49.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:49.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:49.830 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:49.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:49.942 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:48.158181+0000 osd.4 (osd.4) 451 : cluster 0 10.d scrub starts 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:48.416499+0000 mgr.y (mgr.24368) 731 : cluster 0 pgmap v780: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 KiB/s wr, 893 op/s 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:48.619623+0000 osd.1 (osd.1) 367 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:48.619693+0000 osd.1 (osd.1) 368 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:48.849558+0000 mon.a (mon.0) 1546 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:49.163115+0000 osd.1 (osd.1) 369 : cluster 0 5.1d scrub starts 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[21467]: cluster 2024-04-06T03:40:49.165551+0000 osd.1 (osd.1) 370 : cluster 0 5.1d scrub ok 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:48.158181+0000 osd.4 (osd.4) 451 : cluster 0 10.d scrub starts 2024-04-06T03:40:49.943 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:48.416499+0000 mgr.y (mgr.24368) 731 : cluster 0 pgmap v780: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 KiB/s wr, 893 op/s 2024-04-06T03:40:49.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:48.619623+0000 osd.1 (osd.1) 367 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:49.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:48.619693+0000 osd.1 (osd.1) 368 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:49.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:48.849558+0000 mon.a (mon.0) 1546 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-04-06T03:40:49.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:49.163115+0000 osd.1 (osd.1) 369 : cluster 0 5.1d scrub starts 2024-04-06T03:40:49.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:49 smithi050 bash[29221]: cluster 2024-04-06T03:40:49.165551+0000 osd.1 (osd.1) 370 : cluster 0 5.1d scrub ok 2024-04-06T03:40:49.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:49.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:50.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:50.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:48.158181+0000 osd.4 (osd.4) 451 : cluster 0 10.d scrub starts 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:48.416499+0000 mgr.y (mgr.24368) 731 : cluster 0 pgmap v780: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 889 KiB/s wr, 893 op/s 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:48.619623+0000 osd.1 (osd.1) 367 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:48.619693+0000 osd.1 (osd.1) 368 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:48.849558+0000 mon.a (mon.0) 1546 : cluster 0 osdmap e306: 8 total, 8 up, 8 in 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:49.163115+0000 osd.1 (osd.1) 369 : cluster 0 5.1d scrub starts 2024-04-06T03:40:50.074 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:49 smithi153 bash[23852]: cluster 2024-04-06T03:40:49.165551+0000 osd.1 (osd.1) 370 : cluster 0 5.1d scrub ok 2024-04-06T03:40:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:50.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:50.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:50.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:50.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:50.463 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:50.483 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:50.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:50.622 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:50.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:50.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:50.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:50.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:50.887 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:50.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:50.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: cluster 2024-04-06T03:40:49.413389+0000 osd.3 (osd.3) 484 : cluster 0 10.7 scrub starts 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: cluster 2024-04-06T03:40:49.897153+0000 osd.4 (osd.4) 452 : cluster 0 10.d scrub ok 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: audit 2024-04-06T03:40:50.001257+0000 mgr.y (mgr.24368) 732 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: audit 2024-04-06T03:40:50.061963+0000 mon.a (mon.0) 1547 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.146869+0000 osd.1 (osd.1) 371 : cluster 0 4.2 scrub starts 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.149932+0000 osd.1 (osd.1) 372 : cluster 0 4.2 scrub ok 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.417252+0000 mgr.y (mgr.24368) 733 : cluster 0 pgmap v782: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 963 op/s 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: cluster 2024-04-06T03:40:49.413389+0000 osd.3 (osd.3) 484 : cluster 0 10.7 scrub starts 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: cluster 2024-04-06T03:40:49.897153+0000 osd.4 (osd.4) 452 : cluster 0 10.d scrub ok 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: audit 2024-04-06T03:40:50.001257+0000 mgr.y (mgr.24368) 732 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:50.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: audit 2024-04-06T03:40:50.061963+0000 mon.a (mon.0) 1547 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.146869+0000 osd.1 (osd.1) 371 : cluster 0 4.2 scrub starts 2024-04-06T03:40:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.149932+0000 osd.1 (osd.1) 372 : cluster 0 4.2 scrub ok 2024-04-06T03:40:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:50 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.417252+0000 mgr.y (mgr.24368) 733 : cluster 0 pgmap v782: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 963 op/s 2024-04-06T03:40:50.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:51.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:51.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:51.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:51.250 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:51.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: cluster 2024-04-06T03:40:49.413389+0000 osd.3 (osd.3) 484 : cluster 0 10.7 scrub starts 2024-04-06T03:40:51.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: cluster 2024-04-06T03:40:49.897153+0000 osd.4 (osd.4) 452 : cluster 0 10.d scrub ok 2024-04-06T03:40:51.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: audit 2024-04-06T03:40:50.001257+0000 mgr.y (mgr.24368) 732 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:40:51.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: audit 2024-04-06T03:40:50.061963+0000 mon.a (mon.0) 1547 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:40:51.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.146869+0000 osd.1 (osd.1) 371 : cluster 0 4.2 scrub starts 2024-04-06T03:40:51.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.149932+0000 osd.1 (osd.1) 372 : cluster 0 4.2 scrub ok 2024-04-06T03:40:51.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:50 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.417252+0000 mgr.y (mgr.24368) 733 : cluster 0 pgmap v782: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 963 op/s 2024-04-06T03:40:51.273 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:51.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:51.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:51.530 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:51.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:51.662 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:51.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:51.799 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:51.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:51.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:51.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:51.966 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:40:51.988 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:40:52.009 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:40:52.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:52.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:51 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.385230+0000 osd.0 (osd.0) 472 : cluster 0 5.1e scrub starts 2024-04-06T03:40:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:51 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.388485+0000 osd.0 (osd.0) 473 : cluster 0 5.1e scrub ok 2024-04-06T03:40:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:51 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.485822+0000 osd.3 (osd.3) 485 : cluster 0 2.a scrub starts 2024-04-06T03:40:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:51 smithi153 bash[23852]: cluster 2024-04-06T03:40:50.487264+0000 osd.3 (osd.3) 486 : cluster 0 2.a scrub ok 2024-04-06T03:40:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:51 smithi153 bash[23852]: cluster 2024-04-06T03:40:51.549803+0000 osd.1 (osd.1) 373 : cluster 0 6.4 scrub starts 2024-04-06T03:40:52.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:51 smithi153 bash[23852]: cluster 2024-04-06T03:40:51.559679+0000 osd.1 (osd.1) 374 : cluster 0 6.4 scrub ok 2024-04-06T03:40:52.145 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:40:52.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:52.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:52.190 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:40:52.237 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:40:52.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.385230+0000 osd.0 (osd.0) 472 : cluster 0 5.1e scrub starts 2024-04-06T03:40:52.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.388485+0000 osd.0 (osd.0) 473 : cluster 0 5.1e scrub ok 2024-04-06T03:40:52.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.485822+0000 osd.3 (osd.3) 485 : cluster 0 2.a scrub starts 2024-04-06T03:40:52.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[21467]: cluster 2024-04-06T03:40:50.487264+0000 osd.3 (osd.3) 486 : cluster 0 2.a scrub ok 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[21467]: cluster 2024-04-06T03:40:51.549803+0000 osd.1 (osd.1) 373 : cluster 0 6.4 scrub starts 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[21467]: cluster 2024-04-06T03:40:51.559679+0000 osd.1 (osd.1) 374 : cluster 0 6.4 scrub ok 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.385230+0000 osd.0 (osd.0) 472 : cluster 0 5.1e scrub starts 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.388485+0000 osd.0 (osd.0) 473 : cluster 0 5.1e scrub ok 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.485822+0000 osd.3 (osd.3) 485 : cluster 0 2.a scrub starts 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[29221]: cluster 2024-04-06T03:40:50.487264+0000 osd.3 (osd.3) 486 : cluster 0 2.a scrub ok 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[29221]: cluster 2024-04-06T03:40:51.549803+0000 osd.1 (osd.1) 373 : cluster 0 6.4 scrub starts 2024-04-06T03:40:52.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:51 smithi050 bash[29221]: cluster 2024-04-06T03:40:51.559679+0000 osd.1 (osd.1) 374 : cluster 0 6.4 scrub ok 2024-04-06T03:40:52.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:52.280 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:40:52.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:52.322 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:40:52.336 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:40:52.336 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:40:52.336 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:40:52.337 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:40:52.337 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:40:52.337 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:40:52.337 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:40:52.337 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:40:52.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:52.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:52.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:52.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:52.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:52.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:52.706 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:52.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:52.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:52.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: cluster 2024-04-06T03:40:51.558058+0000 osd.0 (osd.0) 474 : cluster 0 6.16 scrub starts 2024-04-06T03:40:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: cluster 2024-04-06T03:40:51.559236+0000 osd.4 (osd.4) 453 : cluster 0 10.1 scrub starts 2024-04-06T03:40:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: cluster 2024-04-06T03:40:51.639600+0000 osd.0 (osd.0) 475 : cluster 0 6.16 scrub ok 2024-04-06T03:40:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: audit 2024-04-06T03:40:51.921214+0000 mon.c (mon.2) 94 : audit 0 from='client.? 172.21.15.50:0/3184018699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:40:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.226952+0000 osd.1 (osd.1) 375 : cluster 0 2.6 scrub starts 2024-04-06T03:40:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.231915+0000 osd.1 (osd.1) 376 : cluster 0 2.6 scrub ok 2024-04-06T03:40:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:52 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.417778+0000 mgr.y (mgr.24368) 734 : cluster 0 pgmap v783: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 831 op/s 2024-04-06T03:40:52.958 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:52.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:52.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: cluster 2024-04-06T03:40:51.558058+0000 osd.0 (osd.0) 474 : cluster 0 6.16 scrub starts 2024-04-06T03:40:52.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: cluster 2024-04-06T03:40:51.559236+0000 osd.4 (osd.4) 453 : cluster 0 10.1 scrub starts 2024-04-06T03:40:52.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: cluster 2024-04-06T03:40:51.639600+0000 osd.0 (osd.0) 475 : cluster 0 6.16 scrub ok 2024-04-06T03:40:52.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: audit 2024-04-06T03:40:51.921214+0000 mon.c (mon.2) 94 : audit 0 from='client.? 172.21.15.50:0/3184018699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.226952+0000 osd.1 (osd.1) 375 : cluster 0 2.6 scrub starts 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.231915+0000 osd.1 (osd.1) 376 : cluster 0 2.6 scrub ok 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.417778+0000 mgr.y (mgr.24368) 734 : cluster 0 pgmap v783: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 831 op/s 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: cluster 2024-04-06T03:40:51.558058+0000 osd.0 (osd.0) 474 : cluster 0 6.16 scrub starts 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: cluster 2024-04-06T03:40:51.559236+0000 osd.4 (osd.4) 453 : cluster 0 10.1 scrub starts 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: cluster 2024-04-06T03:40:51.639600+0000 osd.0 (osd.0) 475 : cluster 0 6.16 scrub ok 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: audit 2024-04-06T03:40:51.921214+0000 mon.c (mon.2) 94 : audit 0 from='client.? 172.21.15.50:0/3184018699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.226952+0000 osd.1 (osd.1) 375 : cluster 0 2.6 scrub starts 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.231915+0000 osd.1 (osd.1) 376 : cluster 0 2.6 scrub ok 2024-04-06T03:40:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:52 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.417778+0000 mgr.y (mgr.24368) 734 : cluster 0 pgmap v783: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 831 op/s 2024-04-06T03:40:53.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:53.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:53.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:53.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:53.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:53.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:53.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:53.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:53.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:53.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:53.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:53.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:53.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:53.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:53.872 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:53.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:53.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:54.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:54.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:54.114 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:53 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.276291+0000 osd.3 (osd.3) 487 : cluster 0 10.7 scrub ok 2024-04-06T03:40:54.114 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:53 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.277920+0000 osd.4 (osd.4) 454 : cluster 0 3.1c scrub starts 2024-04-06T03:40:54.114 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:53 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.283569+0000 osd.4 (osd.4) 455 : cluster 0 3.1c scrub ok 2024-04-06T03:40:54.114 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:53 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.595556+0000 osd.0 (osd.0) 476 : cluster 0 2.13 scrub starts 2024-04-06T03:40:54.114 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:53 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.601980+0000 osd.0 (osd.0) 477 : cluster 0 2.13 scrub ok 2024-04-06T03:40:54.114 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:53 smithi153 bash[23852]: cluster 2024-04-06T03:40:52.814729+0000 osd.4 (osd.4) 456 : cluster 0 10.1 scrub ok 2024-04-06T03:40:54.127 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:54.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:54.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.276291+0000 osd.3 (osd.3) 487 : cluster 0 10.7 scrub ok 2024-04-06T03:40:54.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.277920+0000 osd.4 (osd.4) 454 : cluster 0 3.1c scrub starts 2024-04-06T03:40:54.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.283569+0000 osd.4 (osd.4) 455 : cluster 0 3.1c scrub ok 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.595556+0000 osd.0 (osd.0) 476 : cluster 0 2.13 scrub starts 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.601980+0000 osd.0 (osd.0) 477 : cluster 0 2.13 scrub ok 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[21467]: cluster 2024-04-06T03:40:52.814729+0000 osd.4 (osd.4) 456 : cluster 0 10.1 scrub ok 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.276291+0000 osd.3 (osd.3) 487 : cluster 0 10.7 scrub ok 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.277920+0000 osd.4 (osd.4) 454 : cluster 0 3.1c scrub starts 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.283569+0000 osd.4 (osd.4) 455 : cluster 0 3.1c scrub ok 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.595556+0000 osd.0 (osd.0) 476 : cluster 0 2.13 scrub starts 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.601980+0000 osd.0 (osd.0) 477 : cluster 0 2.13 scrub ok 2024-04-06T03:40:54.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:53 smithi050 bash[29221]: cluster 2024-04-06T03:40:52.814729+0000 osd.4 (osd.4) 456 : cluster 0 10.1 scrub ok 2024-04-06T03:40:54.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:54.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:54.390 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:54.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:54.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:54.666 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:54.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:54.795 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:54.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:54.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:54.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:54.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:55.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:55.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:55.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:55.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:55.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[29221]: cluster 2024-04-06T03:40:53.304086+0000 osd.4 (osd.4) 457 : cluster 0 10.0 scrub starts 2024-04-06T03:40:55.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.232559+0000 osd.7 (osd.7) 491 : cluster 0 2.7 scrub starts 2024-04-06T03:40:55.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.248363+0000 osd.7 (osd.7) 492 : cluster 0 2.7 scrub ok 2024-04-06T03:40:55.157 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.418322+0000 mgr.y (mgr.24368) 735 : cluster 0 pgmap v784: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s 2024-04-06T03:40:55.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[21467]: cluster 2024-04-06T03:40:53.304086+0000 osd.4 (osd.4) 457 : cluster 0 10.0 scrub starts 2024-04-06T03:40:55.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.232559+0000 osd.7 (osd.7) 491 : cluster 0 2.7 scrub starts 2024-04-06T03:40:55.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.248363+0000 osd.7 (osd.7) 492 : cluster 0 2.7 scrub ok 2024-04-06T03:40:55.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:54 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.418322+0000 mgr.y (mgr.24368) 735 : cluster 0 pgmap v784: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s 2024-04-06T03:40:55.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:55.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:55.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:54 smithi153 bash[23852]: cluster 2024-04-06T03:40:53.304086+0000 osd.4 (osd.4) 457 : cluster 0 10.0 scrub starts 2024-04-06T03:40:55.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:54 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.232559+0000 osd.7 (osd.7) 491 : cluster 0 2.7 scrub starts 2024-04-06T03:40:55.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:54 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.248363+0000 osd.7 (osd.7) 492 : cluster 0 2.7 scrub ok 2024-04-06T03:40:55.286 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:54 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.418322+0000 mgr.y (mgr.24368) 735 : cluster 0 pgmap v784: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 821 op/s 2024-04-06T03:40:55.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:55.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:55.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:55.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:55.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:55.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:55.683 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:55.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:55.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:55.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.448556+0000 osd.4 (osd.4) 458 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.448605+0000 osd.4 (osd.4) 459 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.604510+0000 osd.3 (osd.3) 488 : cluster 0 6.1b scrub starts 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.715020+0000 osd.4 (osd.4) 460 : cluster 0 10.0 scrub ok 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.748341+0000 osd.3 (osd.3) 489 : cluster 0 6.1b scrub ok 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:54.910617+0000 mon.a (mon.0) 1548 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:55.179036+0000 osd.7 (osd.7) 493 : cluster 0 3.15 scrub starts 2024-04-06T03:40:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:55 smithi153 bash[23852]: cluster 2024-04-06T03:40:55.181441+0000 osd.7 (osd.7) 494 : cluster 0 3.15 scrub ok 2024-04-06T03:40:56.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:56.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:56.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:56.206 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:56.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.448556+0000 osd.4 (osd.4) 458 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.448605+0000 osd.4 (osd.4) 459 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.604510+0000 osd.3 (osd.3) 488 : cluster 0 6.1b scrub starts 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.715020+0000 osd.4 (osd.4) 460 : cluster 0 10.0 scrub ok 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.748341+0000 osd.3 (osd.3) 489 : cluster 0 6.1b scrub ok 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:54.910617+0000 mon.a (mon.0) 1548 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:55.179036+0000 osd.7 (osd.7) 493 : cluster 0 3.15 scrub starts 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[29221]: cluster 2024-04-06T03:40:55.181441+0000 osd.7 (osd.7) 494 : cluster 0 3.15 scrub ok 2024-04-06T03:40:56.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.448556+0000 osd.4 (osd.4) 458 : cluster 0 purged_snaps scrub starts 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.448605+0000 osd.4 (osd.4) 459 : cluster 0 purged_snaps scrub ok 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.604510+0000 osd.3 (osd.3) 488 : cluster 0 6.1b scrub starts 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.715020+0000 osd.4 (osd.4) 460 : cluster 0 10.0 scrub ok 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.748341+0000 osd.3 (osd.3) 489 : cluster 0 6.1b scrub ok 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:54.910617+0000 mon.a (mon.0) 1548 : cluster 0 osdmap e307: 8 total, 8 up, 8 in 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:55.179036+0000 osd.7 (osd.7) 493 : cluster 0 3.15 scrub starts 2024-04-06T03:40:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:55 smithi050 bash[21467]: cluster 2024-04-06T03:40:55.181441+0000 osd.7 (osd.7) 494 : cluster 0 3.15 scrub ok 2024-04-06T03:40:56.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:56.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:56.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:56.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:56.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:56.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:56.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:56.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:56.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:56.885 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:56.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:56.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:57.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:57.141 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:57.150 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:56 smithi153 bash[23852]: cluster 2024-04-06T03:40:56.211630+0000 osd.7 (osd.7) 495 : cluster 0 3.16 scrub starts 2024-04-06T03:40:57.150 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:56 smithi153 bash[23852]: cluster 2024-04-06T03:40:56.214497+0000 osd.7 (osd.7) 496 : cluster 0 3.16 scrub ok 2024-04-06T03:40:57.150 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:56 smithi153 bash[23852]: cluster 2024-04-06T03:40:56.418845+0000 mgr.y (mgr.24368) 736 : cluster 0 pgmap v786: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 815 op/s 2024-04-06T03:40:57.173 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:57.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:56 smithi050 bash[21467]: cluster 2024-04-06T03:40:56.211630+0000 osd.7 (osd.7) 495 : cluster 0 3.16 scrub starts 2024-04-06T03:40:57.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:56 smithi050 bash[21467]: cluster 2024-04-06T03:40:56.214497+0000 osd.7 (osd.7) 496 : cluster 0 3.16 scrub ok 2024-04-06T03:40:57.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:56 smithi050 bash[21467]: cluster 2024-04-06T03:40:56.418845+0000 mgr.y (mgr.24368) 736 : cluster 0 pgmap v786: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 815 op/s 2024-04-06T03:40:57.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:56 smithi050 bash[29221]: cluster 2024-04-06T03:40:56.211630+0000 osd.7 (osd.7) 495 : cluster 0 3.16 scrub starts 2024-04-06T03:40:57.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:56 smithi050 bash[29221]: cluster 2024-04-06T03:40:56.214497+0000 osd.7 (osd.7) 496 : cluster 0 3.16 scrub ok 2024-04-06T03:40:57.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:56 smithi050 bash[29221]: cluster 2024-04-06T03:40:56.418845+0000 mgr.y (mgr.24368) 736 : cluster 0 pgmap v786: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 815 op/s 2024-04-06T03:40:57.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:57.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:57.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:57.457 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:57.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:57.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:57.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:57.726 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:57.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:57.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:57.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:57.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:57 smithi153 bash[23852]: cluster 2024-04-06T03:40:56.627753+0000 osd.3 (osd.3) 490 : cluster 0 2.1e scrub starts 2024-04-06T03:40:57.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:57 smithi153 bash[23852]: cluster 2024-04-06T03:40:56.694452+0000 osd.3 (osd.3) 491 : cluster 0 2.1e scrub ok 2024-04-06T03:40:57.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:57 smithi153 bash[23852]: cluster 2024-04-06T03:40:57.255782+0000 osd.7 (osd.7) 497 : cluster 0 5.9 scrub starts 2024-04-06T03:40:57.975 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:57 smithi153 bash[23852]: cluster 2024-04-06T03:40:57.261666+0000 osd.7 (osd.7) 498 : cluster 0 5.9 scrub ok 2024-04-06T03:40:57.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:57.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:58.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:58.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:58.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:58.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[21467]: cluster 2024-04-06T03:40:56.627753+0000 osd.3 (osd.3) 490 : cluster 0 2.1e scrub starts 2024-04-06T03:40:58.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[21467]: cluster 2024-04-06T03:40:56.694452+0000 osd.3 (osd.3) 491 : cluster 0 2.1e scrub ok 2024-04-06T03:40:58.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[21467]: cluster 2024-04-06T03:40:57.255782+0000 osd.7 (osd.7) 497 : cluster 0 5.9 scrub starts 2024-04-06T03:40:58.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[21467]: cluster 2024-04-06T03:40:57.261666+0000 osd.7 (osd.7) 498 : cluster 0 5.9 scrub ok 2024-04-06T03:40:58.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[29221]: cluster 2024-04-06T03:40:56.627753+0000 osd.3 (osd.3) 490 : cluster 0 2.1e scrub starts 2024-04-06T03:40:58.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[29221]: cluster 2024-04-06T03:40:56.694452+0000 osd.3 (osd.3) 491 : cluster 0 2.1e scrub ok 2024-04-06T03:40:58.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[29221]: cluster 2024-04-06T03:40:57.255782+0000 osd.7 (osd.7) 497 : cluster 0 5.9 scrub starts 2024-04-06T03:40:58.237 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:57 smithi050 bash[29221]: cluster 2024-04-06T03:40:57.261666+0000 osd.7 (osd.7) 498 : cluster 0 5.9 scrub ok 2024-04-06T03:40:58.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:40:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:40:58.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:58.370 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:40:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:40:58.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:58.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:58.634 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:58.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:58.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:40:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:40:58.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:58.899 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:58.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:59.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:59.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:59.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:59.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:58 smithi153 bash[23852]: cluster 2024-04-06T03:40:57.213165+0000 osd.4 (osd.4) 461 : cluster 0 10.e scrub starts 2024-04-06T03:40:59.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:58 smithi153 bash[23852]: cluster 2024-04-06T03:40:58.419700+0000 mgr.y (mgr.24368) 737 : cluster 0 pgmap v787: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 836 op/s 2024-04-06T03:40:59.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:58 smithi050 bash[21467]: cluster 2024-04-06T03:40:57.213165+0000 osd.4 (osd.4) 461 : cluster 0 10.e scrub starts 2024-04-06T03:40:59.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:58 smithi050 bash[21467]: cluster 2024-04-06T03:40:58.419700+0000 mgr.y (mgr.24368) 737 : cluster 0 pgmap v787: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 836 op/s 2024-04-06T03:40:59.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:58 smithi050 bash[29221]: cluster 2024-04-06T03:40:57.213165+0000 osd.4 (osd.4) 461 : cluster 0 10.e scrub starts 2024-04-06T03:40:59.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:58 smithi050 bash[29221]: cluster 2024-04-06T03:40:58.419700+0000 mgr.y (mgr.24368) 737 : cluster 0 pgmap v787: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 836 op/s 2024-04-06T03:40:59.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:59.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:59.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:40:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:40:59.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:59.441 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:40:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:40:59.489 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:40:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:40:59.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:59.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:40:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:40:59.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:59.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:59.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:59.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:40:59.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:40:59.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:59.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:40:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:40:59.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:40:59.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[21467]: cluster 2024-04-06T03:40:58.189909+0000 osd.4 (osd.4) 462 : cluster 0 2.f scrub starts 2024-04-06T03:40:59.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[21467]: cluster 2024-04-06T03:40:58.191590+0000 osd.4 (osd.4) 463 : cluster 0 2.f scrub ok 2024-04-06T03:40:59.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[21467]: cluster 2024-04-06T03:40:58.453418+0000 osd.4 (osd.4) 464 : cluster 0 10.e scrub ok 2024-04-06T03:40:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[29221]: cluster 2024-04-06T03:40:58.189909+0000 osd.4 (osd.4) 462 : cluster 0 2.f scrub starts 2024-04-06T03:40:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[29221]: cluster 2024-04-06T03:40:58.191590+0000 osd.4 (osd.4) 463 : cluster 0 2.f scrub ok 2024-04-06T03:40:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:40:59 smithi050 bash[29221]: cluster 2024-04-06T03:40:58.453418+0000 osd.4 (osd.4) 464 : cluster 0 10.e scrub ok 2024-04-06T03:41:00.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:00.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:00.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:00.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:41:00.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:00.225 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:41:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:41:00.225 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:59 smithi153 bash[23852]: cluster 2024-04-06T03:40:58.189909+0000 osd.4 (osd.4) 462 : cluster 0 2.f scrub starts 2024-04-06T03:41:00.225 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:59 smithi153 bash[23852]: cluster 2024-04-06T03:40:58.191590+0000 osd.4 (osd.4) 463 : cluster 0 2.f scrub ok 2024-04-06T03:41:00.225 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:40:59 smithi153 bash[23852]: cluster 2024-04-06T03:40:58.453418+0000 osd.4 (osd.4) 464 : cluster 0 10.e scrub ok 2024-04-06T03:41:00.242 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:00.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:00.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:00.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:00.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:00.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:00.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:00.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:00.883 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:00.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:00.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:01.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:01.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:01.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:01.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:00 smithi153 bash[23852]: audit 2024-04-06T03:41:00.009116+0000 mgr.y (mgr.24368) 738 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:01.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:00 smithi153 bash[23852]: cluster 2024-04-06T03:41:00.111648+0000 osd.1 (osd.1) 377 : cluster 0 3.19 scrub starts 2024-04-06T03:41:01.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:00 smithi153 bash[23852]: cluster 2024-04-06T03:41:00.118272+0000 osd.1 (osd.1) 378 : cluster 0 3.19 scrub ok 2024-04-06T03:41:01.169 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:00 smithi153 bash[23852]: cluster 2024-04-06T03:41:00.420332+0000 mgr.y (mgr.24368) 739 : cluster 0 pgmap v788: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 797 op/s 2024-04-06T03:41:01.193 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:01.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[21467]: audit 2024-04-06T03:41:00.009116+0000 mgr.y (mgr.24368) 738 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[21467]: cluster 2024-04-06T03:41:00.111648+0000 osd.1 (osd.1) 377 : cluster 0 3.19 scrub starts 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[21467]: cluster 2024-04-06T03:41:00.118272+0000 osd.1 (osd.1) 378 : cluster 0 3.19 scrub ok 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[21467]: cluster 2024-04-06T03:41:00.420332+0000 mgr.y (mgr.24368) 739 : cluster 0 pgmap v788: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 797 op/s 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[29221]: audit 2024-04-06T03:41:00.009116+0000 mgr.y (mgr.24368) 738 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[29221]: cluster 2024-04-06T03:41:00.111648+0000 osd.1 (osd.1) 377 : cluster 0 3.19 scrub starts 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[29221]: cluster 2024-04-06T03:41:00.118272+0000 osd.1 (osd.1) 378 : cluster 0 3.19 scrub ok 2024-04-06T03:41:01.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:00 smithi050 bash[29221]: cluster 2024-04-06T03:41:00.420332+0000 mgr.y (mgr.24368) 739 : cluster 0 pgmap v788: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 797 op/s 2024-04-06T03:41:01.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:01.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:01.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:01.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:01.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:01.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:01.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:01.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:01.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:01.845 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:01.947 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:01.969 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:01.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:01.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:01 smithi050 bash[21467]: cluster 2024-04-06T03:41:00.851319+0000 osd.2 (osd.2) 381 : cluster 0 10.9 scrub starts 2024-04-06T03:41:01.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:01 smithi050 bash[21467]: cluster 2024-04-06T03:41:01.116761+0000 osd.1 (osd.1) 379 : cluster 0 2.15 scrub starts 2024-04-06T03:41:01.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:01 smithi050 bash[21467]: cluster 2024-04-06T03:41:01.119794+0000 osd.1 (osd.1) 380 : cluster 0 2.15 scrub ok 2024-04-06T03:41:01.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:01 smithi050 bash[29221]: cluster 2024-04-06T03:41:00.851319+0000 osd.2 (osd.2) 381 : cluster 0 10.9 scrub starts 2024-04-06T03:41:01.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:01 smithi050 bash[29221]: cluster 2024-04-06T03:41:01.116761+0000 osd.1 (osd.1) 379 : cluster 0 2.15 scrub starts 2024-04-06T03:41:01.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:01 smithi050 bash[29221]: cluster 2024-04-06T03:41:01.119794+0000 osd.1 (osd.1) 380 : cluster 0 2.15 scrub ok 2024-04-06T03:41:02.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:02.100 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:01 smithi153 bash[23852]: cluster 2024-04-06T03:41:00.851319+0000 osd.2 (osd.2) 381 : cluster 0 10.9 scrub starts 2024-04-06T03:41:02.100 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:01 smithi153 bash[23852]: cluster 2024-04-06T03:41:01.116761+0000 osd.1 (osd.1) 379 : cluster 0 2.15 scrub starts 2024-04-06T03:41:02.100 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:01 smithi153 bash[23852]: cluster 2024-04-06T03:41:01.119794+0000 osd.1 (osd.1) 380 : cluster 0 2.15 scrub ok 2024-04-06T03:41:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.101 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:02.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:02.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:02.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:02.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:02.465 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:02.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:02.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:02.636 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:02.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:02.763 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:02.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:02.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:02.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:02.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[21467]: cluster 2024-04-06T03:41:01.178161+0000 osd.4 (osd.4) 465 : cluster 0 6.17 scrub starts 2024-04-06T03:41:02.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[21467]: cluster 2024-04-06T03:41:01.299944+0000 osd.4 (osd.4) 466 : cluster 0 6.17 scrub ok 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[21467]: cluster 2024-04-06T03:41:01.913496+0000 osd.2 (osd.2) 382 : cluster 0 2.17 scrub starts 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[21467]: cluster 2024-04-06T03:41:01.920508+0000 osd.2 (osd.2) 383 : cluster 0 2.17 scrub ok 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.420939+0000 mgr.y (mgr.24368) 740 : cluster 0 pgmap v789: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 842 op/s 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[29221]: cluster 2024-04-06T03:41:01.178161+0000 osd.4 (osd.4) 465 : cluster 0 6.17 scrub starts 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[29221]: cluster 2024-04-06T03:41:01.299944+0000 osd.4 (osd.4) 466 : cluster 0 6.17 scrub ok 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[29221]: cluster 2024-04-06T03:41:01.913496+0000 osd.2 (osd.2) 382 : cluster 0 2.17 scrub starts 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[29221]: cluster 2024-04-06T03:41:01.920508+0000 osd.2 (osd.2) 383 : cluster 0 2.17 scrub ok 2024-04-06T03:41:02.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:02 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.420939+0000 mgr.y (mgr.24368) 740 : cluster 0 pgmap v789: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 842 op/s 2024-04-06T03:41:02.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:02 smithi153 bash[23852]: cluster 2024-04-06T03:41:01.178161+0000 osd.4 (osd.4) 465 : cluster 0 6.17 scrub starts 2024-04-06T03:41:03.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:02 smithi153 bash[23852]: cluster 2024-04-06T03:41:01.299944+0000 osd.4 (osd.4) 466 : cluster 0 6.17 scrub ok 2024-04-06T03:41:03.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:02 smithi153 bash[23852]: cluster 2024-04-06T03:41:01.913496+0000 osd.2 (osd.2) 382 : cluster 0 2.17 scrub starts 2024-04-06T03:41:03.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:02 smithi153 bash[23852]: cluster 2024-04-06T03:41:01.920508+0000 osd.2 (osd.2) 383 : cluster 0 2.17 scrub ok 2024-04-06T03:41:03.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:02 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.420939+0000 mgr.y (mgr.24368) 740 : cluster 0 pgmap v789: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 842 op/s 2024-04-06T03:41:03.025 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:03.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:03.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.288 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:03.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:03.421 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:03.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:03.553 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:03.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:03.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:03.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:03.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:03.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:03.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.226442+0000 osd.4 (osd.4) 467 : cluster 0 3.3 scrub starts 2024-04-06T03:41:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.234387+0000 osd.4 (osd.4) 468 : cluster 0 3.3 scrub ok 2024-04-06T03:41:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.331077+0000 osd.7 (osd.7) 499 : cluster 0 2.1d scrub starts 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.388336+0000 osd.7 (osd.7) 500 : cluster 0 2.1d scrub ok 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.705277+0000 osd.2 (osd.2) 384 : cluster 0 10.9 scrub ok 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.963640+0000 osd.2 (osd.2) 385 : cluster 0 2.10 scrub starts 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:02.968335+0000 osd.2 (osd.2) 386 : cluster 0 2.10 scrub ok 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:03.047088+0000 osd.1 (osd.1) 381 : cluster 0 6.1 scrub starts 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:03.169736+0000 osd.1 (osd.1) 382 : cluster 0 6.1 scrub ok 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:03.305286+0000 osd.7 (osd.7) 501 : cluster 0 3.11 scrub starts 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:03.308059+0000 osd.7 (osd.7) 502 : cluster 0 3.11 scrub ok 2024-04-06T03:41:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:03.869933+0000 osd.0 (osd.0) 478 : cluster 0 3.1f scrub starts 2024-04-06T03:41:04.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:03 smithi153 bash[23852]: cluster 2024-04-06T03:41:03.880746+0000 osd.0 (osd.0) 479 : cluster 0 3.1f scrub ok 2024-04-06T03:41:04.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:04.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:04.212 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.226442+0000 osd.4 (osd.4) 467 : cluster 0 3.3 scrub starts 2024-04-06T03:41:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.234387+0000 osd.4 (osd.4) 468 : cluster 0 3.3 scrub ok 2024-04-06T03:41:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.331077+0000 osd.7 (osd.7) 499 : cluster 0 2.1d scrub starts 2024-04-06T03:41:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.388336+0000 osd.7 (osd.7) 500 : cluster 0 2.1d scrub ok 2024-04-06T03:41:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.705277+0000 osd.2 (osd.2) 384 : cluster 0 10.9 scrub ok 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.963640+0000 osd.2 (osd.2) 385 : cluster 0 2.10 scrub starts 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:02.968335+0000 osd.2 (osd.2) 386 : cluster 0 2.10 scrub ok 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:03.047088+0000 osd.1 (osd.1) 381 : cluster 0 6.1 scrub starts 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:03.169736+0000 osd.1 (osd.1) 382 : cluster 0 6.1 scrub ok 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:03.305286+0000 osd.7 (osd.7) 501 : cluster 0 3.11 scrub starts 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:03.308059+0000 osd.7 (osd.7) 502 : cluster 0 3.11 scrub ok 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:03.869933+0000 osd.0 (osd.0) 478 : cluster 0 3.1f scrub starts 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:03.880746+0000 osd.0 (osd.0) 479 : cluster 0 3.1f scrub ok 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.226442+0000 osd.4 (osd.4) 467 : cluster 0 3.3 scrub starts 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.234387+0000 osd.4 (osd.4) 468 : cluster 0 3.3 scrub ok 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.331077+0000 osd.7 (osd.7) 499 : cluster 0 2.1d scrub starts 2024-04-06T03:41:04.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.388336+0000 osd.7 (osd.7) 500 : cluster 0 2.1d scrub ok 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.705277+0000 osd.2 (osd.2) 384 : cluster 0 10.9 scrub ok 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.963640+0000 osd.2 (osd.2) 385 : cluster 0 2.10 scrub starts 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:02.968335+0000 osd.2 (osd.2) 386 : cluster 0 2.10 scrub ok 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:03.047088+0000 osd.1 (osd.1) 381 : cluster 0 6.1 scrub starts 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:03.169736+0000 osd.1 (osd.1) 382 : cluster 0 6.1 scrub ok 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:03.305286+0000 osd.7 (osd.7) 501 : cluster 0 3.11 scrub starts 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:03.308059+0000 osd.7 (osd.7) 502 : cluster 0 3.11 scrub ok 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:03.869933+0000 osd.0 (osd.0) 478 : cluster 0 3.1f scrub starts 2024-04-06T03:41:04.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:03 smithi050 bash[29221]: cluster 2024-04-06T03:41:03.880746+0000 osd.0 (osd.0) 479 : cluster 0 3.1f scrub ok 2024-04-06T03:41:04.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.342 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:04.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.472 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:04.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:04.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:04.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:04.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:04.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:04.869 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:04.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:04.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:04.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:04.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:05.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:05.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:04 smithi153 bash[23852]: cluster 2024-04-06T03:41:04.421549+0000 mgr.y (mgr.24368) 741 : cluster 0 pgmap v790: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:41:05.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:04 smithi153 bash[23852]: cluster 2024-04-06T03:41:04.815864+0000 osd.0 (osd.0) 480 : cluster 0 3.10 scrub starts 2024-04-06T03:41:05.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:04 smithi153 bash[23852]: cluster 2024-04-06T03:41:04.817531+0000 osd.0 (osd.0) 481 : cluster 0 3.10 scrub ok 2024-04-06T03:41:05.121 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:05.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:05.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:05.270 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:04.421549+0000 mgr.y (mgr.24368) 741 : cluster 0 pgmap v790: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:41:05.271 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:04.815864+0000 osd.0 (osd.0) 480 : cluster 0 3.10 scrub starts 2024-04-06T03:41:05.271 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[21467]: cluster 2024-04-06T03:41:04.817531+0000 osd.0 (osd.0) 481 : cluster 0 3.10 scrub ok 2024-04-06T03:41:05.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[29221]: cluster 2024-04-06T03:41:04.421549+0000 mgr.y (mgr.24368) 741 : cluster 0 pgmap v790: 148 pgs: 148 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 889 op/s 2024-04-06T03:41:05.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[29221]: cluster 2024-04-06T03:41:04.815864+0000 osd.0 (osd.0) 480 : cluster 0 3.10 scrub starts 2024-04-06T03:41:05.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:04 smithi050 bash[29221]: cluster 2024-04-06T03:41:04.817531+0000 osd.0 (osd.0) 481 : cluster 0 3.10 scrub ok 2024-04-06T03:41:05.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:05.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:05.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:05.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:05.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:05.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:05.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:05.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:05.791 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:41:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:41:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:41:05.792 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:41:05.792 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:41:05.872 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:05.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:05.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:05.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:06.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:06.031 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:05 smithi050 bash[21467]: audit 2024-04-06T03:41:05.084375+0000 mon.a (mon.0) 1549 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:06.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:05 smithi050 bash[29221]: audit 2024-04-06T03:41:05.084375+0000 mon.a (mon.0) 1549 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:06.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 307, 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "0313e520-f3c3-11ee-b649-cb9ed24678a4", 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-06T03:10:26.969128+0000", 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-06T03:40:54.901787+0000", 2024-04-06T03:41:06.121 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-06T03:20:09.072730+0000", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-06T03:19:42.545394+0000", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5804032, 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nodeep-scrub", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 18, 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-06T03:41:06.122 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 10, 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:15:48.096900+0000", 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:41:06.123 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-06T03:41:06.124 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.125 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.126 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.127 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.128 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 6, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 6, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.5, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.129 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:50.011930+0000", 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.130 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "79", 2024-04-06T03:41:06.131 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.132 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.133 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.3200000524520874, 2024-04-06T03:41:06.134 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.3200000524520874, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:51.687715+0000", 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:41:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.136 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "81", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:41:06.137 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.138 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.139 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:41:06.140 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.301921+0000", 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.141 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.142 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "88", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 88, 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.143 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.144 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:41:06.145 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.1699999570846558, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:53.810316+0000", 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.147 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "83", 2024-04-06T03:41:06.148 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.149 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.150 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:41:06.151 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:20:56.321502+0000", 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.152 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-06T03:41:06.153 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "85", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.154 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.155 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.156 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:41:06.157 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 10, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "unique_pool_3", 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-06T03:38:39.661312+0000", 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-06T03:41:06.158 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 16, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 16, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 16, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 16, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 16, 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-06T03:41:06.159 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "287", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-06T03:41:06.160 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-06T03:41:06.161 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-06T03:41:06.162 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1.3300000429153442, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 0.69516563415527344, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 0.93000000715255737, 2024-04-06T03:41:06.163 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "e607f527-4592-4acd-a47f-76575615ae1b", 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-06T03:41:06.164 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 280, 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3531754646 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/3531754646", 2024-04-06T03:41:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/3531754646", 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/3531754646", 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/3531754646", 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "2ae8f784-d36a-4633-8630-2aed04e4ff10", 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0.5613250732421875, 2024-04-06T03:41:06.169 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 280, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.170 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.171 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.172 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-06T03:41:06.173 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 4128410069 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/4128410069", 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/4128410069", 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/4128410069", 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/4128410069", 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:41:06.174 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43f31d09-5c16-4aea-8822-e0d877cc931b", 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 19, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 280, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.178 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1307247904 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/1307247904", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/1307247904", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/1307247904", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/1307247904", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "4c913325-442e-4ee5-94f1-8b5bca0c3240", 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 280, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.181 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-06T03:41:06.182 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1397888508 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/1397888508", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/1397888508", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/1397888508", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/1397888508", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.184 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "50c02bfc-c2f2-4aae-b1d5-c38a86726b48", 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 280, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.185 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6800", 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6801", 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.186 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6802", 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6803", 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6806", 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6807", 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.188 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6804", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6805", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3377683473 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6801/3377683473", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6803/3377683473", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6807/3377683473", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6805/3377683473", 2024-04-06T03:41:06.189 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "c3e8194b-2c20-4ffb-859d-00c33902b241", 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.190 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 46, 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 127, 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6808", 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.191 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6809", 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6810", 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6811", 2024-04-06T03:41:06.192 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6814", 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6815", 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6812", 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6813", 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 357179117 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.194 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6809/357179117", 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6811/357179117", 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6815/357179117", 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6813/357179117", 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "98027d53-4324-44d1-89ce-8da85be58efb", 2024-04-06T03:41:06.195 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 0, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 135, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6816", 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6817", 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6818", 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6819", 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6822", 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6823", 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6820", 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.199 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6821", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 1552538208 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6817/1552538208", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6819/1552538208", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6823/1552538208", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6821/1552538208", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "6b47d95a-1228-4fb1-a315-5371d9b42bbd", 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 280, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6824", 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6825", 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6826", 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6827", 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.203 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6830", 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6831", 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-06T03:41:06.204 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6828", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.153:6829", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2479193924 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.153:6825/2479193924", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.153:6827/2479193924", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.153:6831/2479193924", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.153:6829/2479193924", 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-06T03:41:06.205 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.206 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:39:53.691554+0000", 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:40:48.619696+0000", 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:39:52.850707+0000", 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.208 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:40:20.241925+0000", 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:40:54.448608+0000", 2024-04-06T03:41:06.209 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:40:12.101131+0000", 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.210 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:40:40.927101+0000", 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-06T03:40:15.354513+0000", 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-06T03:41:06.211 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.12", 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "3.4", 2024-04-06T03:41:06.212 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 5, 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 7 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/283275195": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2738019433": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:41:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1268273764": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/567103210": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2891059051": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2170219983": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/4082841962": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/1263219005": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/570623774": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/889382629": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1379780741": "2024-04-07T03:21:47.915758+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2880368026": "2024-04-07T03:10:58.614019+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/132418263": "2024-04-07T03:11:33.442338+0000", 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2044358308": "2024-04-07T03:21:47.915758+0000" 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-06T03:41:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [], 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-06T03:41:06.215 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-06T03:41:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-06T03:41:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-06T03:41:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-06T03:41:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-06T03:41:06.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:41:06.216 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:41:06.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:06.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:05 smithi153 bash[23852]: audit 2024-04-06T03:41:05.084375+0000 mon.a (mon.0) 1549 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:06.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:06.361 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:06.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:06.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:06.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:06.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:06.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:06.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:06.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:06.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:06.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:06.969 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:41:06.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:07.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:06 smithi153 bash[23852]: audit 2024-04-06T03:41:06.115633+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.50:0/1089293343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:41:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:06 smithi153 bash[23852]: audit 2024-04-06T03:41:06.116083+0000 mon.a (mon.0) 1550 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:41:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:06 smithi153 bash[23852]: audit 2024-04-06T03:41:06.120284+0000 mon.c (mon.2) 96 : audit 0 from='client.? 172.21.15.50:0/2508079981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:41:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:06 smithi153 bash[23852]: cluster 2024-04-06T03:41:06.422453+0000 mgr.y (mgr.24368) 742 : cluster 0 pgmap v791: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:41:07.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:07.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:07.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[21467]: audit 2024-04-06T03:41:06.115633+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.50:0/1089293343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[21467]: audit 2024-04-06T03:41:06.116083+0000 mon.a (mon.0) 1550 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[21467]: audit 2024-04-06T03:41:06.120284+0000 mon.c (mon.2) 96 : audit 0 from='client.? 172.21.15.50:0/2508079981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[21467]: cluster 2024-04-06T03:41:06.422453+0000 mgr.y (mgr.24368) 742 : cluster 0 pgmap v791: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[29221]: audit 2024-04-06T03:41:06.115633+0000 mon.c (mon.2) 95 : audit 1 from='client.? 172.21.15.50:0/1089293343' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[29221]: audit 2024-04-06T03:41:06.116083+0000 mon.a (mon.0) 1550 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:41:07.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[29221]: audit 2024-04-06T03:41:06.120284+0000 mon.c (mon.2) 96 : audit 0 from='client.? 172.21.15.50:0/2508079981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-06T03:41:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:06 smithi050 bash[29221]: cluster 2024-04-06T03:41:06.422453+0000 mgr.y (mgr.24368) 742 : cluster 0 pgmap v791: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 869 op/s 2024-04-06T03:41:07.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:07.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:07.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:07.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:07.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:07.550 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:07.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:07.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:07.796 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:07.821 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:07.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:07.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:07.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:08.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:08.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:08.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: cluster 2024-04-06T03:41:06.952358+0000 osd.2 (osd.2) 387 : cluster 0 10.3 scrub starts 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: cluster 2024-04-06T03:41:06.953835+0000 mon.a (mon.0) 1551 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: cluster 2024-04-06T03:41:06.953869+0000 mon.a (mon.0) 1552 : cluster 1 Cluster is now healthy 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: audit 2024-04-06T03:41:06.963257+0000 mon.a (mon.0) 1553 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: cluster 2024-04-06T03:41:06.966869+0000 mon.a (mon.0) 1554 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: cluster 2024-04-06T03:41:07.271891+0000 osd.1 (osd.1) 383 : cluster 0 2.2 scrub starts 2024-04-06T03:41:08.209 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:07 smithi153 bash[23852]: cluster 2024-04-06T03:41:07.273825+0000 osd.1 (osd.1) 384 : cluster 0 2.2 scrub ok 2024-04-06T03:41:08.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: cluster 2024-04-06T03:41:06.952358+0000 osd.2 (osd.2) 387 : cluster 0 10.3 scrub starts 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: cluster 2024-04-06T03:41:06.953835+0000 mon.a (mon.0) 1551 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: cluster 2024-04-06T03:41:06.953869+0000 mon.a (mon.0) 1552 : cluster 1 Cluster is now healthy 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: audit 2024-04-06T03:41:06.963257+0000 mon.a (mon.0) 1553 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: cluster 2024-04-06T03:41:06.966869+0000 mon.a (mon.0) 1554 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: cluster 2024-04-06T03:41:07.271891+0000 osd.1 (osd.1) 383 : cluster 0 2.2 scrub starts 2024-04-06T03:41:08.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[21467]: cluster 2024-04-06T03:41:07.273825+0000 osd.1 (osd.1) 384 : cluster 0 2.2 scrub ok 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: cluster 2024-04-06T03:41:06.952358+0000 osd.2 (osd.2) 387 : cluster 0 10.3 scrub starts 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: cluster 2024-04-06T03:41:06.953835+0000 mon.a (mon.0) 1551 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: cluster 2024-04-06T03:41:06.953869+0000 mon.a (mon.0) 1552 : cluster 1 Cluster is now healthy 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: audit 2024-04-06T03:41:06.963257+0000 mon.a (mon.0) 1553 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: cluster 2024-04-06T03:41:06.966869+0000 mon.a (mon.0) 1554 : cluster 0 osdmap e308: 8 total, 8 up, 8 in 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: cluster 2024-04-06T03:41:07.271891+0000 osd.1 (osd.1) 383 : cluster 0 2.2 scrub starts 2024-04-06T03:41:08.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:07 smithi050 bash[29221]: cluster 2024-04-06T03:41:07.273825+0000 osd.1 (osd.1) 384 : cluster 0 2.2 scrub ok 2024-04-06T03:41:08.329 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:08.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:08.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:08.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:08.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:08.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:08.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:08.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:08.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:08.885 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:08.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:08.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:09.013 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:09.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[21467]: cluster 2024-04-06T03:41:07.075801+0000 osd.4 (osd.4) 469 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[21467]: cluster 2024-04-06T03:41:07.110766+0000 osd.4 (osd.4) 470 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[21467]: cluster 2024-04-06T03:41:08.423384+0000 mgr.y (mgr.24368) 743 : cluster 0 pgmap v793: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 929 op/s 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[21467]: cluster 2024-04-06T03:41:08.465864+0000 osd.3 (osd.3) 492 : cluster 0 5.16 scrub starts 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[21467]: cluster 2024-04-06T03:41:08.471899+0000 osd.3 (osd.3) 493 : cluster 0 5.16 scrub ok 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[29221]: cluster 2024-04-06T03:41:07.075801+0000 osd.4 (osd.4) 469 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[29221]: cluster 2024-04-06T03:41:07.110766+0000 osd.4 (osd.4) 470 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[29221]: cluster 2024-04-06T03:41:08.423384+0000 mgr.y (mgr.24368) 743 : cluster 0 pgmap v793: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 929 op/s 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[29221]: cluster 2024-04-06T03:41:08.465864+0000 osd.3 (osd.3) 492 : cluster 0 5.16 scrub starts 2024-04-06T03:41:09.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:08 smithi050 bash[29221]: cluster 2024-04-06T03:41:08.471899+0000 osd.3 (osd.3) 493 : cluster 0 5.16 scrub ok 2024-04-06T03:41:09.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:09.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:09.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:08 smithi153 bash[23852]: cluster 2024-04-06T03:41:07.075801+0000 osd.4 (osd.4) 469 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:41:09.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:08 smithi153 bash[23852]: cluster 2024-04-06T03:41:07.110766+0000 osd.4 (osd.4) 470 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:41:09.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:08 smithi153 bash[23852]: cluster 2024-04-06T03:41:08.423384+0000 mgr.y (mgr.24368) 743 : cluster 0 pgmap v793: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 929 op/s 2024-04-06T03:41:09.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:08 smithi153 bash[23852]: cluster 2024-04-06T03:41:08.465864+0000 osd.3 (osd.3) 492 : cluster 0 5.16 scrub starts 2024-04-06T03:41:09.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:08 smithi153 bash[23852]: cluster 2024-04-06T03:41:08.471899+0000 osd.3 (osd.3) 493 : cluster 0 5.16 scrub ok 2024-04-06T03:41:09.290 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:09.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:09.407 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:41:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:41:09.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:09.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:09.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:09.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:09.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:09.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:09.844 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:09.947 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:09.976 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:09.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:10.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:10.108 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:10.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:10.220 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:41:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:41:10.221 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:09 smithi153 bash[23852]: cluster 2024-04-06T03:41:09.067217+0000 osd.2 (osd.2) 388 : cluster 0 10.3 scrub ok 2024-04-06T03:41:10.221 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:09 smithi153 bash[23852]: cluster 2024-04-06T03:41:09.290052+0000 osd.1 (osd.1) 385 : cluster 0 6.19 scrub starts 2024-04-06T03:41:10.221 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:09 smithi153 bash[23852]: cluster 2024-04-06T03:41:09.420975+0000 osd.1 (osd.1) 386 : cluster 0 6.19 scrub ok 2024-04-06T03:41:10.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:10.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[21467]: cluster 2024-04-06T03:41:09.067217+0000 osd.2 (osd.2) 388 : cluster 0 10.3 scrub ok 2024-04-06T03:41:10.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[21467]: cluster 2024-04-06T03:41:09.290052+0000 osd.1 (osd.1) 385 : cluster 0 6.19 scrub starts 2024-04-06T03:41:10.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[21467]: cluster 2024-04-06T03:41:09.420975+0000 osd.1 (osd.1) 386 : cluster 0 6.19 scrub ok 2024-04-06T03:41:10.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[29221]: cluster 2024-04-06T03:41:09.067217+0000 osd.2 (osd.2) 388 : cluster 0 10.3 scrub ok 2024-04-06T03:41:10.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[29221]: cluster 2024-04-06T03:41:09.290052+0000 osd.1 (osd.1) 385 : cluster 0 6.19 scrub starts 2024-04-06T03:41:10.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:09 smithi050 bash[29221]: cluster 2024-04-06T03:41:09.420975+0000 osd.1 (osd.1) 386 : cluster 0 6.19 scrub ok 2024-04-06T03:41:10.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:10.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:10.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:10.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:10.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:10.619 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:10.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:10.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:10.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:10.877 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:10.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:10.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:10.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:10 smithi050 bash[21467]: audit 2024-04-06T03:41:10.016970+0000 mgr.y (mgr.24368) 744 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:10.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:10 smithi050 bash[21467]: cluster 2024-04-06T03:41:10.424157+0000 mgr.y (mgr.24368) 745 : cluster 0 pgmap v794: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 900 op/s 2024-04-06T03:41:10.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:10 smithi050 bash[29221]: audit 2024-04-06T03:41:10.016970+0000 mgr.y (mgr.24368) 744 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:10.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:10 smithi050 bash[29221]: cluster 2024-04-06T03:41:10.424157+0000 mgr.y (mgr.24368) 745 : cluster 0 pgmap v794: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 900 op/s 2024-04-06T03:41:11.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:11.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:11.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:11.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:11.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:10 smithi153 bash[23852]: audit 2024-04-06T03:41:10.016970+0000 mgr.y (mgr.24368) 744 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:10 smithi153 bash[23852]: cluster 2024-04-06T03:41:10.424157+0000 mgr.y (mgr.24368) 745 : cluster 0 pgmap v794: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 900 op/s 2024-04-06T03:41:11.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:11.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:11.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:11.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.663 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:11.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:11.797 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:11.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:11.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-06T03:41:11.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:11.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:12.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:12.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:11 smithi153 bash[23852]: cluster 2024-04-06T03:41:10.999537+0000 osd.4 (osd.4) 471 : cluster 0 5.11 scrub starts 2024-04-06T03:41:12.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:11 smithi153 bash[23852]: cluster 2024-04-06T03:41:11.000502+0000 osd.4 (osd.4) 472 : cluster 0 5.11 scrub ok 2024-04-06T03:41:12.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:11 smithi153 bash[23852]: cluster 2024-04-06T03:41:11.020582+0000 osd.2 (osd.2) 389 : cluster 0 2.e scrub starts 2024-04-06T03:41:12.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:11 smithi153 bash[23852]: cluster 2024-04-06T03:41:11.021620+0000 osd.2 (osd.2) 390 : cluster 0 2.e scrub ok 2024-04-06T03:41:12.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:11 smithi153 bash[23852]: cluster 2024-04-06T03:41:11.898106+0000 osd.0 (osd.0) 482 : cluster 0 6.9 scrub starts 2024-04-06T03:41:12.064 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:12.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:12.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[21467]: cluster 2024-04-06T03:41:10.999537+0000 osd.4 (osd.4) 471 : cluster 0 5.11 scrub starts 2024-04-06T03:41:12.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[21467]: cluster 2024-04-06T03:41:11.000502+0000 osd.4 (osd.4) 472 : cluster 0 5.11 scrub ok 2024-04-06T03:41:12.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[21467]: cluster 2024-04-06T03:41:11.020582+0000 osd.2 (osd.2) 389 : cluster 0 2.e scrub starts 2024-04-06T03:41:12.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[21467]: cluster 2024-04-06T03:41:11.021620+0000 osd.2 (osd.2) 390 : cluster 0 2.e scrub ok 2024-04-06T03:41:12.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[21467]: cluster 2024-04-06T03:41:11.898106+0000 osd.0 (osd.0) 482 : cluster 0 6.9 scrub starts 2024-04-06T03:41:12.180 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[29221]: cluster 2024-04-06T03:41:10.999537+0000 osd.4 (osd.4) 471 : cluster 0 5.11 scrub starts 2024-04-06T03:41:12.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[29221]: cluster 2024-04-06T03:41:11.000502+0000 osd.4 (osd.4) 472 : cluster 0 5.11 scrub ok 2024-04-06T03:41:12.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[29221]: cluster 2024-04-06T03:41:11.020582+0000 osd.2 (osd.2) 389 : cluster 0 2.e scrub starts 2024-04-06T03:41:12.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[29221]: cluster 2024-04-06T03:41:11.021620+0000 osd.2 (osd.2) 390 : cluster 0 2.e scrub ok 2024-04-06T03:41:12.181 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:11 smithi050 bash[29221]: cluster 2024-04-06T03:41:11.898106+0000 osd.0 (osd.0) 482 : cluster 0 6.9 scrub starts 2024-04-06T03:41:12.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:12.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:12.351 INFO:tasks.thrashosds.thrasher:j is {'epoch': 307, 'fsid': '0313e520-f3c3-11ee-b649-cb9ed24678a4', 'created': '2024-04-06T03:10:26.969128+0000', 'modified': '2024-04-06T03:40:54.901787+0000', 'last_up_change': '2024-04-06T03:20:09.072730+0000', 'last_in_change': '2024-04-06T03:19:42.545394+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-06T03:15:48.096900+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-06T03:20:50.011930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-06T03:20:51.687715+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-06T03:20:53.301921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 88, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 1.1699999570846558, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-06T03:20:53.810316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-06T03:20:56.321502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-04-06T03:38:39.661312+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1.3300000429153442, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.6951656341552734, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'e607f527-4592-4acd-a47f-76575615ae1b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 3531754646}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 3531754646}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 3531754646}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 3531754646}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 3531754646}]}, 'public_addr': '172.21.15.50:6803/3531754646', 'cluster_addr': '172.21.15.50:6805/3531754646', 'heartbeat_back_addr': '172.21.15.50:6809/3531754646', 'heartbeat_front_addr': '172.21.15.50:6807/3531754646', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2ae8f784-d36a-4633-8630-2aed04e4ff10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5613250732421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 4128410069}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 4128410069}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 4128410069}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 4128410069}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 4128410069}]}, 'public_addr': '172.21.15.50:6811/4128410069', 'cluster_addr': '172.21.15.50:6813/4128410069', 'heartbeat_back_addr': '172.21.15.50:6817/4128410069', 'heartbeat_front_addr': '172.21.15.50:6815/4128410069', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '43f31d09-5c16-4aea-8822-e0d877cc931b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 19, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1307247904}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1307247904}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 1307247904}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1307247904}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1307247904}]}, 'public_addr': '172.21.15.50:6819/1307247904', 'cluster_addr': '172.21.15.50:6821/1307247904', 'heartbeat_back_addr': '172.21.15.50:6825/1307247904', 'heartbeat_front_addr': '172.21.15.50:6823/1307247904', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '4c913325-442e-4ee5-94f1-8b5bca0c3240', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 1397888508}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 1397888508}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 1397888508}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 1397888508}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 1397888508}]}, 'public_addr': '172.21.15.50:6827/1397888508', 'cluster_addr': '172.21.15.50:6829/1397888508', 'heartbeat_back_addr': '172.21.15.50:6833/1397888508', 'heartbeat_front_addr': '172.21.15.50:6831/1397888508', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '50c02bfc-c2f2-4aae-b1d5-c38a86726b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6800', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6801', 'nonce': 3377683473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6802', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6803', 'nonce': 3377683473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6806', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6807', 'nonce': 3377683473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6804', 'nonce': 3377683473}, {'type': 'v1', 'addr': '172.21.15.153:6805', 'nonce': 3377683473}]}, 'public_addr': '172.21.15.153:6801/3377683473', 'cluster_addr': '172.21.15.153:6803/3377683473', 'heartbeat_back_addr': '172.21.15.153:6807/3377683473', 'heartbeat_front_addr': '172.21.15.153:6805/3377683473', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'c3e8194b-2c20-4ffb-859d-00c33902b241', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 46, 'up_thru': 127, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6808', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6809', 'nonce': 357179117}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6810', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6811', 'nonce': 357179117}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6814', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6815', 'nonce': 357179117}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6812', 'nonce': 357179117}, {'type': 'v1', 'addr': '172.21.15.153:6813', 'nonce': 357179117}]}, 'public_addr': '172.21.15.153:6809/357179117', 'cluster_addr': '172.21.15.153:6811/357179117', 'heartbeat_back_addr': '172.21.15.153:6815/357179117', 'heartbeat_front_addr': '172.21.15.153:6813/357179117', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98027d53-4324-44d1-89ce-8da85be58efb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6816', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6817', 'nonce': 1552538208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6818', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6819', 'nonce': 1552538208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6822', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6823', 'nonce': 1552538208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6820', 'nonce': 1552538208}, {'type': 'v1', 'addr': '172.21.15.153:6821', 'nonce': 1552538208}]}, 'public_addr': '172.21.15.153:6817/1552538208', 'cluster_addr': '172.21.15.153:6819/1552538208', 'heartbeat_back_addr': '172.21.15.153:6823/1552538208', 'heartbeat_front_addr': '172.21.15.153:6821/1552538208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '6b47d95a-1228-4fb1-a315-5371d9b42bbd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 280, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6824', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6825', 'nonce': 2479193924}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6826', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6827', 'nonce': 2479193924}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6830', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6831', 'nonce': 2479193924}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6828', 'nonce': 2479193924}, {'type': 'v1', 'addr': '172.21.15.153:6829', 'nonce': 2479193924}]}, 'public_addr': '172.21.15.153:6825/2479193924', 'cluster_addr': '172.21.15.153:6827/2479193924', 'heartbeat_back_addr': '172.21.15.153:6831/2479193924', 'heartbeat_front_addr': '172.21.15.153:6829/2479193924', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:39:53.691554+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:40:48.619696+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:39:52.850707+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:40:20.241925+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:40:54.448608+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:40:12.101131+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:40:40.927101+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-06T03:40:15.354513+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.12', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.4', 'mappings': [{'from': 5, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/283275195': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2738019433': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1268273764': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/567103210': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/2891059051': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2170219983': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:6800/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/4082841962': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6800/1263219005': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/570623774': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:6801/889382629': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:0/1379780741': '2024-04-07T03:21:47.915758+0000', '172.21.15.50:0/2880368026': '2024-04-07T03:10:58.614019+0000', '172.21.15.50:6801/132418263': '2024-04-07T03:11:33.442338+0000', '172.21.15.50:0/2044358308': '2024-04-07T03:21:47.915758+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-06T03:41:12.352 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-06T03:41:12.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:12.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:12.615 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:12.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:12.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:12.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:12.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:12.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:12.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:13.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:13.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:13.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:13.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.007395+0000 osd.0 (osd.0) 483 : cluster 0 6.9 scrub ok 2024-04-06T03:41:13.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.045854+0000 osd.4 (osd.4) 473 : cluster 0 6.1d scrub starts 2024-04-06T03:41:13.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.070402+0000 osd.2 (osd.2) 391 : cluster 0 3.0 scrub starts 2024-04-06T03:41:13.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.093682+0000 osd.2 (osd.2) 392 : cluster 0 3.0 scrub ok 2024-04-06T03:41:13.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.156235+0000 osd.4 (osd.4) 474 : cluster 0 6.1d scrub ok 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.402963+0000 osd.3 (osd.3) 494 : cluster 0 3.b scrub starts 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.410940+0000 osd.3 (osd.3) 495 : cluster 0 3.b scrub ok 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[21467]: cluster 2024-04-06T03:41:12.425279+0000 mgr.y (mgr.24368) 746 : cluster 0 pgmap v795: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 843 op/s 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.007395+0000 osd.0 (osd.0) 483 : cluster 0 6.9 scrub ok 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.045854+0000 osd.4 (osd.4) 473 : cluster 0 6.1d scrub starts 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.070402+0000 osd.2 (osd.2) 391 : cluster 0 3.0 scrub starts 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.093682+0000 osd.2 (osd.2) 392 : cluster 0 3.0 scrub ok 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.156235+0000 osd.4 (osd.4) 474 : cluster 0 6.1d scrub ok 2024-04-06T03:41:13.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.402963+0000 osd.3 (osd.3) 494 : cluster 0 3.b scrub starts 2024-04-06T03:41:13.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.410940+0000 osd.3 (osd.3) 495 : cluster 0 3.b scrub ok 2024-04-06T03:41:13.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:12 smithi050 bash[29221]: cluster 2024-04-06T03:41:12.425279+0000 mgr.y (mgr.24368) 746 : cluster 0 pgmap v795: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 843 op/s 2024-04-06T03:41:13.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:13.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:13.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.007395+0000 osd.0 (osd.0) 483 : cluster 0 6.9 scrub ok 2024-04-06T03:41:13.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.045854+0000 osd.4 (osd.4) 473 : cluster 0 6.1d scrub starts 2024-04-06T03:41:13.291 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.070402+0000 osd.2 (osd.2) 391 : cluster 0 3.0 scrub starts 2024-04-06T03:41:13.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.093682+0000 osd.2 (osd.2) 392 : cluster 0 3.0 scrub ok 2024-04-06T03:41:13.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.156235+0000 osd.4 (osd.4) 474 : cluster 0 6.1d scrub ok 2024-04-06T03:41:13.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.402963+0000 osd.3 (osd.3) 494 : cluster 0 3.b scrub starts 2024-04-06T03:41:13.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.410940+0000 osd.3 (osd.3) 495 : cluster 0 3.b scrub ok 2024-04-06T03:41:13.292 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:12 smithi153 bash[23852]: cluster 2024-04-06T03:41:12.425279+0000 mgr.y (mgr.24368) 746 : cluster 0 pgmap v795: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 843 op/s 2024-04-06T03:41:13.323 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:13.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:13.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:13.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:13.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:13.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:13.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:13.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:13.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:13.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:13.970 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:13.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:14.072 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:14.102 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:14.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:14.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:14.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:14.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.086465+0000 osd.4 (osd.4) 475 : cluster 0 3.a scrub starts 2024-04-06T03:41:14.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.089747+0000 osd.4 (osd.4) 476 : cluster 0 3.a scrub ok 2024-04-06T03:41:14.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.272394+0000 osd.7 (osd.7) 503 : cluster 0 6.15 scrub starts 2024-04-06T03:41:14.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.329531+0000 osd.1 (osd.1) 387 : cluster 0 2.d scrub starts 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.330855+0000 osd.1 (osd.1) 388 : cluster 0 2.d scrub ok 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.914319+0000 osd.0 (osd.0) 484 : cluster 0 5.3 scrub starts 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.918540+0000 osd.0 (osd.0) 485 : cluster 0 5.3 scrub ok 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.086465+0000 osd.4 (osd.4) 475 : cluster 0 3.a scrub starts 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.089747+0000 osd.4 (osd.4) 476 : cluster 0 3.a scrub ok 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.272394+0000 osd.7 (osd.7) 503 : cluster 0 6.15 scrub starts 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.329531+0000 osd.1 (osd.1) 387 : cluster 0 2.d scrub starts 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.330855+0000 osd.1 (osd.1) 388 : cluster 0 2.d scrub ok 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.914319+0000 osd.0 (osd.0) 484 : cluster 0 5.3 scrub starts 2024-04-06T03:41:14.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:13 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.918540+0000 osd.0 (osd.0) 485 : cluster 0 5.3 scrub ok 2024-04-06T03:41:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.086465+0000 osd.4 (osd.4) 475 : cluster 0 3.a scrub starts 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.089747+0000 osd.4 (osd.4) 476 : cluster 0 3.a scrub ok 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.272394+0000 osd.7 (osd.7) 503 : cluster 0 6.15 scrub starts 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.329531+0000 osd.1 (osd.1) 387 : cluster 0 2.d scrub starts 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.330855+0000 osd.1 (osd.1) 388 : cluster 0 2.d scrub ok 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.914319+0000 osd.0 (osd.0) 484 : cluster 0 5.3 scrub starts 2024-04-06T03:41:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:13 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.918540+0000 osd.0 (osd.0) 485 : cluster 0 5.3 scrub ok 2024-04-06T03:41:14.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:14.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:14.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:41:14.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:14.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:14.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:14.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:14.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:14.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:14.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:14.936 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:14.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:15.040 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:15.063 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:15.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:15.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[29221]: cluster 2024-04-06T03:41:13.328784+0000 osd.7 (osd.7) 504 : cluster 0 6.15 scrub ok 2024-04-06T03:41:15.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[29221]: cluster 2024-04-06T03:41:14.276866+0000 osd.7 (osd.7) 505 : cluster 0 2.0 scrub starts 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[29221]: cluster 2024-04-06T03:41:14.282167+0000 osd.7 (osd.7) 506 : cluster 0 2.0 scrub ok 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[29221]: cluster 2024-04-06T03:41:14.356501+0000 osd.3 (osd.3) 496 : cluster 0 10.f scrub starts 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[29221]: cluster 2024-04-06T03:41:14.425879+0000 mgr.y (mgr.24368) 747 : cluster 0 pgmap v796: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 799 op/s 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[21467]: cluster 2024-04-06T03:41:13.328784+0000 osd.7 (osd.7) 504 : cluster 0 6.15 scrub ok 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[21467]: cluster 2024-04-06T03:41:14.276866+0000 osd.7 (osd.7) 505 : cluster 0 2.0 scrub starts 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[21467]: cluster 2024-04-06T03:41:14.282167+0000 osd.7 (osd.7) 506 : cluster 0 2.0 scrub ok 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[21467]: cluster 2024-04-06T03:41:14.356501+0000 osd.3 (osd.3) 496 : cluster 0 10.f scrub starts 2024-04-06T03:41:15.263 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:15 smithi050 bash[21467]: cluster 2024-04-06T03:41:14.425879+0000 mgr.y (mgr.24368) 747 : cluster 0 pgmap v796: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 799 op/s 2024-04-06T03:41:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:15.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:15 smithi153 bash[23852]: cluster 2024-04-06T03:41:13.328784+0000 osd.7 (osd.7) 504 : cluster 0 6.15 scrub ok 2024-04-06T03:41:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:15 smithi153 bash[23852]: cluster 2024-04-06T03:41:14.276866+0000 osd.7 (osd.7) 505 : cluster 0 2.0 scrub starts 2024-04-06T03:41:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:15 smithi153 bash[23852]: cluster 2024-04-06T03:41:14.282167+0000 osd.7 (osd.7) 506 : cluster 0 2.0 scrub ok 2024-04-06T03:41:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:15 smithi153 bash[23852]: cluster 2024-04-06T03:41:14.356501+0000 osd.3 (osd.3) 496 : cluster 0 10.f scrub starts 2024-04-06T03:41:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:15 smithi153 bash[23852]: cluster 2024-04-06T03:41:14.425879+0000 mgr.y (mgr.24368) 747 : cluster 0 pgmap v796: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 799 op/s 2024-04-06T03:41:15.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:15.392 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:15.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:15.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:15.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:15.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:15.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:15.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:15.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:15.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:15.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:16.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:16.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:16.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:16.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[21467]: cluster 2024-04-06T03:41:15.073633+0000 osd.2 (osd.2) 393 : cluster 0 6.14 scrub starts 2024-04-06T03:41:16.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[21467]: cluster 2024-04-06T03:41:15.074789+0000 osd.2 (osd.2) 394 : cluster 0 6.14 scrub ok 2024-04-06T03:41:16.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[21467]: cluster 2024-04-06T03:41:15.466852+0000 osd.2 (osd.2) 395 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:16.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[21467]: cluster 2024-04-06T03:41:15.466967+0000 osd.2 (osd.2) 396 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:16.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[29221]: cluster 2024-04-06T03:41:15.073633+0000 osd.2 (osd.2) 393 : cluster 0 6.14 scrub starts 2024-04-06T03:41:16.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[29221]: cluster 2024-04-06T03:41:15.074789+0000 osd.2 (osd.2) 394 : cluster 0 6.14 scrub ok 2024-04-06T03:41:16.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[29221]: cluster 2024-04-06T03:41:15.466852+0000 osd.2 (osd.2) 395 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:16.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:16 smithi050 bash[29221]: cluster 2024-04-06T03:41:15.466967+0000 osd.2 (osd.2) 396 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:16.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:16 smithi153 bash[23852]: cluster 2024-04-06T03:41:15.073633+0000 osd.2 (osd.2) 393 : cluster 0 6.14 scrub starts 2024-04-06T03:41:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:16 smithi153 bash[23852]: cluster 2024-04-06T03:41:15.074789+0000 osd.2 (osd.2) 394 : cluster 0 6.14 scrub ok 2024-04-06T03:41:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:16 smithi153 bash[23852]: cluster 2024-04-06T03:41:15.466852+0000 osd.2 (osd.2) 395 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:16 smithi153 bash[23852]: cluster 2024-04-06T03:41:15.466967+0000 osd.2 (osd.2) 396 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:16.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:16.371 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:16.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:16.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:16.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:16.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:16.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:16.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:16.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:16.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:17.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:17.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:17.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:17.164 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:17.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:17.277 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:17 smithi153 bash[23852]: cluster 2024-04-06T03:41:16.022239+0000 mon.a (mon.0) 1555 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-04-06T03:41:17.277 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:17 smithi153 bash[23852]: cluster 2024-04-06T03:41:16.426221+0000 mgr.y (mgr.24368) 748 : cluster 0 pgmap v798: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 718 op/s 2024-04-06T03:41:17.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:17 smithi153 bash[23852]: cluster 2024-04-06T03:41:16.509495+0000 osd.3 (osd.3) 497 : cluster 0 10.f scrub ok 2024-04-06T03:41:17.290 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:17.311 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:41:17.350 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:41:17.352 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:41:17.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:41:17.353 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:41:17.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd out 6 2024-04-06T03:41:17.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:17 smithi050 bash[21467]: cluster 2024-04-06T03:41:16.022239+0000 mon.a (mon.0) 1555 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-04-06T03:41:17.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:17 smithi050 bash[21467]: cluster 2024-04-06T03:41:16.426221+0000 mgr.y (mgr.24368) 748 : cluster 0 pgmap v798: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 718 op/s 2024-04-06T03:41:17.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:17 smithi050 bash[21467]: cluster 2024-04-06T03:41:16.509495+0000 osd.3 (osd.3) 497 : cluster 0 10.f scrub ok 2024-04-06T03:41:17.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:17 smithi050 bash[29221]: cluster 2024-04-06T03:41:16.022239+0000 mon.a (mon.0) 1555 : cluster 0 osdmap e309: 8 total, 8 up, 8 in 2024-04-06T03:41:17.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:17 smithi050 bash[29221]: cluster 2024-04-06T03:41:16.426221+0000 mgr.y (mgr.24368) 748 : cluster 0 pgmap v798: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 718 op/s 2024-04-06T03:41:17.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:17 smithi050 bash[29221]: cluster 2024-04-06T03:41:16.509495+0000 osd.3 (osd.3) 497 : cluster 0 10.f scrub ok 2024-04-06T03:41:17.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:17.396 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:41:17.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:17.449 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:41:17.499 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:41:17.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:17.546 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:41:17.562 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:17.590 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:41:17.628 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:41:17.631 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:41:17.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:17.701 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:17.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:17.829 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:17.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:17.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:17.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:17.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:18.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:18.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:18.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:18.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:18.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[21467]: cluster 2024-04-06T03:41:17.067942+0000 osd.4 (osd.4) 477 : cluster 0 3.1a scrub starts 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[21467]: cluster 2024-04-06T03:41:17.072606+0000 osd.4 (osd.4) 478 : cluster 0 3.1a scrub ok 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[21467]: audit 2024-04-06T03:41:17.258421+0000 mon.c (mon.2) 97 : audit 0 from='client.? 172.21.15.50:0/1161636221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[21467]: cluster 2024-04-06T03:41:17.480577+0000 osd.3 (osd.3) 498 : cluster 0 3.18 scrub starts 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[21467]: cluster 2024-04-06T03:41:17.493548+0000 osd.3 (osd.3) 499 : cluster 0 3.18 scrub ok 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[29221]: cluster 2024-04-06T03:41:17.067942+0000 osd.4 (osd.4) 477 : cluster 0 3.1a scrub starts 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[29221]: cluster 2024-04-06T03:41:17.072606+0000 osd.4 (osd.4) 478 : cluster 0 3.1a scrub ok 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[29221]: audit 2024-04-06T03:41:17.258421+0000 mon.c (mon.2) 97 : audit 0 from='client.? 172.21.15.50:0/1161636221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[29221]: cluster 2024-04-06T03:41:17.480577+0000 osd.3 (osd.3) 498 : cluster 0 3.18 scrub starts 2024-04-06T03:41:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:18 smithi050 bash[29221]: cluster 2024-04-06T03:41:17.493548+0000 osd.3 (osd.3) 499 : cluster 0 3.18 scrub ok 2024-04-06T03:41:18.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:18 smithi153 bash[23852]: cluster 2024-04-06T03:41:17.067942+0000 osd.4 (osd.4) 477 : cluster 0 3.1a scrub starts 2024-04-06T03:41:18.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:18 smithi153 bash[23852]: cluster 2024-04-06T03:41:17.072606+0000 osd.4 (osd.4) 478 : cluster 0 3.1a scrub ok 2024-04-06T03:41:18.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:18 smithi153 bash[23852]: audit 2024-04-06T03:41:17.258421+0000 mon.c (mon.2) 97 : audit 0 from='client.? 172.21.15.50:0/1161636221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:41:18.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:18 smithi153 bash[23852]: cluster 2024-04-06T03:41:17.480577+0000 osd.3 (osd.3) 498 : cluster 0 3.18 scrub starts 2024-04-06T03:41:18.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:18 smithi153 bash[23852]: cluster 2024-04-06T03:41:17.493548+0000 osd.3 (osd.3) 499 : cluster 0 3.18 scrub ok 2024-04-06T03:41:18.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:18.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:18.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:18.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:18.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:18.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:18.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:18.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:18.855 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:18.878 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:18.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:18.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:19.006 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:19.109 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:19.133 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:19.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:19.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:19.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:19 smithi153 bash[23852]: cluster 2024-04-06T03:41:18.050038+0000 osd.2 (osd.2) 397 : cluster 0 3.4 scrub starts 2024-04-06T03:41:19.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:19 smithi153 bash[23852]: cluster 2024-04-06T03:41:18.055645+0000 osd.2 (osd.2) 398 : cluster 0 3.4 scrub ok 2024-04-06T03:41:19.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:19 smithi153 bash[23852]: cluster 2024-04-06T03:41:18.426768+0000 mgr.y (mgr.24368) 749 : cluster 0 pgmap v799: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 782 op/s 2024-04-06T03:41:19.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:19 smithi153 bash[23852]: cluster 2024-04-06T03:41:18.430102+0000 osd.3 (osd.3) 500 : cluster 0 5.0 deep-scrub starts 2024-04-06T03:41:19.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:19 smithi153 bash[23852]: cluster 2024-04-06T03:41:18.436217+0000 osd.3 (osd.3) 501 : cluster 0 5.0 deep-scrub ok 2024-04-06T03:41:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:19.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[21467]: cluster 2024-04-06T03:41:18.050038+0000 osd.2 (osd.2) 397 : cluster 0 3.4 scrub starts 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[21467]: cluster 2024-04-06T03:41:18.055645+0000 osd.2 (osd.2) 398 : cluster 0 3.4 scrub ok 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[21467]: cluster 2024-04-06T03:41:18.426768+0000 mgr.y (mgr.24368) 749 : cluster 0 pgmap v799: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 782 op/s 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[21467]: cluster 2024-04-06T03:41:18.430102+0000 osd.3 (osd.3) 500 : cluster 0 5.0 deep-scrub starts 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[21467]: cluster 2024-04-06T03:41:18.436217+0000 osd.3 (osd.3) 501 : cluster 0 5.0 deep-scrub ok 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[29221]: cluster 2024-04-06T03:41:18.050038+0000 osd.2 (osd.2) 397 : cluster 0 3.4 scrub starts 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[29221]: cluster 2024-04-06T03:41:18.055645+0000 osd.2 (osd.2) 398 : cluster 0 3.4 scrub ok 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[29221]: cluster 2024-04-06T03:41:18.426768+0000 mgr.y (mgr.24368) 749 : cluster 0 pgmap v799: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 782 op/s 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[29221]: cluster 2024-04-06T03:41:18.430102+0000 osd.3 (osd.3) 500 : cluster 0 5.0 deep-scrub starts 2024-04-06T03:41:19.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[29221]: cluster 2024-04-06T03:41:18.436217+0000 osd.3 (osd.3) 501 : cluster 0 5.0 deep-scrub ok 2024-04-06T03:41:19.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:19.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:19.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:19.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:19.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:19.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:41:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:41:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:41:19.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:19.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:19.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:19.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:19.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:19.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:20.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:20.039 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:20.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:20.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:20.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:20 smithi050 bash[21467]: cluster 2024-04-06T03:41:19.397763+0000 osd.3 (osd.3) 502 : cluster 0 6.b scrub starts 2024-04-06T03:41:20.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:20 smithi050 bash[29221]: cluster 2024-04-06T03:41:19.397763+0000 osd.3 (osd.3) 502 : cluster 0 6.b scrub starts 2024-04-06T03:41:20.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:20.281 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:20 smithi153 bash[23852]: cluster 2024-04-06T03:41:19.397763+0000 osd.3 (osd.3) 502 : cluster 0 6.b scrub starts 2024-04-06T03:41:20.282 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:41:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:41:20.302 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:20.370 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:20.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:20.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:20.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:20.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:20.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:20.700 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:20.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:20.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:20.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:20.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:21.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:21.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:21.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:21.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:21.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:21.365 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:21.391 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: cluster 2024-04-06T03:41:19.576498+0000 osd.3 (osd.3) 503 : cluster 0 6.b scrub ok 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: audit 2024-04-06T03:41:20.024839+0000 mgr.y (mgr.24368) 750 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: audit 2024-04-06T03:41:20.112818+0000 mon.a (mon.0) 1556 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: audit 2024-04-06T03:41:20.114195+0000 mon.a (mon.0) 1557 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: cluster 2024-04-06T03:41:20.199798+0000 osd.2 (osd.2) 399 : cluster 0 5.a scrub starts 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: cluster 2024-04-06T03:41:20.205335+0000 osd.2 (osd.2) 400 : cluster 0 5.a scrub ok 2024-04-06T03:41:21.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[21467]: cluster 2024-04-06T03:41:20.427456+0000 mgr.y (mgr.24368) 751 : cluster 0 pgmap v800: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 829 op/s 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: cluster 2024-04-06T03:41:19.576498+0000 osd.3 (osd.3) 503 : cluster 0 6.b scrub ok 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: audit 2024-04-06T03:41:20.024839+0000 mgr.y (mgr.24368) 750 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: audit 2024-04-06T03:41:20.112818+0000 mon.a (mon.0) 1556 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: audit 2024-04-06T03:41:20.114195+0000 mon.a (mon.0) 1557 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: cluster 2024-04-06T03:41:20.199798+0000 osd.2 (osd.2) 399 : cluster 0 5.a scrub starts 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: cluster 2024-04-06T03:41:20.205335+0000 osd.2 (osd.2) 400 : cluster 0 5.a scrub ok 2024-04-06T03:41:21.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:21 smithi050 bash[29221]: cluster 2024-04-06T03:41:20.427456+0000 mgr.y (mgr.24368) 751 : cluster 0 pgmap v800: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 829 op/s 2024-04-06T03:41:21.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:21.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:21.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: cluster 2024-04-06T03:41:19.576498+0000 osd.3 (osd.3) 503 : cluster 0 6.b scrub ok 2024-04-06T03:41:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: audit 2024-04-06T03:41:20.024839+0000 mgr.y (mgr.24368) 750 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: audit 2024-04-06T03:41:20.112818+0000 mon.a (mon.0) 1556 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: audit 2024-04-06T03:41:20.114195+0000 mon.a (mon.0) 1557 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: cluster 2024-04-06T03:41:20.199798+0000 osd.2 (osd.2) 399 : cluster 0 5.a scrub starts 2024-04-06T03:41:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: cluster 2024-04-06T03:41:20.205335+0000 osd.2 (osd.2) 400 : cluster 0 5.a scrub ok 2024-04-06T03:41:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:21 smithi153 bash[23852]: cluster 2024-04-06T03:41:20.427456+0000 mgr.y (mgr.24368) 751 : cluster 0 pgmap v800: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 829 op/s 2024-04-06T03:41:21.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:21.613 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:21.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:21.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:21.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:21.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:21.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:21.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:21.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:41:21.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:22.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:22.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:22.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:22.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:22.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:22.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:22 smithi153 bash[23852]: cluster 2024-04-06T03:41:21.001287+0000 osd.0 (osd.0) 486 : cluster 0 5.15 deep-scrub starts 2024-04-06T03:41:22.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:22 smithi153 bash[23852]: cluster 2024-04-06T03:41:21.025387+0000 osd.0 (osd.0) 487 : cluster 0 5.15 deep-scrub ok 2024-04-06T03:41:22.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:22 smithi153 bash[23852]: cluster 2024-04-06T03:41:21.397787+0000 osd.3 (osd.3) 504 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:41:22.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:22 smithi153 bash[23852]: cluster 2024-04-06T03:41:21.406609+0000 osd.3 (osd.3) 505 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:41:22.269 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:22.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:22.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:22.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[21467]: cluster 2024-04-06T03:41:21.001287+0000 osd.0 (osd.0) 486 : cluster 0 5.15 deep-scrub starts 2024-04-06T03:41:22.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[21467]: cluster 2024-04-06T03:41:21.025387+0000 osd.0 (osd.0) 487 : cluster 0 5.15 deep-scrub ok 2024-04-06T03:41:22.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[21467]: cluster 2024-04-06T03:41:21.397787+0000 osd.3 (osd.3) 504 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:41:22.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[21467]: cluster 2024-04-06T03:41:21.406609+0000 osd.3 (osd.3) 505 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:41:22.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[29221]: cluster 2024-04-06T03:41:21.001287+0000 osd.0 (osd.0) 486 : cluster 0 5.15 deep-scrub starts 2024-04-06T03:41:22.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[29221]: cluster 2024-04-06T03:41:21.025387+0000 osd.0 (osd.0) 487 : cluster 0 5.15 deep-scrub ok 2024-04-06T03:41:22.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[29221]: cluster 2024-04-06T03:41:21.397787+0000 osd.3 (osd.3) 504 : cluster 0 2.12 deep-scrub starts 2024-04-06T03:41:22.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:22 smithi050 bash[29221]: cluster 2024-04-06T03:41:21.406609+0000 osd.3 (osd.3) 505 : cluster 0 2.12 deep-scrub ok 2024-04-06T03:41:22.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:22.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:22.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:22.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:22.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:22.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:22.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:22.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:22.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:23.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:23.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:23.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:23.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:23.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:23.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.030774+0000 osd.0 (osd.0) 488 : cluster 0 6.d scrub starts 2024-04-06T03:41:23.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.050592+0000 osd.0 (osd.0) 489 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:23.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.050654+0000 osd.0 (osd.0) 490 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:23.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.123773+0000 mon.a (mon.0) 1558 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.155706+0000 osd.0 (osd.0) 491 : cluster 0 6.d scrub ok 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.204206+0000 osd.1 (osd.1) 389 : cluster 0 5.12 scrub starts 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.205305+0000 osd.1 (osd.1) 390 : cluster 0 5.12 scrub ok 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.427993+0000 mgr.y (mgr.24368) 752 : cluster 0 pgmap v802: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 870 op/s 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.030774+0000 osd.0 (osd.0) 488 : cluster 0 6.d scrub starts 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.050592+0000 osd.0 (osd.0) 489 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.050654+0000 osd.0 (osd.0) 490 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.123773+0000 mon.a (mon.0) 1558 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.155706+0000 osd.0 (osd.0) 491 : cluster 0 6.d scrub ok 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.204206+0000 osd.1 (osd.1) 389 : cluster 0 5.12 scrub starts 2024-04-06T03:41:23.314 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.205305+0000 osd.1 (osd.1) 390 : cluster 0 5.12 scrub ok 2024-04-06T03:41:23.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:23 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.427993+0000 mgr.y (mgr.24368) 752 : cluster 0 pgmap v802: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 870 op/s 2024-04-06T03:41:23.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:23.421 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:23.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:23.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:23.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.030774+0000 osd.0 (osd.0) 488 : cluster 0 6.d scrub starts 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.050592+0000 osd.0 (osd.0) 489 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.050654+0000 osd.0 (osd.0) 490 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.123773+0000 mon.a (mon.0) 1558 : cluster 0 osdmap e310: 8 total, 8 up, 8 in 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.155706+0000 osd.0 (osd.0) 491 : cluster 0 6.d scrub ok 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.204206+0000 osd.1 (osd.1) 389 : cluster 0 5.12 scrub starts 2024-04-06T03:41:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.205305+0000 osd.1 (osd.1) 390 : cluster 0 5.12 scrub ok 2024-04-06T03:41:23.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:23 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.427993+0000 mgr.y (mgr.24368) 752 : cluster 0 pgmap v802: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 870 op/s 2024-04-06T03:41:23.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.576 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:23.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.704 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:23.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:23.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.954 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:23.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:24.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:24.082 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:24.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:24.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:24.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:24.355 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:24.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:24 smithi153 bash[23852]: cluster 2024-04-06T03:41:22.365243+0000 osd.7 (osd.7) 507 : cluster 0 10.c scrub starts 2024-04-06T03:41:24.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:24 smithi153 bash[23852]: cluster 2024-04-06T03:41:23.022487+0000 osd.0 (osd.0) 492 : cluster 0 2.4 scrub starts 2024-04-06T03:41:24.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:24 smithi153 bash[23852]: cluster 2024-04-06T03:41:23.041901+0000 osd.0 (osd.0) 493 : cluster 0 2.4 scrub ok 2024-04-06T03:41:24.358 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:24 smithi153 bash[23852]: cluster 2024-04-06T03:41:23.277464+0000 osd.7 (osd.7) 508 : cluster 0 10.c scrub ok 2024-04-06T03:41:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:24.460 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:24.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[21467]: cluster 2024-04-06T03:41:22.365243+0000 osd.7 (osd.7) 507 : cluster 0 10.c scrub starts 2024-04-06T03:41:24.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[21467]: cluster 2024-04-06T03:41:23.022487+0000 osd.0 (osd.0) 492 : cluster 0 2.4 scrub starts 2024-04-06T03:41:24.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[21467]: cluster 2024-04-06T03:41:23.041901+0000 osd.0 (osd.0) 493 : cluster 0 2.4 scrub ok 2024-04-06T03:41:24.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[21467]: cluster 2024-04-06T03:41:23.277464+0000 osd.7 (osd.7) 508 : cluster 0 10.c scrub ok 2024-04-06T03:41:24.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[29221]: cluster 2024-04-06T03:41:22.365243+0000 osd.7 (osd.7) 507 : cluster 0 10.c scrub starts 2024-04-06T03:41:24.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[29221]: cluster 2024-04-06T03:41:23.022487+0000 osd.0 (osd.0) 492 : cluster 0 2.4 scrub starts 2024-04-06T03:41:24.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[29221]: cluster 2024-04-06T03:41:23.041901+0000 osd.0 (osd.0) 493 : cluster 0 2.4 scrub ok 2024-04-06T03:41:24.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:24 smithi050 bash[29221]: cluster 2024-04-06T03:41:23.277464+0000 osd.7 (osd.7) 508 : cluster 0 10.c scrub ok 2024-04-06T03:41:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.492 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:24.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:24.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:24.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:24.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:24.746 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:24.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:24.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:24.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:25.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:25.142 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:25.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:25.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:25 smithi153 bash[23852]: cluster 2024-04-06T03:41:23.529778+0000 osd.2 (osd.2) 401 : cluster 0 5.8 scrub starts 2024-04-06T03:41:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:25 smithi153 bash[23852]: cluster 2024-04-06T03:41:23.534311+0000 osd.2 (osd.2) 402 : cluster 0 5.8 scrub ok 2024-04-06T03:41:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:25 smithi153 bash[23852]: cluster 2024-04-06T03:41:24.225310+0000 osd.4 (osd.4) 479 : cluster 0 5.c scrub starts 2024-04-06T03:41:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:25 smithi153 bash[23852]: cluster 2024-04-06T03:41:24.231963+0000 osd.4 (osd.4) 480 : cluster 0 5.c scrub ok 2024-04-06T03:41:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:25 smithi153 bash[23852]: cluster 2024-04-06T03:41:24.428650+0000 mgr.y (mgr.24368) 753 : cluster 0 pgmap v803: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 881 op/s 2024-04-06T03:41:25.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:25.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:25.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[21467]: cluster 2024-04-06T03:41:23.529778+0000 osd.2 (osd.2) 401 : cluster 0 5.8 scrub starts 2024-04-06T03:41:25.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[21467]: cluster 2024-04-06T03:41:23.534311+0000 osd.2 (osd.2) 402 : cluster 0 5.8 scrub ok 2024-04-06T03:41:25.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[21467]: cluster 2024-04-06T03:41:24.225310+0000 osd.4 (osd.4) 479 : cluster 0 5.c scrub starts 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[21467]: cluster 2024-04-06T03:41:24.231963+0000 osd.4 (osd.4) 480 : cluster 0 5.c scrub ok 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[21467]: cluster 2024-04-06T03:41:24.428650+0000 mgr.y (mgr.24368) 753 : cluster 0 pgmap v803: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 881 op/s 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[29221]: cluster 2024-04-06T03:41:23.529778+0000 osd.2 (osd.2) 401 : cluster 0 5.8 scrub starts 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[29221]: cluster 2024-04-06T03:41:23.534311+0000 osd.2 (osd.2) 402 : cluster 0 5.8 scrub ok 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[29221]: cluster 2024-04-06T03:41:24.225310+0000 osd.4 (osd.4) 479 : cluster 0 5.c scrub starts 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[29221]: cluster 2024-04-06T03:41:24.231963+0000 osd.4 (osd.4) 480 : cluster 0 5.c scrub ok 2024-04-06T03:41:25.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:25 smithi050 bash[29221]: cluster 2024-04-06T03:41:24.428650+0000 mgr.y (mgr.24368) 753 : cluster 0 pgmap v803: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 881 op/s 2024-04-06T03:41:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:25.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:25.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:25.646 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:25.666 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:25.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:25.802 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:25.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:25.937 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:25.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:26.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:26.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:26.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:26.218 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:26.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:26 smithi153 bash[23852]: cluster 2024-04-06T03:41:25.049429+0000 osd.0 (osd.0) 494 : cluster 0 3.12 scrub starts 2024-04-06T03:41:26.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:26 smithi153 bash[23852]: cluster 2024-04-06T03:41:25.052380+0000 osd.0 (osd.0) 495 : cluster 0 3.12 scrub ok 2024-04-06T03:41:26.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:26 smithi153 bash[23852]: cluster 2024-04-06T03:41:25.171554+0000 osd.4 (osd.4) 481 : cluster 0 2.1a scrub starts 2024-04-06T03:41:26.233 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:26 smithi153 bash[23852]: cluster 2024-04-06T03:41:25.172439+0000 osd.4 (osd.4) 482 : cluster 0 2.1a scrub ok 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[21467]: cluster 2024-04-06T03:41:25.049429+0000 osd.0 (osd.0) 494 : cluster 0 3.12 scrub starts 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[21467]: cluster 2024-04-06T03:41:25.052380+0000 osd.0 (osd.0) 495 : cluster 0 3.12 scrub ok 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[21467]: cluster 2024-04-06T03:41:25.171554+0000 osd.4 (osd.4) 481 : cluster 0 2.1a scrub starts 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[21467]: cluster 2024-04-06T03:41:25.172439+0000 osd.4 (osd.4) 482 : cluster 0 2.1a scrub ok 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[29221]: cluster 2024-04-06T03:41:25.049429+0000 osd.0 (osd.0) 494 : cluster 0 3.12 scrub starts 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[29221]: cluster 2024-04-06T03:41:25.052380+0000 osd.0 (osd.0) 495 : cluster 0 3.12 scrub ok 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[29221]: cluster 2024-04-06T03:41:25.171554+0000 osd.4 (osd.4) 481 : cluster 0 2.1a scrub starts 2024-04-06T03:41:26.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:26 smithi050 bash[29221]: cluster 2024-04-06T03:41:25.172439+0000 osd.4 (osd.4) 482 : cluster 0 2.1a scrub ok 2024-04-06T03:41:26.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:26.358 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:26.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:26.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:26.506 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:26.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:26.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:26.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:26.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:26.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:26.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:26.898 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:26.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:26.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:27.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:27.159 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.072456+0000 osd.0 (osd.0) 496 : cluster 0 5.1b scrub starts 2024-04-06T03:41:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.082943+0000 osd.0 (osd.0) 497 : cluster 0 5.1b scrub ok 2024-04-06T03:41:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.131639+0000 osd.4 (osd.4) 483 : cluster 0 3.1d scrub starts 2024-04-06T03:41:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.135892+0000 osd.4 (osd.4) 484 : cluster 0 3.1d scrub ok 2024-04-06T03:41:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.228872+0000 osd.7 (osd.7) 509 : cluster 0 3.d deep-scrub starts 2024-04-06T03:41:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.233363+0000 osd.7 (osd.7) 510 : cluster 0 3.d deep-scrub ok 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.429218+0000 mgr.y (mgr.24368) 754 : cluster 0 pgmap v804: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 853 op/s 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.466226+0000 osd.3 (osd.3) 506 : cluster 0 3.2 deep-scrub starts 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[21467]: cluster 2024-04-06T03:41:26.470089+0000 osd.3 (osd.3) 507 : cluster 0 3.2 deep-scrub ok 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.072456+0000 osd.0 (osd.0) 496 : cluster 0 5.1b scrub starts 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.082943+0000 osd.0 (osd.0) 497 : cluster 0 5.1b scrub ok 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.131639+0000 osd.4 (osd.4) 483 : cluster 0 3.1d scrub starts 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.135892+0000 osd.4 (osd.4) 484 : cluster 0 3.1d scrub ok 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.228872+0000 osd.7 (osd.7) 509 : cluster 0 3.d deep-scrub starts 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.233363+0000 osd.7 (osd.7) 510 : cluster 0 3.d deep-scrub ok 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.429218+0000 mgr.y (mgr.24368) 754 : cluster 0 pgmap v804: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 853 op/s 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.466226+0000 osd.3 (osd.3) 506 : cluster 0 3.2 deep-scrub starts 2024-04-06T03:41:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:27 smithi050 bash[29221]: cluster 2024-04-06T03:41:26.470089+0000 osd.3 (osd.3) 507 : cluster 0 3.2 deep-scrub ok 2024-04-06T03:41:27.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:27.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:27.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:27.409 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.072456+0000 osd.0 (osd.0) 496 : cluster 0 5.1b scrub starts 2024-04-06T03:41:27.409 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.082943+0000 osd.0 (osd.0) 497 : cluster 0 5.1b scrub ok 2024-04-06T03:41:27.409 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.131639+0000 osd.4 (osd.4) 483 : cluster 0 3.1d scrub starts 2024-04-06T03:41:27.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.135892+0000 osd.4 (osd.4) 484 : cluster 0 3.1d scrub ok 2024-04-06T03:41:27.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.228872+0000 osd.7 (osd.7) 509 : cluster 0 3.d deep-scrub starts 2024-04-06T03:41:27.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.233363+0000 osd.7 (osd.7) 510 : cluster 0 3.d deep-scrub ok 2024-04-06T03:41:27.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.429218+0000 mgr.y (mgr.24368) 754 : cluster 0 pgmap v804: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 853 op/s 2024-04-06T03:41:27.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.466226+0000 osd.3 (osd.3) 506 : cluster 0 3.2 deep-scrub starts 2024-04-06T03:41:27.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:27 smithi153 bash[23852]: cluster 2024-04-06T03:41:26.470089+0000 osd.3 (osd.3) 507 : cluster 0 3.2 deep-scrub ok 2024-04-06T03:41:27.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:27.529 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:27.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:27.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:27.781 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:27.807 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:27.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:27.941 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:27.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:28.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:28.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:28.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:28.216 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:28.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:28.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:28.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:28.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:28 smithi153 bash[23852]: cluster 2024-04-06T03:41:27.991655+0000 osd.0 (osd.0) 498 : cluster 0 5.5 scrub starts 2024-04-06T03:41:28.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:28.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:28 smithi050 bash[21467]: cluster 2024-04-06T03:41:27.991655+0000 osd.0 (osd.0) 498 : cluster 0 5.5 scrub starts 2024-04-06T03:41:28.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:28 smithi050 bash[29221]: cluster 2024-04-06T03:41:27.991655+0000 osd.0 (osd.0) 498 : cluster 0 5.5 scrub starts 2024-04-06T03:41:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:28.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:28.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:28.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:28.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:28.732 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:28.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:28.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:28.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:28.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:29.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:29.144 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:29.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:29.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:29 smithi050 bash[21467]: cluster 2024-04-06T03:41:27.996699+0000 osd.0 (osd.0) 499 : cluster 0 5.5 scrub ok 2024-04-06T03:41:29.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:29 smithi050 bash[21467]: cluster 2024-04-06T03:41:28.429755+0000 mgr.y (mgr.24368) 755 : cluster 0 pgmap v805: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 842 op/s 2024-04-06T03:41:29.257 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:29 smithi050 bash[29221]: cluster 2024-04-06T03:41:27.996699+0000 osd.0 (osd.0) 499 : cluster 0 5.5 scrub ok 2024-04-06T03:41:29.257 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:29 smithi050 bash[29221]: cluster 2024-04-06T03:41:28.429755+0000 mgr.y (mgr.24368) 755 : cluster 0 pgmap v805: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 842 op/s 2024-04-06T03:41:29.274 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:29.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:29.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:29 smithi153 bash[23852]: cluster 2024-04-06T03:41:27.996699+0000 osd.0 (osd.0) 499 : cluster 0 5.5 scrub ok 2024-04-06T03:41:29.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:29 smithi153 bash[23852]: cluster 2024-04-06T03:41:28.429755+0000 mgr.y (mgr.24368) 755 : cluster 0 pgmap v805: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 842 op/s 2024-04-06T03:41:29.413 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:29.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:29.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:29.552 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:29.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:29.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:29.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:41:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:41:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:41:29.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:29.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:29.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:29.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:29.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:30.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:30.070 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:30.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:30.196 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:30.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:41:30.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[23852]: cluster 2024-04-06T03:41:29.116128+0000 osd.7 (osd.7) 511 : cluster 0 6.1c deep-scrub starts 2024-04-06T03:41:30.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[23852]: cluster 2024-04-06T03:41:29.127633+0000 osd.7 (osd.7) 512 : cluster 0 6.1c deep-scrub ok 2024-04-06T03:41:30.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[23852]: cluster 2024-04-06T03:41:29.471898+0000 osd.3 (osd.3) 508 : cluster 0 3.7 scrub starts 2024-04-06T03:41:30.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[23852]: cluster 2024-04-06T03:41:29.483042+0000 osd.3 (osd.3) 509 : cluster 0 3.7 scrub ok 2024-04-06T03:41:30.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[23852]: cluster 2024-04-06T03:41:29.736751+0000 osd.2 (osd.2) 403 : cluster 0 2.1 scrub starts 2024-04-06T03:41:30.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:30 smithi153 bash[23852]: cluster 2024-04-06T03:41:29.749040+0000 osd.2 (osd.2) 404 : cluster 0 2.1 scrub ok 2024-04-06T03:41:30.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:30.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[21467]: cluster 2024-04-06T03:41:29.116128+0000 osd.7 (osd.7) 511 : cluster 0 6.1c deep-scrub starts 2024-04-06T03:41:30.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[21467]: cluster 2024-04-06T03:41:29.127633+0000 osd.7 (osd.7) 512 : cluster 0 6.1c deep-scrub ok 2024-04-06T03:41:30.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[21467]: cluster 2024-04-06T03:41:29.471898+0000 osd.3 (osd.3) 508 : cluster 0 3.7 scrub starts 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[21467]: cluster 2024-04-06T03:41:29.483042+0000 osd.3 (osd.3) 509 : cluster 0 3.7 scrub ok 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[21467]: cluster 2024-04-06T03:41:29.736751+0000 osd.2 (osd.2) 403 : cluster 0 2.1 scrub starts 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[21467]: cluster 2024-04-06T03:41:29.749040+0000 osd.2 (osd.2) 404 : cluster 0 2.1 scrub ok 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[29221]: cluster 2024-04-06T03:41:29.116128+0000 osd.7 (osd.7) 511 : cluster 0 6.1c deep-scrub starts 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[29221]: cluster 2024-04-06T03:41:29.127633+0000 osd.7 (osd.7) 512 : cluster 0 6.1c deep-scrub ok 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[29221]: cluster 2024-04-06T03:41:29.471898+0000 osd.3 (osd.3) 508 : cluster 0 3.7 scrub starts 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[29221]: cluster 2024-04-06T03:41:29.483042+0000 osd.3 (osd.3) 509 : cluster 0 3.7 scrub ok 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[29221]: cluster 2024-04-06T03:41:29.736751+0000 osd.2 (osd.2) 403 : cluster 0 2.1 scrub starts 2024-04-06T03:41:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:30 smithi050 bash[29221]: cluster 2024-04-06T03:41:29.749040+0000 osd.2 (osd.2) 404 : cluster 0 2.1 scrub ok 2024-04-06T03:41:30.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:30.431 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:30.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:30.568 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:30.587 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:30.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:30.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:30.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:30.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:30.845 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:30.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:30.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:30.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:31.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:31.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:31.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:31.237 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:31.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[29221]: audit 2024-04-06T03:41:30.032706+0000 mgr.y (mgr.24368) 756 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:31.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[29221]: cluster 2024-04-06T03:41:30.087272+0000 osd.4 (osd.4) 485 : cluster 0 5.18 scrub starts 2024-04-06T03:41:31.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[29221]: cluster 2024-04-06T03:41:30.088412+0000 osd.4 (osd.4) 486 : cluster 0 5.18 scrub ok 2024-04-06T03:41:31.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[29221]: cluster 2024-04-06T03:41:30.159935+0000 osd.7 (osd.7) 513 : cluster 0 5.1a scrub starts 2024-04-06T03:41:31.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[29221]: cluster 2024-04-06T03:41:30.162017+0000 osd.7 (osd.7) 514 : cluster 0 5.1a scrub ok 2024-04-06T03:41:31.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[29221]: cluster 2024-04-06T03:41:30.430663+0000 mgr.y (mgr.24368) 757 : cluster 0 pgmap v806: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:41:31.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[21467]: audit 2024-04-06T03:41:30.032706+0000 mgr.y (mgr.24368) 756 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:31.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[21467]: cluster 2024-04-06T03:41:30.087272+0000 osd.4 (osd.4) 485 : cluster 0 5.18 scrub starts 2024-04-06T03:41:31.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[21467]: cluster 2024-04-06T03:41:30.088412+0000 osd.4 (osd.4) 486 : cluster 0 5.18 scrub ok 2024-04-06T03:41:31.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[21467]: cluster 2024-04-06T03:41:30.159935+0000 osd.7 (osd.7) 513 : cluster 0 5.1a scrub starts 2024-04-06T03:41:31.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[21467]: cluster 2024-04-06T03:41:30.162017+0000 osd.7 (osd.7) 514 : cluster 0 5.1a scrub ok 2024-04-06T03:41:31.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:31 smithi050 bash[21467]: cluster 2024-04-06T03:41:30.430663+0000 mgr.y (mgr.24368) 757 : cluster 0 pgmap v806: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:41:31.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:31.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:31.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:31.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:31.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:31 smithi153 bash[23852]: audit 2024-04-06T03:41:30.032706+0000 mgr.y (mgr.24368) 756 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:31.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:31 smithi153 bash[23852]: cluster 2024-04-06T03:41:30.087272+0000 osd.4 (osd.4) 485 : cluster 0 5.18 scrub starts 2024-04-06T03:41:31.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:31 smithi153 bash[23852]: cluster 2024-04-06T03:41:30.088412+0000 osd.4 (osd.4) 486 : cluster 0 5.18 scrub ok 2024-04-06T03:41:31.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:31 smithi153 bash[23852]: cluster 2024-04-06T03:41:30.159935+0000 osd.7 (osd.7) 513 : cluster 0 5.1a scrub starts 2024-04-06T03:41:31.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:31 smithi153 bash[23852]: cluster 2024-04-06T03:41:30.162017+0000 osd.7 (osd.7) 514 : cluster 0 5.1a scrub ok 2024-04-06T03:41:31.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:31 smithi153 bash[23852]: cluster 2024-04-06T03:41:30.430663+0000 mgr.y (mgr.24368) 757 : cluster 0 pgmap v806: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:41:31.613 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:31.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:31.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:31.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:31.766 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:31.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:31.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:31.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:31.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:32.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:32.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:32.171 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:32.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.057953+0000 osd.4 (osd.4) 487 : cluster 0 2.11 scrub starts 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.058985+0000 osd.4 (osd.4) 488 : cluster 0 2.11 scrub ok 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.061028+0000 osd.5 (osd.5) 145 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.061080+0000 osd.5 (osd.5) 146 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.174954+0000 mon.a (mon.0) 1559 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.722708+0000 osd.2 (osd.2) 405 : cluster 0 2.3 scrub starts 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.732910+0000 osd.2 (osd.2) 406 : cluster 0 2.3 scrub ok 2024-04-06T03:41:32.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:32 smithi153 bash[23852]: cluster 2024-04-06T03:41:31.993350+0000 osd.0 (osd.0) 500 : cluster 0 6.a scrub starts 2024-04-06T03:41:32.314 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:32.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:32.440 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:32.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.057953+0000 osd.4 (osd.4) 487 : cluster 0 2.11 scrub starts 2024-04-06T03:41:32.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.058985+0000 osd.4 (osd.4) 488 : cluster 0 2.11 scrub ok 2024-04-06T03:41:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.061028+0000 osd.5 (osd.5) 145 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.061080+0000 osd.5 (osd.5) 146 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.174954+0000 mon.a (mon.0) 1559 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-04-06T03:41:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.722708+0000 osd.2 (osd.2) 405 : cluster 0 2.3 scrub starts 2024-04-06T03:41:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.732910+0000 osd.2 (osd.2) 406 : cluster 0 2.3 scrub ok 2024-04-06T03:41:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[29221]: cluster 2024-04-06T03:41:31.993350+0000 osd.0 (osd.0) 500 : cluster 0 6.a scrub starts 2024-04-06T03:41:32.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.057953+0000 osd.4 (osd.4) 487 : cluster 0 2.11 scrub starts 2024-04-06T03:41:32.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.058985+0000 osd.4 (osd.4) 488 : cluster 0 2.11 scrub ok 2024-04-06T03:41:32.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.061028+0000 osd.5 (osd.5) 145 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:32.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.061080+0000 osd.5 (osd.5) 146 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:32.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.174954+0000 mon.a (mon.0) 1559 : cluster 0 osdmap e311: 8 total, 8 up, 8 in 2024-04-06T03:41:32.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.722708+0000 osd.2 (osd.2) 405 : cluster 0 2.3 scrub starts 2024-04-06T03:41:32.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.732910+0000 osd.2 (osd.2) 406 : cluster 0 2.3 scrub ok 2024-04-06T03:41:32.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:32 smithi050 bash[21467]: cluster 2024-04-06T03:41:31.993350+0000 osd.0 (osd.0) 500 : cluster 0 6.a scrub starts 2024-04-06T03:41:32.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:32.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:32.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:32.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:32.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:32.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:32.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:32.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:33.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:33.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:33.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:33.254 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:33.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:33.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:33.512 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:33.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:33.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.045768+0000 osd.4 (osd.4) 489 : cluster 0 2.18 scrub starts 2024-04-06T03:41:33.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.047099+0000 osd.4 (osd.4) 490 : cluster 0 2.18 scrub ok 2024-04-06T03:41:33.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.055613+0000 osd.0 (osd.0) 501 : cluster 0 6.a scrub ok 2024-04-06T03:41:33.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.194578+0000 osd.7 (osd.7) 515 : cluster 0 5.19 scrub starts 2024-04-06T03:41:33.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.196464+0000 osd.7 (osd.7) 516 : cluster 0 5.19 scrub ok 2024-04-06T03:41:33.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.431353+0000 mgr.y (mgr.24368) 758 : cluster 0 pgmap v808: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 816 op/s 2024-04-06T03:41:33.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.719949+0000 osd.2 (osd.2) 407 : cluster 0 2.16 scrub starts 2024-04-06T03:41:33.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:33 smithi153 bash[23852]: cluster 2024-04-06T03:41:32.721041+0000 osd.2 (osd.2) 408 : cluster 0 2.16 scrub ok 2024-04-06T03:41:33.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:33.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.045768+0000 osd.4 (osd.4) 489 : cluster 0 2.18 scrub starts 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.047099+0000 osd.4 (osd.4) 490 : cluster 0 2.18 scrub ok 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.055613+0000 osd.0 (osd.0) 501 : cluster 0 6.a scrub ok 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.194578+0000 osd.7 (osd.7) 515 : cluster 0 5.19 scrub starts 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.196464+0000 osd.7 (osd.7) 516 : cluster 0 5.19 scrub ok 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.431353+0000 mgr.y (mgr.24368) 758 : cluster 0 pgmap v808: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 816 op/s 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.719949+0000 osd.2 (osd.2) 407 : cluster 0 2.16 scrub starts 2024-04-06T03:41:33.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[29221]: cluster 2024-04-06T03:41:32.721041+0000 osd.2 (osd.2) 408 : cluster 0 2.16 scrub ok 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.045768+0000 osd.4 (osd.4) 489 : cluster 0 2.18 scrub starts 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.047099+0000 osd.4 (osd.4) 490 : cluster 0 2.18 scrub ok 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.055613+0000 osd.0 (osd.0) 501 : cluster 0 6.a scrub ok 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.194578+0000 osd.7 (osd.7) 515 : cluster 0 5.19 scrub starts 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.196464+0000 osd.7 (osd.7) 516 : cluster 0 5.19 scrub ok 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.431353+0000 mgr.y (mgr.24368) 758 : cluster 0 pgmap v808: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 816 op/s 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.719949+0000 osd.2 (osd.2) 407 : cluster 0 2.16 scrub starts 2024-04-06T03:41:33.670 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:33 smithi050 bash[21467]: cluster 2024-04-06T03:41:32.721041+0000 osd.2 (osd.2) 408 : cluster 0 2.16 scrub ok 2024-04-06T03:41:33.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:33.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:33.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:33.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:33.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:34.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:34.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:34.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:34.167 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:34.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:34.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:34.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:34.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:34.532 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:34.557 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:34.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:34.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:34.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:34.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:34.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:34.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:34.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:35.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:35.095 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:35.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:34 smithi153 bash[23852]: cluster 2024-04-06T03:41:33.766268+0000 osd.2 (osd.2) 409 : cluster 0 5.b deep-scrub starts 2024-04-06T03:41:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:34 smithi153 bash[23852]: cluster 2024-04-06T03:41:33.770805+0000 osd.2 (osd.2) 410 : cluster 0 5.b deep-scrub ok 2024-04-06T03:41:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:34 smithi153 bash[23852]: cluster 2024-04-06T03:41:34.243908+0000 osd.7 (osd.7) 517 : cluster 0 6.8 scrub starts 2024-04-06T03:41:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:34 smithi153 bash[23852]: cluster 2024-04-06T03:41:34.247535+0000 osd.7 (osd.7) 518 : cluster 0 6.8 scrub ok 2024-04-06T03:41:35.211 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:34 smithi153 bash[23852]: cluster 2024-04-06T03:41:34.432098+0000 mgr.y (mgr.24368) 759 : cluster 0 pgmap v809: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 782 op/s 2024-04-06T03:41:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[21467]: cluster 2024-04-06T03:41:33.766268+0000 osd.2 (osd.2) 409 : cluster 0 5.b deep-scrub starts 2024-04-06T03:41:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[21467]: cluster 2024-04-06T03:41:33.770805+0000 osd.2 (osd.2) 410 : cluster 0 5.b deep-scrub ok 2024-04-06T03:41:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[21467]: cluster 2024-04-06T03:41:34.243908+0000 osd.7 (osd.7) 517 : cluster 0 6.8 scrub starts 2024-04-06T03:41:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[21467]: cluster 2024-04-06T03:41:34.247535+0000 osd.7 (osd.7) 518 : cluster 0 6.8 scrub ok 2024-04-06T03:41:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[21467]: cluster 2024-04-06T03:41:34.432098+0000 mgr.y (mgr.24368) 759 : cluster 0 pgmap v809: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 782 op/s 2024-04-06T03:41:35.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[29221]: cluster 2024-04-06T03:41:33.766268+0000 osd.2 (osd.2) 409 : cluster 0 5.b deep-scrub starts 2024-04-06T03:41:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[29221]: cluster 2024-04-06T03:41:33.770805+0000 osd.2 (osd.2) 410 : cluster 0 5.b deep-scrub ok 2024-04-06T03:41:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[29221]: cluster 2024-04-06T03:41:34.243908+0000 osd.7 (osd.7) 517 : cluster 0 6.8 scrub starts 2024-04-06T03:41:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[29221]: cluster 2024-04-06T03:41:34.247535+0000 osd.7 (osd.7) 518 : cluster 0 6.8 scrub ok 2024-04-06T03:41:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:34 smithi050 bash[29221]: cluster 2024-04-06T03:41:34.432098+0000 mgr.y (mgr.24368) 759 : cluster 0 pgmap v809: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 782 op/s 2024-04-06T03:41:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.242 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:35.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:35.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:35.367 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:35.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:35.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:35.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:35.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:35.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:35.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:35.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:35.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:35.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:35.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[21467]: cluster 2024-04-06T03:41:34.053620+0000 osd.0 (osd.0) 502 : cluster 0 5.2 scrub starts 2024-04-06T03:41:35.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[21467]: cluster 2024-04-06T03:41:34.062044+0000 osd.0 (osd.0) 503 : cluster 0 5.2 scrub ok 2024-04-06T03:41:35.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[21467]: audit 2024-04-06T03:41:35.133588+0000 mon.a (mon.0) 1560 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:35.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[21467]: cluster 2024-04-06T03:41:35.237906+0000 osd.7 (osd.7) 519 : cluster 0 2.8 scrub starts 2024-04-06T03:41:35.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[21467]: cluster 2024-04-06T03:41:35.249699+0000 osd.7 (osd.7) 520 : cluster 0 2.8 scrub ok 2024-04-06T03:41:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[29221]: cluster 2024-04-06T03:41:34.053620+0000 osd.0 (osd.0) 502 : cluster 0 5.2 scrub starts 2024-04-06T03:41:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[29221]: cluster 2024-04-06T03:41:34.062044+0000 osd.0 (osd.0) 503 : cluster 0 5.2 scrub ok 2024-04-06T03:41:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[29221]: audit 2024-04-06T03:41:35.133588+0000 mon.a (mon.0) 1560 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[29221]: cluster 2024-04-06T03:41:35.237906+0000 osd.7 (osd.7) 519 : cluster 0 2.8 scrub starts 2024-04-06T03:41:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:35 smithi050 bash[29221]: cluster 2024-04-06T03:41:35.249699+0000 osd.7 (osd.7) 520 : cluster 0 2.8 scrub ok 2024-04-06T03:41:36.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:36.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:36.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:36.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:36.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:36.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:36.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:35 smithi153 bash[23852]: cluster 2024-04-06T03:41:34.053620+0000 osd.0 (osd.0) 502 : cluster 0 5.2 scrub starts 2024-04-06T03:41:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:35 smithi153 bash[23852]: cluster 2024-04-06T03:41:34.062044+0000 osd.0 (osd.0) 503 : cluster 0 5.2 scrub ok 2024-04-06T03:41:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:35 smithi153 bash[23852]: audit 2024-04-06T03:41:35.133588+0000 mon.a (mon.0) 1560 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:35 smithi153 bash[23852]: cluster 2024-04-06T03:41:35.237906+0000 osd.7 (osd.7) 519 : cluster 0 2.8 scrub starts 2024-04-06T03:41:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:35 smithi153 bash[23852]: cluster 2024-04-06T03:41:35.249699+0000 osd.7 (osd.7) 520 : cluster 0 2.8 scrub ok 2024-04-06T03:41:36.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:36.416 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:36.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:36.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:36.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:36.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:36.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:36.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:36.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:36.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:36.940 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:36.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:37.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:37.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:37.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:36 smithi050 bash[21467]: cluster 2024-04-06T03:41:36.278749+0000 osd.7 (osd.7) 521 : cluster 0 2.5 scrub starts 2024-04-06T03:41:37.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:36 smithi050 bash[21467]: cluster 2024-04-06T03:41:36.282546+0000 osd.7 (osd.7) 522 : cluster 0 2.5 scrub ok 2024-04-06T03:41:37.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:36 smithi050 bash[21467]: cluster 2024-04-06T03:41:36.432796+0000 mgr.y (mgr.24368) 760 : cluster 0 pgmap v810: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 758 op/s 2024-04-06T03:41:37.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:36 smithi050 bash[29221]: cluster 2024-04-06T03:41:36.278749+0000 osd.7 (osd.7) 521 : cluster 0 2.5 scrub starts 2024-04-06T03:41:37.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:36 smithi050 bash[29221]: cluster 2024-04-06T03:41:36.282546+0000 osd.7 (osd.7) 522 : cluster 0 2.5 scrub ok 2024-04-06T03:41:37.083 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:36 smithi050 bash[29221]: cluster 2024-04-06T03:41:36.432796+0000 mgr.y (mgr.24368) 760 : cluster 0 pgmap v810: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 758 op/s 2024-04-06T03:41:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:37.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:37.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:37.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:36 smithi153 bash[23852]: cluster 2024-04-06T03:41:36.278749+0000 osd.7 (osd.7) 521 : cluster 0 2.5 scrub starts 2024-04-06T03:41:37.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:36 smithi153 bash[23852]: cluster 2024-04-06T03:41:36.282546+0000 osd.7 (osd.7) 522 : cluster 0 2.5 scrub ok 2024-04-06T03:41:37.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:36 smithi153 bash[23852]: cluster 2024-04-06T03:41:36.432796+0000 mgr.y (mgr.24368) 760 : cluster 0 pgmap v810: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 758 op/s 2024-04-06T03:41:37.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:37.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:37.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:37.481 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:37.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:37.615 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:37.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:37.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:37.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:37.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:37.895 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:37.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:37.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:37 smithi050 bash[21467]: cluster 2024-04-06T03:41:37.234537+0000 osd.7 (osd.7) 523 : cluster 0 6.5 scrub starts 2024-04-06T03:41:37.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:37 smithi050 bash[21467]: cluster 2024-04-06T03:41:37.293667+0000 osd.7 (osd.7) 524 : cluster 0 6.5 scrub ok 2024-04-06T03:41:37.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:37 smithi050 bash[29221]: cluster 2024-04-06T03:41:37.234537+0000 osd.7 (osd.7) 523 : cluster 0 6.5 scrub starts 2024-04-06T03:41:37.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:37 smithi050 bash[29221]: cluster 2024-04-06T03:41:37.293667+0000 osd.7 (osd.7) 524 : cluster 0 6.5 scrub ok 2024-04-06T03:41:37.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:38.026 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:38.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:38.139 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:37 smithi153 bash[23852]: cluster 2024-04-06T03:41:37.234537+0000 osd.7 (osd.7) 523 : cluster 0 6.5 scrub starts 2024-04-06T03:41:38.139 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:37 smithi153 bash[23852]: cluster 2024-04-06T03:41:37.293667+0000 osd.7 (osd.7) 524 : cluster 0 6.5 scrub ok 2024-04-06T03:41:38.158 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:38.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:38.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:38.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:38.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.553 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:38.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.678 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:38.782 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:38.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:38.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:39.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:39.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:39.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:38 smithi050 bash[21467]: cluster 2024-04-06T03:41:38.433445+0000 mgr.y (mgr.24368) 761 : cluster 0 pgmap v811: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-04-06T03:41:39.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:38 smithi050 bash[21467]: cluster 2024-04-06T03:41:38.453060+0000 osd.3 (osd.3) 510 : cluster 0 6.6 scrub starts 2024-04-06T03:41:39.190 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:38 smithi050 bash[21467]: cluster 2024-04-06T03:41:38.476306+0000 osd.3 (osd.3) 511 : cluster 0 6.6 scrub ok 2024-04-06T03:41:39.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:38 smithi050 bash[29221]: cluster 2024-04-06T03:41:38.433445+0000 mgr.y (mgr.24368) 761 : cluster 0 pgmap v811: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-04-06T03:41:39.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:38 smithi050 bash[29221]: cluster 2024-04-06T03:41:38.453060+0000 osd.3 (osd.3) 510 : cluster 0 6.6 scrub starts 2024-04-06T03:41:39.190 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:38 smithi050 bash[29221]: cluster 2024-04-06T03:41:38.476306+0000 osd.3 (osd.3) 511 : cluster 0 6.6 scrub ok 2024-04-06T03:41:39.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:38 smithi153 bash[23852]: cluster 2024-04-06T03:41:38.433445+0000 mgr.y (mgr.24368) 761 : cluster 0 pgmap v811: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s 2024-04-06T03:41:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:38 smithi153 bash[23852]: cluster 2024-04-06T03:41:38.453060+0000 osd.3 (osd.3) 510 : cluster 0 6.6 scrub starts 2024-04-06T03:41:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:38 smithi153 bash[23852]: cluster 2024-04-06T03:41:38.476306+0000 osd.3 (osd.3) 511 : cluster 0 6.6 scrub ok 2024-04-06T03:41:39.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:39.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:39.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:39.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:39.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:41:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:41:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:41:39.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:39.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:39.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:39.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:39.726 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:39.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:39.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:39.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:39.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:39.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:39 smithi153 bash[23852]: cluster 2024-04-06T03:41:38.597447+0000 osd.7 (osd.7) 525 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:39.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:39 smithi153 bash[23852]: cluster 2024-04-06T03:41:38.597501+0000 osd.7 (osd.7) 526 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:39.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:39 smithi153 bash[23852]: cluster 2024-04-06T03:41:38.958246+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-04-06T03:41:39.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:39 smithi153 bash[23852]: cluster 2024-04-06T03:41:39.445020+0000 osd.3 (osd.3) 512 : cluster 0 5.14 scrub starts 2024-04-06T03:41:39.969 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:39 smithi153 bash[23852]: cluster 2024-04-06T03:41:39.452928+0000 osd.3 (osd.3) 513 : cluster 0 5.14 scrub ok 2024-04-06T03:41:39.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:40.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:40.125 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:38.597447+0000 osd.7 (osd.7) 525 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:38.597501+0000 osd.7 (osd.7) 526 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:38.958246+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:39.445020+0000 osd.3 (osd.3) 512 : cluster 0 5.14 scrub starts 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:39.452928+0000 osd.3 (osd.3) 513 : cluster 0 5.14 scrub ok 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:38.597447+0000 osd.7 (osd.7) 525 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:38.597501+0000 osd.7 (osd.7) 526 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:40.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:38.958246+0000 mon.a (mon.0) 1561 : cluster 0 osdmap e312: 8 total, 8 up, 8 in 2024-04-06T03:41:40.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:39.445020+0000 osd.3 (osd.3) 512 : cluster 0 5.14 scrub starts 2024-04-06T03:41:40.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:39.452928+0000 osd.3 (osd.3) 513 : cluster 0 5.14 scrub ok 2024-04-06T03:41:40.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:40.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:41:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:41:40.258 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:40.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:40.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:40.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:40.508 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:40.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:40.645 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:40.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:40.777 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:40.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:40.913 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:40.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:41.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:41.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:41.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:40 smithi153 bash[23852]: cluster 2024-04-06T03:41:39.147987+0000 osd.0 (osd.0) 504 : cluster 0 6.10 deep-scrub starts 2024-04-06T03:41:41.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:40 smithi153 bash[23852]: cluster 2024-04-06T03:41:39.208196+0000 osd.0 (osd.0) 505 : cluster 0 6.10 deep-scrub ok 2024-04-06T03:41:41.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:40 smithi153 bash[23852]: audit 2024-04-06T03:41:40.040554+0000 mgr.y (mgr.24368) 762 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:41.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:40 smithi153 bash[23852]: cluster 2024-04-06T03:41:40.434644+0000 mgr.y (mgr.24368) 763 : cluster 0 pgmap v813: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:41:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:41.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:41.161 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:39.147987+0000 osd.0 (osd.0) 504 : cluster 0 6.10 deep-scrub starts 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:39.208196+0000 osd.0 (osd.0) 505 : cluster 0 6.10 deep-scrub ok 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: audit 2024-04-06T03:41:40.040554+0000 mgr.y (mgr.24368) 762 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[21467]: cluster 2024-04-06T03:41:40.434644+0000 mgr.y (mgr.24368) 763 : cluster 0 pgmap v813: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:39.147987+0000 osd.0 (osd.0) 504 : cluster 0 6.10 deep-scrub starts 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:39.208196+0000 osd.0 (osd.0) 505 : cluster 0 6.10 deep-scrub ok 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: audit 2024-04-06T03:41:40.040554+0000 mgr.y (mgr.24368) 762 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:41.162 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:40 smithi050 bash[29221]: cluster 2024-04-06T03:41:40.434644+0000 mgr.y (mgr.24368) 763 : cluster 0 pgmap v813: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 847 op/s 2024-04-06T03:41:41.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:41.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:41.318 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:41.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:41.454 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:41.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:41.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:41.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:41.713 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:41.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:41.841 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:41.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:41.972 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:41.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:42.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:42.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:42.229 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:42.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:42.344 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:42 smithi050 bash[21467]: audit 2024-04-06T03:41:42.037688+0000 mon.a (mon.0) 1562 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:41:42.344 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:42 smithi050 bash[29221]: audit 2024-04-06T03:41:42.037688+0000 mon.a (mon.0) 1562 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:41:42.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:42.473 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:42.484 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:42 smithi153 bash[23852]: audit 2024-04-06T03:41:42.037688+0000 mon.a (mon.0) 1562 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:41:42.501 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:42.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:42.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:42.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:42.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:42.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:42.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:42.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:43.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:43.050 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:43.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:43.185 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:43.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:43 smithi050 bash[21467]: cluster 2024-04-06T03:41:42.435316+0000 mgr.y (mgr.24368) 764 : cluster 0 pgmap v814: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 780 op/s 2024-04-06T03:41:43.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:43 smithi050 bash[29221]: cluster 2024-04-06T03:41:42.435316+0000 mgr.y (mgr.24368) 764 : cluster 0 pgmap v814: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 780 op/s 2024-04-06T03:41:43.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:43.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:43 smithi153 bash[23852]: cluster 2024-04-06T03:41:42.435316+0000 mgr.y (mgr.24368) 764 : cluster 0 pgmap v814: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 780 op/s 2024-04-06T03:41:43.313 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:43.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:43.442 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:43.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:43.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:43.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:43.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:43.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:43.834 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:41:43.834 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:41:43.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:43.939 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:43.966 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:43.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:44.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:44.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:44.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:44.239 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:44 smithi153 bash[23852]: cluster 2024-04-06T03:41:43.282471+0000 osd.7 (osd.7) 527 : cluster 0 2.b scrub starts 2024-04-06T03:41:44.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:44 smithi153 bash[23852]: cluster 2024-04-06T03:41:43.284054+0000 osd.7 (osd.7) 528 : cluster 0 2.b scrub ok 2024-04-06T03:41:44.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:44 smithi153 bash[23852]: audit 2024-04-06T03:41:43.479320+0000 mon.a (mon.0) 1563 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:44.240 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:44 smithi153 bash[23852]: audit 2024-04-06T03:41:43.489118+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:44.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:44.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:44.387 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:44.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[21467]: cluster 2024-04-06T03:41:43.282471+0000 osd.7 (osd.7) 527 : cluster 0 2.b scrub starts 2024-04-06T03:41:44.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[21467]: cluster 2024-04-06T03:41:43.284054+0000 osd.7 (osd.7) 528 : cluster 0 2.b scrub ok 2024-04-06T03:41:44.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[21467]: audit 2024-04-06T03:41:43.479320+0000 mon.a (mon.0) 1563 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:44.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[21467]: audit 2024-04-06T03:41:43.489118+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:44.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[21467]: debug 2024-04-06T03:41:44.131+0000 7f702b621700 -1 mon.a@0(leader).osd e312 definitely_dead 0 2024-04-06T03:41:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[29221]: cluster 2024-04-06T03:41:43.282471+0000 osd.7 (osd.7) 527 : cluster 0 2.b scrub starts 2024-04-06T03:41:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[29221]: cluster 2024-04-06T03:41:43.284054+0000 osd.7 (osd.7) 528 : cluster 0 2.b scrub ok 2024-04-06T03:41:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[29221]: audit 2024-04-06T03:41:43.479320+0000 mon.a (mon.0) 1563 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:44 smithi050 bash[29221]: audit 2024-04-06T03:41:43.489118+0000 mon.a (mon.0) 1564 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:44.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:44.522 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:44.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:44.653 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:44.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:44.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:44.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:44.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:44.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:45.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:45.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:45.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:45.138 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:41:45.146 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.6. 2024-04-06T03:41:45.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:45.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:45.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:43.157623+0000 osd.0 (osd.0) 506 : cluster 0 6.18 scrub starts 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:43.229005+0000 osd.0 (osd.0) 507 : cluster 0 6.18 scrub ok 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:44.129926+0000 osd.0 (osd.0) 508 : cluster 0 3.6 deep-scrub starts 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: audit 2024-04-06T03:41:44.134376+0000 mon.a (mon.0) 1565 : audit 1 from='client.? 172.21.15.50:0/1455807420' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:44.134644+0000 mon.a (mon.0) 1566 : cluster 1 Client client.admin marked osd.6 out, while it was still marked up 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: audit 2024-04-06T03:41:44.225573+0000 mon.a (mon.0) 1567 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: audit 2024-04-06T03:41:44.227683+0000 mon.a (mon.0) 1568 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: audit 2024-04-06T03:41:44.236287+0000 mon.a (mon.0) 1569 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:45.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:44.255169+0000 osd.7 (osd.7) 529 : cluster 0 5.4 scrub starts 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:44.258885+0000 osd.7 (osd.7) 530 : cluster 0 5.4 scrub ok 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: cluster 2024-04-06T03:41:44.435880+0000 mgr.y (mgr.24368) 765 : cluster 0 pgmap v815: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 768 op/s 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[29221]: audit 2024-04-06T03:41:44.601774+0000 mon.a (mon.0) 1570 : audit 1 from='client.? 172.21.15.50:0/3631709050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:43.157623+0000 osd.0 (osd.0) 506 : cluster 0 6.18 scrub starts 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:43.229005+0000 osd.0 (osd.0) 507 : cluster 0 6.18 scrub ok 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:44.129926+0000 osd.0 (osd.0) 508 : cluster 0 3.6 deep-scrub starts 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: audit 2024-04-06T03:41:44.134376+0000 mon.a (mon.0) 1565 : audit 1 from='client.? 172.21.15.50:0/1455807420' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:44.134644+0000 mon.a (mon.0) 1566 : cluster 1 Client client.admin marked osd.6 out, while it was still marked up 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: audit 2024-04-06T03:41:44.225573+0000 mon.a (mon.0) 1567 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: audit 2024-04-06T03:41:44.227683+0000 mon.a (mon.0) 1568 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: audit 2024-04-06T03:41:44.236287+0000 mon.a (mon.0) 1569 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:44.255169+0000 osd.7 (osd.7) 529 : cluster 0 5.4 scrub starts 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:44.258885+0000 osd.7 (osd.7) 530 : cluster 0 5.4 scrub ok 2024-04-06T03:41:45.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: cluster 2024-04-06T03:41:44.435880+0000 mgr.y (mgr.24368) 765 : cluster 0 pgmap v815: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 768 op/s 2024-04-06T03:41:45.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:45 smithi050 bash[21467]: audit 2024-04-06T03:41:44.601774+0000 mon.a (mon.0) 1570 : audit 1 from='client.? 172.21.15.50:0/3631709050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:41:45.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:45.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:45.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:45.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:43.157623+0000 osd.0 (osd.0) 506 : cluster 0 6.18 scrub starts 2024-04-06T03:41:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:43.229005+0000 osd.0 (osd.0) 507 : cluster 0 6.18 scrub ok 2024-04-06T03:41:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:44.129926+0000 osd.0 (osd.0) 508 : cluster 0 3.6 deep-scrub starts 2024-04-06T03:41:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: audit 2024-04-06T03:41:44.134376+0000 mon.a (mon.0) 1565 : audit 1 from='client.? 172.21.15.50:0/1455807420' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-04-06T03:41:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:44.134644+0000 mon.a (mon.0) 1566 : cluster 1 Client client.admin marked osd.6 out, while it was still marked up 2024-04-06T03:41:45.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: audit 2024-04-06T03:41:44.225573+0000 mon.a (mon.0) 1567 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:41:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: audit 2024-04-06T03:41:44.227683+0000 mon.a (mon.0) 1568 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:41:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: audit 2024-04-06T03:41:44.236287+0000 mon.a (mon.0) 1569 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:44.255169+0000 osd.7 (osd.7) 529 : cluster 0 5.4 scrub starts 2024-04-06T03:41:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:44.258885+0000 osd.7 (osd.7) 530 : cluster 0 5.4 scrub ok 2024-04-06T03:41:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: cluster 2024-04-06T03:41:44.435880+0000 mgr.y (mgr.24368) 765 : cluster 0 pgmap v815: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 768 op/s 2024-04-06T03:41:45.541 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:45 smithi153 bash[23852]: audit 2024-04-06T03:41:44.601774+0000 mon.a (mon.0) 1570 : audit 1 from='client.? 172.21.15.50:0/3631709050' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:41:45.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:45.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:45.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:45.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:45.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:45.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:45.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:45.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:46.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:46.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:46.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:46.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:46.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: cluster 2024-04-06T03:41:44.156009+0000 osd.0 (osd.0) 509 : cluster 0 3.6 deep-scrub ok 2024-04-06T03:41:46.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: cluster 2024-04-06T03:41:45.107675+0000 osd.0 (osd.0) 510 : cluster 0 3.1 scrub starts 2024-04-06T03:41:46.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: cluster 2024-04-06T03:41:45.112940+0000 osd.0 (osd.0) 511 : cluster 0 3.1 scrub ok 2024-04-06T03:41:46.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: cluster 2024-04-06T03:41:45.117649+0000 mon.a (mon.0) 1571 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:41:46.346 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: audit 2024-04-06T03:41:45.128474+0000 mon.a (mon.0) 1572 : audit 1 from='client.? 172.21.15.50:0/1455807420' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: audit 2024-04-06T03:41:45.128777+0000 mon.a (mon.0) 1573 : audit 1 from='client.? 172.21.15.50:0/3631709050' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[21467]: cluster 2024-04-06T03:41:45.132619+0000 mon.a (mon.0) 1574 : cluster 0 osdmap e313: 8 total, 8 up, 7 in 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: cluster 2024-04-06T03:41:44.156009+0000 osd.0 (osd.0) 509 : cluster 0 3.6 deep-scrub ok 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: cluster 2024-04-06T03:41:45.107675+0000 osd.0 (osd.0) 510 : cluster 0 3.1 scrub starts 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: cluster 2024-04-06T03:41:45.112940+0000 osd.0 (osd.0) 511 : cluster 0 3.1 scrub ok 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: cluster 2024-04-06T03:41:45.117649+0000 mon.a (mon.0) 1571 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: audit 2024-04-06T03:41:45.128474+0000 mon.a (mon.0) 1572 : audit 1 from='client.? 172.21.15.50:0/1455807420' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-06T03:41:46.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: audit 2024-04-06T03:41:45.128777+0000 mon.a (mon.0) 1573 : audit 1 from='client.? 172.21.15.50:0/3631709050' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:41:46.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:46 smithi050 bash[29221]: cluster 2024-04-06T03:41:45.132619+0000 mon.a (mon.0) 1574 : cluster 0 osdmap e313: 8 total, 8 up, 7 in 2024-04-06T03:41:46.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:46.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:46.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:46.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: cluster 2024-04-06T03:41:44.156009+0000 osd.0 (osd.0) 509 : cluster 0 3.6 deep-scrub ok 2024-04-06T03:41:46.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: cluster 2024-04-06T03:41:45.107675+0000 osd.0 (osd.0) 510 : cluster 0 3.1 scrub starts 2024-04-06T03:41:46.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: cluster 2024-04-06T03:41:45.112940+0000 osd.0 (osd.0) 511 : cluster 0 3.1 scrub ok 2024-04-06T03:41:46.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: cluster 2024-04-06T03:41:45.117649+0000 mon.a (mon.0) 1571 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:41:46.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: audit 2024-04-06T03:41:45.128474+0000 mon.a (mon.0) 1572 : audit 1 from='client.? 172.21.15.50:0/1455807420' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-04-06T03:41:46.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: audit 2024-04-06T03:41:45.128777+0000 mon.a (mon.0) 1573 : audit 1 from='client.? 172.21.15.50:0/3631709050' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:41:46.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:46 smithi153 bash[23852]: cluster 2024-04-06T03:41:45.132619+0000 mon.a (mon.0) 1574 : cluster 0 osdmap e313: 8 total, 8 up, 7 in 2024-04-06T03:41:46.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:46.624 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:46.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:46.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:46.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:46.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:46.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:46.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:47.124 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:47.161 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:47.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:47.292 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:47.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:47.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:47 smithi153 bash[23852]: cluster 2024-04-06T03:41:46.133410+0000 mon.a (mon.0) 1575 : cluster 0 osdmap e314: 8 total, 8 up, 7 in 2024-04-06T03:41:47.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:47 smithi153 bash[23852]: cluster 2024-04-06T03:41:46.192340+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:47.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:47 smithi153 bash[23852]: cluster 2024-04-06T03:41:46.192390+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:47.410 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:47 smithi153 bash[23852]: cluster 2024-04-06T03:41:46.436527+0000 mgr.y (mgr.24368) 766 : cluster 0 pgmap v818: 148 pgs: 148 active+clean; 2.6 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 684 B/s rd, 0 B/s wr, 821 op/s 2024-04-06T03:41:47.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:47.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[21467]: cluster 2024-04-06T03:41:46.133410+0000 mon.a (mon.0) 1575 : cluster 0 osdmap e314: 8 total, 8 up, 7 in 2024-04-06T03:41:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[21467]: cluster 2024-04-06T03:41:46.192340+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[21467]: cluster 2024-04-06T03:41:46.192390+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[21467]: cluster 2024-04-06T03:41:46.436527+0000 mgr.y (mgr.24368) 766 : cluster 0 pgmap v818: 148 pgs: 148 active+clean; 2.6 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 684 B/s rd, 0 B/s wr, 821 op/s 2024-04-06T03:41:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[29221]: cluster 2024-04-06T03:41:46.133410+0000 mon.a (mon.0) 1575 : cluster 0 osdmap e314: 8 total, 8 up, 7 in 2024-04-06T03:41:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[29221]: cluster 2024-04-06T03:41:46.192340+0000 osd.6 (osd.6) 103 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[29221]: cluster 2024-04-06T03:41:46.192390+0000 osd.6 (osd.6) 104 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:47 smithi050 bash[29221]: cluster 2024-04-06T03:41:46.436527+0000 mgr.y (mgr.24368) 766 : cluster 0 pgmap v818: 148 pgs: 148 active+clean; 2.6 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 684 B/s rd, 0 B/s wr, 821 op/s 2024-04-06T03:41:47.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:47.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:47.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:47.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:47.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:48.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:48.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:48.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:48.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:48.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:48.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.146760+0000 osd.4 (osd.4) 491 : cluster 0 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 312'9467 2024-04-06T03:41:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.146780+0000 osd.4 (osd.4) 492 : cluster 0 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 312'9552 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.146787+0000 osd.4 (osd.4) 493 : cluster 0 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 312'9746 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.151054+0000 osd.3 (osd.3) 514 : cluster 0 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 312'9668 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.152086+0000 osd.3 (osd.3) 515 : cluster 0 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 312'9830 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.153194+0000 osd.0 (osd.0) 512 : cluster 0 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 312'9752 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.160241+0000 osd.1 (osd.1) 391 : cluster 0 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 312'9883 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[21467]: cluster 2024-04-06T03:41:47.162024+0000 mon.a (mon.0) 1576 : cluster 0 osdmap e315: 8 total, 8 up, 7 in 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.146760+0000 osd.4 (osd.4) 491 : cluster 0 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 312'9467 2024-04-06T03:41:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.146780+0000 osd.4 (osd.4) 492 : cluster 0 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 312'9552 2024-04-06T03:41:48.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.146787+0000 osd.4 (osd.4) 493 : cluster 0 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 312'9746 2024-04-06T03:41:48.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.151054+0000 osd.3 (osd.3) 514 : cluster 0 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 312'9668 2024-04-06T03:41:48.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.152086+0000 osd.3 (osd.3) 515 : cluster 0 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 312'9830 2024-04-06T03:41:48.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.153194+0000 osd.0 (osd.0) 512 : cluster 0 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 312'9752 2024-04-06T03:41:48.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.160241+0000 osd.1 (osd.1) 391 : cluster 0 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 312'9883 2024-04-06T03:41:48.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:48 smithi050 bash[29221]: cluster 2024-04-06T03:41:47.162024+0000 mon.a (mon.0) 1576 : cluster 0 osdmap e315: 8 total, 8 up, 7 in 2024-04-06T03:41:48.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.146760+0000 osd.4 (osd.4) 491 : cluster 0 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 312'9467 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.146780+0000 osd.4 (osd.4) 492 : cluster 0 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 312'9552 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.146787+0000 osd.4 (osd.4) 493 : cluster 0 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 312'9746 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.151054+0000 osd.3 (osd.3) 514 : cluster 0 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 312'9668 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.152086+0000 osd.3 (osd.3) 515 : cluster 0 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 312'9830 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.153194+0000 osd.0 (osd.0) 512 : cluster 0 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 312'9752 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.160241+0000 osd.1 (osd.1) 391 : cluster 0 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 312'9883 2024-04-06T03:41:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:48 smithi153 bash[23852]: cluster 2024-04-06T03:41:47.162024+0000 mon.a (mon.0) 1576 : cluster 0 osdmap e315: 8 total, 8 up, 7 in 2024-04-06T03:41:48.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:48.639 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:48.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:48.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:48.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:48.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:49.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:49.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:49.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:49.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:49.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-06T03:41:49.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:49.266 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:49 smithi153 bash[23852]: cluster 2024-04-06T03:41:48.191746+0000 mon.a (mon.0) 1577 : cluster 0 osdmap e316: 8 total, 8 up, 7 in 2024-04-06T03:41:49.266 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:49 smithi153 bash[23852]: cluster 2024-04-06T03:41:48.437082+0000 mgr.y (mgr.24368) 767 : cluster 0 pgmap v821: 148 pgs: 2 activating+remapped, 1 remapped+peering, 145 active+clean; 2.6 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 435 op/s; 5226/126042 objects misplaced (4.146%); 11 B/s, 5 objects/s recovering 2024-04-06T03:41:49.285 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:49.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:49.420 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:49.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:49 smithi050 bash[21467]: cluster 2024-04-06T03:41:48.191746+0000 mon.a (mon.0) 1577 : cluster 0 osdmap e316: 8 total, 8 up, 7 in 2024-04-06T03:41:49.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:49 smithi050 bash[21467]: cluster 2024-04-06T03:41:48.437082+0000 mgr.y (mgr.24368) 767 : cluster 0 pgmap v821: 148 pgs: 2 activating+remapped, 1 remapped+peering, 145 active+clean; 2.6 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 435 op/s; 5226/126042 objects misplaced (4.146%); 11 B/s, 5 objects/s recovering 2024-04-06T03:41:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:49 smithi050 bash[29221]: cluster 2024-04-06T03:41:48.191746+0000 mon.a (mon.0) 1577 : cluster 0 osdmap e316: 8 total, 8 up, 7 in 2024-04-06T03:41:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:49 smithi050 bash[29221]: cluster 2024-04-06T03:41:48.437082+0000 mgr.y (mgr.24368) 767 : cluster 0 pgmap v821: 148 pgs: 2 activating+remapped, 1 remapped+peering, 145 active+clean; 2.6 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 435 op/s; 5226/126042 objects misplaced (4.146%); 11 B/s, 5 objects/s recovering 2024-04-06T03:41:49.491 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:41:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:41:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:41:49.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:49.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:49.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:49.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:49.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:49.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:49.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:49.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:49.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:50.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:50.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:50.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:50.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:50.300 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:41:50.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:50.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:41:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:41:50.332 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:50 smithi153 bash[23852]: cluster 2024-04-06T03:41:49.186350+0000 osd.3 (osd.3) 516 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:50.332 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:50 smithi153 bash[23852]: cluster 2024-04-06T03:41:49.186432+0000 osd.3 (osd.3) 517 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:50.332 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:50 smithi153 bash[23852]: cluster 2024-04-06T03:41:49.218553+0000 mon.a (mon.0) 1578 : cluster 0 osdmap e317: 8 total, 8 up, 7 in 2024-04-06T03:41:50.332 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:50 smithi153 bash[23852]: audit 2024-04-06T03:41:50.159852+0000 mon.a (mon.0) 1579 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:50.332 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:50 smithi153 bash[23852]: audit 2024-04-06T03:41:50.162155+0000 mon.a (mon.0) 1580 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:50.336 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:41:50.349 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:50.385 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:41:50.429 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:41:50.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:50.471 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:41:50.480 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[21467]: cluster 2024-04-06T03:41:49.186350+0000 osd.3 (osd.3) 516 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[21467]: cluster 2024-04-06T03:41:49.186432+0000 osd.3 (osd.3) 517 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[21467]: cluster 2024-04-06T03:41:49.218553+0000 mon.a (mon.0) 1578 : cluster 0 osdmap e317: 8 total, 8 up, 7 in 2024-04-06T03:41:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[21467]: audit 2024-04-06T03:41:50.159852+0000 mon.a (mon.0) 1579 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[21467]: audit 2024-04-06T03:41:50.162155+0000 mon.a (mon.0) 1580 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[29221]: cluster 2024-04-06T03:41:49.186350+0000 osd.3 (osd.3) 516 : cluster 0 purged_snaps scrub starts 2024-04-06T03:41:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[29221]: cluster 2024-04-06T03:41:49.186432+0000 osd.3 (osd.3) 517 : cluster 0 purged_snaps scrub ok 2024-04-06T03:41:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[29221]: cluster 2024-04-06T03:41:49.218553+0000 mon.a (mon.0) 1578 : cluster 0 osdmap e317: 8 total, 8 up, 7 in 2024-04-06T03:41:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[29221]: audit 2024-04-06T03:41:50.159852+0000 mon.a (mon.0) 1579 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:41:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:50 smithi050 bash[29221]: audit 2024-04-06T03:41:50.162155+0000 mon.a (mon.0) 1580 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:41:50.512 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:41:50.552 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:41:50.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:50.586 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:41:50.588 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:41:50.588 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:41:50.588 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:41:50.588 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:41:50.589 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:41:50.589 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:41:50.589 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:41:50.589 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:41:50.611 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:50.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:50.749 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:50.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:50.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:50.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:50.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:51.022 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:51.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:51.150 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:51.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:41:51.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:51.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:51 smithi153 bash[23852]: audit 2024-04-06T03:41:50.052549+0000 mgr.y (mgr.24368) 768 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:51.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:51 smithi153 bash[23852]: cluster 2024-04-06T03:41:50.220168+0000 mon.a (mon.0) 1581 : cluster 0 osdmap e318: 8 total, 8 up, 7 in 2024-04-06T03:41:51.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:51 smithi153 bash[23852]: audit 2024-04-06T03:41:50.241808+0000 mon.a (mon.0) 1582 : audit 0 from='client.? 172.21.15.50:0/108929120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:41:51.262 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:51 smithi153 bash[23852]: cluster 2024-04-06T03:41:50.437585+0000 mgr.y (mgr.24368) 769 : cluster 0 pgmap v824: 148 pgs: 4 active+remapped+backfilling, 2 activating+remapped, 1 remapped+peering, 141 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 552 op/s; 13801/122139 objects misplaced (11.299%); 22 MiB/s, 370 objects/s recovering 2024-04-06T03:41:51.276 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:51.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[21467]: audit 2024-04-06T03:41:50.052549+0000 mgr.y (mgr.24368) 768 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[21467]: cluster 2024-04-06T03:41:50.220168+0000 mon.a (mon.0) 1581 : cluster 0 osdmap e318: 8 total, 8 up, 7 in 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[21467]: audit 2024-04-06T03:41:50.241808+0000 mon.a (mon.0) 1582 : audit 0 from='client.? 172.21.15.50:0/108929120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[21467]: cluster 2024-04-06T03:41:50.437585+0000 mgr.y (mgr.24368) 769 : cluster 0 pgmap v824: 148 pgs: 4 active+remapped+backfilling, 2 activating+remapped, 1 remapped+peering, 141 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 552 op/s; 13801/122139 objects misplaced (11.299%); 22 MiB/s, 370 objects/s recovering 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[29221]: audit 2024-04-06T03:41:50.052549+0000 mgr.y (mgr.24368) 768 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[29221]: cluster 2024-04-06T03:41:50.220168+0000 mon.a (mon.0) 1581 : cluster 0 osdmap e318: 8 total, 8 up, 7 in 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[29221]: audit 2024-04-06T03:41:50.241808+0000 mon.a (mon.0) 1582 : audit 0 from='client.? 172.21.15.50:0/108929120' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:41:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:51 smithi050 bash[29221]: cluster 2024-04-06T03:41:50.437585+0000 mgr.y (mgr.24368) 769 : cluster 0 pgmap v824: 148 pgs: 4 active+remapped+backfilling, 2 activating+remapped, 1 remapped+peering, 141 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 552 op/s; 13801/122139 objects misplaced (11.299%); 22 MiB/s, 370 objects/s recovering 2024-04-06T03:41:51.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:51.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:51.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:51.692 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:51.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:51.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:51.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:51.957 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:51.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:52.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:52.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:52.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:52.229 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:52.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:52.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:52.525 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:52.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:52.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:52.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:52.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:52.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:52.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:52.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:53.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:53.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:53.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:53.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:53.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:53.595 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:53.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:53 smithi050 bash[21467]: cluster 2024-04-06T03:41:52.438267+0000 mgr.y (mgr.24368) 770 : cluster 0 pgmap v825: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 567 op/s; 14623/118011 objects misplaced (12.391%); 29 MiB/s, 483 objects/s recovering 2024-04-06T03:41:53.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:53 smithi050 bash[29221]: cluster 2024-04-06T03:41:52.438267+0000 mgr.y (mgr.24368) 770 : cluster 0 pgmap v825: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 567 op/s; 14623/118011 objects misplaced (12.391%); 29 MiB/s, 483 objects/s recovering 2024-04-06T03:41:53.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:53.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:53 smithi153 bash[23852]: cluster 2024-04-06T03:41:52.438267+0000 mgr.y (mgr.24368) 770 : cluster 0 pgmap v825: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 567 op/s; 14623/118011 objects misplaced (12.391%); 29 MiB/s, 483 objects/s recovering 2024-04-06T03:41:53.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:53.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:53.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:53.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:54.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:54.121 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:54.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:54.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:54.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.393 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:54.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:54.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:54.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:54.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:54.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:54.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:54.941 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:54.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:55.026 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:41:55.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:41:55.027 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-04-06T03:41:55.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd in 6 2024-04-06T03:41:55.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:55.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:54 smithi153 bash[23852]: cluster 2024-04-06T03:41:54.438751+0000 mgr.y (mgr.24368) 771 : cluster 0 pgmap v826: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 701 op/s; 11158/112890 objects misplaced (9.884%); 51 MiB/s, 826 objects/s recovering 2024-04-06T03:41:55.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:55.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:55.184 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:54 smithi050 bash[21467]: cluster 2024-04-06T03:41:54.438751+0000 mgr.y (mgr.24368) 771 : cluster 0 pgmap v826: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 701 op/s; 11158/112890 objects misplaced (9.884%); 51 MiB/s, 826 objects/s recovering 2024-04-06T03:41:55.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:54 smithi050 bash[29221]: cluster 2024-04-06T03:41:54.438751+0000 mgr.y (mgr.24368) 771 : cluster 0 pgmap v826: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.3 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 655 B/s rd, 0 B/s wr, 701 op/s; 11158/112890 objects misplaced (9.884%); 51 MiB/s, 826 objects/s recovering 2024-04-06T03:41:55.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:55.343 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:55.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:55.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:55.506 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:55.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:55.638 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:55.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:41:55.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:55.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:55.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:55.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:55.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:56.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:56.047 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:56.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:56.177 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:56.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:56.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:56.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:56.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:56.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:56.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:56.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:56.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:56.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:56.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:56.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:56.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:56.980 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:56.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:57.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:57.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:57.189 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:57.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:57.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:57.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:57.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:57.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:57.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:57.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:57.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:57 smithi050 bash[21467]: cluster 2024-04-06T03:41:56.439286+0000 mgr.y (mgr.24368) 772 : cluster 0 pgmap v827: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 692 op/s; 11158/109431 objects misplaced (10.196%); 40 MiB/s, 646 objects/s recovering 2024-04-06T03:41:57.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:57 smithi050 bash[29221]: cluster 2024-04-06T03:41:56.439286+0000 mgr.y (mgr.24368) 772 : cluster 0 pgmap v827: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 692 op/s; 11158/109431 objects misplaced (10.196%); 40 MiB/s, 646 objects/s recovering 2024-04-06T03:41:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:57.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:57.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:57.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:57 smithi153 bash[23852]: cluster 2024-04-06T03:41:56.439286+0000 mgr.y (mgr.24368) 772 : cluster 0 pgmap v827: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 692 op/s; 11158/109431 objects misplaced (10.196%); 40 MiB/s, 646 objects/s recovering 2024-04-06T03:41:57.787 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:57.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:57.921 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:57.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:58.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:58.065 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:58.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:58.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:41:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:41:58.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:58.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:58.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:58.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:58.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:58.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:58.607 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:58.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:58.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:41:58.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:41:58.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:58.862 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:58.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:58.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:58.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:41:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:58.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:41:59.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:59.118 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:41:59.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:59.226 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:41:59.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:41:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:41:59.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:59.391 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:59.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:41:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:41:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:41:59.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:41:59 smithi050 bash[21467]: cluster 2024-04-06T03:41:58.440012+0000 mgr.y (mgr.24368) 773 : cluster 0 pgmap v828: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 999 B/s rd, 0 B/s wr, 759 op/s; 9188/104262 objects misplaced (8.812%); 43 MiB/s, 685 objects/s recovering 2024-04-06T03:41:59.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:41:59 smithi050 bash[29221]: cluster 2024-04-06T03:41:58.440012+0000 mgr.y (mgr.24368) 773 : cluster 0 pgmap v828: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 999 B/s rd, 0 B/s wr, 759 op/s; 9188/104262 objects misplaced (8.812%); 43 MiB/s, 685 objects/s recovering 2024-04-06T03:41:59.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:41:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:41:59.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:59.650 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:41:59 smithi153 bash[23852]: cluster 2024-04-06T03:41:58.440012+0000 mgr.y (mgr.24368) 773 : cluster 0 pgmap v828: 148 pgs: 7 active+remapped+backfilling, 141 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 999 B/s rd, 0 B/s wr, 759 op/s; 9188/104262 objects misplaced (8.812%); 43 MiB/s, 685 objects/s recovering 2024-04-06T03:41:59.669 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:41:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:41:59.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:59.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:41:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:41:59.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:59.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:41:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:41:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:00.033 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:00.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:42:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:42:00.178 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:42:00.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.300 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:00.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:00.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.546 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:00.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:00.675 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:00.675 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:00 smithi153 bash[23852]: cluster 2024-04-06T03:41:59.517451+0000 mon.a (mon.0) 1583 : cluster 0 osdmap e319: 8 total, 8 up, 7 in 2024-04-06T03:42:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:00.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.800 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:00.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:00.925 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:00.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:00.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:00 smithi050 bash[21467]: cluster 2024-04-06T03:41:59.517451+0000 mon.a (mon.0) 1583 : cluster 0 osdmap e319: 8 total, 8 up, 7 in 2024-04-06T03:42:00.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:00 smithi050 bash[29221]: cluster 2024-04-06T03:41:59.517451+0000 mon.a (mon.0) 1583 : cluster 0 osdmap e319: 8 total, 8 up, 7 in 2024-04-06T03:42:01.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:01.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:01.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:01.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:01.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:01.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:01.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:01.434 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:01.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:01.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[21467]: audit 2024-04-06T03:42:00.060439+0000 mgr.y (mgr.24368) 774 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:01.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[21467]: cluster 2024-04-06T03:42:00.110542+0000 osd.1 (osd.1) 392 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:01.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[21467]: cluster 2024-04-06T03:42:00.110592+0000 osd.1 (osd.1) 393 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:01.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[21467]: cluster 2024-04-06T03:42:00.440479+0000 mgr.y (mgr.24368) 775 : cluster 0 pgmap v830: 148 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 141 active+clean; 2.0 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 850 op/s; 5770/96642 objects misplaced (5.970%); 50 MiB/s, 804 objects/s recovering 2024-04-06T03:42:01.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[21467]: cluster 2024-04-06T03:42:00.517613+0000 mon.a (mon.0) 1584 : cluster 0 osdmap e320: 8 total, 8 up, 7 in 2024-04-06T03:42:01.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[29221]: audit 2024-04-06T03:42:00.060439+0000 mgr.y (mgr.24368) 774 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:01.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[29221]: cluster 2024-04-06T03:42:00.110542+0000 osd.1 (osd.1) 392 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:01.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[29221]: cluster 2024-04-06T03:42:00.110592+0000 osd.1 (osd.1) 393 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:01.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[29221]: cluster 2024-04-06T03:42:00.440479+0000 mgr.y (mgr.24368) 775 : cluster 0 pgmap v830: 148 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 141 active+clean; 2.0 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 850 op/s; 5770/96642 objects misplaced (5.970%); 50 MiB/s, 804 objects/s recovering 2024-04-06T03:42:01.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:01 smithi050 bash[29221]: cluster 2024-04-06T03:42:00.517613+0000 mon.a (mon.0) 1584 : cluster 0 osdmap e320: 8 total, 8 up, 7 in 2024-04-06T03:42:01.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:01.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:01.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:01 smithi153 bash[23852]: audit 2024-04-06T03:42:00.060439+0000 mgr.y (mgr.24368) 774 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:01.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:01 smithi153 bash[23852]: cluster 2024-04-06T03:42:00.110542+0000 osd.1 (osd.1) 392 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:01.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:01 smithi153 bash[23852]: cluster 2024-04-06T03:42:00.110592+0000 osd.1 (osd.1) 393 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:01.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:01 smithi153 bash[23852]: cluster 2024-04-06T03:42:00.440479+0000 mgr.y (mgr.24368) 775 : cluster 0 pgmap v830: 148 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 141 active+clean; 2.0 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 850 op/s; 5770/96642 objects misplaced (5.970%); 50 MiB/s, 804 objects/s recovering 2024-04-06T03:42:01.678 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:01 smithi153 bash[23852]: cluster 2024-04-06T03:42:00.517613+0000 mon.a (mon.0) 1584 : cluster 0 osdmap e320: 8 total, 8 up, 7 in 2024-04-06T03:42:01.692 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:01.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:01.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:01.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:01.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:01.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:02.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:02.073 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:02.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:02.195 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:02.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:02.298 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:02.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:02.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:02.442 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:02.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:02.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:02.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:02.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:02 smithi153 bash[23852]: cluster 2024-04-06T03:42:01.517756+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e321: 8 total, 8 up, 7 in 2024-04-06T03:42:02.676 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:02 smithi153 bash[23852]: cluster 2024-04-06T03:42:02.519393+0000 mon.a (mon.0) 1586 : cluster 0 osdmap e322: 8 total, 8 up, 7 in 2024-04-06T03:42:02.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:02.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:02.803 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:02 smithi050 bash[21467]: cluster 2024-04-06T03:42:01.517756+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e321: 8 total, 8 up, 7 in 2024-04-06T03:42:02.803 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:02 smithi050 bash[21467]: cluster 2024-04-06T03:42:02.519393+0000 mon.a (mon.0) 1586 : cluster 0 osdmap e322: 8 total, 8 up, 7 in 2024-04-06T03:42:02.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:02 smithi050 bash[29221]: cluster 2024-04-06T03:42:01.517756+0000 mon.a (mon.0) 1585 : cluster 0 osdmap e321: 8 total, 8 up, 7 in 2024-04-06T03:42:02.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:02 smithi050 bash[29221]: cluster 2024-04-06T03:42:02.519393+0000 mon.a (mon.0) 1586 : cluster 0 osdmap e322: 8 total, 8 up, 7 in 2024-04-06T03:42:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:02.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:02.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:03.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:03.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:03.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:03.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:03.339 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:03.443 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:03.471 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:03.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:03.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:03.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:03.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:03 smithi153 bash[23852]: cluster 2024-04-06T03:42:02.441000+0000 mgr.y (mgr.24368) 776 : cluster 0 pgmap v833: 148 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 141 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 871 op/s; 2043/93762 objects misplaced (2.179%); 81 MiB/s, 1.30k objects/s recovering 2024-04-06T03:42:03.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:03.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:03.862 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:03.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:03.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:03.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:03 smithi050 bash[21467]: cluster 2024-04-06T03:42:02.441000+0000 mgr.y (mgr.24368) 776 : cluster 0 pgmap v833: 148 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 141 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 871 op/s; 2043/93762 objects misplaced (2.179%); 81 MiB/s, 1.30k objects/s recovering 2024-04-06T03:42:03.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:03 smithi050 bash[29221]: cluster 2024-04-06T03:42:02.441000+0000 mgr.y (mgr.24368) 776 : cluster 0 pgmap v833: 148 pgs: 1 active+clean+remapped, 6 active+remapped+backfilling, 141 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 871 op/s; 2043/93762 objects misplaced (2.179%); 81 MiB/s, 1.30k objects/s recovering 2024-04-06T03:42:03.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:03.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:04.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:04.121 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:04.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:04.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:04.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:04.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:04.516 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:04.619 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:04.651 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:04.754 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:04.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:04 smithi153 bash[23852]: cluster 2024-04-06T03:42:03.537832+0000 mon.a (mon.0) 1587 : cluster 0 osdmap e323: 8 total, 8 up, 7 in 2024-04-06T03:42:04.762 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:04 smithi153 bash[23852]: cluster 2024-04-06T03:42:04.541011+0000 mon.a (mon.0) 1588 : cluster 0 osdmap e324: 8 total, 8 up, 7 in 2024-04-06T03:42:04.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:04.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:04.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:04.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:04.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:04 smithi050 bash[21467]: cluster 2024-04-06T03:42:03.537832+0000 mon.a (mon.0) 1587 : cluster 0 osdmap e323: 8 total, 8 up, 7 in 2024-04-06T03:42:04.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:04 smithi050 bash[21467]: cluster 2024-04-06T03:42:04.541011+0000 mon.a (mon.0) 1588 : cluster 0 osdmap e324: 8 total, 8 up, 7 in 2024-04-06T03:42:04.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:04 smithi050 bash[29221]: cluster 2024-04-06T03:42:03.537832+0000 mon.a (mon.0) 1587 : cluster 0 osdmap e323: 8 total, 8 up, 7 in 2024-04-06T03:42:04.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:04 smithi050 bash[29221]: cluster 2024-04-06T03:42:04.541011+0000 mon.a (mon.0) 1588 : cluster 0 osdmap e324: 8 total, 8 up, 7 in 2024-04-06T03:42:05.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:05.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:05.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:05.171 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:05.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:05.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:05.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:05.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:05.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:05.576 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:05.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:05.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:05.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:05 smithi153 bash[23852]: cluster 2024-04-06T03:42:04.442036+0000 mgr.y (mgr.24368) 777 : cluster 0 pgmap v836: 148 pgs: 1 peering, 3 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 355 op/s; 351/92385 objects misplaced (0.380%); 80 MiB/s, 1.28k objects/s recovering 2024-04-06T03:42:05.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:05 smithi153 bash[23852]: audit 2024-04-06T03:42:05.183862+0000 mon.a (mon.0) 1589 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:05.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:05.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:05 smithi050 bash[21467]: cluster 2024-04-06T03:42:04.442036+0000 mgr.y (mgr.24368) 777 : cluster 0 pgmap v836: 148 pgs: 1 peering, 3 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 355 op/s; 351/92385 objects misplaced (0.380%); 80 MiB/s, 1.28k objects/s recovering 2024-04-06T03:42:05.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:05 smithi050 bash[21467]: audit 2024-04-06T03:42:05.183862+0000 mon.a (mon.0) 1589 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:05.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:05 smithi050 bash[29221]: cluster 2024-04-06T03:42:04.442036+0000 mgr.y (mgr.24368) 777 : cluster 0 pgmap v836: 148 pgs: 1 peering, 3 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 355 op/s; 351/92385 objects misplaced (0.380%); 80 MiB/s, 1.28k objects/s recovering 2024-04-06T03:42:05.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:05 smithi050 bash[29221]: audit 2024-04-06T03:42:05.183862+0000 mon.a (mon.0) 1589 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:05.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:05.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:05.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:05.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:06.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:06.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:06.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:06.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:06.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:06.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:06.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:06.488 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:06.590 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:06.615 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:06.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:06.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:06.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:06.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:06.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:06.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:06.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:07.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:07.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:07.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:07.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:07.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:07.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:07.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:07.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:07.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:07.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:07.758 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:07 smithi153 bash[23852]: cluster 2024-04-06T03:42:06.442643+0000 mgr.y (mgr.24368) 778 : cluster 0 pgmap v838: 148 pgs: 1 peering, 3 active+remapped+backfilling, 144 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 344 op/s; 351/91566 objects misplaced (0.383%); 65 MiB/s, 1.04k objects/s recovering 2024-04-06T03:42:07.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:07.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:07.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:07 smithi050 bash[21467]: cluster 2024-04-06T03:42:06.442643+0000 mgr.y (mgr.24368) 778 : cluster 0 pgmap v838: 148 pgs: 1 peering, 3 active+remapped+backfilling, 144 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 344 op/s; 351/91566 objects misplaced (0.383%); 65 MiB/s, 1.04k objects/s recovering 2024-04-06T03:42:07.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:07 smithi050 bash[29221]: cluster 2024-04-06T03:42:06.442643+0000 mgr.y (mgr.24368) 778 : cluster 0 pgmap v838: 148 pgs: 1 peering, 3 active+remapped+backfilling, 144 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 344 op/s; 351/91566 objects misplaced (0.383%); 65 MiB/s, 1.04k objects/s recovering 2024-04-06T03:42:07.905 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:07.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:08.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:08.049 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:08.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:08.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:08.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:08.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:08.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:08.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:08.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:08.571 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:08.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:08.703 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:08.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:08.830 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:08.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:08.966 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:08.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:09.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:09.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:09.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:09.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:09.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:09.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:09.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:09.586 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:42:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:42:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:42:09.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:09.600 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:09 smithi153 bash[23852]: cluster 2024-04-06T03:42:08.443283+0000 mgr.y (mgr.24368) 779 : cluster 0 pgmap v839: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 269 op/s; 21 MiB/s, 332 objects/s recovering 2024-04-06T03:42:09.617 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:09.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:09.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:09.850 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:09.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:09.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:09.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:09.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:09 smithi050 bash[21467]: cluster 2024-04-06T03:42:08.443283+0000 mgr.y (mgr.24368) 779 : cluster 0 pgmap v839: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 269 op/s; 21 MiB/s, 332 objects/s recovering 2024-04-06T03:42:09.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:09 smithi050 bash[29221]: cluster 2024-04-06T03:42:08.443283+0000 mgr.y (mgr.24368) 779 : cluster 0 pgmap v839: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 269 op/s; 21 MiB/s, 332 objects/s recovering 2024-04-06T03:42:10.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:10.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:10.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:10.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:10.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:10.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:42:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:42:10.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:10.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:10.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:10.545 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:10.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:10.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:10.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:10.812 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:10.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:10.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:10.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:11.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.065 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:11.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.196 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:11.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:11.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:11.428 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:11.458 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:11.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:11.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:11.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:11 smithi153 bash[23852]: audit 2024-04-06T03:42:10.068356+0000 mgr.y (mgr.24368) 780 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:11.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:11 smithi153 bash[23852]: cluster 2024-04-06T03:42:10.444544+0000 mgr.y (mgr.24368) 781 : cluster 0 pgmap v840: 148 pgs: 148 active+clean; 1.7 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 502 op/s; 16 MiB/s, 251 objects/s recovering 2024-04-06T03:42:11.722 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:11.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.856 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:11.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:11 smithi050 bash[21467]: audit 2024-04-06T03:42:10.068356+0000 mgr.y (mgr.24368) 780 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:11.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:11 smithi050 bash[21467]: cluster 2024-04-06T03:42:10.444544+0000 mgr.y (mgr.24368) 781 : cluster 0 pgmap v840: 148 pgs: 148 active+clean; 1.7 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 502 op/s; 16 MiB/s, 251 objects/s recovering 2024-04-06T03:42:11.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:11 smithi050 bash[29221]: audit 2024-04-06T03:42:10.068356+0000 mgr.y (mgr.24368) 780 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:11.944 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:11 smithi050 bash[29221]: cluster 2024-04-06T03:42:10.444544+0000 mgr.y (mgr.24368) 781 : cluster 0 pgmap v840: 148 pgs: 148 active+clean; 1.7 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 645 B/s rd, 0 B/s wr, 502 op/s; 16 MiB/s, 251 objects/s recovering 2024-04-06T03:42:11.960 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:11.982 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:11.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:12.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:12.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:12.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:12.249 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:12.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:12.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:12.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:12.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:12.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:12.645 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:12.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:12.776 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:12.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:12.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:12.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:13.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:13.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:13.167 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:13.194 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:13.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:13.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:13.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:13.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:13.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:13.602 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:13.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:13 smithi050 bash[21467]: cluster 2024-04-06T03:42:12.446620+0000 mgr.y (mgr.24368) 782 : cluster 0 pgmap v841: 148 pgs: 148 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 600 op/s; 14 MiB/s, 223 objects/s recovering 2024-04-06T03:42:13.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:13.725 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:13.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:13.838 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:13 smithi153 bash[23852]: cluster 2024-04-06T03:42:12.446620+0000 mgr.y (mgr.24368) 782 : cluster 0 pgmap v841: 148 pgs: 148 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 600 op/s; 14 MiB/s, 223 objects/s recovering 2024-04-06T03:42:13.857 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:13.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:13.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:13 smithi050 bash[29221]: cluster 2024-04-06T03:42:12.446620+0000 mgr.y (mgr.24368) 782 : cluster 0 pgmap v841: 148 pgs: 148 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 600 op/s; 14 MiB/s, 223 objects/s recovering 2024-04-06T03:42:13.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:13.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:13.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:14.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:14.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:14.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:14.243 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:14.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:14.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:14.495 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:14.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:14.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:14.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:14.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:14.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:14.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:14.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:15.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:15.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:15.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:15.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:14 smithi050 bash[21467]: cluster 2024-04-06T03:42:14.447533+0000 mgr.y (mgr.24368) 783 : cluster 0 pgmap v842: 148 pgs: 148 active+clean; 1.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 641 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-04-06T03:42:15.152 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:14 smithi050 bash[29221]: cluster 2024-04-06T03:42:14.447533+0000 mgr.y (mgr.24368) 783 : cluster 0 pgmap v842: 148 pgs: 148 active+clean; 1.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 641 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-04-06T03:42:15.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:15.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:14 smithi153 bash[23852]: cluster 2024-04-06T03:42:14.447533+0000 mgr.y (mgr.24368) 783 : cluster 0 pgmap v842: 148 pgs: 148 active+clean; 1.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 641 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-04-06T03:42:15.300 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:15.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:15.427 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:15.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:15.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:15.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:15.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:15.810 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:15.829 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:15.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:15.951 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:15.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:16.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:16.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:16.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:16.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:16.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:16.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:16.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:16.482 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:16.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:16.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:16.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:16.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:16.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:16.901 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:17.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:17.030 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:17.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:17.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:17.404 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:17.442 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:17.548 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:17.572 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:17.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:17.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:17.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:17 smithi153 bash[23852]: cluster 2024-04-06T03:42:16.448194+0000 mgr.y (mgr.24368) 784 : cluster 0 pgmap v843: 148 pgs: 148 active+clean; 1.4 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 859 B/s rd, 0 B/s wr, 626 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-04-06T03:42:17.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:17.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:17.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:17.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:17 smithi050 bash[21467]: cluster 2024-04-06T03:42:16.448194+0000 mgr.y (mgr.24368) 784 : cluster 0 pgmap v843: 148 pgs: 148 active+clean; 1.4 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 859 B/s rd, 0 B/s wr, 626 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-04-06T03:42:17.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:17 smithi050 bash[29221]: cluster 2024-04-06T03:42:16.448194+0000 mgr.y (mgr.24368) 784 : cluster 0 pgmap v843: 148 pgs: 148 active+clean; 1.4 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 859 B/s rd, 0 B/s wr, 626 op/s; 1.8 MiB/s, 29 objects/s recovering 2024-04-06T03:42:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:17.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:18.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.089 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:18.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:18.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:18.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:18.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:18.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:18.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:18.708 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:18.740 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:18.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.868 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:18.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:18.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:19.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:19.123 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:19.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:19.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:19.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:19.366 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:19.391 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:19.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:19.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:42:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:42:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:42:19.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:19.522 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:19.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:19.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:19.715 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:19 smithi050 bash[21467]: cluster 2024-04-06T03:42:18.448888+0000 mgr.y (mgr.24368) 785 : cluster 0 pgmap v844: 148 pgs: 148 active+clean; 1.3 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-06T03:42:19.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:19 smithi050 bash[29221]: cluster 2024-04-06T03:42:18.448888+0000 mgr.y (mgr.24368) 785 : cluster 0 pgmap v844: 148 pgs: 148 active+clean; 1.3 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-06T03:42:19.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:19.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:19.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:19 smithi153 bash[23852]: cluster 2024-04-06T03:42:18.448888+0000 mgr.y (mgr.24368) 785 : cluster 0 pgmap v844: 148 pgs: 148 active+clean; 1.3 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 748 op/s; 1.8 MiB/s, 28 objects/s recovering 2024-04-06T03:42:19.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:19.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:20.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:20.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:20.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:20.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:20.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:20.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:20.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:20.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:20.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:42:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:42:20.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:20.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:20.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:20.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:20.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:20 smithi153 bash[23852]: audit 2024-04-06T03:42:20.218650+0000 mon.a (mon.0) 1590 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:20.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:20.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:20 smithi050 bash[21467]: audit 2024-04-06T03:42:20.218650+0000 mon.a (mon.0) 1590 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:20.851 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:20.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:20 smithi050 bash[29221]: audit 2024-04-06T03:42:20.218650+0000 mon.a (mon.0) 1590 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:20.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:20.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:20.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:21.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:21.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:21.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:21.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:21.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:21.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:21.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:21.483 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:21.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:21.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:21.638 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:21.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[21467]: cluster 2024-04-06T03:42:20.062993+0000 osd.4 (osd.4) 494 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[21467]: cluster 2024-04-06T03:42:20.063113+0000 osd.4 (osd.4) 495 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[21467]: audit 2024-04-06T03:42:20.076507+0000 mgr.y (mgr.24368) 786 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[21467]: cluster 2024-04-06T03:42:20.449655+0000 mgr.y (mgr.24368) 787 : cluster 0 pgmap v845: 148 pgs: 148 active+clean; 1.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 899 op/s 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[21467]: cluster 2024-04-06T03:42:20.516594+0000 mon.a (mon.0) 1591 : cluster 0 osdmap e325: 8 total, 8 up, 7 in 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[29221]: cluster 2024-04-06T03:42:20.062993+0000 osd.4 (osd.4) 494 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[29221]: cluster 2024-04-06T03:42:20.063113+0000 osd.4 (osd.4) 495 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[29221]: audit 2024-04-06T03:42:20.076507+0000 mgr.y (mgr.24368) 786 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[29221]: cluster 2024-04-06T03:42:20.449655+0000 mgr.y (mgr.24368) 787 : cluster 0 pgmap v845: 148 pgs: 148 active+clean; 1.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 899 op/s 2024-04-06T03:42:21.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:21 smithi050 bash[29221]: cluster 2024-04-06T03:42:20.516594+0000 mon.a (mon.0) 1591 : cluster 0 osdmap e325: 8 total, 8 up, 7 in 2024-04-06T03:42:21.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:21.750 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:21 smithi153 bash[23852]: cluster 2024-04-06T03:42:20.062993+0000 osd.4 (osd.4) 494 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:21.751 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:21 smithi153 bash[23852]: cluster 2024-04-06T03:42:20.063113+0000 osd.4 (osd.4) 495 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:21.751 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:21 smithi153 bash[23852]: audit 2024-04-06T03:42:20.076507+0000 mgr.y (mgr.24368) 786 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:21.751 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:21 smithi153 bash[23852]: cluster 2024-04-06T03:42:20.449655+0000 mgr.y (mgr.24368) 787 : cluster 0 pgmap v845: 148 pgs: 148 active+clean; 1.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 899 op/s 2024-04-06T03:42:21.751 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:21 smithi153 bash[23852]: cluster 2024-04-06T03:42:20.516594+0000 mon.a (mon.0) 1591 : cluster 0 osdmap e325: 8 total, 8 up, 7 in 2024-04-06T03:42:21.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:21.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:21.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:21.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:22.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:22.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:22.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:22.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:22.305 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15306.0:181503 10.f e325/322 10:f645826f:::benchmark_data_smithi050_121293_object82686:head v 325'11233, mlcod=325'11231)", 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.367348+0000", 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010680670999999999, 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010712490999999999, 2024-04-06T03:42:22.379 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367348+0000", 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367348+0000", 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367348+0000", 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9400000000000001e-07 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367353+0000", 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9269999999999996e-06 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367359+0000", 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2029999999999997e-06 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367364+0000", 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2159999999999997e-06 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367917+0000", 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000553186 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367941+0000", 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3340999999999999e-05 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.382 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15306.0:181505 10.7 e325/323 10:e1939ce7:::benchmark_data_smithi050_121293_object82688:head v 325'11422, mlcod=325'11420)", 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.367406+0000", 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010622751, 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010724993, 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367406+0000", 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367406+0000", 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367406+0000", 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7099999999999998e-07 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367410+0000", 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9820000000000002e-06 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367416+0000", 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8409999999999996e-06 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367420+0000", 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7559999999999999e-06 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.370316+0000", 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002895781 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.370330+0000", 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3874e-05 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15306.0:181506 10.f e325/322 10:fe55927c:::benchmark_data_smithi050_121293_object82689:head v 325'11234, mlcod=325'11231)", 2024-04-06T03:42:22.386 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.367453+0000", 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010575644, 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010742846, 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367453+0000", 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367453+0000", 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367453+0000", 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9200000000000001e-07 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367464+0000", 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0499000000000001e-05 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367493+0000", 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9264e-05 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367507+0000", 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4426999999999999e-05 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.372026+0000", 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045190250000000003 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.372045+0000", 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8493000000000001e-05 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181508 10.5 10:a2f65d98:::benchmark_data_smithi050_121293_object82691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.367839+0000", 2024-04-06T03:42:22.390 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.010188825, 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010429384, 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181508 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367839+0000", 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367839+0000", 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367841+0000", 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1799999999999999e-06 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367845+0000", 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.651e-06 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367858+0000", 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2758999999999999e-05 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367868+0000", 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7880000000000001e-06 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.368260+0000", 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039258899999999999 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.368611+0000", 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035020400000000001 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.369061+0000", 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045028100000000002 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373564+0000", 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045027840000000001 2024-04-06T03:42:22.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181512 10.3 10:cc0df8c5:::benchmark_data_smithi050_121293_object82695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.373506+0000", 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0045219450000000003, 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048493199999999998, 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181512 2024-04-06T03:42:22.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373506+0000", 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373506+0000", 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373508+0000", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4130000000000001e-06 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373514+0000", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.964e-06 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373560+0000", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6162000000000003e-05 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373571+0000", 2024-04-06T03:42:22.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1015e-05 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373774+0000", 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020275800000000001 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374133+0000", 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000359582 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-06T03:42:22.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374264+0000", 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131166 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181513 10.9 10.638cb699 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.373687+0000", 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0043411140000000001, 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047435369999999999, 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.399 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181513 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373687+0000", 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373687+0000", 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373688+0000", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.61e-07 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373692+0000", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1840000000000001e-06 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373702+0000", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0434999999999999e-05 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373711+0000", 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8210000000000008e-06 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181514 10.3 10:c9602ff7:::benchmark_data_smithi050_121293_object82697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.373982+0000", 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.0040461359999999997, 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045075009999999997, 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:42:22.402 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181514 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373982+0000", 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373982+0000", 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.373990+0000", 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1769999999999998e-06 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374003+0000", 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2342e-05 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374022+0000", 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9799000000000001e-05 2024-04-06T03:42:22.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374031+0000", 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1249999999999993e-06 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377277+0000", 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003246526 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377405+0000", 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012800600000000001 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377522+0000", 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011651500000000001 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15306.0:181511 10.7 e325/323)", 2024-04-06T03:42:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.374623+0000", 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.003405567, 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039406140000000003, 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374623+0000", 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374623+0000", 2024-04-06T03:42:22.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374624+0000", 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374630+0000", 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6019999999999996e-06 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374671+0000", 2024-04-06T03:42:22.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0933000000000003e-05 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374678+0000", 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1790000000000002e-06 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181485 10.5 10:adda0309:::benchmark_data_smithi050_121293_object82668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.342976+0000", 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.035051976999999998, 2024-04-06T03:42:22.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035640619999999998, 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181485 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.342976+0000", 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.342976+0000", 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.342978+0000", 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7010000000000001e-06 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.342983+0000", 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.186e-06 2024-04-06T03:42:22.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.342995+0000", 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1846e-05 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343003+0000", 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.79e-06 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343236+0000", 2024-04-06T03:42:22.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023300799999999999 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343457+0000", 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022084599999999999 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343569+0000", 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112549 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.360395+0000", 2024-04-06T03:42:22.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016826154999999999 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.360416+0000", 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0526999999999998e-05 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181516 10.9 10.941f6729 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.374742+0000", 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.003286663, 2024-04-06T03:42:22.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039619410000000001, 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181516 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374742+0000", 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374742+0000", 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374742+0000", 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4700000000000001e-07 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374746+0000", 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5710000000000002e-06 2024-04-06T03:42:22.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374757+0000", 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0910999999999999e-05 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.374764+0000", 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.968e-06 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.417 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181486 10.5 10:a2739667:::benchmark_data_smithi050_121293_object82669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.343020+0000", 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.035007986999999997, 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035742031, 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181486 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343020+0000", 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343020+0000", 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343021+0000", 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3700000000000001e-07 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343023+0000", 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6699999999999998e-06 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343030+0000", 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9140000000000002e-06 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343075+0000", 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4984999999999997e-05 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.346329+0000", 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032541520000000002 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.346436+0000", 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000106228 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.346553+0000", 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011779500000000001 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:42:22.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.360424+0000", 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013870357 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367377+0000", 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069530870000000002 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15306.0:181515 10.7 e325/323)", 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.377392+0000", 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00063609199999999999, 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001453661, 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377392+0000", 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377392+0000", 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377393+0000", 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9500000000000001e-07 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377399+0000", 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4629999999999998e-06 2024-04-06T03:42:22.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377411+0000", 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1068e-05 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377419+0000", 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3999999999999992e-06 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:181488 10.5 10:a4c97768:::benchmark_data_smithi050_121293_object82671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:42:22.425 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.343100+0000", 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.034928343000000001, 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035798386000000001, 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 181488 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343100+0000", 2024-04-06T03:42:22.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343100+0000", 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343101+0000", 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6199999999999997e-07 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343105+0000", 2024-04-06T03:42:22.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1309999999999999e-06 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343114+0000", 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8619999999999992e-06 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.343120+0000", 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8589999999999999e-06 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.349832+0000", 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067111540000000004 2024-04-06T03:42:22.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.349918+0000", 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6495000000000004e-05 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.349990+0000", 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1826e-05 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.360429+0000", 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010439086 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.367421+0000", 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069921380000000002 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15306.0:181512 10.3 e325/280)", 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.377473+0000", 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.00055479800000000001, 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015134860000000001, 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.430 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377473+0000", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377473+0000", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377474+0000", 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6400000000000002e-07 2024-04-06T03:42:22.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377478+0000", 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6789999999999998e-06 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377488+0000", 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8779999999999993e-06 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.377494+0000", 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7630000000000003e-06 2024-04-06T03:42:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(client.15306.0:181508 10.5 e325/280)", 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:22.379008+0000", 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4294967295.9990206, 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2086000000000002e-05, 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "queued for pg", 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:42:22.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.379008+0000", 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.379008+0000", 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.379008+0000", 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6000000000000004e-07 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:42:22.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.379012+0000", 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.484e-06 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.379020+0000", 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9680000000000006e-06 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:22.379024+0000", 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8470000000000003e-06 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-06T03:42:22.435 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 15 2024-04-06T03:42:22.436 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:42:22.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:22.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:22 smithi050 bash[21467]: debug 2024-04-06T03:42:22.086+0000 7f702b621700 -1 mon.a@0(leader).osd e325 definitely_dead 0 2024-04-06T03:42:22.466 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:22.545 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.6. 2024-04-06T03:42:22.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:22.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:22.718 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:22 smithi050 bash[21467]: audit 2024-04-06T03:42:22.088504+0000 mon.a (mon.0) 1592 : audit 0 from='client.? 172.21.15.50:0/190986364' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-06T03:42:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:22 smithi050 bash[29221]: audit 2024-04-06T03:42:22.088504+0000 mon.a (mon.0) 1592 : audit 0 from='client.? 172.21.15.50:0/190986364' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-06T03:42:22.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:22 smithi153 bash[23852]: audit 2024-04-06T03:42:22.088504+0000 mon.a (mon.0) 1592 : audit 0 from='client.? 172.21.15.50:0/190986364' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-04-06T03:42:22.809 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:42:22.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:22.830 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:42:22.840 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:22.851 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:42:22.870 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:42:22.894 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:42:22.918 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:42:22.942 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:42:22.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:22.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:22.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:22.967 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:42:22.968 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:42:22.968 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:42:22.968 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:42:22.968 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:42:22.969 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:42:22.969 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:42:22.969 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:42:22.969 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:42:23.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:23.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:23.224 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:23.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:23.352 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:23.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:23.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:23.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:23.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:23.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[21467]: cluster 2024-04-06T03:42:22.254684+0000 osd.0 (osd.0) 513 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[21467]: cluster 2024-04-06T03:42:22.254751+0000 osd.0 (osd.0) 514 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[21467]: cluster 2024-04-06T03:42:22.450221+0000 mgr.y (mgr.24368) 788 : cluster 0 pgmap v847: 148 pgs: 148 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[21467]: audit 2024-04-06T03:42:22.533353+0000 mon.a (mon.0) 1593 : audit 1 from='client.? 172.21.15.50:0/190986364' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[21467]: cluster 2024-04-06T03:42:22.539870+0000 mon.a (mon.0) 1594 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[21467]: audit 2024-04-06T03:42:22.789616+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.50:0/630259053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[29221]: cluster 2024-04-06T03:42:22.254684+0000 osd.0 (osd.0) 513 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:23.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[29221]: cluster 2024-04-06T03:42:22.254751+0000 osd.0 (osd.0) 514 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:23.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[29221]: cluster 2024-04-06T03:42:22.450221+0000 mgr.y (mgr.24368) 788 : cluster 0 pgmap v847: 148 pgs: 148 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:42:23.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[29221]: audit 2024-04-06T03:42:22.533353+0000 mon.a (mon.0) 1593 : audit 1 from='client.? 172.21.15.50:0/190986364' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-06T03:42:23.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[29221]: cluster 2024-04-06T03:42:22.539870+0000 mon.a (mon.0) 1594 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-04-06T03:42:23.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:23 smithi050 bash[29221]: audit 2024-04-06T03:42:22.789616+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.50:0/630259053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:42:23.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:23.749 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:23 smithi153 bash[23852]: cluster 2024-04-06T03:42:22.254684+0000 osd.0 (osd.0) 513 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:23 smithi153 bash[23852]: cluster 2024-04-06T03:42:22.254751+0000 osd.0 (osd.0) 514 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:23 smithi153 bash[23852]: cluster 2024-04-06T03:42:22.450221+0000 mgr.y (mgr.24368) 788 : cluster 0 pgmap v847: 148 pgs: 148 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:42:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:23 smithi153 bash[23852]: audit 2024-04-06T03:42:22.533353+0000 mon.a (mon.0) 1593 : audit 1 from='client.? 172.21.15.50:0/190986364' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-04-06T03:42:23.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:23 smithi153 bash[23852]: cluster 2024-04-06T03:42:22.539870+0000 mon.a (mon.0) 1594 : cluster 0 osdmap e326: 8 total, 8 up, 8 in 2024-04-06T03:42:23.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:23 smithi153 bash[23852]: audit 2024-04-06T03:42:22.789616+0000 mon.c (mon.2) 98 : audit 0 from='client.? 172.21.15.50:0/630259053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:42:23.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:23.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:23.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:23.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:23.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:24.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:24.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:24.224 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:24.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:24.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:24.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:24.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:24.553 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:42:24.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:24.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:24.779 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:24.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[21467]: cluster 2024-04-06T03:42:23.541911+0000 mon.a (mon.0) 1595 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[21467]: audit 2024-04-06T03:42:24.107375+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.50:0/370323708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[21467]: audit 2024-04-06T03:42:24.107891+0000 mon.a (mon.0) 1596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.535944+0000 mon.a (mon.0) 1597 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[21467]: audit 2024-04-06T03:42:24.537949+0000 mon.a (mon.0) 1598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.542082+0000 mon.a (mon.0) 1599 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[29221]: cluster 2024-04-06T03:42:23.541911+0000 mon.a (mon.0) 1595 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[29221]: audit 2024-04-06T03:42:24.107375+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.50:0/370323708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:42:24.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[29221]: audit 2024-04-06T03:42:24.107891+0000 mon.a (mon.0) 1596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:42:24.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.535944+0000 mon.a (mon.0) 1597 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:42:24.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[29221]: audit 2024-04-06T03:42:24.537949+0000 mon.a (mon.0) 1598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:42:24.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:24 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.542082+0000 mon.a (mon.0) 1599 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-04-06T03:42:24.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:24.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:25.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:25.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:24 smithi153 bash[23852]: cluster 2024-04-06T03:42:23.541911+0000 mon.a (mon.0) 1595 : cluster 0 osdmap e327: 8 total, 8 up, 8 in 2024-04-06T03:42:25.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:24 smithi153 bash[23852]: audit 2024-04-06T03:42:24.107375+0000 mon.c (mon.2) 99 : audit 1 from='client.? 172.21.15.50:0/370323708' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:42:25.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:24 smithi153 bash[23852]: audit 2024-04-06T03:42:24.107891+0000 mon.a (mon.0) 1596 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:42:25.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:24 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.535944+0000 mon.a (mon.0) 1597 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:42:25.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:24 smithi153 bash[23852]: audit 2024-04-06T03:42:24.537949+0000 mon.a (mon.0) 1598 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:42:25.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:24 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.542082+0000 mon.a (mon.0) 1599 : cluster 0 osdmap e328: 8 total, 8 up, 8 in 2024-04-06T03:42:25.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:25.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:25.182 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:25.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:25.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:25.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:25.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:25.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:25.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:25.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:25.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.451190+0000 mgr.y (mgr.24368) 789 : cluster 0 pgmap v850: 148 pgs: 4 remapped+peering, 144 active+clean; 1.0 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 797 op/s; 117 B/s, 5 objects/s recovering 2024-04-06T03:42:25.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.542041+0000 osd.1 (osd.1) 394 : cluster 0 10.b starting backfill to osd.6 from (0'0,0'0] MAX to 325'11433 2024-04-06T03:42:25.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.545310+0000 osd.4 (osd.4) 496 : cluster 0 10.d starting backfill to osd.6 from (0'0,0'0] MAX to 325'11053 2024-04-06T03:42:25.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.545458+0000 osd.4 (osd.4) 497 : cluster 0 10.0 starting backfill to osd.6 from (0'0,0'0] MAX to 325'11340 2024-04-06T03:42:25.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.547518+0000 osd.3 (osd.3) 518 : cluster 0 10.f starting backfill to osd.6 from (0'0,0'0] MAX to 325'11250 2024-04-06T03:42:25.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.547552+0000 osd.3 (osd.3) 519 : cluster 0 10.7 starting backfill to osd.6 from (0'0,0'0] MAX to 325'11432 2024-04-06T03:42:25.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.552811+0000 osd.0 (osd.0) 515 : cluster 0 10.a starting backfill to osd.6 from (0'0,0'0] MAX to 325'11348 2024-04-06T03:42:25.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:25 smithi153 bash[23852]: cluster 2024-04-06T03:42:24.555172+0000 osd.4 (osd.4) 498 : cluster 0 10.e starting backfill to osd.6 from (0'0,0'0] MAX to 325'11077 2024-04-06T03:42:25.714 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:25.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:25.829 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.451190+0000 mgr.y (mgr.24368) 789 : cluster 0 pgmap v850: 148 pgs: 4 remapped+peering, 144 active+clean; 1.0 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 797 op/s; 117 B/s, 5 objects/s recovering 2024-04-06T03:42:25.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.542041+0000 osd.1 (osd.1) 394 : cluster 0 10.b starting backfill to osd.6 from (0'0,0'0] MAX to 325'11433 2024-04-06T03:42:25.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.545310+0000 osd.4 (osd.4) 496 : cluster 0 10.d starting backfill to osd.6 from (0'0,0'0] MAX to 325'11053 2024-04-06T03:42:25.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.545458+0000 osd.4 (osd.4) 497 : cluster 0 10.0 starting backfill to osd.6 from (0'0,0'0] MAX to 325'11340 2024-04-06T03:42:25.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.547518+0000 osd.3 (osd.3) 518 : cluster 0 10.f starting backfill to osd.6 from (0'0,0'0] MAX to 325'11250 2024-04-06T03:42:25.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.547552+0000 osd.3 (osd.3) 519 : cluster 0 10.7 starting backfill to osd.6 from (0'0,0'0] MAX to 325'11432 2024-04-06T03:42:25.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.552811+0000 osd.0 (osd.0) 515 : cluster 0 10.a starting backfill to osd.6 from (0'0,0'0] MAX to 325'11348 2024-04-06T03:42:25.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[21467]: cluster 2024-04-06T03:42:24.555172+0000 osd.4 (osd.4) 498 : cluster 0 10.e starting backfill to osd.6 from (0'0,0'0] MAX to 325'11077 2024-04-06T03:42:25.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.451190+0000 mgr.y (mgr.24368) 789 : cluster 0 pgmap v850: 148 pgs: 4 remapped+peering, 144 active+clean; 1.0 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 797 op/s; 117 B/s, 5 objects/s recovering 2024-04-06T03:42:25.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.542041+0000 osd.1 (osd.1) 394 : cluster 0 10.b starting backfill to osd.6 from (0'0,0'0] MAX to 325'11433 2024-04-06T03:42:25.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.545310+0000 osd.4 (osd.4) 496 : cluster 0 10.d starting backfill to osd.6 from (0'0,0'0] MAX to 325'11053 2024-04-06T03:42:25.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.545458+0000 osd.4 (osd.4) 497 : cluster 0 10.0 starting backfill to osd.6 from (0'0,0'0] MAX to 325'11340 2024-04-06T03:42:25.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.547518+0000 osd.3 (osd.3) 518 : cluster 0 10.f starting backfill to osd.6 from (0'0,0'0] MAX to 325'11250 2024-04-06T03:42:25.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.547552+0000 osd.3 (osd.3) 519 : cluster 0 10.7 starting backfill to osd.6 from (0'0,0'0] MAX to 325'11432 2024-04-06T03:42:25.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.552811+0000 osd.0 (osd.0) 515 : cluster 0 10.a starting backfill to osd.6 from (0'0,0'0] MAX to 325'11348 2024-04-06T03:42:25.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:25 smithi050 bash[29221]: cluster 2024-04-06T03:42:24.555172+0000 osd.4 (osd.4) 498 : cluster 0 10.e starting backfill to osd.6 from (0'0,0'0] MAX to 325'11077 2024-04-06T03:42:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:25.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:25.985 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:25.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:26.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:26.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:26.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:26.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:26.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:26.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:26.485 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:26.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:26.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:26.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:26.655 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:26 smithi050 bash[21467]: cluster 2024-04-06T03:42:25.561877+0000 mon.a (mon.0) 1600 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-04-06T03:42:26.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:26 smithi050 bash[29221]: cluster 2024-04-06T03:42:25.561877+0000 mon.a (mon.0) 1600 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-04-06T03:42:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:26.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:26.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:26 smithi153 bash[23852]: cluster 2024-04-06T03:42:25.561877+0000 mon.a (mon.0) 1600 : cluster 0 osdmap e329: 8 total, 8 up, 8 in 2024-04-06T03:42:26.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:26.905 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:26.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:27.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:27.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:27.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:27.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:27.281 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:27.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:27.446 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:27.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:27.577 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:27.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:27.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:27.715 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:27 smithi050 bash[21467]: cluster 2024-04-06T03:42:26.451662+0000 mgr.y (mgr.24368) 790 : cluster 0 pgmap v853: 148 pgs: 5 remapped+peering, 143 active+clean; 1003 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 310 op/s; 505 B/s, 14 objects/s recovering 2024-04-06T03:42:27.715 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:27 smithi050 bash[21467]: cluster 2024-04-06T03:42:26.573790+0000 mon.a (mon.0) 1601 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-04-06T03:42:27.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:27 smithi050 bash[29221]: cluster 2024-04-06T03:42:26.451662+0000 mgr.y (mgr.24368) 790 : cluster 0 pgmap v853: 148 pgs: 5 remapped+peering, 143 active+clean; 1003 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 310 op/s; 505 B/s, 14 objects/s recovering 2024-04-06T03:42:27.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:27 smithi050 bash[29221]: cluster 2024-04-06T03:42:26.573790+0000 mon.a (mon.0) 1601 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-04-06T03:42:27.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:27.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:27.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:27 smithi153 bash[23852]: cluster 2024-04-06T03:42:26.451662+0000 mgr.y (mgr.24368) 790 : cluster 0 pgmap v853: 148 pgs: 5 remapped+peering, 143 active+clean; 1003 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 310 op/s; 505 B/s, 14 objects/s recovering 2024-04-06T03:42:27.955 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:27 smithi153 bash[23852]: cluster 2024-04-06T03:42:26.573790+0000 mon.a (mon.0) 1601 : cluster 0 osdmap e330: 8 total, 8 up, 8 in 2024-04-06T03:42:27.972 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:27.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:28.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:28.102 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:28.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-06T03:42:28.138 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-04-06T03:42:28.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:28.240 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:28.344 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:28.373 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:28.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:28.507 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:28.613 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:28.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:28 smithi050 bash[21467]: cluster 2024-04-06T03:42:28.573308+0000 mon.a (mon.0) 1602 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-06T03:42:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:28 smithi050 bash[29221]: cluster 2024-04-06T03:42:28.573308+0000 mon.a (mon.0) 1602 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-06T03:42:28.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:28.756 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:28 smithi153 bash[23852]: cluster 2024-04-06T03:42:28.573308+0000 mon.a (mon.0) 1602 : cluster 3 Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-04-06T03:42:28.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:28.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:28.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:28.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:29.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.056 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:29.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:29.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:29.301 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:29.329 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:29.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:29.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:42:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:42:29.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:29.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.726 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:29.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:29.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:29 smithi153 bash[23852]: cluster 2024-04-06T03:42:28.452279+0000 mgr.y (mgr.24368) 791 : cluster 0 pgmap v855: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 remapped+peering, 140 active+clean; 948 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 434 op/s; 2628/46155 objects misplaced (5.694%); 420 B/s, 16 objects/s recovering 2024-04-06T03:42:29.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:29 smithi153 bash[23852]: cluster 2024-04-06T03:42:28.882734+0000 osd.2 (osd.2) 411 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:29.841 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:29 smithi153 bash[23852]: cluster 2024-04-06T03:42:28.882800+0000 osd.2 (osd.2) 412 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:29.853 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:29.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:29.963 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[29221]: cluster 2024-04-06T03:42:28.452279+0000 mgr.y (mgr.24368) 791 : cluster 0 pgmap v855: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 remapped+peering, 140 active+clean; 948 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 434 op/s; 2628/46155 objects misplaced (5.694%); 420 B/s, 16 objects/s recovering 2024-04-06T03:42:29.963 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[29221]: cluster 2024-04-06T03:42:28.882734+0000 osd.2 (osd.2) 411 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:29.963 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[29221]: cluster 2024-04-06T03:42:28.882800+0000 osd.2 (osd.2) 412 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:29.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[21467]: cluster 2024-04-06T03:42:28.452279+0000 mgr.y (mgr.24368) 791 : cluster 0 pgmap v855: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 remapped+peering, 140 active+clean; 948 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 434 op/s; 2628/46155 objects misplaced (5.694%); 420 B/s, 16 objects/s recovering 2024-04-06T03:42:29.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[21467]: cluster 2024-04-06T03:42:28.882734+0000 osd.2 (osd.2) 411 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:29.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:29 smithi050 bash[21467]: cluster 2024-04-06T03:42:28.882800+0000 osd.2 (osd.2) 412 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:29.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:29.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:30.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:30.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:30.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:42:30.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:30.214 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:42:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:42:30.226 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:30.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:30.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:30.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:42:30.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:30.468 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:30.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:30.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:30.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:30.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:30.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:30 smithi153 bash[23852]: cluster 2024-04-06T03:42:29.641555+0000 mon.a (mon.0) 1603 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-04-06T03:42:30.874 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:30.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:30.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:30 smithi050 bash[21467]: cluster 2024-04-06T03:42:29.641555+0000 mon.a (mon.0) 1603 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-04-06T03:42:30.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:30 smithi050 bash[29221]: cluster 2024-04-06T03:42:29.641555+0000 mon.a (mon.0) 1603 : cluster 0 osdmap e331: 8 total, 8 up, 8 in 2024-04-06T03:42:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:30.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:31.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:31.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:31.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:31.247 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:31.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:31.376 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:31.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:31.508 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:31.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:31.634 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:31.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:31.750 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:31 smithi153 bash[23852]: audit 2024-04-06T03:42:30.084426+0000 mgr.y (mgr.24368) 792 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:31.750 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:31 smithi153 bash[23852]: cluster 2024-04-06T03:42:30.452835+0000 mgr.y (mgr.24368) 793 : cluster 0 pgmap v857: 148 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 141 active+clean; 792 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 635 op/s; 3496/38670 objects misplaced (9.041%); 27 MiB/s, 433 objects/s recovering 2024-04-06T03:42:31.750 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:31 smithi153 bash[23852]: cluster 2024-04-06T03:42:30.649898+0000 mon.a (mon.0) 1604 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-04-06T03:42:31.766 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:31.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:31.890 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:31.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:31.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:31 smithi050 bash[21467]: audit 2024-04-06T03:42:30.084426+0000 mgr.y (mgr.24368) 792 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:31 smithi050 bash[21467]: cluster 2024-04-06T03:42:30.452835+0000 mgr.y (mgr.24368) 793 : cluster 0 pgmap v857: 148 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 141 active+clean; 792 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 635 op/s; 3496/38670 objects misplaced (9.041%); 27 MiB/s, 433 objects/s recovering 2024-04-06T03:42:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:31 smithi050 bash[21467]: cluster 2024-04-06T03:42:30.649898+0000 mon.a (mon.0) 1604 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-04-06T03:42:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:31 smithi050 bash[29221]: audit 2024-04-06T03:42:30.084426+0000 mgr.y (mgr.24368) 792 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:31 smithi050 bash[29221]: cluster 2024-04-06T03:42:30.452835+0000 mgr.y (mgr.24368) 793 : cluster 0 pgmap v857: 148 pgs: 1 active+clean+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 141 active+clean; 792 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 635 op/s; 3496/38670 objects misplaced (9.041%); 27 MiB/s, 433 objects/s recovering 2024-04-06T03:42:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:31 smithi050 bash[29221]: cluster 2024-04-06T03:42:30.649898+0000 mon.a (mon.0) 1604 : cluster 0 osdmap e332: 8 total, 8 up, 8 in 2024-04-06T03:42:31.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:32.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:32.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:32.146 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:32.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:32.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:32.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:32.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:32.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:32.538 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:32.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:32.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:32.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:32 smithi153 bash[23852]: cluster 2024-04-06T03:42:31.657725+0000 mon.a (mon.0) 1605 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-04-06T03:42:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:32 smithi153 bash[23852]: cluster 2024-04-06T03:42:32.658903+0000 mon.a (mon.0) 1606 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-04-06T03:42:32.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:32.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:32.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:32 smithi050 bash[21467]: cluster 2024-04-06T03:42:31.657725+0000 mon.a (mon.0) 1605 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-04-06T03:42:32.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:32 smithi050 bash[21467]: cluster 2024-04-06T03:42:32.658903+0000 mon.a (mon.0) 1606 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-04-06T03:42:32.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:32 smithi050 bash[29221]: cluster 2024-04-06T03:42:31.657725+0000 mon.a (mon.0) 1605 : cluster 0 osdmap e333: 8 total, 8 up, 8 in 2024-04-06T03:42:32.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:32 smithi050 bash[29221]: cluster 2024-04-06T03:42:32.658903+0000 mon.a (mon.0) 1606 : cluster 0 osdmap e334: 8 total, 8 up, 8 in 2024-04-06T03:42:32.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:32.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:32.968 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:42:33.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:33.141 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:42:33.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:42:33.142 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 7, 6] 2024-04-06T03:42:33.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd out 1 2024-04-06T03:42:33.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:33.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:33.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:33.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:33.464 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:33.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.593 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:33.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:33.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:33 smithi153 bash[23852]: cluster 2024-04-06T03:42:32.454253+0000 mgr.y (mgr.24368) 794 : cluster 0 pgmap v860: 148 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 730 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 745 op/s; 2258/35694 objects misplaced (6.326%); 35 MiB/s, 557 objects/s recovering 2024-04-06T03:42:33.864 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:33.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:33.989 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:42:33.993 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:33.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:34.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:34.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:33 smithi050 bash[21467]: cluster 2024-04-06T03:42:32.454253+0000 mgr.y (mgr.24368) 794 : cluster 0 pgmap v860: 148 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 730 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 745 op/s; 2258/35694 objects misplaced (6.326%); 35 MiB/s, 557 objects/s recovering 2024-04-06T03:42:34.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:33 smithi050 bash[29221]: cluster 2024-04-06T03:42:32.454253+0000 mgr.y (mgr.24368) 794 : cluster 0 pgmap v860: 148 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 141 active+clean; 730 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 745 op/s; 2258/35694 objects misplaced (6.326%); 35 MiB/s, 557 objects/s recovering 2024-04-06T03:42:34.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:34.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:34.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:34.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:34.394 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:34.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:34.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:34.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:34.633 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:34.658 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:34.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:34.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:34.890 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:34.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:34.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:34.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:35.010 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:42:35.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:35.035 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:35.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:35.168 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:35.171 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:34 smithi153 bash[23852]: cluster 2024-04-06T03:42:34.454954+0000 mgr.y (mgr.24368) 795 : cluster 0 pgmap v862: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 145 active+clean; 685 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 518 op/s; 1390/33537 objects misplaced (4.145%); 20 MiB/s, 325 objects/s recovering 2024-04-06T03:42:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:35.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:35.295 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:35.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:35.430 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:35.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:35 smithi050 bash[21467]: cluster 2024-04-06T03:42:34.454954+0000 mgr.y (mgr.24368) 795 : cluster 0 pgmap v862: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 145 active+clean; 685 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 518 op/s; 1390/33537 objects misplaced (4.145%); 20 MiB/s, 325 objects/s recovering 2024-04-06T03:42:35.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:35 smithi050 bash[29221]: cluster 2024-04-06T03:42:34.454954+0000 mgr.y (mgr.24368) 795 : cluster 0 pgmap v862: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 145 active+clean; 685 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 518 op/s; 1390/33537 objects misplaced (4.145%); 20 MiB/s, 325 objects/s recovering 2024-04-06T03:42:35.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:35.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:35.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:35.694 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:35.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:35.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:35.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:35.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:35.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:36.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:36.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:36.241 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:36.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:36.372 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:36.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:36.506 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:36 smithi153 bash[23852]: audit 2024-04-06T03:42:35.247372+0000 mon.a (mon.0) 1607 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:42:36.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:36 smithi153 bash[23852]: audit 2024-04-06T03:42:35.251327+0000 mon.a (mon.0) 1608 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:36.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:36 smithi050 bash[21467]: audit 2024-04-06T03:42:35.247372+0000 mon.a (mon.0) 1607 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:42:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:36 smithi050 bash[21467]: audit 2024-04-06T03:42:35.251327+0000 mon.a (mon.0) 1608 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:36 smithi050 bash[29221]: audit 2024-04-06T03:42:35.247372+0000 mon.a (mon.0) 1607 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:42:36.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:36 smithi050 bash[29221]: audit 2024-04-06T03:42:35.251327+0000 mon.a (mon.0) 1608 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:36.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:36.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:36.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:36.873 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:36.892 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:36.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:36.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:37.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:37.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:37.151 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:37.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:37.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:37.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:37.397 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:37 smithi153 bash[23852]: cluster 2024-04-06T03:42:36.455611+0000 mgr.y (mgr.24368) 796 : cluster 0 pgmap v863: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 145 active+clean; 652 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 373 op/s; 1390/31959 objects misplaced (4.349%); 16 MiB/s, 261 objects/s recovering 2024-04-06T03:42:37.414 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:37.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:37.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:37 smithi050 bash[21467]: cluster 2024-04-06T03:42:36.455611+0000 mgr.y (mgr.24368) 796 : cluster 0 pgmap v863: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 145 active+clean; 652 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 373 op/s; 1390/31959 objects misplaced (4.349%); 16 MiB/s, 261 objects/s recovering 2024-04-06T03:42:37.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:37 smithi050 bash[29221]: cluster 2024-04-06T03:42:36.455611+0000 mgr.y (mgr.24368) 796 : cluster 0 pgmap v863: 148 pgs: 1 peering, 2 active+remapped+backfill_toofull, 145 active+clean; 652 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 373 op/s; 1390/31959 objects misplaced (4.349%); 16 MiB/s, 261 objects/s recovering 2024-04-06T03:42:37.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:37.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:37.690 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:37.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:37.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:37.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:37.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:37.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:38.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:38.065 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:42:38.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:38.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:38.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:38.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:38.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:38.446 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:38.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:38.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:38.589 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:38.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:38.720 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:38.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:38.849 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:38.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:38.988 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:38.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:39.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:39.117 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:39.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:39.252 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:39.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:39.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:39.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:39.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:39.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:39.643 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:39 smithi050 bash[21467]: cluster 2024-04-06T03:42:38.456385+0000 mgr.y (mgr.24368) 797 : cluster 0 pgmap v864: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 560 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 476 op/s; 909/27534 objects misplaced (3.301%); 13 MiB/s, 200 objects/s recovering 2024-04-06T03:42:39.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:42:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:42:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:42:39.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:39 smithi050 bash[29221]: cluster 2024-04-06T03:42:38.456385+0000 mgr.y (mgr.24368) 797 : cluster 0 pgmap v864: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 560 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 476 op/s; 909/27534 objects misplaced (3.301%); 13 MiB/s, 200 objects/s recovering 2024-04-06T03:42:39.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:39.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:39.774 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:39 smithi153 bash[23852]: cluster 2024-04-06T03:42:38.456385+0000 mgr.y (mgr.24368) 797 : cluster 0 pgmap v864: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 560 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 476 op/s; 909/27534 objects misplaced (3.301%); 13 MiB/s, 200 objects/s recovering 2024-04-06T03:42:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:39.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:39.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:39.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:40.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:40.035 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:40.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:40.162 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:40.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:40.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:40.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:42:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:42:40.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:40.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:40.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:40.558 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:40.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:40.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:40.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:40.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:40.832 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:40.936 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:40.979 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:40.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:41.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:41.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:41.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:41.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:41.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:41.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:41.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:41.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:41.504 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:41.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:41.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:41 smithi153 bash[23852]: audit 2024-04-06T03:42:40.092278+0000 mgr.y (mgr.24368) 798 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:41.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:41 smithi153 bash[23852]: cluster 2024-04-06T03:42:40.457069+0000 mgr.y (mgr.24368) 799 : cluster 0 pgmap v865: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 379 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 750 op/s; 909/18876 objects misplaced (4.816%); 11 MiB/s, 177 objects/s recovering 2024-04-06T03:42:41.632 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:41.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:41.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:41.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:41.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:41.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:41.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:41 smithi050 bash[21467]: audit 2024-04-06T03:42:40.092278+0000 mgr.y (mgr.24368) 798 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:41.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:41 smithi050 bash[21467]: cluster 2024-04-06T03:42:40.457069+0000 mgr.y (mgr.24368) 799 : cluster 0 pgmap v865: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 379 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 750 op/s; 909/18876 objects misplaced (4.816%); 11 MiB/s, 177 objects/s recovering 2024-04-06T03:42:41.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:41 smithi050 bash[29221]: audit 2024-04-06T03:42:40.092278+0000 mgr.y (mgr.24368) 798 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:41.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:41 smithi050 bash[29221]: cluster 2024-04-06T03:42:40.457069+0000 mgr.y (mgr.24368) 799 : cluster 0 pgmap v865: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 379 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 750 op/s; 909/18876 objects misplaced (4.816%); 11 MiB/s, 177 objects/s recovering 2024-04-06T03:42:41.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:42.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:42.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:42.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:42.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:42.385 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:42.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:42.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:42.547 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:42.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:42.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:42.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:42.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:42.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:42.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:42.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:42.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:43.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:43.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:43.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:43.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:43.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:43.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:43.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:43.486 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:43.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:43.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:43.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:43 smithi050 bash[21467]: cluster 2024-04-06T03:42:42.457998+0000 mgr.y (mgr.24368) 800 : cluster 0 pgmap v866: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 310 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 672 op/s; 635/15540 objects misplaced (4.086%); 5.2 MiB/s, 83 objects/s recovering 2024-04-06T03:42:43.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:43 smithi050 bash[29221]: cluster 2024-04-06T03:42:42.457998+0000 mgr.y (mgr.24368) 800 : cluster 0 pgmap v866: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 310 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 672 op/s; 635/15540 objects misplaced (4.086%); 5.2 MiB/s, 83 objects/s recovering 2024-04-06T03:42:43.711 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:43.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:43 smithi153 bash[23852]: cluster 2024-04-06T03:42:42.457998+0000 mgr.y (mgr.24368) 800 : cluster 0 pgmap v866: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 310 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 672 op/s; 635/15540 objects misplaced (4.086%); 5.2 MiB/s, 83 objects/s recovering 2024-04-06T03:42:43.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:43.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:43.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:43.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:43.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:44.002 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:44.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:44.127 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:44.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:44.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:44.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:44.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:44.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:44.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:44 smithi050 bash[21467]: audit 2024-04-06T03:42:44.318576+0000 mon.a (mon.0) 1609 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:42:44.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:44 smithi050 bash[29221]: audit 2024-04-06T03:42:44.318576+0000 mon.a (mon.0) 1609 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:42:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:44 smithi153 bash[23852]: audit 2024-04-06T03:42:44.318576+0000 mon.a (mon.0) 1609 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:42:44.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:44.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:44.958 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:44.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:45.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:45.080 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:45.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:45.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:45.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:45.350 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:45.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:45.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:45.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:45.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:45.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[21467]: cluster 2024-04-06T03:42:44.458768+0000 mgr.y (mgr.24368) 801 : cluster 0 pgmap v867: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 191 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 867 B/s rd, 0 B/s wr, 731 op/s; 356/9822 objects misplaced (3.625%); 4.4 MiB/s, 70 objects/s recovering 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[21467]: audit 2024-04-06T03:42:45.261177+0000 mon.a (mon.0) 1610 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[21467]: audit 2024-04-06T03:42:45.261844+0000 mon.a (mon.0) 1611 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[21467]: audit 2024-04-06T03:42:45.266281+0000 mon.a (mon.0) 1612 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[29221]: cluster 2024-04-06T03:42:44.458768+0000 mgr.y (mgr.24368) 801 : cluster 0 pgmap v867: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 191 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 867 B/s rd, 0 B/s wr, 731 op/s; 356/9822 objects misplaced (3.625%); 4.4 MiB/s, 70 objects/s recovering 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[29221]: audit 2024-04-06T03:42:45.261177+0000 mon.a (mon.0) 1610 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[29221]: audit 2024-04-06T03:42:45.261844+0000 mon.a (mon.0) 1611 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:42:45.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:45 smithi050 bash[29221]: audit 2024-04-06T03:42:45.266281+0000 mon.a (mon.0) 1612 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:42:45.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:45 smithi153 bash[23852]: cluster 2024-04-06T03:42:44.458768+0000 mgr.y (mgr.24368) 801 : cluster 0 pgmap v867: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 191 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 867 B/s rd, 0 B/s wr, 731 op/s; 356/9822 objects misplaced (3.625%); 4.4 MiB/s, 70 objects/s recovering 2024-04-06T03:42:45.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:45 smithi153 bash[23852]: audit 2024-04-06T03:42:45.261177+0000 mon.a (mon.0) 1610 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:42:45.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:45 smithi153 bash[23852]: audit 2024-04-06T03:42:45.261844+0000 mon.a (mon.0) 1611 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:42:45.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:45 smithi153 bash[23852]: audit 2024-04-06T03:42:45.266281+0000 mon.a (mon.0) 1612 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:42:45.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:45.871 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:45.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:45.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:45.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:45.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:46.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:46.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:46.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:46.348 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:46.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:46.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:46.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:46.634 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:46.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:46.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:46.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:46.805 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Removed 98814 objects 2024-04-06T03:42:46.805 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Clean up completed and total clean up time :132.178 2024-04-06T03:42:46.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:46.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:46.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:46.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:47.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:47.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:47.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:47.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:47.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:47.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:47.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:47.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:47.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:47.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:47.665 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:47.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:47.796 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:47 smithi153 bash[23852]: cluster 2024-04-06T03:42:46.459281+0000 mgr.y (mgr.24368) 802 : cluster 0 pgmap v868: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 134 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 735 op/s; 356/7086 objects misplaced (5.024%) 2024-04-06T03:42:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:47.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:47.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:47.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:47.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:47 smithi050 bash[21467]: cluster 2024-04-06T03:42:46.459281+0000 mgr.y (mgr.24368) 802 : cluster 0 pgmap v868: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 134 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 735 op/s; 356/7086 objects misplaced (5.024%) 2024-04-06T03:42:47.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:47 smithi050 bash[29221]: cluster 2024-04-06T03:42:46.459281+0000 mgr.y (mgr.24368) 802 : cluster 0 pgmap v868: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 134 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 735 op/s; 356/7086 objects misplaced (5.024%) 2024-04-06T03:42:48.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:48.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:48.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:48.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:48.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:48.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:48.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:48.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:48.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:48.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:48.709 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:48.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:48.843 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:48.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:48.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:48.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:48.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:49.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:49.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:49.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:49.235 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:49.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:49.364 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:49.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:49.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:49.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:49.606 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:42:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:42:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:42:49.624 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:49.728 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:49.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:49 smithi050 bash[29221]: cluster 2024-04-06T03:42:48.460215+0000 mgr.y (mgr.24368) 803 : cluster 0 pgmap v869: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 63 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 786 op/s 2024-04-06T03:42:49.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:49 smithi050 bash[21467]: cluster 2024-04-06T03:42:48.460215+0000 mgr.y (mgr.24368) 803 : cluster 0 pgmap v869: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 63 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 786 op/s 2024-04-06T03:42:49.755 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:49.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:49 smithi153 bash[23852]: cluster 2024-04-06T03:42:48.460215+0000 mgr.y (mgr.24368) 803 : cluster 0 pgmap v869: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 63 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 786 op/s 2024-04-06T03:42:49.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:49.884 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:49.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:49.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:50.012 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:50.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:50.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:50.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:50.248 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-04-06T03:42:50.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-04-06T03:42:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:50.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:42:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:42:50.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:50.396 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:50.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:50.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:50.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:50.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:50 smithi050 bash[21467]: audit 2024-04-06T03:42:50.273890+0000 mon.a (mon.0) 1613 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:50.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:50 smithi050 bash[29221]: audit 2024-04-06T03:42:50.273890+0000 mon.a (mon.0) 1613 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:50.653 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:50.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:50.767 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:50 smithi153 bash[23852]: audit 2024-04-06T03:42:50.273890+0000 mon.a (mon.0) 1613 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:42:50.783 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:50.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:50.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:50.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:50.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:51.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:51.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:51.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:51.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:51.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:51.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:51.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:51.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:51.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:51.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:51.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:51.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:51.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:51 smithi050 bash[21467]: audit 2024-04-06T03:42:50.102966+0000 mgr.y (mgr.24368) 804 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:51.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:51 smithi050 bash[21467]: cluster 2024-04-06T03:42:50.461111+0000 mgr.y (mgr.24368) 805 : cluster 0 pgmap v870: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:42:51.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:51 smithi050 bash[29221]: audit 2024-04-06T03:42:50.102966+0000 mgr.y (mgr.24368) 804 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:51.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:51 smithi050 bash[29221]: cluster 2024-04-06T03:42:50.461111+0000 mgr.y (mgr.24368) 805 : cluster 0 pgmap v870: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:42:51.702 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:51.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:51.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:51.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:51.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:51.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:52.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:51 smithi153 bash[23852]: audit 2024-04-06T03:42:50.102966+0000 mgr.y (mgr.24368) 804 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:42:52.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:51 smithi153 bash[23852]: cluster 2024-04-06T03:42:50.461111+0000 mgr.y (mgr.24368) 805 : cluster 0 pgmap v870: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 746 op/s 2024-04-06T03:42:52.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:52.097 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:52.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:52.229 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:52.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:52.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:52.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:52.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:52.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:52.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:52.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:52.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:52.887 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:52.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:52.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:53.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:53.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:53.145 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:53.248 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:53.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:53.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:53.511 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:53.537 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:53.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:53.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:53.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:53.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:53 smithi153 bash[23852]: cluster 2024-04-06T03:42:52.255306+0000 osd.6 (osd.6) 105 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:53.783 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:53 smithi153 bash[23852]: cluster 2024-04-06T03:42:52.255404+0000 osd.6 (osd.6) 106 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:53.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:53 smithi153 bash[23852]: cluster 2024-04-06T03:42:52.462070+0000 mgr.y (mgr.24368) 806 : cluster 0 pgmap v871: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 505 op/s 2024-04-06T03:42:53.784 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:53 smithi153 bash[23852]: cluster 2024-04-06T03:42:52.569362+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-04-06T03:42:53.798 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:53.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:53.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:53.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:53.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[29221]: cluster 2024-04-06T03:42:52.255306+0000 osd.6 (osd.6) 105 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:53.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[29221]: cluster 2024-04-06T03:42:52.255404+0000 osd.6 (osd.6) 106 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:53.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[29221]: cluster 2024-04-06T03:42:52.462070+0000 mgr.y (mgr.24368) 806 : cluster 0 pgmap v871: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 505 op/s 2024-04-06T03:42:53.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[29221]: cluster 2024-04-06T03:42:52.569362+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-04-06T03:42:53.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[21467]: cluster 2024-04-06T03:42:52.255306+0000 osd.6 (osd.6) 105 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:53.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[21467]: cluster 2024-04-06T03:42:52.255404+0000 osd.6 (osd.6) 106 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:53.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[21467]: cluster 2024-04-06T03:42:52.462070+0000 mgr.y (mgr.24368) 806 : cluster 0 pgmap v871: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 505 op/s 2024-04-06T03:42:53.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:53 smithi050 bash[21467]: cluster 2024-04-06T03:42:52.569362+0000 mon.a (mon.0) 1614 : cluster 0 osdmap e335: 8 total, 8 up, 8 in 2024-04-06T03:42:54.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:54.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:54.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:54.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:54.290 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:54.317 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:54.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:54.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:54.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:54.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:54.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:54.704 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:54.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:54.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:54.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:54.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:54.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:55.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:55.089 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:55.102 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:42:55.107 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:42:55.107 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:42:55.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:54 smithi050 bash[21467]: cluster 2024-04-06T03:42:54.462726+0000 mgr.y (mgr.24368) 807 : cluster 0 pgmap v873: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 305 op/s 2024-04-06T03:42:55.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:54 smithi050 bash[21467]: cluster 2024-04-06T03:42:54.785393+0000 osd.3 (osd.3) 520 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:55.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:54 smithi050 bash[21467]: cluster 2024-04-06T03:42:54.785495+0000 osd.3 (osd.3) 521 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:54 smithi050 bash[29221]: cluster 2024-04-06T03:42:54.462726+0000 mgr.y (mgr.24368) 807 : cluster 0 pgmap v873: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 305 op/s 2024-04-06T03:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:54 smithi050 bash[29221]: cluster 2024-04-06T03:42:54.785393+0000 osd.3 (osd.3) 520 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:55.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:54 smithi050 bash[29221]: cluster 2024-04-06T03:42:54.785495+0000 osd.3 (osd.3) 521 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:55.125 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:42:55.166 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:42:55.188 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:42:55.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:55.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:55.214 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:42:55.239 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:42:55.264 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:42:55.288 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:42:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:42:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:42:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:42:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:42:55.290 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:42:55.291 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:42:55.291 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:42:55.291 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:42:55.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:54 smithi153 bash[23852]: cluster 2024-04-06T03:42:54.462726+0000 mgr.y (mgr.24368) 807 : cluster 0 pgmap v873: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 305 op/s 2024-04-06T03:42:55.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:54 smithi153 bash[23852]: cluster 2024-04-06T03:42:54.785393+0000 osd.3 (osd.3) 520 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:55.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:54 smithi153 bash[23852]: cluster 2024-04-06T03:42:54.785495+0000 osd.3 (osd.3) 521 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:55.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:55.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:55.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: debug 2024-04-06T03:42:55.329+0000 7f702b621700 -1 mon.a@0(leader).osd e336 definitely_dead 0 2024-04-06T03:42:55.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:55.599 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:55.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:55.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:55.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:55.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:55.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:55.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:55.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:55.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: cluster 2024-04-06T03:42:54.957581+0000 mon.a (mon.0) 1615 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-04-06T03:42:55.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: audit 2024-04-06T03:42:55.076464+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.50:0/1081175372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:42:55.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: audit 2024-04-06T03:42:55.333213+0000 mon.c (mon.2) 101 : audit 1 from='client.? 172.21.15.50:0/123257355' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-06T03:42:55.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: audit 2024-04-06T03:42:55.333491+0000 mon.a (mon.0) 1616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-06T03:42:55.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: cluster 2024-04-06T03:42:55.333565+0000 mon.a (mon.0) 1617 : cluster 1 Client client.admin marked osd.1 out, while it was still marked up 2024-04-06T03:42:55.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[21467]: audit 2024-04-06T03:42:55.353968+0000 mon.a (mon.0) 1618 : audit 1 from='client.? 172.21.15.50:0/2802457745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:42:55.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[29221]: cluster 2024-04-06T03:42:54.957581+0000 mon.a (mon.0) 1615 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-04-06T03:42:55.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[29221]: audit 2024-04-06T03:42:55.076464+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.50:0/1081175372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:42:55.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[29221]: audit 2024-04-06T03:42:55.333213+0000 mon.c (mon.2) 101 : audit 1 from='client.? 172.21.15.50:0/123257355' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-06T03:42:55.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[29221]: audit 2024-04-06T03:42:55.333491+0000 mon.a (mon.0) 1616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-06T03:42:55.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[29221]: cluster 2024-04-06T03:42:55.333565+0000 mon.a (mon.0) 1617 : cluster 1 Client client.admin marked osd.1 out, while it was still marked up 2024-04-06T03:42:55.979 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:55 smithi050 bash[29221]: audit 2024-04-06T03:42:55.353968+0000 mon.a (mon.0) 1618 : audit 1 from='client.? 172.21.15.50:0/2802457745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:42:55.984 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:42:55.986 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.1. 2024-04-06T03:42:55.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:56.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:56.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:56.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:56.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:56.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:55 smithi153 bash[23852]: cluster 2024-04-06T03:42:54.957581+0000 mon.a (mon.0) 1615 : cluster 0 osdmap e336: 8 total, 8 up, 8 in 2024-04-06T03:42:56.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:55 smithi153 bash[23852]: audit 2024-04-06T03:42:55.076464+0000 mon.c (mon.2) 100 : audit 0 from='client.? 172.21.15.50:0/1081175372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:42:56.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:55 smithi153 bash[23852]: audit 2024-04-06T03:42:55.333213+0000 mon.c (mon.2) 101 : audit 1 from='client.? 172.21.15.50:0/123257355' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-06T03:42:56.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:55 smithi153 bash[23852]: audit 2024-04-06T03:42:55.333491+0000 mon.a (mon.0) 1616 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-04-06T03:42:56.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:55 smithi153 bash[23852]: cluster 2024-04-06T03:42:55.333565+0000 mon.a (mon.0) 1617 : cluster 1 Client client.admin marked osd.1 out, while it was still marked up 2024-04-06T03:42:56.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:55 smithi153 bash[23852]: audit 2024-04-06T03:42:55.353968+0000 mon.a (mon.0) 1618 : audit 1 from='client.? 172.21.15.50:0/2802457745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:42:56.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:56.389 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:56.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:56.521 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:56.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:56.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:56.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:56.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:56.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:56.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:56.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:57.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:57.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: cluster 2024-04-06T03:42:55.373894+0000 osd.7 (osd.7) 531 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:57.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: cluster 2024-04-06T03:42:55.373970+0000 osd.7 (osd.7) 532 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:57.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: cluster 2024-04-06T03:42:55.953659+0000 mon.a (mon.0) 1619 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:42:57.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: audit 2024-04-06T03:42:55.961372+0000 mon.a (mon.0) 1620 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-06T03:42:57.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: audit 2024-04-06T03:42:55.961523+0000 mon.a (mon.0) 1621 : audit 1 from='client.? 172.21.15.50:0/2802457745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:42:57.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: cluster 2024-04-06T03:42:55.967604+0000 mon.a (mon.0) 1622 : cluster 0 osdmap e337: 8 total, 8 up, 7 in 2024-04-06T03:42:57.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:56 smithi153 bash[23852]: cluster 2024-04-06T03:42:56.463477+0000 mgr.y (mgr.24368) 808 : cluster 0 pgmap v876: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:42:57.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:57.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:57.181 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:57.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:56 smithi050 bash[21467]: cluster 2024-04-06T03:42:55.373894+0000 osd.7 (osd.7) 531 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:56 smithi050 bash[21467]: cluster 2024-04-06T03:42:55.373970+0000 osd.7 (osd.7) 532 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:56 smithi050 bash[21467]: cluster 2024-04-06T03:42:55.953659+0000 mon.a (mon.0) 1619 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[21467]: audit 2024-04-06T03:42:55.961372+0000 mon.a (mon.0) 1620 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[21467]: audit 2024-04-06T03:42:55.961523+0000 mon.a (mon.0) 1621 : audit 1 from='client.? 172.21.15.50:0/2802457745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[21467]: cluster 2024-04-06T03:42:55.967604+0000 mon.a (mon.0) 1622 : cluster 0 osdmap e337: 8 total, 8 up, 7 in 2024-04-06T03:42:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[21467]: cluster 2024-04-06T03:42:56.463477+0000 mgr.y (mgr.24368) 808 : cluster 0 pgmap v876: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: cluster 2024-04-06T03:42:55.373894+0000 osd.7 (osd.7) 531 : cluster 0 purged_snaps scrub starts 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: cluster 2024-04-06T03:42:55.373970+0000 osd.7 (osd.7) 532 : cluster 0 purged_snaps scrub ok 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: cluster 2024-04-06T03:42:55.953659+0000 mon.a (mon.0) 1619 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: audit 2024-04-06T03:42:55.961372+0000 mon.a (mon.0) 1620 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: audit 2024-04-06T03:42:55.961523+0000 mon.a (mon.0) 1621 : audit 1 from='client.? 172.21.15.50:0/2802457745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: cluster 2024-04-06T03:42:55.967604+0000 mon.a (mon.0) 1622 : cluster 0 osdmap e337: 8 total, 8 up, 7 in 2024-04-06T03:42:57.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:57 smithi050 bash[29221]: cluster 2024-04-06T03:42:56.463477+0000 mgr.y (mgr.24368) 808 : cluster 0 pgmap v876: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:42:57.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:57.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:57.407 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:57.435 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:42:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:42:57.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-06T03:42:57.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:57.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:57.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:57.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:57.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:57.869 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:57.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:57.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:57.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:42:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:42:58.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:58.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:58.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:58.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:42:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:42:58.355 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:56.844200+0000 osd.2 (osd.2) 413 : cluster 0 5.d scrub starts 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:56.849032+0000 osd.2 (osd.2) 414 : cluster 0 5.d scrub ok 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:57.051119+0000 mon.a (mon.0) 1623 : cluster 0 osdmap e338: 8 total, 8 up, 7 in 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:57.052476+0000 osd.4 (osd.4) 499 : cluster 0 5.f scrub starts 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:57.104683+0000 osd.4 (osd.4) 500 : cluster 0 5.f scrub ok 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:57.219199+0000 osd.7 (osd.7) 533 : cluster 0 3.e scrub starts 2024-04-06T03:42:58.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:58 smithi153 bash[23852]: cluster 2024-04-06T03:42:57.221604+0000 osd.7 (osd.7) 534 : cluster 0 3.e scrub ok 2024-04-06T03:42:58.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:58.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:56.844200+0000 osd.2 (osd.2) 413 : cluster 0 5.d scrub starts 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:56.849032+0000 osd.2 (osd.2) 414 : cluster 0 5.d scrub ok 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:57.051119+0000 mon.a (mon.0) 1623 : cluster 0 osdmap e338: 8 total, 8 up, 7 in 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:57.052476+0000 osd.4 (osd.4) 499 : cluster 0 5.f scrub starts 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:57.104683+0000 osd.4 (osd.4) 500 : cluster 0 5.f scrub ok 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:57.219199+0000 osd.7 (osd.7) 533 : cluster 0 3.e scrub starts 2024-04-06T03:42:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[29221]: cluster 2024-04-06T03:42:57.221604+0000 osd.7 (osd.7) 534 : cluster 0 3.e scrub ok 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:56.844200+0000 osd.2 (osd.2) 413 : cluster 0 5.d scrub starts 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:56.849032+0000 osd.2 (osd.2) 414 : cluster 0 5.d scrub ok 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:57.051119+0000 mon.a (mon.0) 1623 : cluster 0 osdmap e338: 8 total, 8 up, 7 in 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:57.052476+0000 osd.4 (osd.4) 499 : cluster 0 5.f scrub starts 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:57.104683+0000 osd.4 (osd.4) 500 : cluster 0 5.f scrub ok 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:57.219199+0000 osd.7 (osd.7) 533 : cluster 0 3.e scrub starts 2024-04-06T03:42:58.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:58 smithi050 bash[21467]: cluster 2024-04-06T03:42:57.221604+0000 osd.7 (osd.7) 534 : cluster 0 3.e scrub ok 2024-04-06T03:42:58.511 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:58.614 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:58.639 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:42:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:42:58.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:42:58.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:42:58.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:58.897 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:58.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:58.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:59.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:59.027 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:59.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:59.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:59.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:59.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.052508+0000 mon.a (mon.0) 1624 : cluster 0 osdmap e339: 8 total, 8 up, 7 in 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.053051+0000 osd.4 (osd.4) 501 : cluster 0 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 334'12382 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.053756+0000 osd.4 (osd.4) 502 : cluster 0 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 334'12456 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.057094+0000 osd.0 (osd.0) 516 : cluster 0 10.b starting backfill to osd.4 from (304'6465,331'11683] MIN to 334'12408 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.092609+0000 osd.4 (osd.4) 503 : cluster 0 5.1f scrub starts 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.110253+0000 osd.4 (osd.4) 504 : cluster 0 5.1f scrub ok 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.463977+0000 mgr.y (mgr.24368) 809 : cluster 0 pgmap v879: 148 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 181 B/s, 8 objects/s recovering 2024-04-06T03:42:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:42:59 smithi153 bash[23852]: cluster 2024-04-06T03:42:58.945572+0000 mon.a (mon.0) 1625 : cluster 0 osdmap e340: 8 total, 8 up, 7 in 2024-04-06T03:42:59.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:59.325 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:42:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.052508+0000 mon.a (mon.0) 1624 : cluster 0 osdmap e339: 8 total, 8 up, 7 in 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.053051+0000 osd.4 (osd.4) 501 : cluster 0 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 334'12382 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.053756+0000 osd.4 (osd.4) 502 : cluster 0 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 334'12456 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.057094+0000 osd.0 (osd.0) 516 : cluster 0 10.b starting backfill to osd.4 from (304'6465,331'11683] MIN to 334'12408 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.092609+0000 osd.4 (osd.4) 503 : cluster 0 5.1f scrub starts 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.110253+0000 osd.4 (osd.4) 504 : cluster 0 5.1f scrub ok 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.463977+0000 mgr.y (mgr.24368) 809 : cluster 0 pgmap v879: 148 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 181 B/s, 8 objects/s recovering 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[21467]: cluster 2024-04-06T03:42:58.945572+0000 mon.a (mon.0) 1625 : cluster 0 osdmap e340: 8 total, 8 up, 7 in 2024-04-06T03:42:59.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.052508+0000 mon.a (mon.0) 1624 : cluster 0 osdmap e339: 8 total, 8 up, 7 in 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.053051+0000 osd.4 (osd.4) 501 : cluster 0 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 334'12382 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.053756+0000 osd.4 (osd.4) 502 : cluster 0 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 334'12456 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.057094+0000 osd.0 (osd.0) 516 : cluster 0 10.b starting backfill to osd.4 from (304'6465,331'11683] MIN to 334'12408 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.092609+0000 osd.4 (osd.4) 503 : cluster 0 5.1f scrub starts 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.110253+0000 osd.4 (osd.4) 504 : cluster 0 5.1f scrub ok 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.463977+0000 mgr.y (mgr.24368) 809 : cluster 0 pgmap v879: 148 pgs: 1 remapped+peering, 1 active+remapped+backfill_toofull, 146 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 181 B/s, 8 objects/s recovering 2024-04-06T03:42:59.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:42:59 smithi050 bash[29221]: cluster 2024-04-06T03:42:58.945572+0000 mon.a (mon.0) 1625 : cluster 0 osdmap e340: 8 total, 8 up, 7 in 2024-04-06T03:42:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:59.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:59.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:42:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:42:59.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:42:59.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:59.593 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:42:59.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:59.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:59.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:42:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:42:59.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:59.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:42:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:42:59.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:42:59.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:00.008 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:00.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.050687+0000 osd.3 (osd.3) 522 : cluster 0 3.8 scrub starts 2024-04-06T03:43:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.057329+0000 mon.a (mon.0) 1626 : cluster 3 Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-06T03:43:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.057803+0000 osd.3 (osd.3) 523 : cluster 0 3.8 scrub ok 2024-04-06T03:43:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.680472+0000 osd.5 (osd.5) 147 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.680581+0000 osd.5 (osd.5) 148 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.950449+0000 mon.a (mon.0) 1627 : cluster 0 osdmap e341: 8 total, 8 up, 7 in 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.050687+0000 osd.3 (osd.3) 522 : cluster 0 3.8 scrub starts 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.057329+0000 mon.a (mon.0) 1626 : cluster 3 Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.057803+0000 osd.3 (osd.3) 523 : cluster 0 3.8 scrub ok 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.680472+0000 osd.5 (osd.5) 147 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.680581+0000 osd.5 (osd.5) 148 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:00 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.950449+0000 mon.a (mon.0) 1627 : cluster 0 osdmap e341: 8 total, 8 up, 7 in 2024-04-06T03:43:00.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:00.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:00.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:00.274 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:00.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:43:00.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.050687+0000 osd.3 (osd.3) 522 : cluster 0 3.8 scrub starts 2024-04-06T03:43:00.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.057329+0000 mon.a (mon.0) 1626 : cluster 3 Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-04-06T03:43:00.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.057803+0000 osd.3 (osd.3) 523 : cluster 0 3.8 scrub ok 2024-04-06T03:43:00.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.680472+0000 osd.5 (osd.5) 147 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:00.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.680581+0000 osd.5 (osd.5) 148 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:00.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:00 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.950449+0000 mon.a (mon.0) 1627 : cluster 0 osdmap e341: 8 total, 8 up, 7 in 2024-04-06T03:43:00.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:00.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:00.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:00.535 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:00.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:00.666 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:00.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:00.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:00.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:00.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:00.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:00.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:01.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:01.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:01.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:01.189 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:01.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:01 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.891806+0000 osd.0 (osd.0) 517 : cluster 0 2.1f scrub starts 2024-04-06T03:43:01.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:01 smithi153 bash[23852]: cluster 2024-04-06T03:42:59.893980+0000 osd.0 (osd.0) 518 : cluster 0 2.1f scrub ok 2024-04-06T03:43:01.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:01 smithi153 bash[23852]: audit 2024-04-06T03:43:00.113686+0000 mgr.y (mgr.24368) 810 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:01.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:01 smithi153 bash[23852]: cluster 2024-04-06T03:43:00.464963+0000 mgr.y (mgr.24368) 811 : cluster 0 pgmap v882: 148 pgs: 2 peering, 1 remapped+peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 457 B/s, 22 objects/s recovering 2024-04-06T03:43:01.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:43:01.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:01.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:01.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:01.438 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:43:01 smithi153 bash[51781]: logger=cleanup t=2024-04-06T03:43:01.195044371Z level=info msg="Completed cleanup jobs" duration=35.638153ms 2024-04-06T03:43:01.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.891806+0000 osd.0 (osd.0) 517 : cluster 0 2.1f scrub starts 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[29221]: cluster 2024-04-06T03:42:59.893980+0000 osd.0 (osd.0) 518 : cluster 0 2.1f scrub ok 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[29221]: audit 2024-04-06T03:43:00.113686+0000 mgr.y (mgr.24368) 810 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[29221]: cluster 2024-04-06T03:43:00.464963+0000 mgr.y (mgr.24368) 811 : cluster 0 pgmap v882: 148 pgs: 2 peering, 1 remapped+peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 457 B/s, 22 objects/s recovering 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.891806+0000 osd.0 (osd.0) 517 : cluster 0 2.1f scrub starts 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[21467]: cluster 2024-04-06T03:42:59.893980+0000 osd.0 (osd.0) 518 : cluster 0 2.1f scrub ok 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[21467]: audit 2024-04-06T03:43:00.113686+0000 mgr.y (mgr.24368) 810 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:01 smithi050 bash[21467]: cluster 2024-04-06T03:43:00.464963+0000 mgr.y (mgr.24368) 811 : cluster 0 pgmap v882: 148 pgs: 2 peering, 1 remapped+peering, 1 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 457 B/s, 22 objects/s recovering 2024-04-06T03:43:01.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:01.584 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:01.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:01.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:01.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:01.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:01.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:01.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:01.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:02.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:02.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:02.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:02.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:02.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:02.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:02.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:02.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:02.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:02.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:02.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:02.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:03.023 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:03.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:03.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:03.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:03.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:03.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:03.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:03.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:03.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:03.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:03.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[21467]: cluster 2024-04-06T03:43:02.078830+0000 osd.3 (osd.3) 524 : cluster 0 4.0 scrub starts 2024-04-06T03:43:03.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[21467]: cluster 2024-04-06T03:43:02.082107+0000 osd.3 (osd.3) 525 : cluster 0 4.0 scrub ok 2024-04-06T03:43:03.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[21467]: cluster 2024-04-06T03:43:02.465560+0000 mgr.y (mgr.24368) 812 : cluster 0 pgmap v883: 148 pgs: 2 peering, 2 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 755 B/s rd, 0 op/s; 337 B/s, 16 objects/s recovering 2024-04-06T03:43:03.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.184880+0000 osd.7 (osd.7) 535 : cluster 0 5.10 scrub starts 2024-04-06T03:43:03.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.187361+0000 osd.7 (osd.7) 536 : cluster 0 5.10 scrub ok 2024-04-06T03:43:03.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[29221]: cluster 2024-04-06T03:43:02.078830+0000 osd.3 (osd.3) 524 : cluster 0 4.0 scrub starts 2024-04-06T03:43:03.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[29221]: cluster 2024-04-06T03:43:02.082107+0000 osd.3 (osd.3) 525 : cluster 0 4.0 scrub ok 2024-04-06T03:43:03.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[29221]: cluster 2024-04-06T03:43:02.465560+0000 mgr.y (mgr.24368) 812 : cluster 0 pgmap v883: 148 pgs: 2 peering, 2 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 755 B/s rd, 0 op/s; 337 B/s, 16 objects/s recovering 2024-04-06T03:43:03.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.184880+0000 osd.7 (osd.7) 535 : cluster 0 5.10 scrub starts 2024-04-06T03:43:03.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:03 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.187361+0000 osd.7 (osd.7) 536 : cluster 0 5.10 scrub ok 2024-04-06T03:43:03.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:03.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:03.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:03 smithi153 bash[23852]: cluster 2024-04-06T03:43:02.078830+0000 osd.3 (osd.3) 524 : cluster 0 4.0 scrub starts 2024-04-06T03:43:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:03 smithi153 bash[23852]: cluster 2024-04-06T03:43:02.082107+0000 osd.3 (osd.3) 525 : cluster 0 4.0 scrub ok 2024-04-06T03:43:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:03 smithi153 bash[23852]: cluster 2024-04-06T03:43:02.465560+0000 mgr.y (mgr.24368) 812 : cluster 0 pgmap v883: 148 pgs: 2 peering, 2 active+remapped+backfill_toofull, 144 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 755 B/s rd, 0 op/s; 337 B/s, 16 objects/s recovering 2024-04-06T03:43:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:03 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.184880+0000 osd.7 (osd.7) 535 : cluster 0 5.10 scrub starts 2024-04-06T03:43:03.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:03 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.187361+0000 osd.7 (osd.7) 536 : cluster 0 5.10 scrub ok 2024-04-06T03:43:03.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:03.920 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:03.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:03.968 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_3' removed 2024-04-06T03:43:04.023 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:04.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:04.165 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:04.198 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:04.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:04.236 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:43:04.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 0 1 2024-04-06T03:43:04.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:04.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:04.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:04.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:04.551 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:04.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:04.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:04.709 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:04.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.108014+0000 osd.3 (osd.3) 526 : cluster 0 6.c scrub starts 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.110473+0000 osd.3 (osd.3) 527 : cluster 0 6.c scrub ok 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: audit 2024-04-06T03:43:03.649412+0000 mon.c (mon.2) 102 : audit 1 from='client.? 172.21.15.50:0/1530584444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: audit 2024-04-06T03:43:03.649983+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.891070+0000 osd.2 (osd.2) 415 : cluster 0 2.c scrub starts 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.893452+0000 osd.2 (osd.2) 416 : cluster 0 2.c scrub ok 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: audit 2024-04-06T03:43:03.939912+0000 mon.a (mon.0) 1629 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:43:04.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:04 smithi153 bash[23852]: cluster 2024-04-06T03:43:03.943319+0000 mon.a (mon.0) 1630 : cluster 0 osdmap e342: 8 total, 8 up, 7 in 2024-04-06T03:43:04.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:04.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:04.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.108014+0000 osd.3 (osd.3) 526 : cluster 0 6.c scrub starts 2024-04-06T03:43:04.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.110473+0000 osd.3 (osd.3) 527 : cluster 0 6.c scrub ok 2024-04-06T03:43:04.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: audit 2024-04-06T03:43:03.649412+0000 mon.c (mon.2) 102 : audit 1 from='client.? 172.21.15.50:0/1530584444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: audit 2024-04-06T03:43:03.649983+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.891070+0000 osd.2 (osd.2) 415 : cluster 0 2.c scrub starts 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.893452+0000 osd.2 (osd.2) 416 : cluster 0 2.c scrub ok 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: audit 2024-04-06T03:43:03.939912+0000 mon.a (mon.0) 1629 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[21467]: cluster 2024-04-06T03:43:03.943319+0000 mon.a (mon.0) 1630 : cluster 0 osdmap e342: 8 total, 8 up, 7 in 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.108014+0000 osd.3 (osd.3) 526 : cluster 0 6.c scrub starts 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.110473+0000 osd.3 (osd.3) 527 : cluster 0 6.c scrub ok 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: audit 2024-04-06T03:43:03.649412+0000 mon.c (mon.2) 102 : audit 1 from='client.? 172.21.15.50:0/1530584444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: audit 2024-04-06T03:43:03.649983+0000 mon.a (mon.0) 1628 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:43:04.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.891070+0000 osd.2 (osd.2) 415 : cluster 0 2.c scrub starts 2024-04-06T03:43:04.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.893452+0000 osd.2 (osd.2) 416 : cluster 0 2.c scrub ok 2024-04-06T03:43:04.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: audit 2024-04-06T03:43:03.939912+0000 mon.a (mon.0) 1629 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:43:04.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:04 smithi050 bash[29221]: cluster 2024-04-06T03:43:03.943319+0000 mon.a (mon.0) 1630 : cluster 0 osdmap e342: 8 total, 8 up, 7 in 2024-04-06T03:43:04.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:04.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:04.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:04.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:05.055 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-04-06T03:43:05.055 INFO:tasks.radosbench:Beginning radosbench... 2024-04-06T03:43:05.056 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-04-06T03:43:05.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool create unique_pool_4 16 2024-04-06T03:43:05.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:05.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:05.208 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:05.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:05.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:05.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:05.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:05.474 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:05.475 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:05.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:05.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:05.632 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:05.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.152891+0000 osd.3 (osd.3) 528 : cluster 0 6.1f scrub starts 2024-04-06T03:43:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.325558+0000 osd.3 (osd.3) 529 : cluster 0 6.1f scrub ok 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.466143+0000 mgr.y (mgr.24368) 813 : cluster 0 pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.878032+0000 osd.2 (osd.2) 417 : cluster 0 5.6 scrub starts 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.879774+0000 osd.2 (osd.2) 418 : cluster 0 5.6 scrub ok 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.940022+0000 mon.a (mon.0) 1631 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.988533+0000 osd.0 (osd.0) 519 : cluster 0 5.7 scrub starts 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: cluster 2024-04-06T03:43:04.990124+0000 osd.0 (osd.0) 520 : cluster 0 5.7 scrub ok 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: audit 2024-04-06T03:43:05.293389+0000 mon.a (mon.0) 1632 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[21467]: audit 2024-04-06T03:43:05.294527+0000 mon.a (mon.0) 1633 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.152891+0000 osd.3 (osd.3) 528 : cluster 0 6.1f scrub starts 2024-04-06T03:43:05.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.325558+0000 osd.3 (osd.3) 529 : cluster 0 6.1f scrub ok 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.466143+0000 mgr.y (mgr.24368) 813 : cluster 0 pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.878032+0000 osd.2 (osd.2) 417 : cluster 0 5.6 scrub starts 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.879774+0000 osd.2 (osd.2) 418 : cluster 0 5.6 scrub ok 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.940022+0000 mon.a (mon.0) 1631 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.988533+0000 osd.0 (osd.0) 519 : cluster 0 5.7 scrub starts 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: cluster 2024-04-06T03:43:04.990124+0000 osd.0 (osd.0) 520 : cluster 0 5.7 scrub ok 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: audit 2024-04-06T03:43:05.293389+0000 mon.a (mon.0) 1632 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:05.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:05 smithi050 bash[29221]: audit 2024-04-06T03:43:05.294527+0000 mon.a (mon.0) 1633 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:05.764 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:05.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.152891+0000 osd.3 (osd.3) 528 : cluster 0 6.1f scrub starts 2024-04-06T03:43:05.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.325558+0000 osd.3 (osd.3) 529 : cluster 0 6.1f scrub ok 2024-04-06T03:43:05.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.466143+0000 mgr.y (mgr.24368) 813 : cluster 0 pgmap v885: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T03:43:05.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.878032+0000 osd.2 (osd.2) 417 : cluster 0 5.6 scrub starts 2024-04-06T03:43:05.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.879774+0000 osd.2 (osd.2) 418 : cluster 0 5.6 scrub ok 2024-04-06T03:43:05.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.940022+0000 mon.a (mon.0) 1631 : cluster 1 Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-04-06T03:43:05.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.988533+0000 osd.0 (osd.0) 519 : cluster 0 5.7 scrub starts 2024-04-06T03:43:05.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: cluster 2024-04-06T03:43:04.990124+0000 osd.0 (osd.0) 520 : cluster 0 5.7 scrub ok 2024-04-06T03:43:05.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: audit 2024-04-06T03:43:05.293389+0000 mon.a (mon.0) 1632 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:05.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:05 smithi153 bash[23852]: audit 2024-04-06T03:43:05.294527+0000 mon.a (mon.0) 1633 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:05.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:05.896 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:05.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:05.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:06.026 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:06.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:06.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:06.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:06.260 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:06.289 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:06.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:06.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:06.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:06.543 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:06.553 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:06.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:06.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[29221]: cluster 2024-04-06T03:43:05.232082+0000 osd.4 (osd.4) 505 : cluster 0 6.1a scrub starts 2024-04-06T03:43:06.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[29221]: cluster 2024-04-06T03:43:05.271754+0000 osd.4 (osd.4) 506 : cluster 0 6.1a scrub ok 2024-04-06T03:43:06.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[29221]: cluster 2024-04-06T03:43:05.940395+0000 osd.0 (osd.0) 521 : cluster 0 3.1b scrub starts 2024-04-06T03:43:06.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[29221]: cluster 2024-04-06T03:43:05.943533+0000 osd.0 (osd.0) 522 : cluster 0 3.1b scrub ok 2024-04-06T03:43:06.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[21467]: cluster 2024-04-06T03:43:05.232082+0000 osd.4 (osd.4) 505 : cluster 0 6.1a scrub starts 2024-04-06T03:43:06.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[21467]: cluster 2024-04-06T03:43:05.271754+0000 osd.4 (osd.4) 506 : cluster 0 6.1a scrub ok 2024-04-06T03:43:06.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[21467]: cluster 2024-04-06T03:43:05.940395+0000 osd.0 (osd.0) 521 : cluster 0 3.1b scrub starts 2024-04-06T03:43:06.668 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:06 smithi050 bash[21467]: cluster 2024-04-06T03:43:05.943533+0000 osd.0 (osd.0) 522 : cluster 0 3.1b scrub ok 2024-04-06T03:43:06.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:06.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:06.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:06 smithi153 bash[23852]: cluster 2024-04-06T03:43:05.232082+0000 osd.4 (osd.4) 505 : cluster 0 6.1a scrub starts 2024-04-06T03:43:06.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:06 smithi153 bash[23852]: cluster 2024-04-06T03:43:05.271754+0000 osd.4 (osd.4) 506 : cluster 0 6.1a scrub ok 2024-04-06T03:43:06.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:06 smithi153 bash[23852]: cluster 2024-04-06T03:43:05.940395+0000 osd.0 (osd.0) 521 : cluster 0 3.1b scrub starts 2024-04-06T03:43:06.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:06 smithi153 bash[23852]: cluster 2024-04-06T03:43:05.943533+0000 osd.0 (osd.0) 522 : cluster 0 3.1b scrub ok 2024-04-06T03:43:06.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:06.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:06.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:06.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:07.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:07.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:07.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:07.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:07.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:07.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:07.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:07.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:07.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:07.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:07.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:07.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:07 smithi153 bash[23852]: cluster 2024-04-06T03:43:06.467150+0000 mgr.y (mgr.24368) 814 : cluster 0 pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 544 B/s rd, 0 op/s 2024-04-06T03:43:07.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:07 smithi153 bash[23852]: cluster 2024-04-06T03:43:06.902911+0000 osd.2 (osd.2) 419 : cluster 0 6.f scrub starts 2024-04-06T03:43:07.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:07 smithi153 bash[23852]: cluster 2024-04-06T03:43:07.049541+0000 osd.2 (osd.2) 420 : cluster 0 6.f scrub ok 2024-04-06T03:43:07.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:07 smithi153 bash[23852]: cluster 2024-04-06T03:43:07.282726+0000 osd.7 (osd.7) 537 : cluster 0 3.f scrub starts 2024-04-06T03:43:07.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:07 smithi153 bash[23852]: cluster 2024-04-06T03:43:07.287828+0000 osd.7 (osd.7) 538 : cluster 0 3.f scrub ok 2024-04-06T03:43:07.723 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:07.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:07.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:07.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[21467]: cluster 2024-04-06T03:43:06.467150+0000 mgr.y (mgr.24368) 814 : cluster 0 pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 544 B/s rd, 0 op/s 2024-04-06T03:43:07.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[21467]: cluster 2024-04-06T03:43:06.902911+0000 osd.2 (osd.2) 419 : cluster 0 6.f scrub starts 2024-04-06T03:43:07.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[21467]: cluster 2024-04-06T03:43:07.049541+0000 osd.2 (osd.2) 420 : cluster 0 6.f scrub ok 2024-04-06T03:43:07.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[21467]: cluster 2024-04-06T03:43:07.282726+0000 osd.7 (osd.7) 537 : cluster 0 3.f scrub starts 2024-04-06T03:43:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[21467]: cluster 2024-04-06T03:43:07.287828+0000 osd.7 (osd.7) 538 : cluster 0 3.f scrub ok 2024-04-06T03:43:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[29221]: cluster 2024-04-06T03:43:06.467150+0000 mgr.y (mgr.24368) 814 : cluster 0 pgmap v886: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 544 B/s rd, 0 op/s 2024-04-06T03:43:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[29221]: cluster 2024-04-06T03:43:06.902911+0000 osd.2 (osd.2) 419 : cluster 0 6.f scrub starts 2024-04-06T03:43:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[29221]: cluster 2024-04-06T03:43:07.049541+0000 osd.2 (osd.2) 420 : cluster 0 6.f scrub ok 2024-04-06T03:43:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[29221]: cluster 2024-04-06T03:43:07.282726+0000 osd.7 (osd.7) 537 : cluster 0 3.f scrub starts 2024-04-06T03:43:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:07 smithi050 bash[29221]: cluster 2024-04-06T03:43:07.287828+0000 osd.7 (osd.7) 538 : cluster 0 3.f scrub ok 2024-04-06T03:43:07.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:07.988 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:07.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:08.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:08.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:08.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:08.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:08.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:08.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:08.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:08.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:08.637 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:08.740 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:08.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:08 smithi153 bash[23852]: cluster 2024-04-06T03:43:08.262009+0000 osd.7 (osd.7) 539 : cluster 0 6.12 scrub starts 2024-04-06T03:43:08.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:08 smithi153 bash[23852]: cluster 2024-04-06T03:43:08.294061+0000 osd.7 (osd.7) 540 : cluster 0 6.12 scrub ok 2024-04-06T03:43:08.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:08 smithi050 bash[29221]: cluster 2024-04-06T03:43:08.262009+0000 osd.7 (osd.7) 539 : cluster 0 6.12 scrub starts 2024-04-06T03:43:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:08 smithi050 bash[29221]: cluster 2024-04-06T03:43:08.294061+0000 osd.7 (osd.7) 540 : cluster 0 6.12 scrub ok 2024-04-06T03:43:08.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:08 smithi050 bash[21467]: cluster 2024-04-06T03:43:08.262009+0000 osd.7 (osd.7) 539 : cluster 0 6.12 scrub starts 2024-04-06T03:43:08.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:08 smithi050 bash[21467]: cluster 2024-04-06T03:43:08.294061+0000 osd.7 (osd.7) 540 : cluster 0 6.12 scrub ok 2024-04-06T03:43:08.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:08.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:09.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:09.033 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:09.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:09.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:09.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:09.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:09.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:09.423 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:43:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:43:09.423 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:09.527 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:09.554 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:09.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:09.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:09 smithi153 bash[23852]: cluster 2024-04-06T03:43:08.468024+0000 mgr.y (mgr.24368) 815 : cluster 0 pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:43:09.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:09 smithi153 bash[23852]: cluster 2024-04-06T03:43:08.966857+0000 osd.2 (osd.2) 421 : cluster 0 6.1 scrub starts 2024-04-06T03:43:09.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:09 smithi153 bash[23852]: cluster 2024-04-06T03:43:08.974291+0000 osd.0 (osd.0) 523 : cluster 0 3.17 scrub starts 2024-04-06T03:43:09.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:09 smithi153 bash[23852]: cluster 2024-04-06T03:43:08.976932+0000 osd.0 (osd.0) 524 : cluster 0 3.17 scrub ok 2024-04-06T03:43:09.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:09 smithi153 bash[23852]: cluster 2024-04-06T03:43:09.002650+0000 osd.2 (osd.2) 422 : cluster 0 6.1 scrub ok 2024-04-06T03:43:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:09.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:09.811 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:09.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:09.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[21467]: cluster 2024-04-06T03:43:08.468024+0000 mgr.y (mgr.24368) 815 : cluster 0 pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:43:09.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[21467]: cluster 2024-04-06T03:43:08.966857+0000 osd.2 (osd.2) 421 : cluster 0 6.1 scrub starts 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[21467]: cluster 2024-04-06T03:43:08.974291+0000 osd.0 (osd.0) 523 : cluster 0 3.17 scrub starts 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[21467]: cluster 2024-04-06T03:43:08.976932+0000 osd.0 (osd.0) 524 : cluster 0 3.17 scrub ok 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[21467]: cluster 2024-04-06T03:43:09.002650+0000 osd.2 (osd.2) 422 : cluster 0 6.1 scrub ok 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[29221]: cluster 2024-04-06T03:43:08.468024+0000 mgr.y (mgr.24368) 815 : cluster 0 pgmap v887: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[29221]: cluster 2024-04-06T03:43:08.966857+0000 osd.2 (osd.2) 421 : cluster 0 6.1 scrub starts 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[29221]: cluster 2024-04-06T03:43:08.974291+0000 osd.0 (osd.0) 523 : cluster 0 3.17 scrub starts 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[29221]: cluster 2024-04-06T03:43:08.976932+0000 osd.0 (osd.0) 524 : cluster 0 3.17 scrub ok 2024-04-06T03:43:09.927 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:09 smithi050 bash[29221]: cluster 2024-04-06T03:43:09.002650+0000 osd.2 (osd.2) 422 : cluster 0 6.1 scrub ok 2024-04-06T03:43:09.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:09.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:10.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:10.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:10.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:10.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:10.317 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:43:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:43:10.331 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:10.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:10.460 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:10.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:10.585 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:10.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:10.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:10 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.001838+0000 osd.0 (osd.0) 525 : cluster 0 2.13 scrub starts 2024-04-06T03:43:10.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:10 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.003396+0000 osd.0 (osd.0) 526 : cluster 0 2.13 scrub ok 2024-04-06T03:43:10.700 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:10 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.271664+0000 osd.7 (osd.7) 541 : cluster 0 3.13 scrub starts 2024-04-06T03:43:10.700 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:10 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.274723+0000 osd.7 (osd.7) 542 : cluster 0 3.13 scrub ok 2024-04-06T03:43:10.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:10.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.001838+0000 osd.0 (osd.0) 525 : cluster 0 2.13 scrub starts 2024-04-06T03:43:10.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.003396+0000 osd.0 (osd.0) 526 : cluster 0 2.13 scrub ok 2024-04-06T03:43:10.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.271664+0000 osd.7 (osd.7) 541 : cluster 0 3.13 scrub starts 2024-04-06T03:43:10.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.274723+0000 osd.7 (osd.7) 542 : cluster 0 3.13 scrub ok 2024-04-06T03:43:10.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.001838+0000 osd.0 (osd.0) 525 : cluster 0 2.13 scrub starts 2024-04-06T03:43:10.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.003396+0000 osd.0 (osd.0) 526 : cluster 0 2.13 scrub ok 2024-04-06T03:43:10.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.271664+0000 osd.7 (osd.7) 541 : cluster 0 3.13 scrub starts 2024-04-06T03:43:10.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:10 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.274723+0000 osd.7 (osd.7) 542 : cluster 0 3.13 scrub ok 2024-04-06T03:43:10.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:10.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:10.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:10.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:11.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:11.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:11.241 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:11.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:11.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:11.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:11.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:11.630 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:11.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.093763+0000 osd.3 (osd.3) 530 : cluster 0 3.5 scrub starts 2024-04-06T03:43:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.098998+0000 osd.3 (osd.3) 531 : cluster 0 3.5 scrub ok 2024-04-06T03:43:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.120402+0000 osd.1 (osd.1) 395 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.120529+0000 osd.1 (osd.1) 396 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: audit 2024-04-06T03:43:10.125150+0000 mgr.y (mgr.24368) 816 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.468798+0000 mgr.y (mgr.24368) 817 : cluster 0 pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:10.557299+0000 mon.a (mon.0) 1634 : cluster 0 osdmap e343: 8 total, 8 up, 7 in 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:11.272544+0000 osd.7 (osd.7) 543 : cluster 0 1.0 scrub starts 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[21467]: cluster 2024-04-06T03:43:11.279582+0000 osd.7 (osd.7) 544 : cluster 0 1.0 scrub ok 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.093763+0000 osd.3 (osd.3) 530 : cluster 0 3.5 scrub starts 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.098998+0000 osd.3 (osd.3) 531 : cluster 0 3.5 scrub ok 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.120402+0000 osd.1 (osd.1) 395 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:11.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.120529+0000 osd.1 (osd.1) 396 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:11.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: audit 2024-04-06T03:43:10.125150+0000 mgr.y (mgr.24368) 816 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:11.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.468798+0000 mgr.y (mgr.24368) 817 : cluster 0 pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:43:11.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:10.557299+0000 mon.a (mon.0) 1634 : cluster 0 osdmap e343: 8 total, 8 up, 7 in 2024-04-06T03:43:11.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:11.272544+0000 osd.7 (osd.7) 543 : cluster 0 1.0 scrub starts 2024-04-06T03:43:11.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:11 smithi050 bash[29221]: cluster 2024-04-06T03:43:11.279582+0000 osd.7 (osd.7) 544 : cluster 0 1.0 scrub ok 2024-04-06T03:43:11.759 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:11.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.093763+0000 osd.3 (osd.3) 530 : cluster 0 3.5 scrub starts 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.098998+0000 osd.3 (osd.3) 531 : cluster 0 3.5 scrub ok 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.120402+0000 osd.1 (osd.1) 395 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.120529+0000 osd.1 (osd.1) 396 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: audit 2024-04-06T03:43:10.125150+0000 mgr.y (mgr.24368) 816 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.468798+0000 mgr.y (mgr.24368) 817 : cluster 0 pgmap v888: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:43:11.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:10.557299+0000 mon.a (mon.0) 1634 : cluster 0 osdmap e343: 8 total, 8 up, 7 in 2024-04-06T03:43:11.875 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:11.272544+0000 osd.7 (osd.7) 543 : cluster 0 1.0 scrub starts 2024-04-06T03:43:11.875 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:11 smithi153 bash[23852]: cluster 2024-04-06T03:43:11.279582+0000 osd.7 (osd.7) 544 : cluster 0 1.0 scrub ok 2024-04-06T03:43:11.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:11.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:11.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:12.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:12.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:12.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:12.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:12.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:12.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:12.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:12.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:12.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:12.644 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:12.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:12.771 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:12.772 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:12.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:12.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:12.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:12.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:12.932 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:12.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:13.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:13.063 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:13.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:13.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:13.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:13.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:13.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:13.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:13.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:13.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:13.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:13.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:13.820 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:13.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:13 smithi153 bash[23852]: cluster 2024-04-06T03:43:12.469538+0000 mgr.y (mgr.24368) 818 : cluster 0 pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 600 B/s rd, 0 op/s 2024-04-06T03:43:13.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:13 smithi153 bash[23852]: cluster 2024-04-06T03:43:13.317147+0000 osd.7 (osd.7) 545 : cluster 0 2.1c scrub starts 2024-04-06T03:43:13.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:13 smithi153 bash[23852]: cluster 2024-04-06T03:43:13.319713+0000 osd.7 (osd.7) 546 : cluster 0 2.1c scrub ok 2024-04-06T03:43:13.848 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:13.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:13.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:13.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:14.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:14.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:14.113 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:13 smithi050 bash[29221]: cluster 2024-04-06T03:43:12.469538+0000 mgr.y (mgr.24368) 818 : cluster 0 pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 600 B/s rd, 0 op/s 2024-04-06T03:43:14.113 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:13 smithi050 bash[29221]: cluster 2024-04-06T03:43:13.317147+0000 osd.7 (osd.7) 545 : cluster 0 2.1c scrub starts 2024-04-06T03:43:14.113 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:13 smithi050 bash[29221]: cluster 2024-04-06T03:43:13.319713+0000 osd.7 (osd.7) 546 : cluster 0 2.1c scrub ok 2024-04-06T03:43:14.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:13 smithi050 bash[21467]: cluster 2024-04-06T03:43:12.469538+0000 mgr.y (mgr.24368) 818 : cluster 0 pgmap v890: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 600 B/s rd, 0 op/s 2024-04-06T03:43:14.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:13 smithi050 bash[21467]: cluster 2024-04-06T03:43:13.317147+0000 osd.7 (osd.7) 545 : cluster 0 2.1c scrub starts 2024-04-06T03:43:14.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:13 smithi050 bash[21467]: cluster 2024-04-06T03:43:13.319713+0000 osd.7 (osd.7) 546 : cluster 0 2.1c scrub ok 2024-04-06T03:43:14.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:14.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:14.346 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:14.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:14.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:14.609 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:14.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:14.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:14.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:14.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:14.900 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:14.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:15.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:15.029 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:15.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:14 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.056265+0000 osd.0 (osd.0) 527 : cluster 0 6.0 scrub starts 2024-04-06T03:43:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:14 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.128951+0000 osd.0 (osd.0) 528 : cluster 0 6.0 scrub ok 2024-04-06T03:43:15.144 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:14 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.345561+0000 osd.7 (osd.7) 547 : cluster 0 2.7 scrub starts 2024-04-06T03:43:15.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:14 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.470587+0000 mgr.y (mgr.24368) 819 : cluster 0 pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:15.160 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:15.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.056265+0000 osd.0 (osd.0) 527 : cluster 0 6.0 scrub starts 2024-04-06T03:43:15.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.128951+0000 osd.0 (osd.0) 528 : cluster 0 6.0 scrub ok 2024-04-06T03:43:15.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.345561+0000 osd.7 (osd.7) 547 : cluster 0 2.7 scrub starts 2024-04-06T03:43:15.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.470587+0000 mgr.y (mgr.24368) 819 : cluster 0 pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:15.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.056265+0000 osd.0 (osd.0) 527 : cluster 0 6.0 scrub starts 2024-04-06T03:43:15.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.128951+0000 osd.0 (osd.0) 528 : cluster 0 6.0 scrub ok 2024-04-06T03:43:15.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.345561+0000 osd.7 (osd.7) 547 : cluster 0 2.7 scrub starts 2024-04-06T03:43:15.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:14 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.470587+0000 mgr.y (mgr.24368) 819 : cluster 0 pgmap v891: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:15.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:15.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:15.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:15.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:15.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:15.553 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:15.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:15.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:15.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:15.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:15.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:15.945 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:15.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:16.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:16.076 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:16.139 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:3536 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94633190265216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:35:55.062300+0000", 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 441.07445061599998, 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0313558300000001, 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.140 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3536 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062300+0000", 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062300+0000", 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062301+0000", 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6700000000000001e-06 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062308+0000", 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7560000000000002e-06 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062320+0000", 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2036e-05 2024-04-06T03:43:16.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062329+0000", 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5399999999999996e-06 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062393+0000", 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4819000000000004e-05 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.062400+0000", 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5919999999999997e-06 2024-04-06T03:43:16.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.068306+0000", 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005905881 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:55.068314+0000", 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6399999999999997e-06 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.085618+0000", 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.017304116 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.086745+0000", 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001126911 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.090114+0000", 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033693909999999998 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.090225+0000", 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011146600000000001 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:35:56.093655+0000", 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034300120000000001 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:1784 10.5 10:a357ad56:::benchmark_data_smithi050_121293_object1783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-06T03:43:16.146 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:05.293559+0000", 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 250.84319131999999, 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54544321799999995, 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1784 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293559+0000", 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293559+0000", 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293560+0000", 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.167e-06 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293582+0000", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1722e-05 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293593+0000", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1117999999999999e-05 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293602+0000", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0129999999999999e-06 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293655+0000", 2024-04-06T03:43:16.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2834999999999998e-05 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293776+0000", 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121413 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.293932+0000", 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015556899999999999 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.294506+0000", 2024-04-06T03:43:16.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00057395700000000005 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.296905+0000", 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023993830000000002 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.838715+0000", 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54181025599999999 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.838879+0000", 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016350499999999999 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.839002+0000", 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012328 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:1788 10.5 10:ab1b2d09:::benchmark_data_smithi050_121293_object1787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:05.295589+0000", 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 250.841160938, 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54581453700000004, 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.152 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1788 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295589+0000", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295589+0000", 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295591+0000", 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1990000000000001e-06 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295613+0000", 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2126000000000001e-05 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295626+0000", 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3491e-05 2024-04-06T03:43:16.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295635+0000", 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5120000000000008e-06 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295692+0000", 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7195999999999999e-05 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295812+0000", 2024-04-06T03:43:16.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120596 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.295938+0000", 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012574099999999999 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.296439+0000", 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050093100000000003 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.297104+0000", 2024-04-06T03:43:16.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066479799999999997 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.841111+0000", 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54400658499999999 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.841287+0000", 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017646900000000001 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.841404+0000", 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011689299999999999 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:1798 10.5 10:ab979aa2:::benchmark_data_smithi050_121293_object1797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:05.302506+0000", 2024-04-06T03:43:16.158 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 250.83424436999999, 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53987650899999995, 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1798 2024-04-06T03:43:16.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302506+0000", 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302506+0000", 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302507+0000", 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.525e-06 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302532+0000", 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4298e-05 2024-04-06T03:43:16.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302546+0000", 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4166e-05 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302555+0000", 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6449999999999994e-06 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302625+0000", 2024-04-06T03:43:16.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0246999999999993e-05 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.302865+0000", 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000240499 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.303063+0000", 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00019775399999999999 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.303942+0000", 2024-04-06T03:43:16.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087912199999999998 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.304368+0000", 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042550699999999999 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.842120+0000", 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.537752538 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.842262+0000", 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000142032 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.842382+0000", 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120176 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:1807 10.3 10:c3381f53:::benchmark_data_smithi050_121293_object1806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-04-06T03:43:16.165 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:39:05.308812+0000", 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 250.827938117, 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53315130200000005, 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1807 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308812+0000", 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308812+0000", 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308813+0000", 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1519999999999999e-06 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308848+0000", 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5057e-05 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308861+0000", 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2347000000000001e-05 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308870+0000", 2024-04-06T03:43:16.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1989999999999995e-06 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.308978+0000", 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107842 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309198+0000", 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000219798 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.309364+0000", 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000166761 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.310095+0000", 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073103299999999996 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.311226+0000", 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011302879999999999 2024-04-06T03:43:16.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.841642+0000", 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53041627499999999 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.841843+0000", 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020137 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:39:05.841963+0000", 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012018 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.172 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84284 10.5 10:ab1670b1:::benchmark_data_smithi050_121293_object84283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.251785+0000", 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 178.884965053, 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.553444669, 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.173 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84284 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251785+0000", 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251785+0000", 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251787+0000", 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.654e-06 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251806+0000", 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8978999999999999e-05 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251821+0000", 2024-04-06T03:43:16.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4772e-05 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251828+0000", 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7729999999999999e-06 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251896+0000", 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7665999999999999e-05 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.252160+0000", 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026426100000000001 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.252567+0000", 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040631400000000001 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.256697+0000", 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041303599999999996 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.259869+0000", 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031716029999999998 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.804993+0000", 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54512430300000003 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805108+0000", 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115173 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805230+0000", 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121811 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84296 10.5 10:a00a2d50:::benchmark_data_smithi050_121293_object84295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.260163+0000", 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 178.87658713100001, 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54529861700000004, 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.179 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84296 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260163+0000", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260163+0000", 2024-04-06T03:43:16.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260165+0000", 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.353e-06 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260185+0000", 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0602e-05 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260198+0000", 2024-04-06T03:43:16.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3125999999999999e-05 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260206+0000", 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.038e-06 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260262+0000", 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5720999999999997e-05 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260386+0000", 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012419699999999999 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260536+0000", 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014956799999999999 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.263569+0000", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003033277 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.264076+0000", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050717899999999996 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805305+0000", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54122838200000001 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805395+0000", 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0693999999999996e-05 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805462+0000", 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.648e-05 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84317 10.5 10:a61baf96:::benchmark_data_smithi050_121293_object84316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.281228+0000", 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 178.855522729, 2024-04-06T03:43:16.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53607865799999999, 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84317 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281228+0000", 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281228+0000", 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281229+0000", 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.077e-06 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.187 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281252+0000", 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3617000000000001e-05 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281265+0000", 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3216e-05 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281273+0000", 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9289999999999998e-06 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281333+0000", 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9434000000000001e-05 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281465+0000", 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013207599999999999 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281609+0000", 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014411299999999999 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288194+0000", 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006585105 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.289260+0000", 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010654449999999999 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817035+0000", 2024-04-06T03:43:16.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52777511399999999 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817192+0000", 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000156973 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817306+0000", 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114559 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84322 10.5 10:a7999a3b:::benchmark_data_smithi050_121293_object84321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.285007+0000", 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 178.85174319699999, 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53267631500000001, 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.192 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84322 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285007+0000", 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285007+0000", 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285008+0000", 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.392e-06 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285029+0000", 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0820999999999999e-05 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285041+0000", 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1715e-05 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285050+0000", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8109999999999992e-06 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285128+0000", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8628999999999995e-05 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285273+0000", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000144752 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285417+0000", 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014338400000000001 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288378+0000", 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029612089999999998 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.291212+0000", 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002834016 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817432+0000", 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52621990600000002 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817580+0000", 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148221 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817683+0000", 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103459 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:125694 10.5 10:a235a722:::benchmark_data_smithi050_121293_object26877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:05.968814+0000", 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 130.167935951, 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.517048438, 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 125694 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.198 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968814+0000", 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968814+0000", 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968815+0000", 2024-04-06T03:43:16.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8799999999999998e-07 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968817+0000", 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8899999999999999e-06 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968821+0000", 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.25e-06 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968825+0000", 2024-04-06T03:43:16.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8619999999999996e-06 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.971804+0000", 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029791119999999999 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.971917+0000", 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112909 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.971989+0000", 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2778e-05 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.975226+0000", 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032362430000000002 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.979463+0000", 2024-04-06T03:43:16.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042373840000000003 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485763+0000", 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50629953500000002 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485834+0000", 2024-04-06T03:43:16.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1685000000000002e-05 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485863+0000", 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8501999999999999e-05 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:125712 10.5 10:ae747d95:::benchmark_data_smithi050_121293_object26895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:05.987354+0000", 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 130.14939612800001, 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49859803699999999, 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.204 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 125712 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987354+0000", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987354+0000", 2024-04-06T03:43:16.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987355+0000", 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7300000000000005e-07 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987359+0000", 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1910000000000002e-06 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987389+0000", 2024-04-06T03:43:16.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9692e-05 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987398+0000", 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2669999999999995e-06 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987442+0000", 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3679e-05 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987514+0000", 2024-04-06T03:43:16.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2622999999999999e-05 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.987580+0000", 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5135000000000006e-05 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.990454+0000", 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028741769999999999 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.991973+0000", 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001518929 2024-04-06T03:43:16.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485900+0000", 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49392755300000002 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485929+0000", 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8753999999999999e-05 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485952+0000", 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3264e-05 2024-04-06T03:43:16.209 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:132557 10.5 10:ab2329a2:::benchmark_data_smithi050_121293_object33740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:15.033163+0000", 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 121.103586799, 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50213324599999998, 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.210 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 132557 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033163+0000", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033163+0000", 2024-04-06T03:43:16.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033165+0000", 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.122e-06 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033170+0000", 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7370000000000003e-06 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033181+0000", 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0574000000000001e-05 2024-04-06T03:43:16.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033187+0000", 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1659999999999999e-06 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033245+0000", 2024-04-06T03:43:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7890999999999999e-05 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033383+0000", 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137961 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033511+0000", 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012850700000000001 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.039189+0000", 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056775660000000002 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.044714+0000", 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0055248290000000002 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535199+0000", 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49048494599999998 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535250+0000", 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0898e-05 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535297+0000", 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7049000000000003e-05 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:132558 10.3 10:c450a2ec:::benchmark_data_smithi050_121293_object33741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-06T03:43:16.216 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:15.037802+0000", 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 121.09894828500001, 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49737507399999997, 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 132558 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037802+0000", 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037802+0000", 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037803+0000", 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.406e-06 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037808+0000", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9479999999999999e-06 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037822+0000", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3611999999999999e-05 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037831+0000", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6680000000000007e-06 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037889+0000", 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8350000000000002e-05 2024-04-06T03:43:16.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.037971+0000", 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1686000000000004e-05 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.038037+0000", 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6681999999999996e-05 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.041026+0000", 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029886259999999999 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.041109+0000", 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3360000000000001e-05 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535055+0000", 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49394543600000002 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535129+0000", 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3973000000000005e-05 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.221 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535177+0000", 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8327e-05 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:165868 10.5 10:afb65d11:::benchmark_data_smithi050_121293_object67051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:58.597166+0000", 2024-04-06T03:43:16.222 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 77.539583884999999, 2024-04-06T03:43:16.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49994584800000003, 2024-04-06T03:43:16.223 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.223 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.223 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.223 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.224 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.224 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.224 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 165868 2024-04-06T03:43:16.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.224 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597166+0000", 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597166+0000", 2024-04-06T03:43:16.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597168+0000", 2024-04-06T03:43:16.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2139999999999999e-06 2024-04-06T03:43:16.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597172+0000", 2024-04-06T03:43:16.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8280000000000002e-06 2024-04-06T03:43:16.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597182+0000", 2024-04-06T03:43:16.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1109999999999997e-06 2024-04-06T03:43:16.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597187+0000", 2024-04-06T03:43:16.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3159999999999996e-06 2024-04-06T03:43:16.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597232+0000", 2024-04-06T03:43:16.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4932000000000003e-05 2024-04-06T03:43:16.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597310+0000", 2024-04-06T03:43:16.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8149e-05 2024-04-06T03:43:16.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:43:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597421+0000", 2024-04-06T03:43:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011133699999999999 2024-04-06T03:43:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.606930+0000", 2024-04-06T03:43:16.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095090890000000001 2024-04-06T03:43:16.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.234 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.080788+0000", 2024-04-06T03:43:16.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47385771799999998 2024-04-06T03:43:16.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.096979+0000", 2024-04-06T03:43:16.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016190623000000001 2024-04-06T03:43:16.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.097055+0000", 2024-04-06T03:43:16.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6276e-05 2024-04-06T03:43:16.236 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.236 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.236 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.236 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.097112+0000", 2024-04-06T03:43:16.236 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7255e-05 2024-04-06T03:43:16.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.237 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.237 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.237 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:171114 10.3 10:c26c9224:::benchmark_data_smithi050_121293_object72297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-04-06T03:43:16.238 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:09.004447+0000", 2024-04-06T03:43:16.238 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 67.132303445999995, 2024-04-06T03:43:16.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.58491213500000006, 2024-04-06T03:43:16.238 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.238 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.239 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:43:16.239 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.239 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:43:16.239 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:43:16.239 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 171114 2024-04-06T03:43:16.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.240 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004447+0000", 2024-04-06T03:43:16.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004447+0000", 2024-04-06T03:43:16.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004448+0000", 2024-04-06T03:43:16.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0860000000000001e-06 2024-04-06T03:43:16.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004454+0000", 2024-04-06T03:43:16.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7080000000000002e-06 2024-04-06T03:43:16.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.244 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.244 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004466+0000", 2024-04-06T03:43:16.244 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2658e-05 2024-04-06T03:43:16.244 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.244 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.245 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004475+0000", 2024-04-06T03:43:16.245 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4630000000000001e-06 2024-04-06T03:43:16.245 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.245 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.245 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.246 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.008731+0000", 2024-04-06T03:43:16.246 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004256592 2024-04-06T03:43:16.246 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.246 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.246 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.247 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.008822+0000", 2024-04-06T03:43:16.247 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0902999999999993e-05 2024-04-06T03:43:16.247 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.247 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.247 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-06T03:43:16.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.008908+0000", 2024-04-06T03:43:16.248 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5931000000000001e-05 2024-04-06T03:43:16.248 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.248 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.248 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:43:16.248 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.010809+0000", 2024-04-06T03:43:16.249 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019007760000000001 2024-04-06T03:43:16.249 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.249 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.249 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.249 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.011786+0000", 2024-04-06T03:43:16.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097677300000000001 2024-04-06T03:43:16.250 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.250 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.250 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:43:16.250 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.459408+0000", 2024-04-06T03:43:16.250 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.44762270700000001 2024-04-06T03:43:16.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.251 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.251 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:43:16.251 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.589303+0000", 2024-04-06T03:43:16.251 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12989435099999999 2024-04-06T03:43:16.251 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.589359+0000", 2024-04-06T03:43:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6187000000000003e-05 2024-04-06T03:43:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.252 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:4838 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94633190265216 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-04-06T03:43:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:57.241733+0000", 2024-04-06T03:43:16.253 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.895016956999999, 2024-04-06T03:43:16.254 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024350209999999999, 2024-04-06T03:43:16.254 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.254 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.254 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:43:16.254 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.254 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:43:16.255 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:43:16.255 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4838 2024-04-06T03:43:16.255 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.255 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.255 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241733+0000", 2024-04-06T03:43:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.256 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241733+0000", 2024-04-06T03:43:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.257 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241735+0000", 2024-04-06T03:43:16.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8780000000000001e-06 2024-04-06T03:43:16.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.258 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.258 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.258 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241744+0000", 2024-04-06T03:43:16.258 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8799999999999997e-06 2024-04-06T03:43:16.259 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.259 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.259 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241773+0000", 2024-04-06T03:43:16.259 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8462999999999998e-05 2024-04-06T03:43:16.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.260 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.260 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.260 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241787+0000", 2024-04-06T03:43:16.260 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4674e-05 2024-04-06T03:43:16.260 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.241901+0000", 2024-04-06T03:43:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113751 2024-04-06T03:43:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.261 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.262 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.262 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.242041+0000", 2024-04-06T03:43:16.262 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013955599999999999 2024-04-06T03:43:16.262 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.262 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:57.244168+0000", 2024-04-06T03:43:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002127819 2024-04-06T03:43:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.263 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.264 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.264 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.264 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:4842 5.d 5:b4812045:::notify.4:head [watch ping cookie 94633190264064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-04-06T03:43:16.264 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:43:02.244638+0000", 2024-04-06T03:43:16.264 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.892112086999999, 2024-04-06T03:43:16.264 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012425559999999999, 2024-04-06T03:43:16.265 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.265 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.265 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:43:16.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:43:16.265 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:43:16.266 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4842 2024-04-06T03:43:16.266 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.266 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.266 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.266 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.244638+0000", 2024-04-06T03:43:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.244638+0000", 2024-04-06T03:43:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.244641+0000", 2024-04-06T03:43:16.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6419999999999999e-06 2024-04-06T03:43:16.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.244655+0000", 2024-04-06T03:43:16.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3807e-05 2024-04-06T03:43:16.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.244684+0000", 2024-04-06T03:43:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8863000000000001e-05 2024-04-06T03:43:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.244706+0000", 2024-04-06T03:43:16.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2453999999999999e-05 2024-04-06T03:43:16.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.245641+0000", 2024-04-06T03:43:16.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093549200000000001 2024-04-06T03:43:16.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.272 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.272 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.245708+0000", 2024-04-06T03:43:16.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6483999999999996e-05 2024-04-06T03:43:16.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:02.245881+0000", 2024-04-06T03:43:16.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000172814 2024-04-06T03:43:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:4852 5.d 5:b4812045:::notify.4:head [watch ping cookie 94633190264064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:43:07.248882+0000", 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.8878683249999995, 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010233480000000001, 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.275 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4852 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.248882+0000", 2024-04-06T03:43:16.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.248882+0000", 2024-04-06T03:43:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.278 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.248885+0000", 2024-04-06T03:43:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0939999999999999e-06 2024-04-06T03:43:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.248898+0000", 2024-04-06T03:43:16.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3406e-05 2024-04-06T03:43:16.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.248936+0000", 2024-04-06T03:43:16.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7978999999999997e-05 2024-04-06T03:43:16.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.248960+0000", 2024-04-06T03:43:16.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3808999999999999e-05 2024-04-06T03:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249241+0000", 2024-04-06T03:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000280986 2024-04-06T03:43:16.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249536+0000", 2024-04-06T03:43:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029511799999999998 2024-04-06T03:43:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249905+0000", 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036895599999999998 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:4853 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94633190265216 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-04-06T03:43:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:43:07.249028+0000", 2024-04-06T03:43:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.887722642, 2024-04-06T03:43:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016632699999999999, 2024-04-06T03:43:16.285 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.286 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.286 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:43:16.286 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.286 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:43:16.286 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:43:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4853 2024-04-06T03:43:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249028+0000", 2024-04-06T03:43:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249028+0000", 2024-04-06T03:43:16.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249029+0000", 2024-04-06T03:43:16.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3370000000000001e-06 2024-04-06T03:43:16.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249038+0000", 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3160000000000003e-06 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249064+0000", 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5775999999999999e-05 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249082+0000", 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8113000000000001e-05 2024-04-06T03:43:16.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.249244+0000", 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000161352 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.250388+0000", 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011440739999999999 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:07.250691+0000", 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030330199999999998 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.291 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:4863 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94633190265216 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:43:12.252792+0000", 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.8839587789999999, 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001311836, 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 4863 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.292 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.252792+0000", 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.252792+0000", 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.252794+0000", 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0329999999999998e-06 2024-04-06T03:43:16.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.252803+0000", 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3030000000000005e-06 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.252829+0000", 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6395999999999999e-05 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.252849+0000", 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9496000000000001e-05 2024-04-06T03:43:16.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.253669+0000", 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082053400000000004 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.253924+0000", 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025502899999999999 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:43:12.254103+0000", 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017904499999999999 2024-04-06T03:43:16.295 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.296 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.296 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:43:16.296 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:43:16.296 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:43:16.296 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:16.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.068782+0000 osd.4 (osd.4) 507 : cluster 0 6.e scrub starts 2024-04-06T03:43:16.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.126241+0000 osd.4 (osd.4) 508 : cluster 0 6.e scrub ok 2024-04-06T03:43:16.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[21467]: cluster 2024-04-06T03:43:14.347866+0000 osd.7 (osd.7) 548 : cluster 0 2.7 scrub ok 2024-04-06T03:43:16.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[21467]: cluster 2024-04-06T03:43:15.086636+0000 osd.0 (osd.0) 529 : cluster 0 6.16 scrub starts 2024-04-06T03:43:16.297 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[21467]: cluster 2024-04-06T03:43:15.163092+0000 osd.0 (osd.0) 530 : cluster 0 6.16 scrub ok 2024-04-06T03:43:16.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.068782+0000 osd.4 (osd.4) 507 : cluster 0 6.e scrub starts 2024-04-06T03:43:16.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.126241+0000 osd.4 (osd.4) 508 : cluster 0 6.e scrub ok 2024-04-06T03:43:16.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[29221]: cluster 2024-04-06T03:43:14.347866+0000 osd.7 (osd.7) 548 : cluster 0 2.7 scrub ok 2024-04-06T03:43:16.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[29221]: cluster 2024-04-06T03:43:15.086636+0000 osd.0 (osd.0) 529 : cluster 0 6.16 scrub starts 2024-04-06T03:43:16.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:15 smithi050 bash[29221]: cluster 2024-04-06T03:43:15.163092+0000 osd.0 (osd.0) 530 : cluster 0 6.16 scrub ok 2024-04-06T03:43:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:43:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:43:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:43:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:43:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:43:16.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:43:16.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:15 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.068782+0000 osd.4 (osd.4) 507 : cluster 0 6.e scrub starts 2024-04-06T03:43:16.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:15 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.126241+0000 osd.4 (osd.4) 508 : cluster 0 6.e scrub ok 2024-04-06T03:43:16.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:15 smithi153 bash[23852]: cluster 2024-04-06T03:43:14.347866+0000 osd.7 (osd.7) 548 : cluster 0 2.7 scrub ok 2024-04-06T03:43:16.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:15 smithi153 bash[23852]: cluster 2024-04-06T03:43:15.086636+0000 osd.0 (osd.0) 529 : cluster 0 6.16 scrub starts 2024-04-06T03:43:16.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:15 smithi153 bash[23852]: cluster 2024-04-06T03:43:15.163092+0000 osd.0 (osd.0) 530 : cluster 0 6.16 scrub ok 2024-04-06T03:43:16.310 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:43:16.324 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:16.333 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:43:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:43:16.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:16.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:16.556 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:16.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:16.708 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:16.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:16.840 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:16.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:16.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:16.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:17.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:17.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:16 smithi153 bash[23852]: cluster 2024-04-06T03:43:15.991868+0000 osd.2 (osd.2) 423 : cluster 0 3.0 scrub starts 2024-04-06T03:43:17.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:16 smithi153 bash[23852]: cluster 2024-04-06T03:43:15.996359+0000 osd.2 (osd.2) 424 : cluster 0 3.0 scrub ok 2024-04-06T03:43:17.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:16 smithi153 bash[23852]: audit 2024-04-06T03:43:16.137605+0000 mon.c (mon.2) 103 : audit 0 from='client.? 172.21.15.50:0/1312597150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:43:17.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:16 smithi153 bash[23852]: cluster 2024-04-06T03:43:16.471197+0000 mgr.y (mgr.24368) 820 : cluster 0 pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:17.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:17.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:17.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[21467]: cluster 2024-04-06T03:43:15.991868+0000 osd.2 (osd.2) 423 : cluster 0 3.0 scrub starts 2024-04-06T03:43:17.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[21467]: cluster 2024-04-06T03:43:15.996359+0000 osd.2 (osd.2) 424 : cluster 0 3.0 scrub ok 2024-04-06T03:43:17.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[21467]: audit 2024-04-06T03:43:16.137605+0000 mon.c (mon.2) 103 : audit 0 from='client.? 172.21.15.50:0/1312597150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:43:17.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[21467]: cluster 2024-04-06T03:43:16.471197+0000 mgr.y (mgr.24368) 820 : cluster 0 pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:17.210 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[29221]: cluster 2024-04-06T03:43:15.991868+0000 osd.2 (osd.2) 423 : cluster 0 3.0 scrub starts 2024-04-06T03:43:17.210 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[29221]: cluster 2024-04-06T03:43:15.996359+0000 osd.2 (osd.2) 424 : cluster 0 3.0 scrub ok 2024-04-06T03:43:17.211 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[29221]: audit 2024-04-06T03:43:16.137605+0000 mon.c (mon.2) 103 : audit 0 from='client.? 172.21.15.50:0/1312597150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:43:17.211 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:16 smithi050 bash[29221]: cluster 2024-04-06T03:43:16.471197+0000 mgr.y (mgr.24368) 820 : cluster 0 pgmap v892: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:17.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:17.352 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:17.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:17.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:17.585 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:17.613 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:17.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:17.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:17.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:17.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:17.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:17.981 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:43:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:18.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:17 smithi050 bash[21467]: audit 2024-04-06T03:43:16.975939+0000 mon.a (mon.0) 1635 : audit 1 from='client.? 172.21.15.50:0/1003207840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:43:18.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:17 smithi050 bash[29221]: audit 2024-04-06T03:43:16.975939+0000 mon.a (mon.0) 1635 : audit 1 from='client.? 172.21.15.50:0/1003207840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:43:18.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:18.138 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:18.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:18.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:18.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:17 smithi153 bash[23852]: audit 2024-04-06T03:43:16.975939+0000 mon.a (mon.0) 1635 : audit 1 from='client.? 172.21.15.50:0/1003207840' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:43:18.372 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:18.400 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:18.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:18.533 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:18.637 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:18.664 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:18.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:18.796 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:18.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:18.927 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:18.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:19.030 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:19.056 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:19.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:19.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:19.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.117754+0000 osd.4 (osd.4) 509 : cluster 0 6.2 scrub starts 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.119209+0000 osd.4 (osd.4) 510 : cluster 0 6.2 scrub ok 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.929370+0000 osd.2 (osd.2) 425 : cluster 0 6.14 scrub starts 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.931629+0000 osd.2 (osd.2) 426 : cluster 0 6.14 scrub ok 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.962613+0000 mon.a (mon.0) 1636 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.962670+0000 mon.a (mon.0) 1637 : cluster 1 Cluster is now healthy 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: audit 2024-04-06T03:43:17.973335+0000 mon.a (mon.0) 1638 : audit 1 from='client.? 172.21.15.50:0/1003207840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:17.979000+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e344: 8 total, 8 up, 7 in 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.133340+0000 osd.0 (osd.0) 531 : cluster 0 3.1f scrub starts 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.135322+0000 osd.0 (osd.0) 532 : cluster 0 3.1f scrub ok 2024-04-06T03:43:19.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:18 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.471844+0000 mgr.y (mgr.24368) 821 : cluster 0 pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.117754+0000 osd.4 (osd.4) 509 : cluster 0 6.2 scrub starts 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.119209+0000 osd.4 (osd.4) 510 : cluster 0 6.2 scrub ok 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.929370+0000 osd.2 (osd.2) 425 : cluster 0 6.14 scrub starts 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.931629+0000 osd.2 (osd.2) 426 : cluster 0 6.14 scrub ok 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.962613+0000 mon.a (mon.0) 1636 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.962670+0000 mon.a (mon.0) 1637 : cluster 1 Cluster is now healthy 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: audit 2024-04-06T03:43:17.973335+0000 mon.a (mon.0) 1638 : audit 1 from='client.? 172.21.15.50:0/1003207840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:17.979000+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e344: 8 total, 8 up, 7 in 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.133340+0000 osd.0 (osd.0) 531 : cluster 0 3.1f scrub starts 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.135322+0000 osd.0 (osd.0) 532 : cluster 0 3.1f scrub ok 2024-04-06T03:43:19.303 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.471844+0000 mgr.y (mgr.24368) 821 : cluster 0 pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.117754+0000 osd.4 (osd.4) 509 : cluster 0 6.2 scrub starts 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.119209+0000 osd.4 (osd.4) 510 : cluster 0 6.2 scrub ok 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.929370+0000 osd.2 (osd.2) 425 : cluster 0 6.14 scrub starts 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.931629+0000 osd.2 (osd.2) 426 : cluster 0 6.14 scrub ok 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.962613+0000 mon.a (mon.0) 1636 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.962670+0000 mon.a (mon.0) 1637 : cluster 1 Cluster is now healthy 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: audit 2024-04-06T03:43:17.973335+0000 mon.a (mon.0) 1638 : audit 1 from='client.? 172.21.15.50:0/1003207840' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:43:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:17.979000+0000 mon.a (mon.0) 1639 : cluster 0 osdmap e344: 8 total, 8 up, 7 in 2024-04-06T03:43:19.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.133340+0000 osd.0 (osd.0) 531 : cluster 0 3.1f scrub starts 2024-04-06T03:43:19.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.135322+0000 osd.0 (osd.0) 532 : cluster 0 3.1f scrub ok 2024-04-06T03:43:19.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.471844+0000 mgr.y (mgr.24368) 821 : cluster 0 pgmap v894: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:19.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:19.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:19.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:19.491 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:43:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:43:19.555 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:19.585 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:19.689 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:19.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:19.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:19.848 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:19.951 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:19.977 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:19.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:20.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-06T03:43:20.080 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:20.108 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:20.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:20.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:20.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.141858+0000 osd.4 (osd.4) 511 : cluster 0 5.1 scrub starts 2024-04-06T03:43:20.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.143365+0000 osd.4 (osd.4) 512 : cluster 0 5.1 scrub ok 2024-04-06T03:43:20.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.912306+0000 osd.2 (osd.2) 427 : cluster 0 2.e scrub starts 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:18.914505+0000 osd.2 (osd.2) 428 : cluster 0 2.e scrub ok 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:19.074760+0000 osd.3 (osd.3) 532 : cluster 0 3.c scrub starts 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[21467]: cluster 2024-04-06T03:43:19.078325+0000 osd.3 (osd.3) 533 : cluster 0 3.c scrub ok 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.141858+0000 osd.4 (osd.4) 511 : cluster 0 5.1 scrub starts 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.143365+0000 osd.4 (osd.4) 512 : cluster 0 5.1 scrub ok 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.912306+0000 osd.2 (osd.2) 427 : cluster 0 2.e scrub starts 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:18.914505+0000 osd.2 (osd.2) 428 : cluster 0 2.e scrub ok 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:19.074760+0000 osd.3 (osd.3) 532 : cluster 0 3.c scrub starts 2024-04-06T03:43:20.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:19 smithi050 bash[29221]: cluster 2024-04-06T03:43:19.078325+0000 osd.3 (osd.3) 533 : cluster 0 3.c scrub ok 2024-04-06T03:43:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:20.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:43:20.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:19 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.141858+0000 osd.4 (osd.4) 511 : cluster 0 5.1 scrub starts 2024-04-06T03:43:20.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:19 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.143365+0000 osd.4 (osd.4) 512 : cluster 0 5.1 scrub ok 2024-04-06T03:43:20.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:19 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.912306+0000 osd.2 (osd.2) 427 : cluster 0 2.e scrub starts 2024-04-06T03:43:20.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:19 smithi153 bash[23852]: cluster 2024-04-06T03:43:18.914505+0000 osd.2 (osd.2) 428 : cluster 0 2.e scrub ok 2024-04-06T03:43:20.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:19 smithi153 bash[23852]: cluster 2024-04-06T03:43:19.074760+0000 osd.3 (osd.3) 532 : cluster 0 3.c scrub starts 2024-04-06T03:43:20.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:19 smithi153 bash[23852]: cluster 2024-04-06T03:43:19.078325+0000 osd.3 (osd.3) 533 : cluster 0 3.c scrub ok 2024-04-06T03:43:20.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:20.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:20.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:20.493 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:20.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:20.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:20.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:20.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:20.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:20.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:20.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:20.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:20.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:21.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[23852]: cluster 2024-04-06T03:43:19.188892+0000 osd.4 (osd.4) 513 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:43:21.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[23852]: cluster 2024-04-06T03:43:19.191402+0000 osd.4 (osd.4) 514 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:43:21.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[23852]: audit 2024-04-06T03:43:20.132593+0000 mgr.y (mgr.24368) 822 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:21.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[23852]: audit 2024-04-06T03:43:20.311827+0000 mon.a (mon.0) 1640 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:21.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[23852]: audit 2024-04-06T03:43:20.312903+0000 mon.a (mon.0) 1641 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:21.000 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:20 smithi153 bash[23852]: cluster 2024-04-06T03:43:20.473129+0000 mgr.y (mgr.24368) 823 : cluster 0 pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:43:21.016 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:21.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:21.145 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[21467]: cluster 2024-04-06T03:43:19.188892+0000 osd.4 (osd.4) 513 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:43:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[21467]: cluster 2024-04-06T03:43:19.191402+0000 osd.4 (osd.4) 514 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[21467]: audit 2024-04-06T03:43:20.132593+0000 mgr.y (mgr.24368) 822 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[21467]: audit 2024-04-06T03:43:20.311827+0000 mon.a (mon.0) 1640 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[21467]: audit 2024-04-06T03:43:20.312903+0000 mon.a (mon.0) 1641 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[21467]: cluster 2024-04-06T03:43:20.473129+0000 mgr.y (mgr.24368) 823 : cluster 0 pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[29221]: cluster 2024-04-06T03:43:19.188892+0000 osd.4 (osd.4) 513 : cluster 0 4.1 deep-scrub starts 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[29221]: cluster 2024-04-06T03:43:19.191402+0000 osd.4 (osd.4) 514 : cluster 0 4.1 deep-scrub ok 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[29221]: audit 2024-04-06T03:43:20.132593+0000 mgr.y (mgr.24368) 822 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[29221]: audit 2024-04-06T03:43:20.311827+0000 mon.a (mon.0) 1640 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[29221]: audit 2024-04-06T03:43:20.312903+0000 mon.a (mon.0) 1641 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:20 smithi050 bash[29221]: cluster 2024-04-06T03:43:20.473129+0000 mgr.y (mgr.24368) 823 : cluster 0 pgmap v895: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:43:21.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:21.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:21.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:21.409 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:21.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:21.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:21.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:21.672 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:21.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:21.803 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:21.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:21.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:21.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:22.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:22.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:22.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:22.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:22.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:22.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[29221]: cluster 2024-04-06T03:43:20.946140+0000 osd.2 (osd.2) 429 : cluster 0 5.a scrub starts 2024-04-06T03:43:22.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[29221]: cluster 2024-04-06T03:43:20.948150+0000 osd.2 (osd.2) 430 : cluster 0 5.a scrub ok 2024-04-06T03:43:22.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[29221]: cluster 2024-04-06T03:43:21.033289+0000 osd.3 (osd.3) 534 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:43:22.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[29221]: cluster 2024-04-06T03:43:21.171505+0000 osd.3 (osd.3) 535 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:43:22.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[21467]: cluster 2024-04-06T03:43:20.946140+0000 osd.2 (osd.2) 429 : cluster 0 5.a scrub starts 2024-04-06T03:43:22.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[21467]: cluster 2024-04-06T03:43:20.948150+0000 osd.2 (osd.2) 430 : cluster 0 5.a scrub ok 2024-04-06T03:43:22.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[21467]: cluster 2024-04-06T03:43:21.033289+0000 osd.3 (osd.3) 534 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:43:22.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:21 smithi050 bash[21467]: cluster 2024-04-06T03:43:21.171505+0000 osd.3 (osd.3) 535 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:43:22.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:22.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:21 smithi153 bash[23852]: cluster 2024-04-06T03:43:20.946140+0000 osd.2 (osd.2) 429 : cluster 0 5.a scrub starts 2024-04-06T03:43:22.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:21 smithi153 bash[23852]: cluster 2024-04-06T03:43:20.948150+0000 osd.2 (osd.2) 430 : cluster 0 5.a scrub ok 2024-04-06T03:43:22.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:21 smithi153 bash[23852]: cluster 2024-04-06T03:43:21.033289+0000 osd.3 (osd.3) 534 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:43:22.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:21 smithi153 bash[23852]: cluster 2024-04-06T03:43:21.171505+0000 osd.3 (osd.3) 535 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:43:22.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:22.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:43:22.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:22.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:22.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:22.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:22.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:22.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:22.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:22.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:22.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:23.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:23.065 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[29221]: cluster 2024-04-06T03:43:22.123463+0000 osd.0 (osd.0) 533 : cluster 0 3.10 scrub starts 2024-04-06T03:43:23.065 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[29221]: cluster 2024-04-06T03:43:22.127028+0000 osd.0 (osd.0) 534 : cluster 0 3.10 scrub ok 2024-04-06T03:43:23.065 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[29221]: cluster 2024-04-06T03:43:22.282567+0000 osd.7 (osd.7) 549 : cluster 0 3.16 deep-scrub starts 2024-04-06T03:43:23.065 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[29221]: cluster 2024-04-06T03:43:22.294350+0000 osd.7 (osd.7) 550 : cluster 0 3.16 deep-scrub ok 2024-04-06T03:43:23.065 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[29221]: cluster 2024-04-06T03:43:22.473713+0000 mgr.y (mgr.24368) 824 : cluster 0 pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:23.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[21467]: cluster 2024-04-06T03:43:22.123463+0000 osd.0 (osd.0) 533 : cluster 0 3.10 scrub starts 2024-04-06T03:43:23.066 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[21467]: cluster 2024-04-06T03:43:22.127028+0000 osd.0 (osd.0) 534 : cluster 0 3.10 scrub ok 2024-04-06T03:43:23.066 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[21467]: cluster 2024-04-06T03:43:22.282567+0000 osd.7 (osd.7) 549 : cluster 0 3.16 deep-scrub starts 2024-04-06T03:43:23.066 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[21467]: cluster 2024-04-06T03:43:22.294350+0000 osd.7 (osd.7) 550 : cluster 0 3.16 deep-scrub ok 2024-04-06T03:43:23.066 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:22 smithi050 bash[21467]: cluster 2024-04-06T03:43:22.473713+0000 mgr.y (mgr.24368) 824 : cluster 0 pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:23.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:23.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:22 smithi153 bash[23852]: cluster 2024-04-06T03:43:22.123463+0000 osd.0 (osd.0) 533 : cluster 0 3.10 scrub starts 2024-04-06T03:43:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:22 smithi153 bash[23852]: cluster 2024-04-06T03:43:22.127028+0000 osd.0 (osd.0) 534 : cluster 0 3.10 scrub ok 2024-04-06T03:43:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:22 smithi153 bash[23852]: cluster 2024-04-06T03:43:22.282567+0000 osd.7 (osd.7) 549 : cluster 0 3.16 deep-scrub starts 2024-04-06T03:43:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:22 smithi153 bash[23852]: cluster 2024-04-06T03:43:22.294350+0000 osd.7 (osd.7) 550 : cluster 0 3.16 deep-scrub ok 2024-04-06T03:43:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:22 smithi153 bash[23852]: cluster 2024-04-06T03:43:22.473713+0000 mgr.y (mgr.24368) 824 : cluster 0 pgmap v896: 132 pgs: 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:23.208 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:23.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:23.332 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:23.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:23.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:23.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:23.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:23.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:23.693 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:23.719 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:23.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:23.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:43:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:23.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:23.978 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:23.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_4' created 2024-04-06T03:43:24.014 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-06T03:43:24.080 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:24.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.221 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:23 smithi153 bash[23852]: audit 2024-04-06T03:43:23.196650+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.50:0/2563357290' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-06T03:43:24.221 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:23 smithi153 bash[23852]: audit 2024-04-06T03:43:23.197127+0000 mon.a (mon.0) 1642 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-06T03:43:24.221 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:23 smithi153 bash[23852]: cluster 2024-04-06T03:43:23.246811+0000 osd.7 (osd.7) 551 : cluster 0 6.3 scrub starts 2024-04-06T03:43:24.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:23 smithi153 bash[23852]: cluster 2024-04-06T03:43:23.337141+0000 osd.7 (osd.7) 552 : cluster 0 6.3 scrub ok 2024-04-06T03:43:24.222 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:23 smithi153 bash[23852]: audit 2024-04-06T03:43:23.646917+0000 mon.a (mon.0) 1643 : audit 1 from='client.? 172.21.15.50:0/3364797705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:43:24.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:24.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[21467]: audit 2024-04-06T03:43:23.196650+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.50:0/2563357290' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-06T03:43:24.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[21467]: audit 2024-04-06T03:43:23.197127+0000 mon.a (mon.0) 1642 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-06T03:43:24.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[21467]: cluster 2024-04-06T03:43:23.246811+0000 osd.7 (osd.7) 551 : cluster 0 6.3 scrub starts 2024-04-06T03:43:24.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[21467]: cluster 2024-04-06T03:43:23.337141+0000 osd.7 (osd.7) 552 : cluster 0 6.3 scrub ok 2024-04-06T03:43:24.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[21467]: audit 2024-04-06T03:43:23.646917+0000 mon.a (mon.0) 1643 : audit 1 from='client.? 172.21.15.50:0/3364797705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:43:24.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[29221]: audit 2024-04-06T03:43:23.196650+0000 mon.c (mon.2) 104 : audit 1 from='client.? 172.21.15.50:0/2563357290' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-06T03:43:24.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[29221]: audit 2024-04-06T03:43:23.197127+0000 mon.a (mon.0) 1642 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-04-06T03:43:24.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[29221]: cluster 2024-04-06T03:43:23.246811+0000 osd.7 (osd.7) 551 : cluster 0 6.3 scrub starts 2024-04-06T03:43:24.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[29221]: cluster 2024-04-06T03:43:23.337141+0000 osd.7 (osd.7) 552 : cluster 0 6.3 scrub ok 2024-04-06T03:43:24.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:23 smithi050 bash[29221]: audit 2024-04-06T03:43:23.646917+0000 mon.a (mon.0) 1643 : audit 1 from='client.? 172.21.15.50:0/3364797705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:43:24.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:24.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:24.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:24.499 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:24.603 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.629 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:24.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:24.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:24.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:24.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:24.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:25.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:25.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:25.146 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:23.224215+0000 osd.4 (osd.4) 515 : cluster 0 5.1c scrub starts 2024-04-06T03:43:25.146 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:23.225638+0000 osd.4 (osd.4) 516 : cluster 0 5.1c scrub ok 2024-04-06T03:43:25.146 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:23.985948+0000 osd.2 (osd.2) 431 : cluster 0 2.17 scrub starts 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:23.988129+0000 osd.2 (osd.2) 432 : cluster 0 2.17 scrub ok 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: audit 2024-04-06T03:43:24.000541+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: audit 2024-04-06T03:43:24.000798+0000 mon.a (mon.0) 1645 : audit 1 from='client.? 172.21.15.50:0/3364797705' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:24.007065+0000 mon.a (mon.0) 1646 : cluster 0 osdmap e345: 8 total, 8 up, 7 in 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:24.203382+0000 osd.7 (osd.7) 553 : cluster 0 2.9 scrub starts 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:24.206521+0000 osd.7 (osd.7) 554 : cluster 0 2.9 scrub ok 2024-04-06T03:43:25.147 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:25 smithi153 bash[23852]: cluster 2024-04-06T03:43:24.474733+0000 mgr.y (mgr.24368) 825 : cluster 0 pgmap v898: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:25.162 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:25.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:23.224215+0000 osd.4 (osd.4) 515 : cluster 0 5.1c scrub starts 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:23.225638+0000 osd.4 (osd.4) 516 : cluster 0 5.1c scrub ok 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:23.985948+0000 osd.2 (osd.2) 431 : cluster 0 2.17 scrub starts 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:23.988129+0000 osd.2 (osd.2) 432 : cluster 0 2.17 scrub ok 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: audit 2024-04-06T03:43:24.000541+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: audit 2024-04-06T03:43:24.000798+0000 mon.a (mon.0) 1645 : audit 1 from='client.? 172.21.15.50:0/3364797705' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:24.007065+0000 mon.a (mon.0) 1646 : cluster 0 osdmap e345: 8 total, 8 up, 7 in 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:24.203382+0000 osd.7 (osd.7) 553 : cluster 0 2.9 scrub starts 2024-04-06T03:43:25.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:24.206521+0000 osd.7 (osd.7) 554 : cluster 0 2.9 scrub ok 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[29221]: cluster 2024-04-06T03:43:24.474733+0000 mgr.y (mgr.24368) 825 : cluster 0 pgmap v898: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:23.224215+0000 osd.4 (osd.4) 515 : cluster 0 5.1c scrub starts 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:23.225638+0000 osd.4 (osd.4) 516 : cluster 0 5.1c scrub ok 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:23.985948+0000 osd.2 (osd.2) 431 : cluster 0 2.17 scrub starts 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:23.988129+0000 osd.2 (osd.2) 432 : cluster 0 2.17 scrub ok 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: audit 2024-04-06T03:43:24.000541+0000 mon.a (mon.0) 1644 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: audit 2024-04-06T03:43:24.000798+0000 mon.a (mon.0) 1645 : audit 1 from='client.? 172.21.15.50:0/3364797705' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:24.007065+0000 mon.a (mon.0) 1646 : cluster 0 osdmap e345: 8 total, 8 up, 7 in 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:24.203382+0000 osd.7 (osd.7) 553 : cluster 0 2.9 scrub starts 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:24.206521+0000 osd.7 (osd.7) 554 : cluster 0 2.9 scrub ok 2024-04-06T03:43:25.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:25 smithi050 bash[21467]: cluster 2024-04-06T03:43:24.474733+0000 mgr.y (mgr.24368) 825 : cluster 0 pgmap v898: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:25.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:25.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:25.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:25.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:25.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 2 1 2024-04-06T03:43:25.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-04-06T03:43:25.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:25.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:25.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:25.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:25.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:26.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:26.076 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:26.152 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:26.171 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[21467]: cluster 2024-04-06T03:43:24.252274+0000 osd.4 (osd.4) 517 : cluster 0 3.9 scrub starts 2024-04-06T03:43:26.171 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[21467]: cluster 2024-04-06T03:43:24.254828+0000 osd.4 (osd.4) 518 : cluster 0 3.9 scrub ok 2024-04-06T03:43:26.171 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[21467]: cluster 2024-04-06T03:43:25.001800+0000 mon.a (mon.0) 1647 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[21467]: cluster 2024-04-06T03:43:25.011369+0000 mon.a (mon.0) 1648 : cluster 0 osdmap e346: 8 total, 8 up, 7 in 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[21467]: cluster 2024-04-06T03:43:25.153360+0000 osd.0 (osd.0) 535 : cluster 0 2.2 scrub starts 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[21467]: cluster 2024-04-06T03:43:25.155440+0000 osd.0 (osd.0) 536 : cluster 0 2.2 scrub ok 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[29221]: cluster 2024-04-06T03:43:24.252274+0000 osd.4 (osd.4) 517 : cluster 0 3.9 scrub starts 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[29221]: cluster 2024-04-06T03:43:24.254828+0000 osd.4 (osd.4) 518 : cluster 0 3.9 scrub ok 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[29221]: cluster 2024-04-06T03:43:25.001800+0000 mon.a (mon.0) 1647 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[29221]: cluster 2024-04-06T03:43:25.011369+0000 mon.a (mon.0) 1648 : cluster 0 osdmap e346: 8 total, 8 up, 7 in 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[29221]: cluster 2024-04-06T03:43:25.153360+0000 osd.0 (osd.0) 535 : cluster 0 2.2 scrub starts 2024-04-06T03:43:26.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:26 smithi050 bash[29221]: cluster 2024-04-06T03:43:25.155440+0000 osd.0 (osd.0) 536 : cluster 0 2.2 scrub ok 2024-04-06T03:43:26.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:26.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:26.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:26 smithi153 bash[23852]: cluster 2024-04-06T03:43:24.252274+0000 osd.4 (osd.4) 517 : cluster 0 3.9 scrub starts 2024-04-06T03:43:26.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:26 smithi153 bash[23852]: cluster 2024-04-06T03:43:24.254828+0000 osd.4 (osd.4) 518 : cluster 0 3.9 scrub ok 2024-04-06T03:43:26.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:26 smithi153 bash[23852]: cluster 2024-04-06T03:43:25.001800+0000 mon.a (mon.0) 1647 : cluster 3 Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-06T03:43:26.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:26 smithi153 bash[23852]: cluster 2024-04-06T03:43:25.011369+0000 mon.a (mon.0) 1648 : cluster 0 osdmap e346: 8 total, 8 up, 7 in 2024-04-06T03:43:26.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:26 smithi153 bash[23852]: cluster 2024-04-06T03:43:25.153360+0000 osd.0 (osd.0) 535 : cluster 0 2.2 scrub starts 2024-04-06T03:43:26.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:26 smithi153 bash[23852]: cluster 2024-04-06T03:43:25.155440+0000 osd.0 (osd.0) 536 : cluster 0 2.2 scrub ok 2024-04-06T03:43:26.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:26.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:26.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:26.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:26.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:26.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:26.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:26.734 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:26.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:26.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:26.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:26.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:27.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:25.994222+0000 osd.2 (osd.2) 433 : cluster 0 2.10 scrub starts 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:25.995916+0000 osd.2 (osd.2) 434 : cluster 0 2.10 scrub ok 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.023848+0000 mon.a (mon.0) 1649 : cluster 0 osdmap e347: 8 total, 8 up, 7 in 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.131743+0000 osd.0 (osd.0) 537 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.134274+0000 osd.0 (osd.0) 538 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.161608+0000 osd.3 (osd.3) 536 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.163308+0000 osd.3 (osd.3) 537 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.199353+0000 osd.4 (osd.4) 519 : cluster 0 3.14 scrub starts 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.202025+0000 osd.4 (osd.4) 520 : cluster 0 3.14 scrub ok 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.227745+0000 osd.7 (osd.7) 555 : cluster 0 3.11 scrub starts 2024-04-06T03:43:27.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.230131+0000 osd.7 (osd.7) 556 : cluster 0 3.11 scrub ok 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[21467]: cluster 2024-04-06T03:43:26.475302+0000 mgr.y (mgr.24368) 826 : cluster 0 pgmap v901: 148 pgs: 1 creating+activating, 6 creating+peering, 7 unknown, 134 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:25.994222+0000 osd.2 (osd.2) 433 : cluster 0 2.10 scrub starts 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:25.995916+0000 osd.2 (osd.2) 434 : cluster 0 2.10 scrub ok 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.023848+0000 mon.a (mon.0) 1649 : cluster 0 osdmap e347: 8 total, 8 up, 7 in 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.131743+0000 osd.0 (osd.0) 537 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.134274+0000 osd.0 (osd.0) 538 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.161608+0000 osd.3 (osd.3) 536 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.163308+0000 osd.3 (osd.3) 537 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.199353+0000 osd.4 (osd.4) 519 : cluster 0 3.14 scrub starts 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.202025+0000 osd.4 (osd.4) 520 : cluster 0 3.14 scrub ok 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.227745+0000 osd.7 (osd.7) 555 : cluster 0 3.11 scrub starts 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.230131+0000 osd.7 (osd.7) 556 : cluster 0 3.11 scrub ok 2024-04-06T03:43:27.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:27 smithi050 bash[29221]: cluster 2024-04-06T03:43:26.475302+0000 mgr.y (mgr.24368) 826 : cluster 0 pgmap v901: 148 pgs: 1 creating+activating, 6 creating+peering, 7 unknown, 134 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:27.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:27.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:27.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:25.994222+0000 osd.2 (osd.2) 433 : cluster 0 2.10 scrub starts 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:25.995916+0000 osd.2 (osd.2) 434 : cluster 0 2.10 scrub ok 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.023848+0000 mon.a (mon.0) 1649 : cluster 0 osdmap e347: 8 total, 8 up, 7 in 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.131743+0000 osd.0 (osd.0) 537 : cluster 0 5.1e deep-scrub starts 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.134274+0000 osd.0 (osd.0) 538 : cluster 0 5.1e deep-scrub ok 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.161608+0000 osd.3 (osd.3) 536 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:43:27.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.163308+0000 osd.3 (osd.3) 537 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:43:27.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.199353+0000 osd.4 (osd.4) 519 : cluster 0 3.14 scrub starts 2024-04-06T03:43:27.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.202025+0000 osd.4 (osd.4) 520 : cluster 0 3.14 scrub ok 2024-04-06T03:43:27.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.227745+0000 osd.7 (osd.7) 555 : cluster 0 3.11 scrub starts 2024-04-06T03:43:27.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.230131+0000 osd.7 (osd.7) 556 : cluster 0 3.11 scrub ok 2024-04-06T03:43:27.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:27 smithi153 bash[23852]: cluster 2024-04-06T03:43:26.475302+0000 mgr.y (mgr.24368) 826 : cluster 0 pgmap v901: 148 pgs: 1 creating+activating, 6 creating+peering, 7 unknown, 134 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:27.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:27.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:27.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:27.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:27.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:27.637 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:27.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:27.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:27.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:27.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:27.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:28.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:28.029 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:28.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:28.143 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:28 smithi050 bash[21467]: cluster 2024-04-06T03:43:27.167608+0000 osd.0 (osd.0) 539 : cluster 0 5.15 scrub starts 2024-04-06T03:43:28.144 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:28 smithi050 bash[21467]: cluster 2024-04-06T03:43:27.170702+0000 osd.0 (osd.0) 540 : cluster 0 5.15 scrub ok 2024-04-06T03:43:28.144 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:28 smithi050 bash[29221]: cluster 2024-04-06T03:43:27.167608+0000 osd.0 (osd.0) 539 : cluster 0 5.15 scrub starts 2024-04-06T03:43:28.144 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:28 smithi050 bash[29221]: cluster 2024-04-06T03:43:27.170702+0000 osd.0 (osd.0) 540 : cluster 0 5.15 scrub ok 2024-04-06T03:43:28.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:28.192 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:28.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:28.276 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:28 smithi153 bash[23852]: cluster 2024-04-06T03:43:27.167608+0000 osd.0 (osd.0) 539 : cluster 0 5.15 scrub starts 2024-04-06T03:43:28.276 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:28 smithi153 bash[23852]: cluster 2024-04-06T03:43:27.170702+0000 osd.0 (osd.0) 540 : cluster 0 5.15 scrub ok 2024-04-06T03:43:28.292 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:28.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:28.421 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:28.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:28.554 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:28.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:28.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:28.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:28.814 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:28.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:28.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:28.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:29.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:29.074 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:29.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:29.204 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:29.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[21467]: cluster 2024-04-06T03:43:27.912423+0000 osd.2 (osd.2) 435 : cluster 0 3.4 scrub starts 2024-04-06T03:43:29.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[21467]: cluster 2024-04-06T03:43:27.916050+0000 osd.2 (osd.2) 436 : cluster 0 3.4 scrub ok 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[21467]: cluster 2024-04-06T03:43:28.168640+0000 osd.3 (osd.3) 538 : cluster 0 2.19 scrub starts 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[21467]: cluster 2024-04-06T03:43:28.171176+0000 osd.3 (osd.3) 539 : cluster 0 2.19 scrub ok 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[21467]: cluster 2024-04-06T03:43:28.476589+0000 mgr.y (mgr.24368) 827 : cluster 0 pgmap v902: 148 pgs: 1 creating+activating, 6 creating+peering, 141 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[29221]: cluster 2024-04-06T03:43:27.912423+0000 osd.2 (osd.2) 435 : cluster 0 3.4 scrub starts 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[29221]: cluster 2024-04-06T03:43:27.916050+0000 osd.2 (osd.2) 436 : cluster 0 3.4 scrub ok 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[29221]: cluster 2024-04-06T03:43:28.168640+0000 osd.3 (osd.3) 538 : cluster 0 2.19 scrub starts 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[29221]: cluster 2024-04-06T03:43:28.171176+0000 osd.3 (osd.3) 539 : cluster 0 2.19 scrub ok 2024-04-06T03:43:29.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[29221]: cluster 2024-04-06T03:43:28.476589+0000 mgr.y (mgr.24368) 827 : cluster 0 pgmap v902: 148 pgs: 1 creating+activating, 6 creating+peering, 141 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:43:29.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:29.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:29.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:29 smithi153 bash[23852]: cluster 2024-04-06T03:43:27.912423+0000 osd.2 (osd.2) 435 : cluster 0 3.4 scrub starts 2024-04-06T03:43:29.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:29 smithi153 bash[23852]: cluster 2024-04-06T03:43:27.916050+0000 osd.2 (osd.2) 436 : cluster 0 3.4 scrub ok 2024-04-06T03:43:29.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:29 smithi153 bash[23852]: cluster 2024-04-06T03:43:28.168640+0000 osd.3 (osd.3) 538 : cluster 0 2.19 scrub starts 2024-04-06T03:43:29.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:29 smithi153 bash[23852]: cluster 2024-04-06T03:43:28.171176+0000 osd.3 (osd.3) 539 : cluster 0 2.19 scrub ok 2024-04-06T03:43:29.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:29 smithi153 bash[23852]: cluster 2024-04-06T03:43:28.476589+0000 mgr.y (mgr.24368) 827 : cluster 0 pgmap v902: 148 pgs: 1 creating+activating, 6 creating+peering, 141 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:43:29.463 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:29.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:29.578 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:43:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:43:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:43:29.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:29.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:29.727 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:29.759 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:29.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:29.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:29.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:29.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:30.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:30.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:30.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:30.232 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:30.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:30.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.047914+0000 osd.4 (osd.4) 521 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.048065+0000 osd.4 (osd.4) 522 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.171544+0000 osd.3 (osd.3) 540 : cluster 0 6.11 scrub starts 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.175732+0000 osd.4 (osd.4) 523 : cluster 0 2.6 scrub starts 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.178870+0000 osd.4 (osd.4) 524 : cluster 0 2.6 scrub ok 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.199403+0000 osd.0 (osd.0) 541 : cluster 0 2.15 scrub starts 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.201569+0000 osd.0 (osd.0) 542 : cluster 0 2.15 scrub ok 2024-04-06T03:43:30.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:30 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.215496+0000 osd.3 (osd.3) 541 : cluster 0 6.11 scrub ok 2024-04-06T03:43:30.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:30.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:30.511 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:30.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:30.645 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.047914+0000 osd.4 (osd.4) 521 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.048065+0000 osd.4 (osd.4) 522 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.171544+0000 osd.3 (osd.3) 540 : cluster 0 6.11 scrub starts 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.175732+0000 osd.4 (osd.4) 523 : cluster 0 2.6 scrub starts 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.178870+0000 osd.4 (osd.4) 524 : cluster 0 2.6 scrub ok 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.199403+0000 osd.0 (osd.0) 541 : cluster 0 2.15 scrub starts 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.201569+0000 osd.0 (osd.0) 542 : cluster 0 2.15 scrub ok 2024-04-06T03:43:30.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.215496+0000 osd.3 (osd.3) 541 : cluster 0 6.11 scrub ok 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.047914+0000 osd.4 (osd.4) 521 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.048065+0000 osd.4 (osd.4) 522 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.171544+0000 osd.3 (osd.3) 540 : cluster 0 6.11 scrub starts 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.175732+0000 osd.4 (osd.4) 523 : cluster 0 2.6 scrub starts 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.178870+0000 osd.4 (osd.4) 524 : cluster 0 2.6 scrub ok 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.199403+0000 osd.0 (osd.0) 541 : cluster 0 2.15 scrub starts 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.201569+0000 osd.0 (osd.0) 542 : cluster 0 2.15 scrub ok 2024-04-06T03:43:30.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:30 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.215496+0000 osd.3 (osd.3) 541 : cluster 0 6.11 scrub ok 2024-04-06T03:43:30.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:30.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:30.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:30.900 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:30.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:31.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:31.029 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:31.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:31.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.939674+0000 osd.2 (osd.2) 437 : cluster 0 2.3 scrub starts 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:29.941519+0000 osd.2 (osd.2) 438 : cluster 0 2.3 scrub ok 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:30.068325+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e348: 8 total, 8 up, 7 in 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: audit 2024-04-06T03:43:30.143770+0000 mgr.y (mgr.24368) 828 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:30.204731+0000 osd.3 (osd.3) 542 : cluster 0 5.17 scrub starts 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:30.206383+0000 osd.4 (osd.4) 525 : cluster 0 5.e scrub starts 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:30.210506+0000 osd.4 (osd.4) 526 : cluster 0 5.e scrub ok 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:30.232257+0000 osd.3 (osd.3) 543 : cluster 0 5.17 scrub ok 2024-04-06T03:43:31.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:31 smithi153 bash[23852]: cluster 2024-04-06T03:43:30.477639+0000 mgr.y (mgr.24368) 829 : cluster 0 pgmap v904: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:31.160 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.939674+0000 osd.2 (osd.2) 437 : cluster 0 2.3 scrub starts 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:29.941519+0000 osd.2 (osd.2) 438 : cluster 0 2.3 scrub ok 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:30.068325+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e348: 8 total, 8 up, 7 in 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: audit 2024-04-06T03:43:30.143770+0000 mgr.y (mgr.24368) 828 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:30.204731+0000 osd.3 (osd.3) 542 : cluster 0 5.17 scrub starts 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:30.206383+0000 osd.4 (osd.4) 525 : cluster 0 5.e scrub starts 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:30.210506+0000 osd.4 (osd.4) 526 : cluster 0 5.e scrub ok 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:30.232257+0000 osd.3 (osd.3) 543 : cluster 0 5.17 scrub ok 2024-04-06T03:43:31.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[29221]: cluster 2024-04-06T03:43:30.477639+0000 mgr.y (mgr.24368) 829 : cluster 0 pgmap v904: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.939674+0000 osd.2 (osd.2) 437 : cluster 0 2.3 scrub starts 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:29.941519+0000 osd.2 (osd.2) 438 : cluster 0 2.3 scrub ok 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:30.068325+0000 mon.a (mon.0) 1650 : cluster 0 osdmap e348: 8 total, 8 up, 7 in 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: audit 2024-04-06T03:43:30.143770+0000 mgr.y (mgr.24368) 828 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:30.204731+0000 osd.3 (osd.3) 542 : cluster 0 5.17 scrub starts 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:30.206383+0000 osd.4 (osd.4) 525 : cluster 0 5.e scrub starts 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:30.210506+0000 osd.4 (osd.4) 526 : cluster 0 5.e scrub ok 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:30.232257+0000 osd.3 (osd.3) 543 : cluster 0 5.17 scrub ok 2024-04-06T03:43:31.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:31 smithi050 bash[21467]: cluster 2024-04-06T03:43:30.477639+0000 mgr.y (mgr.24368) 829 : cluster 0 pgmap v904: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:31.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:31.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:31.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:31.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:31.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:31.555 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:31.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:31.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:31.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:31.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:31.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:31.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:31.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:32.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:32.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:32.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:32.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:32.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.344 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:32.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.471 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:32.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.601 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:32.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:32.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:32.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:32.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:32.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:33.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:33.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:33.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:33.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:33.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:33.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:33.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:33.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:33.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:33.636 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:33.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:33.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.150444+0000 osd.3 (osd.3) 544 : cluster 0 2.a scrub starts 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.152473+0000 osd.3 (osd.3) 545 : cluster 0 2.a scrub ok 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.208427+0000 osd.4 (osd.4) 527 : cluster 0 4.2 scrub starts 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.211726+0000 osd.4 (osd.4) 528 : cluster 0 4.2 scrub ok 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.478547+0000 mgr.y (mgr.24368) 830 : cluster 0 pgmap v905: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 685 B/s rd, 0 op/s 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.888201+0000 osd.2 (osd.2) 439 : cluster 0 2.16 scrub starts 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:32.890803+0000 osd.2 (osd.2) 440 : cluster 0 2.16 scrub ok 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:33.092609+0000 osd.0 (osd.0) 543 : cluster 0 5.3 scrub starts 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[21467]: cluster 2024-04-06T03:43:33.096075+0000 osd.0 (osd.0) 544 : cluster 0 5.3 scrub ok 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.150444+0000 osd.3 (osd.3) 544 : cluster 0 2.a scrub starts 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.152473+0000 osd.3 (osd.3) 545 : cluster 0 2.a scrub ok 2024-04-06T03:43:33.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.208427+0000 osd.4 (osd.4) 527 : cluster 0 4.2 scrub starts 2024-04-06T03:43:33.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.211726+0000 osd.4 (osd.4) 528 : cluster 0 4.2 scrub ok 2024-04-06T03:43:33.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.478547+0000 mgr.y (mgr.24368) 830 : cluster 0 pgmap v905: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 685 B/s rd, 0 op/s 2024-04-06T03:43:33.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.888201+0000 osd.2 (osd.2) 439 : cluster 0 2.16 scrub starts 2024-04-06T03:43:33.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:32.890803+0000 osd.2 (osd.2) 440 : cluster 0 2.16 scrub ok 2024-04-06T03:43:33.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:33.092609+0000 osd.0 (osd.0) 543 : cluster 0 5.3 scrub starts 2024-04-06T03:43:33.744 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:33 smithi050 bash[29221]: cluster 2024-04-06T03:43:33.096075+0000 osd.0 (osd.0) 544 : cluster 0 5.3 scrub ok 2024-04-06T03:43:33.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.150444+0000 osd.3 (osd.3) 544 : cluster 0 2.a scrub starts 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.152473+0000 osd.3 (osd.3) 545 : cluster 0 2.a scrub ok 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.208427+0000 osd.4 (osd.4) 527 : cluster 0 4.2 scrub starts 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.211726+0000 osd.4 (osd.4) 528 : cluster 0 4.2 scrub ok 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.478547+0000 mgr.y (mgr.24368) 830 : cluster 0 pgmap v905: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 685 B/s rd, 0 op/s 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.888201+0000 osd.2 (osd.2) 439 : cluster 0 2.16 scrub starts 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:32.890803+0000 osd.2 (osd.2) 440 : cluster 0 2.16 scrub ok 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:33.092609+0000 osd.0 (osd.0) 543 : cluster 0 5.3 scrub starts 2024-04-06T03:43:33.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:33 smithi153 bash[23852]: cluster 2024-04-06T03:43:33.096075+0000 osd.0 (osd.0) 544 : cluster 0 5.3 scrub ok 2024-04-06T03:43:33.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:33.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:33.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:34.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:34.032 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:34.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:34.160 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:34.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:34.292 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:34.390 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:43:34.390 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:43:34.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:34.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:34.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:34.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:34.649 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[29221]: cluster 2024-04-06T03:43:33.108969+0000 osd.3 (osd.3) 546 : cluster 0 6.1b deep-scrub starts 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[29221]: cluster 2024-04-06T03:43:33.261292+0000 osd.3 (osd.3) 547 : cluster 0 6.1b deep-scrub ok 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[29221]: cluster 2024-04-06T03:43:34.094866+0000 osd.0 (osd.0) 545 : cluster 0 2.4 scrub starts 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[29221]: cluster 2024-04-06T03:43:34.097796+0000 osd.0 (osd.0) 546 : cluster 0 2.4 scrub ok 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[21467]: cluster 2024-04-06T03:43:33.108969+0000 osd.3 (osd.3) 546 : cluster 0 6.1b deep-scrub starts 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[21467]: cluster 2024-04-06T03:43:33.261292+0000 osd.3 (osd.3) 547 : cluster 0 6.1b deep-scrub ok 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[21467]: cluster 2024-04-06T03:43:34.094866+0000 osd.0 (osd.0) 545 : cluster 0 2.4 scrub starts 2024-04-06T03:43:34.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:34 smithi050 bash[21467]: cluster 2024-04-06T03:43:34.097796+0000 osd.0 (osd.0) 546 : cluster 0 2.4 scrub ok 2024-04-06T03:43:34.675 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:34.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:34 smithi153 bash[23852]: cluster 2024-04-06T03:43:33.108969+0000 osd.3 (osd.3) 546 : cluster 0 6.1b deep-scrub starts 2024-04-06T03:43:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:34 smithi153 bash[23852]: cluster 2024-04-06T03:43:33.261292+0000 osd.3 (osd.3) 547 : cluster 0 6.1b deep-scrub ok 2024-04-06T03:43:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:34 smithi153 bash[23852]: cluster 2024-04-06T03:43:34.094866+0000 osd.0 (osd.0) 545 : cluster 0 2.4 scrub starts 2024-04-06T03:43:34.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:34 smithi153 bash[23852]: cluster 2024-04-06T03:43:34.097796+0000 osd.0 (osd.0) 546 : cluster 0 2.4 scrub ok 2024-04-06T03:43:34.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:34.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:34.937 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:34.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:35.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:35.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:35.174 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:35.201 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:35.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:35.332 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:35.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.462 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:35.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:34.075361+0000 osd.3 (osd.3) 548 : cluster 0 6.13 scrub starts 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:34.209611+0000 osd.3 (osd.3) 549 : cluster 0 6.13 scrub ok 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:34.479692+0000 mgr.y (mgr.24368) 831 : cluster 0 pgmap v906: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:35.076647+0000 osd.0 (osd.0) 547 : cluster 0 6.9 scrub starts 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:35.178307+0000 osd.0 (osd.0) 548 : cluster 0 6.9 scrub ok 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:35.223200+0000 osd.7 (osd.7) 557 : cluster 0 5.9 scrub starts 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: cluster 2024-04-06T03:43:35.224530+0000 osd.7 (osd.7) 558 : cluster 0 5.9 scrub ok 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: audit 2024-04-06T03:43:35.333570+0000 mon.a (mon.0) 1651 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:35.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[29221]: audit 2024-04-06T03:43:35.335083+0000 mon.a (mon.0) 1652 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:34.075361+0000 osd.3 (osd.3) 548 : cluster 0 6.13 scrub starts 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:34.209611+0000 osd.3 (osd.3) 549 : cluster 0 6.13 scrub ok 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:34.479692+0000 mgr.y (mgr.24368) 831 : cluster 0 pgmap v906: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:35.076647+0000 osd.0 (osd.0) 547 : cluster 0 6.9 scrub starts 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:35.178307+0000 osd.0 (osd.0) 548 : cluster 0 6.9 scrub ok 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:35.223200+0000 osd.7 (osd.7) 557 : cluster 0 5.9 scrub starts 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: cluster 2024-04-06T03:43:35.224530+0000 osd.7 (osd.7) 558 : cluster 0 5.9 scrub ok 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: audit 2024-04-06T03:43:35.333570+0000 mon.a (mon.0) 1651 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:35.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:35 smithi050 bash[21467]: audit 2024-04-06T03:43:35.335083+0000 mon.a (mon.0) 1652 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:35.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:35.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:34.075361+0000 osd.3 (osd.3) 548 : cluster 0 6.13 scrub starts 2024-04-06T03:43:35.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:34.209611+0000 osd.3 (osd.3) 549 : cluster 0 6.13 scrub ok 2024-04-06T03:43:35.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:34.479692+0000 mgr.y (mgr.24368) 831 : cluster 0 pgmap v906: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:35.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:35.076647+0000 osd.0 (osd.0) 547 : cluster 0 6.9 scrub starts 2024-04-06T03:43:35.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:35.178307+0000 osd.0 (osd.0) 548 : cluster 0 6.9 scrub ok 2024-04-06T03:43:35.713 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:35.223200+0000 osd.7 (osd.7) 557 : cluster 0 5.9 scrub starts 2024-04-06T03:43:35.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: cluster 2024-04-06T03:43:35.224530+0000 osd.7 (osd.7) 558 : cluster 0 5.9 scrub ok 2024-04-06T03:43:35.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: audit 2024-04-06T03:43:35.333570+0000 mon.a (mon.0) 1651 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:35.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:35 smithi153 bash[23852]: audit 2024-04-06T03:43:35.335083+0000 mon.a (mon.0) 1652 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:35.731 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:35.834 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.860 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:35.964 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:35.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:36.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:36.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:36.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:36.245 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:36.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:36.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:36.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:36.508 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:36.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:36.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:36 smithi050 bash[21467]: cluster 2024-04-06T03:43:35.188958+0000 osd.4 (osd.4) 529 : cluster 0 6.1e scrub starts 2024-04-06T03:43:36.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:36 smithi050 bash[21467]: cluster 2024-04-06T03:43:35.230012+0000 osd.4 (osd.4) 530 : cluster 0 6.1e scrub ok 2024-04-06T03:43:36.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:36 smithi050 bash[29221]: cluster 2024-04-06T03:43:35.188958+0000 osd.4 (osd.4) 529 : cluster 0 6.1e scrub starts 2024-04-06T03:43:36.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:36 smithi050 bash[29221]: cluster 2024-04-06T03:43:35.230012+0000 osd.4 (osd.4) 530 : cluster 0 6.1e scrub ok 2024-04-06T03:43:36.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:36.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:36.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:36.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:36 smithi153 bash[23852]: cluster 2024-04-06T03:43:35.188958+0000 osd.4 (osd.4) 529 : cluster 0 6.1e scrub starts 2024-04-06T03:43:36.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:36 smithi153 bash[23852]: cluster 2024-04-06T03:43:35.230012+0000 osd.4 (osd.4) 530 : cluster 0 6.1e scrub ok 2024-04-06T03:43:36.866 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:36.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:36.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:36.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:36.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:37.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:37.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:37.148 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:37.251 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:37.278 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:37.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:37.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:37.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:37.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:37.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:37.665 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:37.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[21467]: cluster 2024-04-06T03:43:36.182682+0000 osd.4 (osd.4) 531 : cluster 0 5.1d scrub starts 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[21467]: cluster 2024-04-06T03:43:36.185029+0000 osd.4 (osd.4) 532 : cluster 0 5.1d scrub ok 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[21467]: cluster 2024-04-06T03:43:36.480819+0000 mgr.y (mgr.24368) 832 : cluster 0 pgmap v907: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[21467]: cluster 2024-04-06T03:43:37.049884+0000 osd.0 (osd.0) 549 : cluster 0 3.12 scrub starts 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[21467]: cluster 2024-04-06T03:43:37.053837+0000 osd.0 (osd.0) 550 : cluster 0 3.12 scrub ok 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[29221]: cluster 2024-04-06T03:43:36.182682+0000 osd.4 (osd.4) 531 : cluster 0 5.1d scrub starts 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[29221]: cluster 2024-04-06T03:43:36.185029+0000 osd.4 (osd.4) 532 : cluster 0 5.1d scrub ok 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[29221]: cluster 2024-04-06T03:43:36.480819+0000 mgr.y (mgr.24368) 832 : cluster 0 pgmap v907: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[29221]: cluster 2024-04-06T03:43:37.049884+0000 osd.0 (osd.0) 549 : cluster 0 3.12 scrub starts 2024-04-06T03:43:37.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:37 smithi050 bash[29221]: cluster 2024-04-06T03:43:37.053837+0000 osd.0 (osd.0) 550 : cluster 0 3.12 scrub ok 2024-04-06T03:43:37.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:37.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:37.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:37.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:37 smithi153 bash[23852]: cluster 2024-04-06T03:43:36.182682+0000 osd.4 (osd.4) 531 : cluster 0 5.1d scrub starts 2024-04-06T03:43:37.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:37 smithi153 bash[23852]: cluster 2024-04-06T03:43:36.185029+0000 osd.4 (osd.4) 532 : cluster 0 5.1d scrub ok 2024-04-06T03:43:37.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:37 smithi153 bash[23852]: cluster 2024-04-06T03:43:36.480819+0000 mgr.y (mgr.24368) 832 : cluster 0 pgmap v907: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:37.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:37 smithi153 bash[23852]: cluster 2024-04-06T03:43:37.049884+0000 osd.0 (osd.0) 549 : cluster 0 3.12 scrub starts 2024-04-06T03:43:37.914 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:37 smithi153 bash[23852]: cluster 2024-04-06T03:43:37.053837+0000 osd.0 (osd.0) 550 : cluster 0 3.12 scrub ok 2024-04-06T03:43:37.932 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:37.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:37.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-06T03:43:38.034 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:38.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:38.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:38.189 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:38.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:38.320 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:38.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:38.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:38.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:38.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:38.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:38.712 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:38.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:38 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.244701+0000 osd.7 (osd.7) 559 : cluster 0 3.15 scrub starts 2024-04-06T03:43:38.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:38 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.248244+0000 osd.7 (osd.7) 560 : cluster 0 3.15 scrub ok 2024-04-06T03:43:38.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:38 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.244701+0000 osd.7 (osd.7) 559 : cluster 0 3.15 scrub starts 2024-04-06T03:43:38.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:38 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.248244+0000 osd.7 (osd.7) 560 : cluster 0 3.15 scrub ok 2024-04-06T03:43:38.815 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:38.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:38 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.244701+0000 osd.7 (osd.7) 559 : cluster 0 3.15 scrub starts 2024-04-06T03:43:38.827 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:38 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.248244+0000 osd.7 (osd.7) 560 : cluster 0 3.15 scrub ok 2024-04-06T03:43:38.843 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:38.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:38.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:39.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:39.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:39.213 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:39.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:39.316 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:43:39.336 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:43:39.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:39.357 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:43:39.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:39.377 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:43:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:43:39.424 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:43:39.449 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:43:39.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:39.474 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:43:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:43:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:43:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:43:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:43:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:43:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:43:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:43:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:43:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:39.560 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:43:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:43:39.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:39.627 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:39.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:39 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.481772+0000 mgr.y (mgr.24368) 833 : cluster 0 pgmap v908: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:39.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:39 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.851010+0000 osd.2 (osd.2) 441 : cluster 0 2.1 scrub starts 2024-04-06T03:43:39.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:39 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.853450+0000 osd.2 (osd.2) 442 : cluster 0 2.1 scrub ok 2024-04-06T03:43:39.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:39 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.983685+0000 osd.0 (osd.0) 551 : cluster 0 5.5 scrub starts 2024-04-06T03:43:39.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:39 smithi153 bash[23852]: cluster 2024-04-06T03:43:38.985725+0000 osd.0 (osd.0) 552 : cluster 0 5.5 scrub ok 2024-04-06T03:43:39.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:39 smithi153 bash[23852]: audit 2024-04-06T03:43:39.294645+0000 mon.a (mon.0) 1653 : audit 0 from='client.? 172.21.15.50:0/176729911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:43:39.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.481772+0000 mgr.y (mgr.24368) 833 : cluster 0 pgmap v908: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.851010+0000 osd.2 (osd.2) 441 : cluster 0 2.1 scrub starts 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.853450+0000 osd.2 (osd.2) 442 : cluster 0 2.1 scrub ok 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.983685+0000 osd.0 (osd.0) 551 : cluster 0 5.5 scrub starts 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21467]: cluster 2024-04-06T03:43:38.985725+0000 osd.0 (osd.0) 552 : cluster 0 5.5 scrub ok 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[21467]: audit 2024-04-06T03:43:39.294645+0000 mon.a (mon.0) 1653 : audit 0 from='client.? 172.21.15.50:0/176729911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.481772+0000 mgr.y (mgr.24368) 833 : cluster 0 pgmap v908: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.851010+0000 osd.2 (osd.2) 441 : cluster 0 2.1 scrub starts 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.853450+0000 osd.2 (osd.2) 442 : cluster 0 2.1 scrub ok 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.983685+0000 osd.0 (osd.0) 551 : cluster 0 5.5 scrub starts 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[29221]: cluster 2024-04-06T03:43:38.985725+0000 osd.0 (osd.0) 552 : cluster 0 5.5 scrub ok 2024-04-06T03:43:39.867 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:39 smithi050 bash[29221]: audit 2024-04-06T03:43:39.294645+0000 mon.a (mon.0) 1653 : audit 0 from='client.? 172.21.15.50:0/176729911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:43:39.883 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:39.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:39.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:40.013 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:40.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:40.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:40.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:40.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:40.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:40.299 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:43:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:43:40.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:40.408 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:40.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:40.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:40.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:40.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:40.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[29221]: cluster 2024-04-06T03:43:39.137928+0000 osd.4 (osd.4) 533 : cluster 0 3.3 scrub starts 2024-04-06T03:43:40.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[29221]: cluster 2024-04-06T03:43:39.143526+0000 osd.4 (osd.4) 534 : cluster 0 3.3 scrub ok 2024-04-06T03:43:40.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[29221]: cluster 2024-04-06T03:43:39.843106+0000 osd.2 (osd.2) 443 : cluster 0 5.8 scrub starts 2024-04-06T03:43:40.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[29221]: cluster 2024-04-06T03:43:39.845615+0000 osd.2 (osd.2) 444 : cluster 0 5.8 scrub ok 2024-04-06T03:43:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[21467]: cluster 2024-04-06T03:43:39.137928+0000 osd.4 (osd.4) 533 : cluster 0 3.3 scrub starts 2024-04-06T03:43:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[21467]: cluster 2024-04-06T03:43:39.143526+0000 osd.4 (osd.4) 534 : cluster 0 3.3 scrub ok 2024-04-06T03:43:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[21467]: cluster 2024-04-06T03:43:39.843106+0000 osd.2 (osd.2) 443 : cluster 0 5.8 scrub starts 2024-04-06T03:43:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:40 smithi050 bash[21467]: cluster 2024-04-06T03:43:39.845615+0000 osd.2 (osd.2) 444 : cluster 0 5.8 scrub ok 2024-04-06T03:43:40.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:40 smithi153 bash[23852]: cluster 2024-04-06T03:43:39.137928+0000 osd.4 (osd.4) 533 : cluster 0 3.3 scrub starts 2024-04-06T03:43:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:40 smithi153 bash[23852]: cluster 2024-04-06T03:43:39.143526+0000 osd.4 (osd.4) 534 : cluster 0 3.3 scrub ok 2024-04-06T03:43:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:40 smithi153 bash[23852]: cluster 2024-04-06T03:43:39.843106+0000 osd.2 (osd.2) 443 : cluster 0 5.8 scrub starts 2024-04-06T03:43:40.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:40 smithi153 bash[23852]: cluster 2024-04-06T03:43:39.845615+0000 osd.2 (osd.2) 444 : cluster 0 5.8 scrub ok 2024-04-06T03:43:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:40.901 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:40.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:40.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:41.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:41.058 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:41.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:41.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:41.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:41.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:41.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:41.445 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:41.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:41.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:41.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:41.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:41 smithi153 bash[23852]: audit 2024-04-06T03:43:40.154908+0000 mgr.y (mgr.24368) 834 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:41.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:41 smithi153 bash[23852]: cluster 2024-04-06T03:43:40.483015+0000 mgr.y (mgr.24368) 835 : cluster 0 pgmap v909: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 982 B/s rd, 0 op/s 2024-04-06T03:43:41.707 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:41.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:41.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:41 smithi050 bash[21467]: audit 2024-04-06T03:43:40.154908+0000 mgr.y (mgr.24368) 834 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:41.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:41 smithi050 bash[21467]: cluster 2024-04-06T03:43:40.483015+0000 mgr.y (mgr.24368) 835 : cluster 0 pgmap v909: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 982 B/s rd, 0 op/s 2024-04-06T03:43:41.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:41 smithi050 bash[29221]: audit 2024-04-06T03:43:40.154908+0000 mgr.y (mgr.24368) 834 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:41.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:41 smithi050 bash[29221]: cluster 2024-04-06T03:43:40.483015+0000 mgr.y (mgr.24368) 835 : cluster 0 pgmap v909: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 982 B/s rd, 0 op/s 2024-04-06T03:43:41.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:41.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:41.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:41.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:42.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:42.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:42.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:42.326 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:42.354 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:42.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:42.484 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:42.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:42.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[29221]: cluster 2024-04-06T03:43:41.781627+0000 osd.2 (osd.2) 445 : cluster 0 5.b scrub starts 2024-04-06T03:43:42.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[29221]: cluster 2024-04-06T03:43:41.784147+0000 osd.2 (osd.2) 446 : cluster 0 5.b scrub ok 2024-04-06T03:43:42.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[29221]: cluster 2024-04-06T03:43:41.943479+0000 osd.0 (osd.0) 553 : cluster 0 6.d scrub starts 2024-04-06T03:43:42.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[29221]: cluster 2024-04-06T03:43:42.121030+0000 osd.0 (osd.0) 554 : cluster 0 6.d scrub ok 2024-04-06T03:43:42.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[29221]: cluster 2024-04-06T03:43:42.229253+0000 osd.7 (osd.7) 561 : cluster 0 2.0 scrub starts 2024-04-06T03:43:42.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[29221]: cluster 2024-04-06T03:43:42.231550+0000 osd.7 (osd.7) 562 : cluster 0 2.0 scrub ok 2024-04-06T03:43:42.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[21467]: cluster 2024-04-06T03:43:41.781627+0000 osd.2 (osd.2) 445 : cluster 0 5.b scrub starts 2024-04-06T03:43:42.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[21467]: cluster 2024-04-06T03:43:41.784147+0000 osd.2 (osd.2) 446 : cluster 0 5.b scrub ok 2024-04-06T03:43:42.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[21467]: cluster 2024-04-06T03:43:41.943479+0000 osd.0 (osd.0) 553 : cluster 0 6.d scrub starts 2024-04-06T03:43:42.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[21467]: cluster 2024-04-06T03:43:42.121030+0000 osd.0 (osd.0) 554 : cluster 0 6.d scrub ok 2024-04-06T03:43:42.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[21467]: cluster 2024-04-06T03:43:42.229253+0000 osd.7 (osd.7) 561 : cluster 0 2.0 scrub starts 2024-04-06T03:43:42.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:42 smithi050 bash[21467]: cluster 2024-04-06T03:43:42.231550+0000 osd.7 (osd.7) 562 : cluster 0 2.0 scrub ok 2024-04-06T03:43:42.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:42.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:42.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:42 smithi153 bash[23852]: cluster 2024-04-06T03:43:41.781627+0000 osd.2 (osd.2) 445 : cluster 0 5.b scrub starts 2024-04-06T03:43:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:42 smithi153 bash[23852]: cluster 2024-04-06T03:43:41.784147+0000 osd.2 (osd.2) 446 : cluster 0 5.b scrub ok 2024-04-06T03:43:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:42 smithi153 bash[23852]: cluster 2024-04-06T03:43:41.943479+0000 osd.0 (osd.0) 553 : cluster 0 6.d scrub starts 2024-04-06T03:43:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:42 smithi153 bash[23852]: cluster 2024-04-06T03:43:42.121030+0000 osd.0 (osd.0) 554 : cluster 0 6.d scrub ok 2024-04-06T03:43:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:42 smithi153 bash[23852]: cluster 2024-04-06T03:43:42.229253+0000 osd.7 (osd.7) 561 : cluster 0 2.0 scrub starts 2024-04-06T03:43:42.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:42 smithi153 bash[23852]: cluster 2024-04-06T03:43:42.231550+0000 osd.7 (osd.7) 562 : cluster 0 2.0 scrub ok 2024-04-06T03:43:42.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:42.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:42.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:42.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:43.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:43.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:43.139 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:43.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:43.270 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:43.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:43.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:43.505 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:43.529 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:43.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:43.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:43 smithi050 bash[21467]: cluster 2024-04-06T03:43:41.983429+0000 osd.3 (osd.3) 550 : cluster 0 2.14 scrub starts 2024-04-06T03:43:43.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:43 smithi050 bash[21467]: cluster 2024-04-06T03:43:41.985328+0000 osd.3 (osd.3) 551 : cluster 0 2.14 scrub ok 2024-04-06T03:43:43.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:43 smithi050 bash[21467]: cluster 2024-04-06T03:43:42.483985+0000 mgr.y (mgr.24368) 836 : cluster 0 pgmap v910: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:43.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:43 smithi050 bash[29221]: cluster 2024-04-06T03:43:41.983429+0000 osd.3 (osd.3) 550 : cluster 0 2.14 scrub starts 2024-04-06T03:43:43.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:43 smithi050 bash[29221]: cluster 2024-04-06T03:43:41.985328+0000 osd.3 (osd.3) 551 : cluster 0 2.14 scrub ok 2024-04-06T03:43:43.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:43 smithi050 bash[29221]: cluster 2024-04-06T03:43:42.483985+0000 mgr.y (mgr.24368) 836 : cluster 0 pgmap v910: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:43.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:43.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:43.776 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:43 smithi153 bash[23852]: cluster 2024-04-06T03:43:41.983429+0000 osd.3 (osd.3) 550 : cluster 0 2.14 scrub starts 2024-04-06T03:43:43.776 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:43 smithi153 bash[23852]: cluster 2024-04-06T03:43:41.985328+0000 osd.3 (osd.3) 551 : cluster 0 2.14 scrub ok 2024-04-06T03:43:43.776 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:43 smithi153 bash[23852]: cluster 2024-04-06T03:43:42.483985+0000 mgr.y (mgr.24368) 836 : cluster 0 pgmap v910: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:43:43.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:43.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:43.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:43.922 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:43.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:44.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:44.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:44.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:44.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:44.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:44.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:44.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:44.451 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:44.554 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:44.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:44.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:44.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:44 smithi153 bash[23852]: cluster 2024-04-06T03:43:42.961777+0000 osd.3 (osd.3) 552 : cluster 0 6.4 scrub starts 2024-04-06T03:43:44.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:44 smithi153 bash[23852]: cluster 2024-04-06T03:43:42.964715+0000 osd.3 (osd.3) 553 : cluster 0 6.4 scrub ok 2024-04-06T03:43:44.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:44 smithi153 bash[23852]: cluster 2024-04-06T03:43:43.047540+0000 osd.4 (osd.4) 535 : cluster 0 3.a scrub starts 2024-04-06T03:43:44.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:44 smithi153 bash[23852]: cluster 2024-04-06T03:43:43.052865+0000 osd.4 (osd.4) 536 : cluster 0 3.a scrub ok 2024-04-06T03:43:44.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:44.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[21467]: cluster 2024-04-06T03:43:42.961777+0000 osd.3 (osd.3) 552 : cluster 0 6.4 scrub starts 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[21467]: cluster 2024-04-06T03:43:42.964715+0000 osd.3 (osd.3) 553 : cluster 0 6.4 scrub ok 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[21467]: cluster 2024-04-06T03:43:43.047540+0000 osd.4 (osd.4) 535 : cluster 0 3.a scrub starts 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[21467]: cluster 2024-04-06T03:43:43.052865+0000 osd.4 (osd.4) 536 : cluster 0 3.a scrub ok 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[29221]: cluster 2024-04-06T03:43:42.961777+0000 osd.3 (osd.3) 552 : cluster 0 6.4 scrub starts 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[29221]: cluster 2024-04-06T03:43:42.964715+0000 osd.3 (osd.3) 553 : cluster 0 6.4 scrub ok 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[29221]: cluster 2024-04-06T03:43:43.047540+0000 osd.4 (osd.4) 535 : cluster 0 3.a scrub starts 2024-04-06T03:43:44.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:44 smithi050 bash[29221]: cluster 2024-04-06T03:43:43.052865+0000 osd.4 (osd.4) 536 : cluster 0 3.a scrub ok 2024-04-06T03:43:44.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:43:44.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:44.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:44.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:44.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:44.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:45.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:45.105 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:45.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:45.236 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:45.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:45.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:45.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:45.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:45.585 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:45.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: cluster 2024-04-06T03:43:44.096109+0000 osd.4 (osd.4) 537 : cluster 0 6.1d scrub starts 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: cluster 2024-04-06T03:43:44.166657+0000 osd.4 (osd.4) 538 : cluster 0 6.1d scrub ok 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: cluster 2024-04-06T03:43:44.485253+0000 mgr.y (mgr.24368) 837 : cluster 0 pgmap v911: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: cluster 2024-04-06T03:43:44.916999+0000 osd.2 (osd.2) 447 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: cluster 2024-04-06T03:43:44.917078+0000 osd.2 (osd.2) 448 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: audit 2024-04-06T03:43:45.316037+0000 mon.a (mon.0) 1654 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: audit 2024-04-06T03:43:45.408156+0000 mon.c (mon.2) 105 : audit 1 from='client.? 172.21.15.50:0/303314591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:43:45.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[29221]: audit 2024-04-06T03:43:45.408567+0000 mon.a (mon.0) 1655 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:43:45.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: cluster 2024-04-06T03:43:44.096109+0000 osd.4 (osd.4) 537 : cluster 0 6.1d scrub starts 2024-04-06T03:43:45.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: cluster 2024-04-06T03:43:44.166657+0000 osd.4 (osd.4) 538 : cluster 0 6.1d scrub ok 2024-04-06T03:43:45.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: cluster 2024-04-06T03:43:44.485253+0000 mgr.y (mgr.24368) 837 : cluster 0 pgmap v911: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:45.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: cluster 2024-04-06T03:43:44.916999+0000 osd.2 (osd.2) 447 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:45.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: cluster 2024-04-06T03:43:44.917078+0000 osd.2 (osd.2) 448 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:45.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: audit 2024-04-06T03:43:45.316037+0000 mon.a (mon.0) 1654 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:43:45.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: audit 2024-04-06T03:43:45.408156+0000 mon.c (mon.2) 105 : audit 1 from='client.? 172.21.15.50:0/303314591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:43:45.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:45 smithi050 bash[21467]: audit 2024-04-06T03:43:45.408567+0000 mon.a (mon.0) 1655 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:43:45.609 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:43:45.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:45.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:45.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:45.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:45.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:45.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:45.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:45.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:46.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: cluster 2024-04-06T03:43:44.096109+0000 osd.4 (osd.4) 537 : cluster 0 6.1d scrub starts 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: cluster 2024-04-06T03:43:44.166657+0000 osd.4 (osd.4) 538 : cluster 0 6.1d scrub ok 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: cluster 2024-04-06T03:43:44.485253+0000 mgr.y (mgr.24368) 837 : cluster 0 pgmap v911: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: cluster 2024-04-06T03:43:44.916999+0000 osd.2 (osd.2) 447 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: cluster 2024-04-06T03:43:44.917078+0000 osd.2 (osd.2) 448 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: audit 2024-04-06T03:43:45.316037+0000 mon.a (mon.0) 1654 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:43:46.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: audit 2024-04-06T03:43:45.408156+0000 mon.c (mon.2) 105 : audit 1 from='client.? 172.21.15.50:0/303314591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:43:46.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:45 smithi153 bash[23852]: audit 2024-04-06T03:43:45.408567+0000 mon.a (mon.0) 1655 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:43:46.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:46.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:46.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:46.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:46.254 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:46.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:46.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:46.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:46.609 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-06T03:43:46.610 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-04-06T03:43:46.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:46.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:46.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:46.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: cluster 2024-04-06T03:43:45.588620+0000 mon.a (mon.0) 1656 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:43:46.753 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: audit 2024-04-06T03:43:45.598059+0000 mon.a (mon.0) 1657 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: cluster 2024-04-06T03:43:45.603083+0000 mon.a (mon.0) 1658 : cluster 0 osdmap e349: 8 total, 8 up, 7 in 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: audit 2024-04-06T03:43:45.818283+0000 mon.a (mon.0) 1659 : audit 1 from='client.? 172.21.15.50:0/435820507' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: audit 2024-04-06T03:43:45.841377+0000 mon.a (mon.0) 1660 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: audit 2024-04-06T03:43:45.848926+0000 mon.a (mon.0) 1661 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: audit 2024-04-06T03:43:45.876223+0000 mon.a (mon.0) 1662 : audit 1 from='client.? 172.21.15.50:0/2805859429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: cluster 2024-04-06T03:43:45.915026+0000 osd.0 (osd.0) 555 : cluster 0 6.10 deep-scrub starts 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: cluster 2024-04-06T03:43:45.956968+0000 osd.0 (osd.0) 556 : cluster 0 6.10 deep-scrub ok 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: cluster 2024-04-06T03:43:46.257041+0000 osd.0 (osd.0) 557 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:46.754 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:46 smithi153 bash[23852]: cluster 2024-04-06T03:43:46.257118+0000 osd.0 (osd.0) 558 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:46.768 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:46.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:46.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:46.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: cluster 2024-04-06T03:43:45.588620+0000 mon.a (mon.0) 1656 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: audit 2024-04-06T03:43:45.598059+0000 mon.a (mon.0) 1657 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: cluster 2024-04-06T03:43:45.603083+0000 mon.a (mon.0) 1658 : cluster 0 osdmap e349: 8 total, 8 up, 7 in 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: audit 2024-04-06T03:43:45.818283+0000 mon.a (mon.0) 1659 : audit 1 from='client.? 172.21.15.50:0/435820507' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: audit 2024-04-06T03:43:45.841377+0000 mon.a (mon.0) 1660 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: audit 2024-04-06T03:43:45.848926+0000 mon.a (mon.0) 1661 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: audit 2024-04-06T03:43:45.876223+0000 mon.a (mon.0) 1662 : audit 1 from='client.? 172.21.15.50:0/2805859429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: cluster 2024-04-06T03:43:45.915026+0000 osd.0 (osd.0) 555 : cluster 0 6.10 deep-scrub starts 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: cluster 2024-04-06T03:43:45.956968+0000 osd.0 (osd.0) 556 : cluster 0 6.10 deep-scrub ok 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: cluster 2024-04-06T03:43:46.257041+0000 osd.0 (osd.0) 557 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:46.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[21467]: cluster 2024-04-06T03:43:46.257118+0000 osd.0 (osd.0) 558 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: cluster 2024-04-06T03:43:45.588620+0000 mon.a (mon.0) 1656 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: audit 2024-04-06T03:43:45.598059+0000 mon.a (mon.0) 1657 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: cluster 2024-04-06T03:43:45.603083+0000 mon.a (mon.0) 1658 : cluster 0 osdmap e349: 8 total, 8 up, 7 in 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: audit 2024-04-06T03:43:45.818283+0000 mon.a (mon.0) 1659 : audit 1 from='client.? 172.21.15.50:0/435820507' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: audit 2024-04-06T03:43:45.841377+0000 mon.a (mon.0) 1660 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: audit 2024-04-06T03:43:45.848926+0000 mon.a (mon.0) 1661 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: audit 2024-04-06T03:43:45.876223+0000 mon.a (mon.0) 1662 : audit 1 from='client.? 172.21.15.50:0/2805859429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: cluster 2024-04-06T03:43:45.915026+0000 osd.0 (osd.0) 555 : cluster 0 6.10 deep-scrub starts 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: cluster 2024-04-06T03:43:45.956968+0000 osd.0 (osd.0) 556 : cluster 0 6.10 deep-scrub ok 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: cluster 2024-04-06T03:43:46.257041+0000 osd.0 (osd.0) 557 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:46.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:46 smithi050 bash[29221]: cluster 2024-04-06T03:43:46.257118+0000 osd.0 (osd.0) 558 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:47.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:47.029 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:47.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:47.155 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:47.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:47.286 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:47.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:47.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:47.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:47.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:47.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:47.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:47.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:47 smithi153 bash[23852]: cluster 2024-04-06T03:43:45.967949+0000 osd.3 (osd.3) 554 : cluster 0 2.1e scrub starts 2024-04-06T03:43:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:47 smithi153 bash[23852]: cluster 2024-04-06T03:43:46.486033+0000 mgr.y (mgr.24368) 838 : cluster 0 pgmap v913: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:47 smithi153 bash[23852]: audit 2024-04-06T03:43:46.600082+0000 mon.a (mon.0) 1663 : audit 1 from='client.? 172.21.15.50:0/435820507' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:43:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:47 smithi153 bash[23852]: audit 2024-04-06T03:43:46.600191+0000 mon.a (mon.0) 1664 : audit 1 from='client.? 172.21.15.50:0/2805859429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:43:47.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:47 smithi153 bash[23852]: cluster 2024-04-06T03:43:46.604492+0000 mon.a (mon.0) 1665 : cluster 0 osdmap e350: 8 total, 8 up, 7 in 2024-04-06T03:43:47.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:47.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:47.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[21467]: cluster 2024-04-06T03:43:45.967949+0000 osd.3 (osd.3) 554 : cluster 0 2.1e scrub starts 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[21467]: cluster 2024-04-06T03:43:46.486033+0000 mgr.y (mgr.24368) 838 : cluster 0 pgmap v913: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[21467]: audit 2024-04-06T03:43:46.600082+0000 mon.a (mon.0) 1663 : audit 1 from='client.? 172.21.15.50:0/435820507' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[21467]: audit 2024-04-06T03:43:46.600191+0000 mon.a (mon.0) 1664 : audit 1 from='client.? 172.21.15.50:0/2805859429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[21467]: cluster 2024-04-06T03:43:46.604492+0000 mon.a (mon.0) 1665 : cluster 0 osdmap e350: 8 total, 8 up, 7 in 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[29221]: cluster 2024-04-06T03:43:45.967949+0000 osd.3 (osd.3) 554 : cluster 0 2.1e scrub starts 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[29221]: cluster 2024-04-06T03:43:46.486033+0000 mgr.y (mgr.24368) 838 : cluster 0 pgmap v913: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:43:47.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[29221]: audit 2024-04-06T03:43:46.600082+0000 mon.a (mon.0) 1663 : audit 1 from='client.? 172.21.15.50:0/435820507' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-04-06T03:43:47.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[29221]: audit 2024-04-06T03:43:46.600191+0000 mon.a (mon.0) 1664 : audit 1 from='client.? 172.21.15.50:0/2805859429' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:43:47.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:47 smithi050 bash[29221]: cluster 2024-04-06T03:43:46.604492+0000 mon.a (mon.0) 1665 : cluster 0 osdmap e350: 8 total, 8 up, 7 in 2024-04-06T03:43:48.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:48.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:48.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:48.204 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:48.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:48.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:48.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:48.464 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:48.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:48.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:48.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:48 smithi153 bash[23852]: cluster 2024-04-06T03:43:48.602052+0000 mon.a (mon.0) 1666 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:43:48.724 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:48.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:48.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:48.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:48.983 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:48.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:48 smithi050 bash[21467]: cluster 2024-04-06T03:43:48.602052+0000 mon.a (mon.0) 1666 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:43:48.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:48 smithi050 bash[29221]: cluster 2024-04-06T03:43:48.602052+0000 mon.a (mon.0) 1666 : cluster 1 Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-06T03:43:49.086 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:49.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:49.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:49.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:49.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:49.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:49.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:49.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:49.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:49.617 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:43:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:43:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:43:49.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:49.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:49.761 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:49.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:49.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:49 smithi050 bash[29221]: cluster 2024-04-06T03:43:48.487222+0000 mgr.y (mgr.24368) 839 : cluster 0 pgmap v915: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:49.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:49 smithi050 bash[21467]: cluster 2024-04-06T03:43:48.487222+0000 mgr.y (mgr.24368) 839 : cluster 0 pgmap v915: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:49.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:49.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:49.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:50.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:50.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:49 smithi153 bash[23852]: cluster 2024-04-06T03:43:48.487222+0000 mgr.y (mgr.24368) 839 : cluster 0 pgmap v915: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:50.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 3 1 2024-04-06T03:43:50.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:50.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:50.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:50.286 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:50.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:43:50.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:50.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:50.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:50.530 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:43:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:43:50.542 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:50.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:50.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:50.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:50.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:50.908 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:50.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:50.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:51.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:51.044 DEBUG:teuthology.orchestra.run.smithi050:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-04-06T03:43:51.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:51.060 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-04-06T03:43:51.061 INFO:teuthology.orchestra.run:waiting for 3000 2024-04-06T03:43:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:51.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:51.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:51.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:hints = 1 2024-04-06T03:43:51.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-04-06T03:43:51.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object prefix: benchmark_data_smithi050_138698 2024-04-06T03:43:51.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:43:51.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 0 0 0 0 0 0 - 0 2024-04-06T03:43:51.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:51.425 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:51.458 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[21467]: audit 2024-04-06T03:43:50.164338+0000 mgr.y (mgr.24368) 840 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[21467]: audit 2024-04-06T03:43:50.354132+0000 mon.a (mon.0) 1667 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[21467]: audit 2024-04-06T03:43:50.355512+0000 mon.a (mon.0) 1668 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[21467]: cluster 2024-04-06T03:43:50.488072+0000 mgr.y (mgr.24368) 841 : cluster 0 pgmap v916: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[29221]: audit 2024-04-06T03:43:50.164338+0000 mgr.y (mgr.24368) 840 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[29221]: audit 2024-04-06T03:43:50.354132+0000 mon.a (mon.0) 1667 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[29221]: audit 2024-04-06T03:43:50.355512+0000 mon.a (mon.0) 1668 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:51 smithi050 bash[29221]: cluster 2024-04-06T03:43:50.488072+0000 mgr.y (mgr.24368) 841 : cluster 0 pgmap v916: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:51.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:51.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:51.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:51.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:51.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:51.856 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:51 smithi153 bash[23852]: audit 2024-04-06T03:43:50.164338+0000 mgr.y (mgr.24368) 840 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:43:51.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:51 smithi153 bash[23852]: audit 2024-04-06T03:43:50.354132+0000 mon.a (mon.0) 1667 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:51.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:51 smithi153 bash[23852]: audit 2024-04-06T03:43:50.355512+0000 mon.a (mon.0) 1668 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:43:51.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:51 smithi153 bash[23852]: cluster 2024-04-06T03:43:50.488072+0000 mgr.y (mgr.24368) 841 : cluster 0 pgmap v916: 148 pgs: 148 active+clean; 582 KiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:43:51.870 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:51.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:51.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:52.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:52.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:52.136 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:52.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:52.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 1 16 2556 2540 158.741 158.75 0.00485533 0.00626991 2024-04-06T03:43:52.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:52.377 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:52.413 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:52.520 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:52.548 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:52.560 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:52.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:52.688 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:52.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:52.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:52.926 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:52.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:52.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:53.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:53.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:53.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:53.225 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:53.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 2 16 5356 5340 166.862 175 0.00510133 0.00597527 2024-04-06T03:43:53.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:53.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:53 smithi050 bash[21467]: audit 2024-04-06T03:43:52.242395+0000 mon.a (mon.0) 1669 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:53.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:53 smithi050 bash[21467]: audit 2024-04-06T03:43:52.251123+0000 mon.a (mon.0) 1670 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:53.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:53 smithi050 bash[21467]: cluster 2024-04-06T03:43:52.488757+0000 mgr.y (mgr.24368) 842 : cluster 0 pgmap v917: 148 pgs: 148 active+clean; 10 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 1.2 MiB/s wr, 20 op/s 2024-04-06T03:43:53.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:53 smithi050 bash[29221]: audit 2024-04-06T03:43:52.242395+0000 mon.a (mon.0) 1669 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:53.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:53 smithi050 bash[29221]: audit 2024-04-06T03:43:52.251123+0000 mon.a (mon.0) 1670 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:53.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:53 smithi050 bash[29221]: cluster 2024-04-06T03:43:52.488757+0000 mgr.y (mgr.24368) 842 : cluster 0 pgmap v917: 148 pgs: 148 active+clean; 10 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 1.2 MiB/s wr, 20 op/s 2024-04-06T03:43:53.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:53.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:53.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:53.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:53 smithi153 bash[23852]: audit 2024-04-06T03:43:52.242395+0000 mon.a (mon.0) 1669 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:53.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:53 smithi153 bash[23852]: audit 2024-04-06T03:43:52.251123+0000 mon.a (mon.0) 1670 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:53.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:53 smithi153 bash[23852]: cluster 2024-04-06T03:43:52.488757+0000 mgr.y (mgr.24368) 842 : cluster 0 pgmap v917: 148 pgs: 148 active+clean; 10 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 1.2 MiB/s wr, 20 op/s 2024-04-06T03:43:53.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:53.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:53.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:53.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:53.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:53.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:53.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:53.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:53.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:54.007 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:54.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:54.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:54.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:54.244 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 3 16 7750 7734 161.111 149.625 0.00144779 0.00599092 2024-04-06T03:43:54.265 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:54.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:54.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:54.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:54.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:54.590 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:54.592 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:43:54.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:54.657 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:54.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:54.783 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:54.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:54.910 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:54.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:55.012 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:55.022 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:54 smithi153 bash[23852]: cluster 2024-04-06T03:43:54.489561+0000 mgr.y (mgr.24368) 843 : cluster 0 pgmap v918: 148 pgs: 148 active+clean; 150 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 270 op/s 2024-04-06T03:43:55.037 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:55.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:55.150 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:54 smithi050 bash[21467]: cluster 2024-04-06T03:43:54.489561+0000 mgr.y (mgr.24368) 843 : cluster 0 pgmap v918: 148 pgs: 148 active+clean; 150 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 270 op/s 2024-04-06T03:43:55.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:54 smithi050 bash[29221]: cluster 2024-04-06T03:43:54.489561+0000 mgr.y (mgr.24368) 843 : cluster 0 pgmap v918: 148 pgs: 148 active+clean; 150 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 270 op/s 2024-04-06T03:43:55.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:55.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 4 16 9630 9614 150.206 117.5 0.00152503 0.00664163 2024-04-06T03:43:55.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:55.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:43:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:43:55.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:55.414 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:55.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:55.543 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:55.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:55.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:55.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:55.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:55.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:55.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:56.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:56.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:56.229 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:56.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 5 16 11594 11578 144.713 122.75 0.00424078 0.00690118 2024-04-06T03:43:56.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:56.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:56.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:56.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:56.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:56.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:56.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:56.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:56.859 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:56.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:56.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:56.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:57.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:57.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:57.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:57.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 6 16 14014 13998 145.8 151.25 0.00625962 0.00685011 2024-04-06T03:43:57.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:57.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:57.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:57.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:57.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:57.527 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:57.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:57 smithi050 bash[21467]: cluster 2024-04-06T03:43:56.490573+0000 mgr.y (mgr.24368) 844 : cluster 0 pgmap v919: 148 pgs: 148 active+clean; 381 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 609 op/s 2024-04-06T03:43:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:57 smithi050 bash[21467]: cluster 2024-04-06T03:43:57.191447+0000 osd.7 (osd.7) 563 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:57.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:57 smithi050 bash[21467]: cluster 2024-04-06T03:43:57.191511+0000 osd.7 (osd.7) 564 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:57.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:57 smithi050 bash[29221]: cluster 2024-04-06T03:43:56.490573+0000 mgr.y (mgr.24368) 844 : cluster 0 pgmap v919: 148 pgs: 148 active+clean; 381 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 609 op/s 2024-04-06T03:43:57.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:57 smithi050 bash[29221]: cluster 2024-04-06T03:43:57.191447+0000 osd.7 (osd.7) 563 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:57.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:57 smithi050 bash[29221]: cluster 2024-04-06T03:43:57.191511+0000 osd.7 (osd.7) 564 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:57.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:57.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:57.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:57 smithi153 bash[23852]: cluster 2024-04-06T03:43:56.490573+0000 mgr.y (mgr.24368) 844 : cluster 0 pgmap v919: 148 pgs: 148 active+clean; 381 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 38 MiB/s wr, 609 op/s 2024-04-06T03:43:57.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:57 smithi153 bash[23852]: cluster 2024-04-06T03:43:57.191447+0000 osd.7 (osd.7) 563 : cluster 0 purged_snaps scrub starts 2024-04-06T03:43:57.769 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:57 smithi153 bash[23852]: cluster 2024-04-06T03:43:57.191511+0000 osd.7 (osd.7) 564 : cluster 0 purged_snaps scrub ok 2024-04-06T03:43:57.785 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:57.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:57.916 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:57.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:58.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:58.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:58.156 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:58.185 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:58.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 7 16 16220 16204 144.667 137.875 0.0102373 0.00690308 2024-04-06T03:43:58.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:58.310 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:43:58.310 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:43:58.311 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:43:58.311 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:43:58.311 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:43:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:43:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:43:58.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:58.456 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:43:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:43:58.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:58.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:58.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:58.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:58 smithi153 bash[23852]: cluster 2024-04-06T03:43:57.550760+0000 mon.a (mon.0) 1671 : cluster 0 osdmap e351: 8 total, 8 up, 7 in 2024-04-06T03:43:58.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:58 smithi153 bash[23852]: audit 2024-04-06T03:43:58.439778+0000 mon.a (mon.0) 1672 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:58.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:58 smithi153 bash[23852]: audit 2024-04-06T03:43:58.451159+0000 mon.a (mon.0) 1673 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:58.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:58.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:58.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:58.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:58.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:58 smithi050 bash[29221]: cluster 2024-04-06T03:43:57.550760+0000 mon.a (mon.0) 1671 : cluster 0 osdmap e351: 8 total, 8 up, 7 in 2024-04-06T03:43:58.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:58 smithi050 bash[29221]: audit 2024-04-06T03:43:58.439778+0000 mon.a (mon.0) 1672 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:58.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:43:58 smithi050 bash[29221]: audit 2024-04-06T03:43:58.451159+0000 mon.a (mon.0) 1673 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:58.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:58.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:43:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:43:59.102 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:59.127 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:43:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:43:59.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:59.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:58 smithi050 bash[21467]: cluster 2024-04-06T03:43:57.550760+0000 mon.a (mon.0) 1671 : cluster 0 osdmap e351: 8 total, 8 up, 7 in 2024-04-06T03:43:59.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:58 smithi050 bash[21467]: audit 2024-04-06T03:43:58.439778+0000 mon.a (mon.0) 1672 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:59.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:58 smithi050 bash[21467]: audit 2024-04-06T03:43:58.451159+0000 mon.a (mon.0) 1673 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:43:59.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 8 15 18266 18251 142.575 127.938 0.0105312 0.00700685 2024-04-06T03:43:59.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:59.359 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:59.390 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:43:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:43:59.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:59.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:59.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:59.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:43:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:43:59.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:59.779 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:59.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:59.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:43:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:43:59.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:43:59.916 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:43:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:43:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:00.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:00.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:59 smithi153 bash[23852]: cluster 2024-04-06T03:43:58.491282+0000 mgr.y (mgr.24368) 845 : cluster 0 pgmap v921: 148 pgs: 148 active+clean; 693 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:44:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:59 smithi153 bash[23852]: audit 2024-04-06T03:43:59.285875+0000 mon.a (mon.0) 1674 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:44:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:59 smithi153 bash[23852]: audit 2024-04-06T03:43:59.288252+0000 mon.a (mon.0) 1675 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:44:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:43:59 smithi153 bash[23852]: audit 2024-04-06T03:43:59.296442+0000 mon.a (mon.0) 1676 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:44:00.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:00.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:00.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 9 16 19664 19648 136.434 87.3125 0.00498161 0.0073227 2024-04-06T03:44:00.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:00.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:00.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:00.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:59 smithi050 bash[21467]: cluster 2024-04-06T03:43:58.491282+0000 mgr.y (mgr.24368) 845 : cluster 0 pgmap v921: 148 pgs: 148 active+clean; 693 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:44:00.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:59 smithi050 bash[21467]: audit 2024-04-06T03:43:59.285875+0000 mon.a (mon.0) 1674 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:44:00.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:59 smithi050 bash[21467]: audit 2024-04-06T03:43:59.288252+0000 mon.a (mon.0) 1675 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:44:00.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:43:59 smithi050 bash[21467]: audit 2024-04-06T03:43:59.296442+0000 mon.a (mon.0) 1676 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:44:00.434 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:00.538 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:00.549 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:44:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:44:00.565 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:00.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:00.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:00 smithi050 bash[29221]: cluster 2024-04-06T03:43:58.491282+0000 mgr.y (mgr.24368) 845 : cluster 0 pgmap v921: 148 pgs: 148 active+clean; 693 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-04-06T03:44:00.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:00 smithi050 bash[29221]: audit 2024-04-06T03:43:59.285875+0000 mon.a (mon.0) 1674 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:44:00.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:00 smithi050 bash[29221]: audit 2024-04-06T03:43:59.288252+0000 mon.a (mon.0) 1675 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:44:00.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:00 smithi050 bash[29221]: audit 2024-04-06T03:43:59.296442+0000 mon.a (mon.0) 1676 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:44:00.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:00.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:00.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:00.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:00.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:00.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:01.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:01.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:01.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:01.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:01.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 10 16 21547 21531 134.558 117.688 0.00348218 0.00742653 2024-04-06T03:44:01.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:01.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:01.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:01.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:01.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:01.610 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:01.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:01.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:01 smithi050 bash[29221]: audit 2024-04-06T03:44:00.171956+0000 mgr.y (mgr.24368) 846 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:01.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:01 smithi050 bash[29221]: cluster 2024-04-06T03:44:00.492092+0000 mgr.y (mgr.24368) 847 : cluster 0 pgmap v922: 148 pgs: 148 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-04-06T03:44:01.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:01 smithi050 bash[21467]: audit 2024-04-06T03:44:00.171956+0000 mgr.y (mgr.24368) 846 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:01.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:01 smithi050 bash[21467]: cluster 2024-04-06T03:44:00.492092+0000 mgr.y (mgr.24368) 847 : cluster 0 pgmap v922: 148 pgs: 148 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-04-06T03:44:01.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:01.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:01.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:01 smithi153 bash[23852]: audit 2024-04-06T03:44:00.171956+0000 mgr.y (mgr.24368) 846 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:01.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:01 smithi153 bash[23852]: cluster 2024-04-06T03:44:00.492092+0000 mgr.y (mgr.24368) 847 : cluster 0 pgmap v922: 148 pgs: 148 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-04-06T03:44:01.871 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:01.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:01.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:01.998 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:02.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:02.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:02.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:02.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 11 16 22637 22621 128.518 68.125 0.00658883 0.00777388 2024-04-06T03:44:02.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:02.371 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:02.403 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:02.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:02.532 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:02.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:02.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:02.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:02.789 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:02.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:02.924 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:02.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:03.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:03.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:03.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:03.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 12 16 24481 24465 127.411 115.25 0.00511453 0.00783144 2024-04-06T03:44:03.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:03.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:03.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:03.437 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:03.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:03.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:03.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:03.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:03.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:03 smithi153 bash[23852]: cluster 2024-04-06T03:44:02.492677+0000 mgr.y (mgr.24368) 848 : cluster 0 pgmap v923: 148 pgs: 148 active+clean; 1.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-04-06T03:44:03.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:03.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:03.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:03.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:03 smithi050 bash[21467]: cluster 2024-04-06T03:44:02.492677+0000 mgr.y (mgr.24368) 848 : cluster 0 pgmap v923: 148 pgs: 148 active+clean; 1.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-04-06T03:44:03.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:03 smithi050 bash[29221]: cluster 2024-04-06T03:44:02.492677+0000 mgr.y (mgr.24368) 848 : cluster 0 pgmap v923: 148 pgs: 148 active+clean; 1.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-04-06T03:44:03.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:03.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:04.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:04.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:04.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:04.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:04.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 13 16 25998 25982 124.903 94.8125 0.00462509 0.00800015 2024-04-06T03:44:04.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:04.352 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:04.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:04.479 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:04.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-06T03:44:04.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:04.637 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:04.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:04.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:04.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:04.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:04.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:05.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:05.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:05.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:05.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:05.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 14 16 27914 27898 124.534 119.75 0.00942289 0.00802488 2024-04-06T03:44:05.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:05.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:05.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:05.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:05.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:05.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:05.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:05.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:05.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:05.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:05.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:05.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:05 smithi153 bash[23852]: cluster 2024-04-06T03:44:04.493271+0000 mgr.y (mgr.24368) 849 : cluster 0 pgmap v924: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-04-06T03:44:05.938 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:05 smithi153 bash[23852]: audit 2024-04-06T03:44:05.371412+0000 mon.a (mon.0) 1677 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:05.963 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:05.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:06.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:06.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:06.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:06.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:06.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:05 smithi050 bash[21467]: cluster 2024-04-06T03:44:04.493271+0000 mgr.y (mgr.24368) 849 : cluster 0 pgmap v924: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-04-06T03:44:06.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:05 smithi050 bash[21467]: audit 2024-04-06T03:44:05.371412+0000 mon.a (mon.0) 1677 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:06.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:05 smithi050 bash[29221]: cluster 2024-04-06T03:44:04.493271+0000 mgr.y (mgr.24368) 849 : cluster 0 pgmap v924: 148 pgs: 148 active+clean; 1.3 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-04-06T03:44:06.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:05 smithi050 bash[29221]: audit 2024-04-06T03:44:05.371412+0000 mon.a (mon.0) 1677 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:06.245 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 15 16 29073 29057 121.061 72.4375 0.00537738 0.00825461 2024-04-06T03:44:06.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:06.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:06.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:06.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:06.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:06.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:06.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:06.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:06.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:06.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:06.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:06.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:07.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:07.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:07.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:07.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 16 16 31012 30996 121.068 121.188 0.0117589 0.00825299 2024-04-06T03:44:07.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:07.284 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:07.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:07.409 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:07.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:07.551 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:07.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:07.685 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:07.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:07 smithi050 bash[21467]: cluster 2024-04-06T03:44:06.494132+0000 mgr.y (mgr.24368) 850 : cluster 0 pgmap v925: 148 pgs: 148 active+clean; 1.5 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-04-06T03:44:07.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:07 smithi050 bash[29221]: cluster 2024-04-06T03:44:06.494132+0000 mgr.y (mgr.24368) 850 : cluster 0 pgmap v925: 148 pgs: 148 active+clean; 1.5 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-04-06T03:44:07.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:07.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:07 smithi153 bash[23852]: cluster 2024-04-06T03:44:06.494132+0000 mgr.y (mgr.24368) 850 : cluster 0 pgmap v925: 148 pgs: 148 active+clean; 1.5 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-04-06T03:44:07.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:07.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:07.947 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:07.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:08.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:08.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.206 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:08.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 17 16 32652 32636 119.975 102.5 0.00245518 0.00831105 2024-04-06T03:44:08.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:08.332 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:08.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:08.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:08.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:08.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.735 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:08.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:08.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:08.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:08 smithi050 bash[21467]: cluster 2024-04-06T03:44:07.163009+0000 osd.6 (osd.6) 107 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:08.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:08 smithi050 bash[21467]: cluster 2024-04-06T03:44:07.163075+0000 osd.6 (osd.6) 108 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:08.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:08 smithi050 bash[21467]: cluster 2024-04-06T03:44:07.577608+0000 mon.a (mon.0) 1678 : cluster 0 osdmap e352: 8 total, 8 up, 7 in 2024-04-06T03:44:08.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:08 smithi050 bash[29221]: cluster 2024-04-06T03:44:07.163009+0000 osd.6 (osd.6) 107 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:08.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:08 smithi050 bash[29221]: cluster 2024-04-06T03:44:07.163075+0000 osd.6 (osd.6) 108 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:08.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:08 smithi050 bash[29221]: cluster 2024-04-06T03:44:07.577608+0000 mon.a (mon.0) 1678 : cluster 0 osdmap e352: 8 total, 8 up, 7 in 2024-04-06T03:44:08.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:08.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:08.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:08.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:08.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:08 smithi153 bash[23852]: cluster 2024-04-06T03:44:07.163009+0000 osd.6 (osd.6) 107 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:08.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:08 smithi153 bash[23852]: cluster 2024-04-06T03:44:07.163075+0000 osd.6 (osd.6) 108 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:08.978 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:08 smithi153 bash[23852]: cluster 2024-04-06T03:44:07.577608+0000 mon.a (mon.0) 1678 : cluster 0 osdmap e352: 8 total, 8 up, 7 in 2024-04-06T03:44:09.002 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:09.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:09.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:09.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:09.239 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:09.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 18 16 33933 33917 117.757 80.0625 0.00527693 0.00848597 2024-04-06T03:44:09.270 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:09.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:09.443 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:44:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:09.493 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:09.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:09.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:09.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:09.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:09.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:09.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:09 smithi153 bash[23852]: cluster 2024-04-06T03:44:08.302927+0000 osd.5 (osd.5) 149 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:09.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:09 smithi153 bash[23852]: cluster 2024-04-06T03:44:08.303063+0000 osd.5 (osd.5) 150 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:09.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:09 smithi153 bash[23852]: cluster 2024-04-06T03:44:08.494554+0000 mgr.y (mgr.24368) 851 : cluster 0 pgmap v927: 148 pgs: 148 active+clean; 1.8 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-04-06T03:44:09.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:09 smithi153 bash[23852]: cluster 2024-04-06T03:44:08.590671+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e353: 8 total, 8 up, 7 in 2024-04-06T03:44:09.888 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:09.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:09.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[21467]: cluster 2024-04-06T03:44:08.302927+0000 osd.5 (osd.5) 149 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:09.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[21467]: cluster 2024-04-06T03:44:08.303063+0000 osd.5 (osd.5) 150 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:09.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[21467]: cluster 2024-04-06T03:44:08.494554+0000 mgr.y (mgr.24368) 851 : cluster 0 pgmap v927: 148 pgs: 148 active+clean; 1.8 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-04-06T03:44:09.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[21467]: cluster 2024-04-06T03:44:08.590671+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e353: 8 total, 8 up, 7 in 2024-04-06T03:44:09.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[29221]: cluster 2024-04-06T03:44:08.302927+0000 osd.5 (osd.5) 149 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:09.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[29221]: cluster 2024-04-06T03:44:08.303063+0000 osd.5 (osd.5) 150 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:09.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[29221]: cluster 2024-04-06T03:44:08.494554+0000 mgr.y (mgr.24368) 851 : cluster 0 pgmap v927: 148 pgs: 148 active+clean; 1.8 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 111 MiB/s wr, 1.78k op/s 2024-04-06T03:44:09.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:09 smithi050 bash[29221]: cluster 2024-04-06T03:44:08.590671+0000 mon.a (mon.0) 1679 : cluster 0 osdmap e353: 8 total, 8 up, 7 in 2024-04-06T03:44:09.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:10.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:10.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:10.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 19 15 35707 35692 117.398 110.938 0.00727925 0.00851294 2024-04-06T03:44:10.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:10.283 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:10.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:10.399 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:44:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:44:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:10.535 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:10.561 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:10.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:10.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:10.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:10.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:10.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:10.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:10.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:11.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:11.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:11.206 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:11.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:44:11.246977+0000 min lat: 0.00125308 max lat: 0.329568 avg lat: 0.00864231 2024-04-06T03:44:11.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:44:11.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 20 16 37021 37005 115.631 82.0625 0.00784295 0.00864231 2024-04-06T03:44:11.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:11.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:11.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:11.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:11.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:11.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:11.587 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:11.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:11.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:11.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:11.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:11.843 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:11.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:11.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:11.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:12.036 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:11 smithi050 bash[21467]: audit 2024-04-06T03:44:10.180178+0000 mgr.y (mgr.24368) 852 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:12.036 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:11 smithi050 bash[21467]: cluster 2024-04-06T03:44:10.495210+0000 mgr.y (mgr.24368) 853 : cluster 0 pgmap v929: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-04-06T03:44:12.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:11 smithi050 bash[29221]: audit 2024-04-06T03:44:10.180178+0000 mgr.y (mgr.24368) 852 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:12.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:11 smithi050 bash[29221]: cluster 2024-04-06T03:44:10.495210+0000 mgr.y (mgr.24368) 853 : cluster 0 pgmap v929: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-04-06T03:44:12.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:11 smithi153 bash[23852]: audit 2024-04-06T03:44:10.180178+0000 mgr.y (mgr.24368) 852 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:12.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:11 smithi153 bash[23852]: cluster 2024-04-06T03:44:10.495210+0000 mgr.y (mgr.24368) 853 : cluster 0 pgmap v929: 148 pgs: 148 active+clean; 2.0 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.98k op/s 2024-04-06T03:44:12.078 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:12.096 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:12.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:12.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 21 16 38672 38656 115.038 103.188 0.0168122 0.0086874 2024-04-06T03:44:12.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:12.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:12.487 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:12.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:12.613 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:12.715 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:12.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:12.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:12.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:12.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:12.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:12.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:13.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:13.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:13.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:13.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 22 16 39691 39675 112.703 63.6875 0.00766347 0.00886453 2024-04-06T03:44:13.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:13.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:13.373 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:13.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:13.502 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:13.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:13.617 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:13 smithi153 bash[23852]: cluster 2024-04-06T03:44:12.495978+0000 mgr.y (mgr.24368) 854 : cluster 0 pgmap v930: 148 pgs: 148 active+clean; 2.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-04-06T03:44:13.636 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:13.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:13.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:13.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:13.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:13 smithi050 bash[29221]: cluster 2024-04-06T03:44:12.495978+0000 mgr.y (mgr.24368) 854 : cluster 0 pgmap v930: 148 pgs: 148 active+clean; 2.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-04-06T03:44:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:13 smithi050 bash[21467]: cluster 2024-04-06T03:44:12.495978+0000 mgr.y (mgr.24368) 854 : cluster 0 pgmap v930: 148 pgs: 148 active+clean; 2.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 94 MiB/s wr, 1.50k op/s 2024-04-06T03:44:13.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:13.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:13.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:14.019 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:14.122 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:14.160 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:14.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 23 16 40786 40770 110.778 68.4375 0.00823666 0.00902214 2024-04-06T03:44:14.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:14.293 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:14.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:14.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:14.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:14.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:14.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:14.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:14.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:14.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:14.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:14.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:14.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:15.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:15.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:15.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:15.208 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:15.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:15 smithi050 bash[29221]: cluster 2024-04-06T03:44:14.496704+0000 mgr.y (mgr.24368) 855 : cluster 0 pgmap v931: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-06T03:44:15.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:14 smithi050 bash[21467]: cluster 2024-04-06T03:44:14.496704+0000 mgr.y (mgr.24368) 855 : cluster 0 pgmap v931: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-06T03:44:15.246 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 24 16 41915 41899 109.102 70.5625 0.00296192 0.00911032 2024-04-06T03:44:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:14 smithi153 bash[23852]: cluster 2024-04-06T03:44:14.496704+0000 mgr.y (mgr.24368) 855 : cluster 0 pgmap v931: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-04-06T03:44:15.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:15.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:15.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:15.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:15.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:15.601 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:15.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:15.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:15.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:15.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:15.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:15.993 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:15.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:16.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:16.118 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:16.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:16.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 25 16 42945 42929 107.308 64.375 0.0117225 0.00931321 2024-04-06T03:44:16.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:16.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:16.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:16.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:16.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:16.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:16.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:16.783 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:16.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:16.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:16.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:17.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:17.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:17.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:17.177 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:17.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 26 16 43827 43811 105.301 55.125 0.00274838 0.0094818 2024-04-06T03:44:17.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:17.301 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:17.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:17.434 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:17.459 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:44:17.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:17.570 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:17.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:17.687 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:17 smithi050 bash[21467]: cluster 2024-04-06T03:44:16.497327+0000 mgr.y (mgr.24368) 856 : cluster 0 pgmap v932: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 573 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-06T03:44:17.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:17 smithi050 bash[29221]: cluster 2024-04-06T03:44:16.497327+0000 mgr.y (mgr.24368) 856 : cluster 0 pgmap v932: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 573 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-06T03:44:17.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:17.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:17.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:17 smithi153 bash[23852]: cluster 2024-04-06T03:44:16.497327+0000 mgr.y (mgr.24368) 856 : cluster 0 pgmap v932: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 573 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-04-06T03:44:17.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:17.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:17.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:17.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:18.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:18.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:18.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:18.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 27 16 45292 45276 104.792 91.5625 0.0133197 0.0095379 2024-04-06T03:44:18.330 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:18.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:18.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:18.480 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:18.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:18.610 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:18.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:18.743 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:18.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:18.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:18.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:18.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:19.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:19.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:19.131 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:19.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 28 16 46636 46620 104.049 84 0.00503992 0.00956295 2024-04-06T03:44:19.256 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:19.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.393 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:19.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:19.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:19.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:19 smithi153 bash[23852]: cluster 2024-04-06T03:44:18.497934+0000 mgr.y (mgr.24368) 857 : cluster 0 pgmap v933: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-04-06T03:44:19.636 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:19 smithi153 bash[23852]: audit 2024-04-06T03:44:19.501368+0000 mon.a (mon.0) 1680 : audit 0 from='client.? 172.21.15.50:0/511255835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:44:19.656 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:19.691 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:44:19.729 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:44:19.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:19 smithi050 bash[21467]: cluster 2024-04-06T03:44:18.497934+0000 mgr.y (mgr.24368) 857 : cluster 0 pgmap v933: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-04-06T03:44:19.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:19 smithi050 bash[21467]: audit 2024-04-06T03:44:19.501368+0000 mon.a (mon.0) 1680 : audit 0 from='client.? 172.21.15.50:0/511255835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:44:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:19 smithi050 bash[29221]: cluster 2024-04-06T03:44:18.497934+0000 mgr.y (mgr.24368) 857 : cluster 0 pgmap v933: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-04-06T03:44:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:19 smithi050 bash[29221]: audit 2024-04-06T03:44:19.501368+0000 mon.a (mon.0) 1680 : audit 0 from='client.? 172.21.15.50:0/511255835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:44:19.742 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:44:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:44:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:19.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:19.772 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:44:19.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:19.808 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:44:19.852 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:44:19.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.898 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:44:19.915 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:19.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:19.936 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:44:19.980 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:44:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:44:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:44:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:44:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:44:19.982 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:44:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:44:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:44:19.983 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:44:20.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.046 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:20.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:20.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:20.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:20.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 29 16 47744 47728 102.849 69.25 0.00320386 0.00971497 2024-04-06T03:44:20.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:20.300 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:20.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:20.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:20.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:20.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:20.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:44:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:44:20.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:20.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:20.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:20.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:20 smithi050 bash[21467]: audit 2024-04-06T03:44:20.391149+0000 mon.a (mon.0) 1681 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:20.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:20 smithi050 bash[29221]: audit 2024-04-06T03:44:20.391149+0000 mon.a (mon.0) 1681 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:20.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:20.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:20.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:20.940 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:20 smithi153 bash[23852]: audit 2024-04-06T03:44:20.391149+0000 mon.a (mon.0) 1681 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:20.958 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:20.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:21.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:21.080 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:21.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:21.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:21.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 30 16 48882 48866 101.791 71.125 0.00734547 0.0098175 2024-04-06T03:44:21.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:21.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:21.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:21.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:21.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:21.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:21.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:21 smithi153 bash[23852]: audit 2024-04-06T03:44:20.188418+0000 mgr.y (mgr.24368) 858 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:21.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:21 smithi153 bash[23852]: cluster 2024-04-06T03:44:20.498564+0000 mgr.y (mgr.24368) 859 : cluster 0 pgmap v934: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 859 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-04-06T03:44:21.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:21.858 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:21.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:21 smithi050 bash[21467]: audit 2024-04-06T03:44:20.188418+0000 mgr.y (mgr.24368) 858 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:21.945 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:21 smithi050 bash[21467]: cluster 2024-04-06T03:44:20.498564+0000 mgr.y (mgr.24368) 859 : cluster 0 pgmap v934: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 859 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-04-06T03:44:21.945 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:21 smithi050 bash[29221]: audit 2024-04-06T03:44:20.188418+0000 mgr.y (mgr.24368) 858 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:21.945 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:21 smithi050 bash[29221]: cluster 2024-04-06T03:44:20.498564+0000 mgr.y (mgr.24368) 859 : cluster 0 pgmap v934: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 859 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-04-06T03:44:21.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:22.007 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:22.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:22.137 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:22.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:22.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:22.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 31 16 49426 49410 99.6042 34 0.00228736 0.00999954 2024-04-06T03:44:22.262 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:22.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:22.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:22.498 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:22.526 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:22.602 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:44:22.605 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-06T03:44:22.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:22.658 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:22.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:22.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:22 smithi153 bash[23852]: cluster 2024-04-06T03:44:21.419419+0000 osd.3 (osd.3) 555 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:22.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:22 smithi153 bash[23852]: cluster 2024-04-06T03:44:21.419559+0000 osd.3 (osd.3) 556 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:22.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:22 smithi153 bash[23852]: cluster 2024-04-06T03:44:21.582972+0000 mon.a (mon.0) 1682 : cluster 0 osdmap e354: 8 total, 8 up, 7 in 2024-04-06T03:44:22.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:22 smithi153 bash[23852]: audit 2024-04-06T03:44:21.946416+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.50:0/169551151' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:44:22.772 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:22 smithi153 bash[23852]: audit 2024-04-06T03:44:21.947000+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:44:22.773 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:22 smithi153 bash[23852]: audit 2024-04-06T03:44:22.064266+0000 mon.a (mon.0) 1684 : audit 1 from='client.? 172.21.15.50:0/997652829' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:44:22.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:22.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:22.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:23.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[21467]: cluster 2024-04-06T03:44:21.419419+0000 osd.3 (osd.3) 555 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:23.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[21467]: cluster 2024-04-06T03:44:21.419559+0000 osd.3 (osd.3) 556 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:23.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[21467]: cluster 2024-04-06T03:44:21.582972+0000 mon.a (mon.0) 1682 : cluster 0 osdmap e354: 8 total, 8 up, 7 in 2024-04-06T03:44:23.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[21467]: audit 2024-04-06T03:44:21.946416+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.50:0/169551151' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:44:23.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[21467]: audit 2024-04-06T03:44:21.947000+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:44:23.023 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[21467]: audit 2024-04-06T03:44:22.064266+0000 mon.a (mon.0) 1684 : audit 1 from='client.? 172.21.15.50:0/997652829' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:44:23.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[29221]: cluster 2024-04-06T03:44:21.419419+0000 osd.3 (osd.3) 555 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:23.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[29221]: cluster 2024-04-06T03:44:21.419559+0000 osd.3 (osd.3) 556 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:23.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[29221]: cluster 2024-04-06T03:44:21.582972+0000 mon.a (mon.0) 1682 : cluster 0 osdmap e354: 8 total, 8 up, 7 in 2024-04-06T03:44:23.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[29221]: audit 2024-04-06T03:44:21.946416+0000 mon.c (mon.2) 106 : audit 1 from='client.? 172.21.15.50:0/169551151' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:44:23.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[29221]: audit 2024-04-06T03:44:21.947000+0000 mon.a (mon.0) 1683 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:44:23.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:22 smithi050 bash[29221]: audit 2024-04-06T03:44:22.064266+0000 mon.a (mon.0) 1684 : audit 1 from='client.? 172.21.15.50:0/997652829' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:44:23.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:23.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:23.169 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:23.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:23.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 32 16 50650 50634 98.882 76.5 0.00655554 0.0101082 2024-04-06T03:44:23.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:23.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:23.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:23.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:23.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:23.596 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:23.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:23.719 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:23.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:23.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[21467]: cluster 2024-04-06T03:44:22.499263+0000 mgr.y (mgr.24368) 860 : cluster 0 pgmap v936: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-06T03:44:23.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[21467]: cluster 2024-04-06T03:44:22.579434+0000 mon.a (mon.0) 1685 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:44:23.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[21467]: audit 2024-04-06T03:44:22.582598+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:44:23.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[21467]: audit 2024-04-06T03:44:22.582702+0000 mon.a (mon.0) 1687 : audit 1 from='client.? 172.21.15.50:0/997652829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:44:23.937 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[21467]: cluster 2024-04-06T03:44:22.589349+0000 mon.a (mon.0) 1688 : cluster 0 osdmap e355: 8 total, 8 up, 7 in 2024-04-06T03:44:23.937 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[29221]: cluster 2024-04-06T03:44:22.499263+0000 mgr.y (mgr.24368) 860 : cluster 0 pgmap v936: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-06T03:44:23.938 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[29221]: cluster 2024-04-06T03:44:22.579434+0000 mon.a (mon.0) 1685 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:44:23.938 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[29221]: audit 2024-04-06T03:44:22.582598+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:44:23.938 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[29221]: audit 2024-04-06T03:44:22.582702+0000 mon.a (mon.0) 1687 : audit 1 from='client.? 172.21.15.50:0/997652829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:44:23.938 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:23 smithi050 bash[29221]: cluster 2024-04-06T03:44:22.589349+0000 mon.a (mon.0) 1688 : cluster 0 osdmap e355: 8 total, 8 up, 7 in 2024-04-06T03:44:23.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:23.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:23.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:24.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:23 smithi153 bash[23852]: cluster 2024-04-06T03:44:22.499263+0000 mgr.y (mgr.24368) 860 : cluster 0 pgmap v936: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-04-06T03:44:24.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:23 smithi153 bash[23852]: cluster 2024-04-06T03:44:22.579434+0000 mon.a (mon.0) 1685 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:44:24.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:23 smithi153 bash[23852]: audit 2024-04-06T03:44:22.582598+0000 mon.a (mon.0) 1686 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:44:24.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:23 smithi153 bash[23852]: audit 2024-04-06T03:44:22.582702+0000 mon.a (mon.0) 1687 : audit 1 from='client.? 172.21.15.50:0/997652829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:44:24.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:23 smithi153 bash[23852]: cluster 2024-04-06T03:44:22.589349+0000 mon.a (mon.0) 1688 : cluster 0 osdmap e355: 8 total, 8 up, 7 in 2024-04-06T03:44:24.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:24.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:24.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:24.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 33 16 51681 51665 97.8381 64.4375 0.00853748 0.0102153 2024-04-06T03:44:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:24.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:24.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:24.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:24.513 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:24.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:24.642 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:24.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:24.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:24.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:24.898 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:24.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:25.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:25.012 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:24 smithi050 bash[21467]: cluster 2024-04-06T03:44:24.500104+0000 mgr.y (mgr.24368) 861 : cluster 0 pgmap v938: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-06T03:44:25.012 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:24 smithi050 bash[29221]: cluster 2024-04-06T03:44:24.500104+0000 mgr.y (mgr.24368) 861 : cluster 0 pgmap v938: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-06T03:44:25.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:25.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:25.143 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:24 smithi153 bash[23852]: cluster 2024-04-06T03:44:24.500104+0000 mgr.y (mgr.24368) 861 : cluster 0 pgmap v938: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-04-06T03:44:25.166 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:25.248 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 34 16 52808 52792 97.032 70.4375 0.00581202 0.0103012 2024-04-06T03:44:25.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:25.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:25.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:25.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:25.521 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:25.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:44:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:25.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:25.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:25.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:25.926 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:25.950 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:25.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:26.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:26.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:26.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:26.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:26.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 35 16 53765 53749 95.9685 59.8125 0.0140863 0.0104145 2024-04-06T03:44:26.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:26.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:26.468 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:26.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:26.611 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:26.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:26.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:26.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:26.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:26.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:26.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:27.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:27.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:27.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:27.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 36 16 55041 55025 95.5178 79.75 0.00797914 0.010464 2024-04-06T03:44:27.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:27.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:27.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:27.418 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:27.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:27.565 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:27.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 4 1 2024-04-06T03:44:27.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:27.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:27 smithi153 bash[23852]: cluster 2024-04-06T03:44:26.500633+0000 mgr.y (mgr.24368) 862 : cluster 0 pgmap v939: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:44:27.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:27.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:27.847 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:27 smithi050 bash[29221]: cluster 2024-04-06T03:44:26.500633+0000 mgr.y (mgr.24368) 862 : cluster 0 pgmap v939: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:44:27.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:27 smithi050 bash[21467]: cluster 2024-04-06T03:44:26.500633+0000 mgr.y (mgr.24368) 862 : cluster 0 pgmap v939: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-04-06T03:44:27.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:27.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:27.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:27.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:28.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:28.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:28.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:28.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.239 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:28.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 37 16 56384 56368 95.2047 83.9375 0.00523304 0.0104987 2024-04-06T03:44:28.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:28.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:28.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:28.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:28.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:28.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.777 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:28.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:28.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:29.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:29.041 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:29.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:29.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:29.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 38 16 57762 57746 94.9655 86.125 0.00626148 0.0105253 2024-04-06T03:44:29.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:29.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:29.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:29.423 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:44:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:44:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:29.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:29.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:29.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:29.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:44:29.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.706 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:29 smithi050 bash[21467]: cluster 2024-04-06T03:44:28.501357+0000 mgr.y (mgr.24368) 863 : cluster 0 pgmap v940: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 977 op/s 2024-04-06T03:44:29.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:29 smithi050 bash[29221]: cluster 2024-04-06T03:44:28.501357+0000 mgr.y (mgr.24368) 863 : cluster 0 pgmap v940: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 977 op/s 2024-04-06T03:44:29.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:29.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:29 smithi153 bash[23852]: cluster 2024-04-06T03:44:28.501357+0000 mgr.y (mgr.24368) 863 : cluster 0 pgmap v940: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 977 op/s 2024-04-06T03:44:29.829 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:29.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:29.961 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:29.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:30.066 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:30.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:30.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:30.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:30.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 39 16 59060 59044 94.6105 81.125 0.0109671 0.0105644 2024-04-06T03:44:30.298 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:44:30.298 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:44:30.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:30.361 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:30.374 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:44:30.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:30.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:44:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:44:30.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:30.624 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:30.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:30.760 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:30.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:30.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:30.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:31.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:31.040 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:31.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:31.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:31.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:44:31.249983+0000 min lat: 0.00125308 max lat: 0.443907 avg lat: 0.010603 2024-04-06T03:44:31.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:44:31.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 40 16 60350 60334 94.2607 80.625 0.0232608 0.010603 2024-04-06T03:44:31.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:31.305 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:31.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:31.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:31.434 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:31.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:31.569 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:31.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:31.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:31 smithi050 bash[21467]: audit 2024-04-06T03:44:30.195970+0000 mgr.y (mgr.24368) 864 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:31.684 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:31 smithi050 bash[21467]: cluster 2024-04-06T03:44:30.502020+0000 mgr.y (mgr.24368) 865 : cluster 0 pgmap v941: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-06T03:44:31.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:31 smithi050 bash[29221]: audit 2024-04-06T03:44:30.195970+0000 mgr.y (mgr.24368) 864 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:31.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:31 smithi050 bash[29221]: cluster 2024-04-06T03:44:30.502020+0000 mgr.y (mgr.24368) 865 : cluster 0 pgmap v941: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-06T03:44:31.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:31.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:31.830 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:31.836 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:31 smithi153 bash[23852]: audit 2024-04-06T03:44:30.195970+0000 mgr.y (mgr.24368) 864 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:31.836 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:31 smithi153 bash[23852]: cluster 2024-04-06T03:44:30.502020+0000 mgr.y (mgr.24368) 865 : cluster 0 pgmap v941: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-04-06T03:44:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:31.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:31.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:31.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:32.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:32.106 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:32.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:32.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:32.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 41 16 61630 61614 93.9126 80 0.018866 0.0106427 2024-04-06T03:44:32.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:32.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:32.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:32.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:32.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:32.619 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:32.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:32.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:32.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:32.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:32.876 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:32.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:32.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:33.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:33.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:33.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:33.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:33.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 42 16 62454 62438 92.9027 51.5 0.00271393 0.0107377 2024-04-06T03:44:33.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:33.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:33.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:33.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:33.490 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:33.523 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:33.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:33.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:33.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:33 smithi050 bash[21467]: cluster 2024-04-06T03:44:32.502559+0000 mgr.y (mgr.24368) 866 : cluster 0 pgmap v942: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-06T03:44:33.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:33 smithi050 bash[29221]: cluster 2024-04-06T03:44:32.502559+0000 mgr.y (mgr.24368) 866 : cluster 0 pgmap v942: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-06T03:44:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:33.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:33.785 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:33 smithi153 bash[23852]: cluster 2024-04-06T03:44:32.502559+0000 mgr.y (mgr.24368) 866 : cluster 0 pgmap v942: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-04-06T03:44:33.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-06T03:44:33.888 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:33.967 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:33.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:34.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:34.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:34.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:34.236 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:34.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 43 16 63490 63474 92.2479 64.75 0.011922 0.0108348 2024-04-06T03:44:34.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:34.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:34.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:34.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:34.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:34.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:34.628 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:34.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:34.759 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:34.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:34.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:34.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:34.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:34.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:35.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:35.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:35.247 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:35.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 44 16 64396 64380 91.4381 56.625 0.00655266 0.0108598 2024-04-06T03:44:35.257 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:34 smithi153 bash[23852]: cluster 2024-04-06T03:44:34.503301+0000 mgr.y (mgr.24368) 867 : cluster 0 pgmap v943: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-06T03:44:35.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:35.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:35.397 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:34 smithi050 bash[21467]: cluster 2024-04-06T03:44:34.503301+0000 mgr.y (mgr.24368) 867 : cluster 0 pgmap v943: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-06T03:44:35.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:34 smithi050 bash[29221]: cluster 2024-04-06T03:44:34.503301+0000 mgr.y (mgr.24368) 867 : cluster 0 pgmap v943: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-04-06T03:44:35.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:35.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:35.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:35.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.668 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:35.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:35.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:35.802 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:35.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:35.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:35.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:36.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:36.062 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:36.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:36.200 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:36.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:35 smithi050 bash[21467]: audit 2024-04-06T03:44:35.413932+0000 mon.a (mon.0) 1689 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:36.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:35 smithi050 bash[29221]: audit 2024-04-06T03:44:35.413932+0000 mon.a (mon.0) 1689 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:36.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 45 16 65615 65599 91.099 76.1875 0.00409592 0.0109721 2024-04-06T03:44:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:35 smithi153 bash[23852]: audit 2024-04-06T03:44:35.413932+0000 mon.a (mon.0) 1689 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:36.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:36.330 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:36.377 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:44:36.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:36.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:36.595 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:36.699 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:36.738 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:36.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:36.868 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:36.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:36.993 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:36.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:36.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:37.095 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:37.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:36 smithi153 bash[23852]: cluster 2024-04-06T03:44:35.231959+0000 osd.1 (osd.1) 397 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:37.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:36 smithi153 bash[23852]: cluster 2024-04-06T03:44:35.232057+0000 osd.1 (osd.1) 398 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:37.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:36 smithi153 bash[23852]: cluster 2024-04-06T03:44:35.984640+0000 mon.a (mon.0) 1690 : cluster 0 osdmap e356: 8 total, 8 up, 7 in 2024-04-06T03:44:37.107 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:36 smithi153 bash[23852]: cluster 2024-04-06T03:44:36.503740+0000 mgr.y (mgr.24368) 868 : cluster 0 pgmap v945: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-06T03:44:37.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:37.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[21467]: cluster 2024-04-06T03:44:35.231959+0000 osd.1 (osd.1) 397 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[21467]: cluster 2024-04-06T03:44:35.232057+0000 osd.1 (osd.1) 398 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[21467]: cluster 2024-04-06T03:44:35.984640+0000 mon.a (mon.0) 1690 : cluster 0 osdmap e356: 8 total, 8 up, 7 in 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[21467]: cluster 2024-04-06T03:44:36.503740+0000 mgr.y (mgr.24368) 868 : cluster 0 pgmap v945: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[29221]: cluster 2024-04-06T03:44:35.231959+0000 osd.1 (osd.1) 397 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[29221]: cluster 2024-04-06T03:44:35.232057+0000 osd.1 (osd.1) 398 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:37.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[29221]: cluster 2024-04-06T03:44:35.984640+0000 mon.a (mon.0) 1690 : cluster 0 osdmap e356: 8 total, 8 up, 7 in 2024-04-06T03:44:37.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:36 smithi050 bash[29221]: cluster 2024-04-06T03:44:36.503740+0000 mgr.y (mgr.24368) 868 : cluster 0 pgmap v945: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-04-06T03:44:37.249 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 46 16 66670 66654 90.5519 65.9375 0.0097316 0.0110373 2024-04-06T03:44:37.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:37.366 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:37.400 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:37.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:37.538 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:37.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:37.666 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:37.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:37.806 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:37.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:37.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:37.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:38.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:38.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:38.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:38.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:38.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 47 16 67870 67854 90.2208 75 0.0050137 0.0110737 2024-04-06T03:44:38.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:38.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:38.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:38.471 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:38.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:38.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:38.608 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:38.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:38.712 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:44:38.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:38.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:38.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:38.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:38.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:38.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:44:38.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:38.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:39.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:39.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:39.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 48 16 68825 68809 89.5846 59.6875 0.0114115 0.0111576 2024-04-06T03:44:39.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:39.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:39.385 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:39.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:39.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:39.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:39.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:39 smithi153 bash[23852]: cluster 2024-04-06T03:44:38.456251+0000 osd.4 (osd.4) 539 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:39.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:39 smithi153 bash[23852]: cluster 2024-04-06T03:44:38.456342+0000 osd.4 (osd.4) 540 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:39.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:39 smithi153 bash[23852]: cluster 2024-04-06T03:44:38.504328+0000 mgr.y (mgr.24368) 869 : cluster 0 pgmap v946: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-04-06T03:44:39.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:39 smithi153 bash[23852]: cluster 2024-04-06T03:44:38.516698+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e357: 8 total, 8 up, 7 in 2024-04-06T03:44:39.657 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:39.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:39.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:39.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:39.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:39.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:40.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:40.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:40.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:40.169 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:40.241 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:44:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[21467]: cluster 2024-04-06T03:44:38.456251+0000 osd.4 (osd.4) 539 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[21467]: cluster 2024-04-06T03:44:38.456342+0000 osd.4 (osd.4) 540 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[21467]: cluster 2024-04-06T03:44:38.504328+0000 mgr.y (mgr.24368) 869 : cluster 0 pgmap v946: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[21467]: cluster 2024-04-06T03:44:38.516698+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e357: 8 total, 8 up, 7 in 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[29221]: cluster 2024-04-06T03:44:38.456251+0000 osd.4 (osd.4) 539 : cluster 0 purged_snaps scrub starts 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[29221]: cluster 2024-04-06T03:44:38.456342+0000 osd.4 (osd.4) 540 : cluster 0 purged_snaps scrub ok 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[29221]: cluster 2024-04-06T03:44:38.504328+0000 mgr.y (mgr.24368) 869 : cluster 0 pgmap v946: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-04-06T03:44:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:39 smithi050 bash[29221]: cluster 2024-04-06T03:44:38.516698+0000 mon.a (mon.0) 1691 : cluster 0 osdmap e357: 8 total, 8 up, 7 in 2024-04-06T03:44:40.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 49 16 69815 69799 89.0189 61.875 0.00986785 0.0112015 2024-04-06T03:44:40.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:40.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:40.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:40.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:40.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:40.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:44:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:44:40.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:40.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:40.680 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:40.783 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:40.811 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:40.914 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:40.946 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:40.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:41.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:41.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:41.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:41.201 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:41.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 50 16 70207 70191 87.7285 24.5 0.00333485 0.0113752 2024-04-06T03:44:41.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:41.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:41.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:41.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:41.599 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:41.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:41.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:41 smithi050 bash[21467]: audit 2024-04-06T03:44:40.204127+0000 mgr.y (mgr.24368) 870 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:41.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:41 smithi050 bash[21467]: cluster 2024-04-06T03:44:40.505662+0000 mgr.y (mgr.24368) 871 : cluster 0 pgmap v948: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-04-06T03:44:41.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:41 smithi050 bash[29221]: audit 2024-04-06T03:44:40.204127+0000 mgr.y (mgr.24368) 870 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:41.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:41 smithi050 bash[29221]: cluster 2024-04-06T03:44:40.505662+0000 mgr.y (mgr.24368) 871 : cluster 0 pgmap v948: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-04-06T03:44:41.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:41 smithi153 bash[23852]: audit 2024-04-06T03:44:40.204127+0000 mgr.y (mgr.24368) 870 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:41 smithi153 bash[23852]: cluster 2024-04-06T03:44:40.505662+0000 mgr.y (mgr.24368) 871 : cluster 0 pgmap v948: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-04-06T03:44:41.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:41.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:41.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:41.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:42.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:42.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:42.168 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:44:42.202 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:44:42.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:42.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 51 16 71028 71012 87.0144 51.3125 0.0130271 0.0114875 2024-04-06T03:44:42.252 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:44:42.265 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:42.294 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:44:42.330 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:44:42.361 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:44:42.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:42.399 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:44:42.432 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:44:42.436 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:44:42.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:42.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:42 smithi050 bash[21467]: audit 2024-04-06T03:44:42.125620+0000 mon.c (mon.2) 107 : audit 0 from='client.? 172.21.15.50:0/2785187683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:44:42.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:42 smithi050 bash[29221]: audit 2024-04-06T03:44:42.125620+0000 mon.c (mon.2) 107 : audit 0 from='client.? 172.21.15.50:0/2785187683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:44:42.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:42.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:42.774 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:42.776 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:42 smithi153 bash[23852]: audit 2024-04-06T03:44:42.125620+0000 mon.c (mon.2) 107 : audit 0 from='client.? 172.21.15.50:0/2785187683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:44:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:42.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:42.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:42.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:43.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:43.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:43.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:43.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:43.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 52 16 71755 71739 86.2148 45.4375 0.00327435 0.0115873 2024-04-06T03:44:43.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:43.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:43.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:43.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:43.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:43 smithi153 bash[23852]: cluster 2024-04-06T03:44:42.506184+0000 mgr.y (mgr.24368) 872 : cluster 0 pgmap v949: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-06T03:44:43.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:43.568 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:43.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:43.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:43.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:43.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:43 smithi050 bash[21467]: cluster 2024-04-06T03:44:42.506184+0000 mgr.y (mgr.24368) 872 : cluster 0 pgmap v949: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-06T03:44:43.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:43 smithi050 bash[29221]: cluster 2024-04-06T03:44:42.506184+0000 mgr.y (mgr.24368) 872 : cluster 0 pgmap v949: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-06T03:44:43.840 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:43.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:43.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:43.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:44.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:44.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:44.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:44.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:44.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 53 15 72644 72629 85.6375 55.625 0.0148017 0.0116725 2024-04-06T03:44:44.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:44.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:44.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:44.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:44.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:44.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:44.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:44.769 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:44.771 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:44:44.771 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:44:44.771 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.771 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:73096 11.e 11:7496419b:::benchmark_data_smithi050_138698_object73095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-04-06T03:44:44.771 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:44.744868+0000", 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0093026250000000001, 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.009338608, 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 73096 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:44:44.772 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.744868+0000", 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.744868+0000", 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.773 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.744871+0000", 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2440000000000002e-06 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.744960+0000", 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8731000000000001e-05 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:44:44.774 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745009+0000", 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9863e-05 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745035+0000", 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5103e-05 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745612+0000", 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00057744199999999995 2024-04-06T03:44:44.775 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.746451+0000", 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000839392 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,7", 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.746712+0000", 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026037599999999999 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.776 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:73099 11.e 11:75f24257:::benchmark_data_smithi050_138698_object73098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:44.745139+0000", 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0090314450000000008, 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0091574970000000006, 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "waiting for sub ops", 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 73099 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:44:44.777 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745139+0000", 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745139+0000", 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745145+0000", 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0159999999999997e-06 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:44:44.778 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745200+0000", 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5231e-05 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745220+0000", 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9862000000000001e-05 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.745235+0000", 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4679e-05 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:44:44.779 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.751112+0000", 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0058774439999999999 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.751274+0000", 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016185099999999999 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,7", 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.751449+0000", 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017469999999999999 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.780 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:73093 11.6 e357/345 11:66de3db1:::benchmark_data_smithi050_138698_object73092:head v 357'4672, mlcod=357'4668)", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:44.752256+0000", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.0019144170000000001, 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002153075, 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752256+0000", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752256+0000", 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.781 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752256+0000", 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.22e-07 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752291+0000", 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5024000000000002e-05 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:44:44.782 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752298+0000", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8709999999999999e-06 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752303+0000", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3009999999999997e-06 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752354+0000", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0696999999999999e-05 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.752376+0000", 2024-04-06T03:44:44.783 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2055000000000001e-05 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:73085 11.b e357/345 11:d648862e:::benchmark_data_smithi050_138698_object73084:head v 357'4481, mlcod=357'4478)", 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:44.753738+0000", 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.00043255399999999998, 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00074810800000000004, 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:44:44.784 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753738+0000", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753738+0000", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753739+0000", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2399999999999996e-07 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753778+0000", 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9912e-05 2024-04-06T03:44:44.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753791+0000", 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2649999999999999e-05 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753799+0000", 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0790000000000001e-06 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753855+0000", 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5488000000000001e-05 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.786 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753873+0000", 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8498000000000001e-05 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:73097 11.b e357/345)", 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:44.753821+0000", 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.00034915599999999999, 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00073883399999999995, 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "queued for pg", 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.787 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753821+0000", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753821+0000", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753821+0000", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8999999999999998e-07 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753851+0000", 2024-04-06T03:44:44.788 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9397e-05 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753858+0000", 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3000000000000004e-06 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.753864+0000", 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5670000000000002e-06 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.789 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:73100 11.7 11:edad3046:::benchmark_data_smithi050_138698_object73099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:44.754008+0000", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 0.00016263900000000001, 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00062376399999999996, 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "reached pg", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 73100 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754008+0000", 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.790 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754008+0000", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754009+0000", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4270000000000001e-06 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754052+0000", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2642000000000003e-05 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:44:44.791 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754064+0000", 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2211e-05 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754072+0000", 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.439e-06 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:44.754144+0000", 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1990999999999996e-05 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 6 2024-04-06T03:44:44.792 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:44:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:44.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:44.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:44.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:44.992 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:44 smithi153 bash[23852]: cluster 2024-04-06T03:44:44.506955+0000 mgr.y (mgr.24368) 873 : cluster 0 pgmap v950: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 996 op/s 2024-04-06T03:44:45.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:45.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:44 smithi050 bash[21467]: cluster 2024-04-06T03:44:44.506955+0000 mgr.y (mgr.24368) 873 : cluster 0 pgmap v950: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 996 op/s 2024-04-06T03:44:45.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:44 smithi050 bash[29221]: cluster 2024-04-06T03:44:44.506955+0000 mgr.y (mgr.24368) 873 : cluster 0 pgmap v950: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 996 op/s 2024-04-06T03:44:45.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:45.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:45.183 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:45.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 54 16 73600 73584 85.1569 59.6875 0.0104368 0.0117378 2024-04-06T03:44:45.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:45.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:45.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:45.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:45.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.591 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:45.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.726 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:45.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.852 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:45.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-06T03:44:45.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.982 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:45.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:46.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:46.116 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:46.219 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:46.250 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 55 16 74330 74314 84.438 45.625 0.00292044 0.0117998 2024-04-06T03:44:46.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:46.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:46.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:46.476 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:46.507 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:46.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:46.653 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:46.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:46.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:46.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:46.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:46.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:47.018 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:47.040 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:47.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.182 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:47.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 56 16 75318 75302 84.0327 61.75 0.0046644 0.011889 2024-04-06T03:44:47.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:47.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:47.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:47.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.562 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:47.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:47.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:47.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:47.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:47 smithi153 bash[23852]: cluster 2024-04-06T03:44:46.507397+0000 mgr.y (mgr.24368) 874 : cluster 0 pgmap v951: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 921 op/s 2024-04-06T03:44:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:47.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:47.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:47 smithi050 bash[29221]: cluster 2024-04-06T03:44:46.507397+0000 mgr.y (mgr.24368) 874 : cluster 0 pgmap v951: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 921 op/s 2024-04-06T03:44:47.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:47 smithi050 bash[21467]: cluster 2024-04-06T03:44:46.507397+0000 mgr.y (mgr.24368) 874 : cluster 0 pgmap v951: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 921 op/s 2024-04-06T03:44:48.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:48.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:48.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:48.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 57 16 75987 75971 83.292 41.8125 0.0154474 0.0120006 2024-04-06T03:44:48.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:48.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:48.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:48.477 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:48.583 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:48.620 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:48.724 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:48.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:48.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:44:48.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:48.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:49.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:49.171 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:49.199 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:49.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 58 16 77073 77057 83.026 67.875 0.00895352 0.0120395 2024-04-06T03:44:49.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.338 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:44:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:44:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:49.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:49.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:49.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:49.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:49 smithi050 bash[21467]: cluster 2024-04-06T03:44:48.507939+0000 mgr.y (mgr.24368) 875 : cluster 0 pgmap v952: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s 2024-04-06T03:44:49.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:49 smithi050 bash[29221]: cluster 2024-04-06T03:44:48.507939+0000 mgr.y (mgr.24368) 875 : cluster 0 pgmap v952: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s 2024-04-06T03:44:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:49.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:49.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:49.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:49.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:49.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:49.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:50.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:49 smithi153 bash[23852]: cluster 2024-04-06T03:44:48.507939+0000 mgr.y (mgr.24368) 875 : cluster 0 pgmap v952: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 56 MiB/s wr, 893 op/s 2024-04-06T03:44:50.096 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:50.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:50.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:50.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 59 16 78122 78106 82.73 65.5625 0.00354633 0.0120814 2024-04-06T03:44:50.267 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:50.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:50.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:50.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:50.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:44:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:44:50.541 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:50.599 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-06T03:44:50.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:50 smithi050 bash[21467]: audit 2024-04-06T03:44:49.983484+0000 mon.c (mon.2) 108 : audit 1 from='client.? 172.21.15.50:0/2456842854' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:44:50.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:50 smithi050 bash[21467]: audit 2024-04-06T03:44:49.984320+0000 mon.a (mon.0) 1692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:44:50.645 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:50 smithi050 bash[21467]: audit 2024-04-06T03:44:50.433196+0000 mon.a (mon.0) 1693 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:50.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:50 smithi050 bash[29221]: audit 2024-04-06T03:44:49.983484+0000 mon.c (mon.2) 108 : audit 1 from='client.? 172.21.15.50:0/2456842854' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:44:50.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:50 smithi050 bash[29221]: audit 2024-04-06T03:44:49.984320+0000 mon.a (mon.0) 1692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:44:50.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:50 smithi050 bash[29221]: audit 2024-04-06T03:44:50.433196+0000 mon.a (mon.0) 1693 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:50.647 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:50.673 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:50.714 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:44:50.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:50 smithi153 bash[23852]: audit 2024-04-06T03:44:49.983484+0000 mon.c (mon.2) 108 : audit 1 from='client.? 172.21.15.50:0/2456842854' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:44:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:50 smithi153 bash[23852]: audit 2024-04-06T03:44:49.984320+0000 mon.a (mon.0) 1692 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:44:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:50 smithi153 bash[23852]: audit 2024-04-06T03:44:50.433196+0000 mon.a (mon.0) 1693 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:44:50.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:50.904 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:50.945 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:50.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:51.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:51.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:51.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:51.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:51.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:44:51.252038+0000 min lat: 0.00125308 max lat: 0.443907 avg lat: 0.0121384 2024-04-06T03:44:51.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:44:51.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 60 16 79085 79069 82.3542 60.1875 0.00809461 0.0121384 2024-04-06T03:44:51.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:51.346 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:51.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:51.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:51.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:51.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:51.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:51.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:51.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:51.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[21467]: audit 2024-04-06T03:44:50.212129+0000 mgr.y (mgr.24368) 876 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:51.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[21467]: cluster 2024-04-06T03:44:50.508694+0000 mgr.y (mgr.24368) 877 : cluster 0 pgmap v953: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 950 op/s 2024-04-06T03:44:51.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[21467]: audit 2024-04-06T03:44:50.568763+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:44:51.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[21467]: cluster 2024-04-06T03:44:50.572473+0000 mon.a (mon.0) 1695 : cluster 0 osdmap e358: 8 total, 8 up, 7 in 2024-04-06T03:44:51.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[29221]: audit 2024-04-06T03:44:50.212129+0000 mgr.y (mgr.24368) 876 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:51.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[29221]: cluster 2024-04-06T03:44:50.508694+0000 mgr.y (mgr.24368) 877 : cluster 0 pgmap v953: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 950 op/s 2024-04-06T03:44:51.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[29221]: audit 2024-04-06T03:44:50.568763+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:44:51.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:51 smithi050 bash[29221]: cluster 2024-04-06T03:44:50.572473+0000 mon.a (mon.0) 1695 : cluster 0 osdmap e358: 8 total, 8 up, 7 in 2024-04-06T03:44:51.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:51.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:51.984 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:51.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:51 smithi153 bash[23852]: audit 2024-04-06T03:44:50.212129+0000 mgr.y (mgr.24368) 876 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:44:51.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:51 smithi153 bash[23852]: cluster 2024-04-06T03:44:50.508694+0000 mgr.y (mgr.24368) 877 : cluster 0 pgmap v953: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 950 op/s 2024-04-06T03:44:51.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:51 smithi153 bash[23852]: audit 2024-04-06T03:44:50.568763+0000 mon.a (mon.0) 1694 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:44:51.994 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:51 smithi153 bash[23852]: cluster 2024-04-06T03:44:50.572473+0000 mon.a (mon.0) 1695 : cluster 0 osdmap e358: 8 total, 8 up, 7 in 2024-04-06T03:44:52.008 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:52.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:52.137 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:52.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:52.251 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 61 16 79964 79948 81.9047 54.9375 0.00476091 0.0121742 2024-04-06T03:44:52.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:52.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:52.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:52.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:52.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:52.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:52.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:52.666 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:52.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:52.809 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:52.913 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:52.944 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:52.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:53.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:53.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:53.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 5 1 2024-04-06T03:44:53.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:53.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:53.252 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 62 16 80985 80969 81.6108 63.8125 0.0159893 0.0122478 2024-04-06T03:44:53.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:53.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:53.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:53.486 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:53.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:53.625 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:53.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:53.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:53 smithi153 bash[23852]: cluster 2024-04-06T03:44:52.509755+0000 mgr.y (mgr.24368) 878 : cluster 0 pgmap v955: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 832 op/s 2024-04-06T03:44:53.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:53.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:53 smithi050 bash[21467]: cluster 2024-04-06T03:44:52.509755+0000 mgr.y (mgr.24368) 878 : cluster 0 pgmap v955: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 832 op/s 2024-04-06T03:44:53.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:53 smithi050 bash[29221]: cluster 2024-04-06T03:44:52.509755+0000 mgr.y (mgr.24368) 878 : cluster 0 pgmap v955: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 832 op/s 2024-04-06T03:44:53.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:53.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:53.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:54.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:54.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:54.165 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:54.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 63 16 82015 81999 81.3371 64.375 0.013439 0.0122897 2024-04-06T03:44:54.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:54.300 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:54.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:54.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:54.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:54.568 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:54.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:54.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:54.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:54.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:54.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:54.987 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:44:55.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:54 smithi153 bash[23852]: audit 2024-04-06T03:44:54.069306+0000 mon.a (mon.0) 1696 : audit 1 from='client.? 172.21.15.50:0/1515715443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:44:55.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:54 smithi153 bash[23852]: cluster 2024-04-06T03:44:54.511170+0000 mgr.y (mgr.24368) 879 : cluster 0 pgmap v956: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s 2024-04-06T03:44:55.070 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:55.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:55.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:55.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:54 smithi050 bash[21467]: audit 2024-04-06T03:44:54.069306+0000 mon.a (mon.0) 1696 : audit 1 from='client.? 172.21.15.50:0/1515715443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:44:55.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:54 smithi050 bash[21467]: cluster 2024-04-06T03:44:54.511170+0000 mgr.y (mgr.24368) 879 : cluster 0 pgmap v956: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s 2024-04-06T03:44:55.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:54 smithi050 bash[29221]: audit 2024-04-06T03:44:54.069306+0000 mon.a (mon.0) 1696 : audit 1 from='client.? 172.21.15.50:0/1515715443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:44:55.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:54 smithi050 bash[29221]: cluster 2024-04-06T03:44:54.511170+0000 mgr.y (mgr.24368) 879 : cluster 0 pgmap v956: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s 2024-04-06T03:44:55.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:55.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 64 16 83151 83135 81.1755 71 0.0101806 0.0123143 2024-04-06T03:44:55.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:55.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:55.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:55.537 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:55.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:55.679 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:55.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:55.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:55.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:55.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:56.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:56.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:55 smithi050 bash[21467]: cluster 2024-04-06T03:44:54.974078+0000 mon.a (mon.0) 1697 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:44:56.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:55 smithi050 bash[21467]: audit 2024-04-06T03:44:54.979035+0000 mon.a (mon.0) 1698 : audit 1 from='client.? 172.21.15.50:0/1515715443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:44:56.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:55 smithi050 bash[21467]: cluster 2024-04-06T03:44:54.983119+0000 mon.a (mon.0) 1699 : cluster 0 osdmap e359: 8 total, 8 up, 7 in 2024-04-06T03:44:56.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:55 smithi050 bash[29221]: cluster 2024-04-06T03:44:54.974078+0000 mon.a (mon.0) 1697 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:44:56.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:55 smithi050 bash[29221]: audit 2024-04-06T03:44:54.979035+0000 mon.a (mon.0) 1698 : audit 1 from='client.? 172.21.15.50:0/1515715443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:44:56.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:55 smithi050 bash[29221]: cluster 2024-04-06T03:44:54.983119+0000 mon.a (mon.0) 1699 : cluster 0 osdmap e359: 8 total, 8 up, 7 in 2024-04-06T03:44:56.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:56.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:56.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:56.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 65 16 84103 84087 80.8419 59.5 0.00291959 0.0123353 2024-04-06T03:44:56.285 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:55 smithi153 bash[23852]: cluster 2024-04-06T03:44:54.974078+0000 mon.a (mon.0) 1697 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:44:56.285 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:55 smithi153 bash[23852]: audit 2024-04-06T03:44:54.979035+0000 mon.a (mon.0) 1698 : audit 1 from='client.? 172.21.15.50:0/1515715443' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:44:56.285 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:55 smithi153 bash[23852]: cluster 2024-04-06T03:44:54.983119+0000 mon.a (mon.0) 1699 : cluster 0 osdmap e359: 8 total, 8 up, 7 in 2024-04-06T03:44:56.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:56.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:56.429 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:44:56.429 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:44:56.429 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:44:56.429 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:44:56.430 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:44:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:56.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:56.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:56.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:56.720 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:56.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:56.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:56.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:56.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:57.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:57.102 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:56 smithi050 bash[21467]: cluster 2024-04-06T03:44:56.511969+0000 mgr.y (mgr.24368) 880 : cluster 0 pgmap v958: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s 2024-04-06T03:44:57.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:56 smithi050 bash[21467]: cluster 2024-04-06T03:44:56.585652+0000 osd.3 (osd.3) 557 : cluster 0 11.1 scrub starts 2024-04-06T03:44:57.103 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:56 smithi050 bash[29221]: cluster 2024-04-06T03:44:56.511969+0000 mgr.y (mgr.24368) 880 : cluster 0 pgmap v958: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s 2024-04-06T03:44:57.103 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:56 smithi050 bash[29221]: cluster 2024-04-06T03:44:56.585652+0000 osd.3 (osd.3) 557 : cluster 0 11.1 scrub starts 2024-04-06T03:44:57.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:57.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:57.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:57 smithi153 bash[23852]: cluster 2024-04-06T03:44:56.511969+0000 mgr.y (mgr.24368) 880 : cluster 0 pgmap v958: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 917 op/s 2024-04-06T03:44:57.231 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:57 smithi153 bash[23852]: cluster 2024-04-06T03:44:56.585652+0000 osd.3 (osd.3) 557 : cluster 0 11.1 scrub starts 2024-04-06T03:44:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:57.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 66 16 84731 84715 80.2117 39.25 0.00481169 0.0124618 2024-04-06T03:44:57.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:57.391 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:57.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:57.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:44:57.591 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-06T03:44:57.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:57.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:57.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:57.829 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:44:57.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:57.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:58.069 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:58.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:58.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:58.230 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:58.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:57 smithi050 bash[21467]: cluster 2024-04-06T03:44:57.398953+0000 osd.3 (osd.3) 558 : cluster 0 2.1e scrub starts 2024-04-06T03:44:58.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:57 smithi050 bash[21467]: cluster 2024-04-06T03:44:57.401280+0000 osd.3 (osd.3) 559 : cluster 0 2.1e scrub ok 2024-04-06T03:44:58.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:57 smithi050 bash[29221]: cluster 2024-04-06T03:44:57.398953+0000 osd.3 (osd.3) 558 : cluster 0 2.1e scrub starts 2024-04-06T03:44:58.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:57 smithi050 bash[29221]: cluster 2024-04-06T03:44:57.401280+0000 osd.3 (osd.3) 559 : cluster 0 2.1e scrub ok 2024-04-06T03:44:58.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 67 16 85758 85742 79.9725 64.1875 0.0123246 0.0124994 2024-04-06T03:44:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:57 smithi153 bash[23852]: cluster 2024-04-06T03:44:57.398953+0000 osd.3 (osd.3) 558 : cluster 0 2.1e scrub starts 2024-04-06T03:44:58.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:57 smithi153 bash[23852]: cluster 2024-04-06T03:44:57.401280+0000 osd.3 (osd.3) 559 : cluster 0 2.1e scrub ok 2024-04-06T03:44:58.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:58.359 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:58.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:58.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:58.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:58.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:58.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:58.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:44:58.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:44:58.846 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:58.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:44:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:58.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:44:58.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:59.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:59.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:59.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:57.325575+0000 osd.4 (osd.4) 541 : cluster 0 3.1c scrub starts 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:57.339345+0000 osd.4 (osd.4) 542 : cluster 0 3.1c scrub ok 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:57.587476+0000 osd.2 (osd.2) 449 : cluster 0 5.d scrub starts 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:57.607941+0000 osd.2 (osd.2) 450 : cluster 0 5.d scrub ok 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.331215+0000 osd.3 (osd.3) 560 : cluster 0 11.1 scrub ok 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.406763+0000 osd.3 (osd.3) 561 : cluster 0 3.1e scrub starts 2024-04-06T03:44:59.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.408950+0000 osd.3 (osd.3) 562 : cluster 0 3.1e scrub ok 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.512756+0000 mgr.y (mgr.24368) 881 : cluster 0 pgmap v959: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:57.325575+0000 osd.4 (osd.4) 541 : cluster 0 3.1c scrub starts 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:57.339345+0000 osd.4 (osd.4) 542 : cluster 0 3.1c scrub ok 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:57.587476+0000 osd.2 (osd.2) 449 : cluster 0 5.d scrub starts 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:57.607941+0000 osd.2 (osd.2) 450 : cluster 0 5.d scrub ok 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.331215+0000 osd.3 (osd.3) 560 : cluster 0 11.1 scrub ok 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.406763+0000 osd.3 (osd.3) 561 : cluster 0 3.1e scrub starts 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.408950+0000 osd.3 (osd.3) 562 : cluster 0 3.1e scrub ok 2024-04-06T03:44:59.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.512756+0000 mgr.y (mgr.24368) 881 : cluster 0 pgmap v959: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s 2024-04-06T03:44:59.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:59.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 68 16 86243 86227 79.2422 30.3125 0.0142848 0.0126141 2024-04-06T03:44:59.281 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:44:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:44:59.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:57.325575+0000 osd.4 (osd.4) 541 : cluster 0 3.1c scrub starts 2024-04-06T03:44:59.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:57.339345+0000 osd.4 (osd.4) 542 : cluster 0 3.1c scrub ok 2024-04-06T03:44:59.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:57.587476+0000 osd.2 (osd.2) 449 : cluster 0 5.d scrub starts 2024-04-06T03:44:59.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:57.607941+0000 osd.2 (osd.2) 450 : cluster 0 5.d scrub ok 2024-04-06T03:44:59.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.331215+0000 osd.3 (osd.3) 560 : cluster 0 11.1 scrub ok 2024-04-06T03:44:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.406763+0000 osd.3 (osd.3) 561 : cluster 0 3.1e scrub starts 2024-04-06T03:44:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.408950+0000 osd.3 (osd.3) 562 : cluster 0 3.1e scrub ok 2024-04-06T03:44:59.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:44:59 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.512756+0000 mgr.y (mgr.24368) 881 : cluster 0 pgmap v959: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s 2024-04-06T03:44:59.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:59.414 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:44:59.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:44:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:44:59.427 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:44:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:44:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:44:59.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:44:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:44:59.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:59.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:44:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:44:59.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:44:59.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:44:59.917 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:44:59.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:44:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:44:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:00.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:00.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:00.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.343114+0000 osd.4 (osd.4) 543 : cluster 0 6.17 scrub starts 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.475079+0000 osd.4 (osd.4) 544 : cluster 0 6.17 scrub ok 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.595579+0000 osd.2 (osd.2) 451 : cluster 0 5.6 scrub starts 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[21467]: cluster 2024-04-06T03:44:58.598006+0000 osd.2 (osd.2) 452 : cluster 0 5.6 scrub ok 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[21467]: audit 2024-04-06T03:44:59.391393+0000 mon.a (mon.0) 1700 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.343114+0000 osd.4 (osd.4) 543 : cluster 0 6.17 scrub starts 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.475079+0000 osd.4 (osd.4) 544 : cluster 0 6.17 scrub ok 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.595579+0000 osd.2 (osd.2) 451 : cluster 0 5.6 scrub starts 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[29221]: cluster 2024-04-06T03:44:58.598006+0000 osd.2 (osd.2) 452 : cluster 0 5.6 scrub ok 2024-04-06T03:45:00.200 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:00 smithi050 bash[29221]: audit 2024-04-06T03:44:59.391393+0000 mon.a (mon.0) 1700 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:45:00.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:00.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:00 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.343114+0000 osd.4 (osd.4) 543 : cluster 0 6.17 scrub starts 2024-04-06T03:45:00.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:00 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.475079+0000 osd.4 (osd.4) 544 : cluster 0 6.17 scrub ok 2024-04-06T03:45:00.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:00 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.595579+0000 osd.2 (osd.2) 451 : cluster 0 5.6 scrub starts 2024-04-06T03:45:00.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:00 smithi153 bash[23852]: cluster 2024-04-06T03:44:58.598006+0000 osd.2 (osd.2) 452 : cluster 0 5.6 scrub ok 2024-04-06T03:45:00.223 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:00 smithi153 bash[23852]: audit 2024-04-06T03:44:59.391393+0000 mon.a (mon.0) 1700 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:45:00.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 69 16 87149 87133 78.9143 56.625 0.0103933 0.0126668 2024-04-06T03:45:00.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:45:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:45:00.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:00.346 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:00.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:00.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:00.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:00.614 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:00.721 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:00.748 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:00.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:00.882 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:00.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:00.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:01.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:01.119 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:01.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:01.250 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:01.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 70 16 88203 88187 78.728 65.875 0.0188638 0.0126975 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: cluster 2024-04-06T03:44:59.370503+0000 osd.4 (osd.4) 545 : cluster 0 2.f scrub starts 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: cluster 2024-04-06T03:44:59.371918+0000 osd.4 (osd.4) 546 : cluster 0 2.f scrub ok 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: cluster 2024-04-06T03:44:59.427085+0000 osd.7 (osd.7) 565 : cluster 0 6.15 scrub starts 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: cluster 2024-04-06T03:44:59.520725+0000 osd.7 (osd.7) 566 : cluster 0 6.15 scrub ok 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: audit 2024-04-06T03:45:00.223146+0000 mgr.y (mgr.24368) 882 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: cluster 2024-04-06T03:45:00.517045+0000 mgr.y (mgr.24368) 883 : cluster 0 pgmap v960: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 939 op/s 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: audit 2024-04-06T03:45:00.792945+0000 mon.a (mon.0) 1701 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: audit 2024-04-06T03:45:00.793824+0000 mon.a (mon.0) 1702 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:45:01.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:01 smithi153 bash[23852]: audit 2024-04-06T03:45:00.800220+0000 mon.a (mon.0) 1703 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:01.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:01.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:01.392 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: cluster 2024-04-06T03:44:59.370503+0000 osd.4 (osd.4) 545 : cluster 0 2.f scrub starts 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: cluster 2024-04-06T03:44:59.371918+0000 osd.4 (osd.4) 546 : cluster 0 2.f scrub ok 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: cluster 2024-04-06T03:44:59.427085+0000 osd.7 (osd.7) 565 : cluster 0 6.15 scrub starts 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: cluster 2024-04-06T03:44:59.520725+0000 osd.7 (osd.7) 566 : cluster 0 6.15 scrub ok 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: audit 2024-04-06T03:45:00.223146+0000 mgr.y (mgr.24368) 882 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: cluster 2024-04-06T03:45:00.517045+0000 mgr.y (mgr.24368) 883 : cluster 0 pgmap v960: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 939 op/s 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: audit 2024-04-06T03:45:00.792945+0000 mon.a (mon.0) 1701 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: audit 2024-04-06T03:45:00.793824+0000 mon.a (mon.0) 1702 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[21467]: audit 2024-04-06T03:45:00.800220+0000 mon.a (mon.0) 1703 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: cluster 2024-04-06T03:44:59.370503+0000 osd.4 (osd.4) 545 : cluster 0 2.f scrub starts 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: cluster 2024-04-06T03:44:59.371918+0000 osd.4 (osd.4) 546 : cluster 0 2.f scrub ok 2024-04-06T03:45:01.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: cluster 2024-04-06T03:44:59.427085+0000 osd.7 (osd.7) 565 : cluster 0 6.15 scrub starts 2024-04-06T03:45:01.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: cluster 2024-04-06T03:44:59.520725+0000 osd.7 (osd.7) 566 : cluster 0 6.15 scrub ok 2024-04-06T03:45:01.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: audit 2024-04-06T03:45:00.223146+0000 mgr.y (mgr.24368) 882 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:01.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: cluster 2024-04-06T03:45:00.517045+0000 mgr.y (mgr.24368) 883 : cluster 0 pgmap v960: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 59 MiB/s wr, 939 op/s 2024-04-06T03:45:01.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: audit 2024-04-06T03:45:00.792945+0000 mon.a (mon.0) 1701 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:45:01.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: audit 2024-04-06T03:45:00.793824+0000 mon.a (mon.0) 1702 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:45:01.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:01 smithi050 bash[29221]: audit 2024-04-06T03:45:00.800220+0000 mon.a (mon.0) 1703 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:01.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:01.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:01.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:01.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:01.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:01.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:01.775 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:01.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:01.919 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:01.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:02.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:02.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:02 smithi153 bash[23852]: cluster 2024-04-06T03:45:01.567415+0000 osd.3 (osd.3) 563 : cluster 0 2.1b scrub starts 2024-04-06T03:45:02.031 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:02 smithi153 bash[23852]: cluster 2024-04-06T03:45:01.585113+0000 osd.3 (osd.3) 564 : cluster 0 2.1b scrub ok 2024-04-06T03:45:02.042 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:02.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:02 smithi050 bash[21467]: cluster 2024-04-06T03:45:01.567415+0000 osd.3 (osd.3) 563 : cluster 0 2.1b scrub starts 2024-04-06T03:45:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:02 smithi050 bash[21467]: cluster 2024-04-06T03:45:01.585113+0000 osd.3 (osd.3) 564 : cluster 0 2.1b scrub ok 2024-04-06T03:45:02.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:02 smithi050 bash[29221]: cluster 2024-04-06T03:45:01.567415+0000 osd.3 (osd.3) 563 : cluster 0 2.1b scrub starts 2024-04-06T03:45:02.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:02 smithi050 bash[29221]: cluster 2024-04-06T03:45:01.585113+0000 osd.3 (osd.3) 564 : cluster 0 2.1b scrub ok 2024-04-06T03:45:02.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:02.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 71 15 88810 88795 78.1543 38 0.0177284 0.0127902 2024-04-06T03:45:02.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:02.306 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:02.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:02.421 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:45:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:02.544 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:02.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:02.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:02.689 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:02.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:02.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:02.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:02.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:02.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:03.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:03.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:03.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:03.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:03.213 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:03.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:01.546792+0000 osd.4 (osd.4) 547 : cluster 0 2.d scrub starts 2024-04-06T03:45:03.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:01.559221+0000 osd.4 (osd.4) 548 : cluster 0 2.d scrub ok 2024-04-06T03:45:03.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:01.573056+0000 osd.2 (osd.2) 453 : cluster 0 6.1 scrub starts 2024-04-06T03:45:03.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:01.597113+0000 osd.2 (osd.2) 454 : cluster 0 6.1 scrub ok 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.000348+0000 osd.0 (osd.0) 559 : cluster 0 5.1b scrub starts 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.132891+0000 osd.0 (osd.0) 560 : cluster 0 5.1b scrub ok 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.517500+0000 mgr.y (mgr.24368) 884 : cluster 0 pgmap v961: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 931 op/s 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:01.546792+0000 osd.4 (osd.4) 547 : cluster 0 2.d scrub starts 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:01.559221+0000 osd.4 (osd.4) 548 : cluster 0 2.d scrub ok 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:01.573056+0000 osd.2 (osd.2) 453 : cluster 0 6.1 scrub starts 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:01.597113+0000 osd.2 (osd.2) 454 : cluster 0 6.1 scrub ok 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.000348+0000 osd.0 (osd.0) 559 : cluster 0 5.1b scrub starts 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.132891+0000 osd.0 (osd.0) 560 : cluster 0 5.1b scrub ok 2024-04-06T03:45:03.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:03 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.517500+0000 mgr.y (mgr.24368) 884 : cluster 0 pgmap v961: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 931 op/s 2024-04-06T03:45:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:03.253 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 72 16 89677 89661 77.8205 54.125 0.00468053 0.0128438 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:01.546792+0000 osd.4 (osd.4) 547 : cluster 0 2.d scrub starts 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:01.559221+0000 osd.4 (osd.4) 548 : cluster 0 2.d scrub ok 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:01.573056+0000 osd.2 (osd.2) 453 : cluster 0 6.1 scrub starts 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:01.597113+0000 osd.2 (osd.2) 454 : cluster 0 6.1 scrub ok 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.000348+0000 osd.0 (osd.0) 559 : cluster 0 5.1b scrub starts 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.132891+0000 osd.0 (osd.0) 560 : cluster 0 5.1b scrub ok 2024-04-06T03:45:03.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:03 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.517500+0000 mgr.y (mgr.24368) 884 : cluster 0 pgmap v961: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 58 MiB/s wr, 931 op/s 2024-04-06T03:45:03.321 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:03.354 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:03.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:03.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:03.594 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:03.621 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:03.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:03.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:03.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:03.870 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:03.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:03.991 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:03.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:04.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:04.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:04.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:04.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:04 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.405871+0000 osd.7 (osd.7) 567 : cluster 0 3.d scrub starts 2024-04-06T03:45:04.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:04 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.411562+0000 osd.7 (osd.7) 568 : cluster 0 3.d scrub ok 2024-04-06T03:45:04.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:04 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.453781+0000 osd.4 (osd.4) 549 : cluster 0 5.c scrub starts 2024-04-06T03:45:04.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:04 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.458836+0000 osd.4 (osd.4) 550 : cluster 0 5.c scrub ok 2024-04-06T03:45:04.125 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:04 smithi153 bash[23852]: cluster 2024-04-06T03:45:02.724390+0000 osd.2 (osd.2) 455 : cluster 0 11.c scrub starts 2024-04-06T03:45:04.222 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:04.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.405871+0000 osd.7 (osd.7) 567 : cluster 0 3.d scrub starts 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.411562+0000 osd.7 (osd.7) 568 : cluster 0 3.d scrub ok 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.453781+0000 osd.4 (osd.4) 549 : cluster 0 5.c scrub starts 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.458836+0000 osd.4 (osd.4) 550 : cluster 0 5.c scrub ok 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[21467]: cluster 2024-04-06T03:45:02.724390+0000 osd.2 (osd.2) 455 : cluster 0 11.c scrub starts 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.405871+0000 osd.7 (osd.7) 567 : cluster 0 3.d scrub starts 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.411562+0000 osd.7 (osd.7) 568 : cluster 0 3.d scrub ok 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.453781+0000 osd.4 (osd.4) 549 : cluster 0 5.c scrub starts 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.458836+0000 osd.4 (osd.4) 550 : cluster 0 5.c scrub ok 2024-04-06T03:45:04.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:04 smithi050 bash[29221]: cluster 2024-04-06T03:45:02.724390+0000 osd.2 (osd.2) 455 : cluster 0 11.c scrub starts 2024-04-06T03:45:04.249 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:04.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 73 16 90348 90332 77.329 41.9375 0.0219566 0.0129271 2024-04-06T03:45:04.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:04.381 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:04.486 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:04.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:04.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:04.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:04.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:04.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:04.898 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:04.922 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:04.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:05.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:05.061 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:05.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:05.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:05 smithi153 bash[23852]: cluster 2024-04-06T03:45:03.992908+0000 osd.2 (osd.2) 456 : cluster 0 11.c scrub ok 2024-04-06T03:45:05.178 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:05 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.004656+0000 osd.0 (osd.0) 561 : cluster 0 3.6 scrub starts 2024-04-06T03:45:05.179 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:05 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.006793+0000 osd.0 (osd.0) 562 : cluster 0 3.6 scrub ok 2024-04-06T03:45:05.179 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:05 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.518076+0000 mgr.y (mgr.24368) 885 : cluster 0 pgmap v962: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 829 op/s 2024-04-06T03:45:05.193 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:05.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 74 16 91395 91379 77.1682 65.4375 0.00484149 0.0129425 2024-04-06T03:45:05.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[21467]: cluster 2024-04-06T03:45:03.992908+0000 osd.2 (osd.2) 456 : cluster 0 11.c scrub ok 2024-04-06T03:45:05.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.004656+0000 osd.0 (osd.0) 561 : cluster 0 3.6 scrub starts 2024-04-06T03:45:05.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.006793+0000 osd.0 (osd.0) 562 : cluster 0 3.6 scrub ok 2024-04-06T03:45:05.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.518076+0000 mgr.y (mgr.24368) 885 : cluster 0 pgmap v962: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 829 op/s 2024-04-06T03:45:05.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[29221]: cluster 2024-04-06T03:45:03.992908+0000 osd.2 (osd.2) 456 : cluster 0 11.c scrub ok 2024-04-06T03:45:05.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.004656+0000 osd.0 (osd.0) 561 : cluster 0 3.6 scrub starts 2024-04-06T03:45:05.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.006793+0000 osd.0 (osd.0) 562 : cluster 0 3.6 scrub ok 2024-04-06T03:45:05.297 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:05 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.518076+0000 mgr.y (mgr.24368) 885 : cluster 0 pgmap v962: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 829 op/s 2024-04-06T03:45:05.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:05.329 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:05.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:05.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:05.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:05.583 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:05.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:05.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:05.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:05.840 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:05.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:05.977 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:05.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:06.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:06.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.398853+0000 osd.7 (osd.7) 569 : cluster 0 2.1d scrub starts 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.425902+0000 osd.4 (osd.4) 551 : cluster 0 3.1a scrub starts 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.428717+0000 osd.4 (osd.4) 552 : cluster 0 3.1a scrub ok 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: cluster 2024-04-06T03:45:04.452653+0000 osd.7 (osd.7) 570 : cluster 0 2.1d scrub ok 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: cluster 2024-04-06T03:45:05.029733+0000 osd.0 (osd.0) 563 : cluster 0 6.18 scrub starts 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: cluster 2024-04-06T03:45:05.101245+0000 osd.0 (osd.0) 564 : cluster 0 6.18 scrub ok 2024-04-06T03:45:06.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[21467]: audit 2024-04-06T03:45:05.449851+0000 mon.a (mon.0) 1704 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.398853+0000 osd.7 (osd.7) 569 : cluster 0 2.1d scrub starts 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.425902+0000 osd.4 (osd.4) 551 : cluster 0 3.1a scrub starts 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.428717+0000 osd.4 (osd.4) 552 : cluster 0 3.1a scrub ok 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: cluster 2024-04-06T03:45:04.452653+0000 osd.7 (osd.7) 570 : cluster 0 2.1d scrub ok 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: cluster 2024-04-06T03:45:05.029733+0000 osd.0 (osd.0) 563 : cluster 0 6.18 scrub starts 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: cluster 2024-04-06T03:45:05.101245+0000 osd.0 (osd.0) 564 : cluster 0 6.18 scrub ok 2024-04-06T03:45:06.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:06 smithi050 bash[29221]: audit 2024-04-06T03:45:05.449851+0000 mon.a (mon.0) 1704 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:06.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 75 16 92055 92039 76.6891 41.25 0.0192853 0.0130351 2024-04-06T03:45:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.398853+0000 osd.7 (osd.7) 569 : cluster 0 2.1d scrub starts 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.425902+0000 osd.4 (osd.4) 551 : cluster 0 3.1a scrub starts 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.428717+0000 osd.4 (osd.4) 552 : cluster 0 3.1a scrub ok 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: cluster 2024-04-06T03:45:04.452653+0000 osd.7 (osd.7) 570 : cluster 0 2.1d scrub ok 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: cluster 2024-04-06T03:45:05.029733+0000 osd.0 (osd.0) 563 : cluster 0 6.18 scrub starts 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: cluster 2024-04-06T03:45:05.101245+0000 osd.0 (osd.0) 564 : cluster 0 6.18 scrub ok 2024-04-06T03:45:06.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:06 smithi153 bash[23852]: audit 2024-04-06T03:45:05.449851+0000 mon.a (mon.0) 1704 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:06.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:06.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.525 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:06.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:06.656 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:06.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:06.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:06.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:06.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:06.936 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:07.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:07.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:07.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:07.173 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:07.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 76 15 93105 93090 76.5442 65.6875 0.0128797 0.0130591 2024-04-06T03:45:07.276 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:07.285 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:07 smithi153 bash[23852]: cluster 2024-04-06T03:45:06.518801+0000 mgr.y (mgr.24368) 886 : cluster 0 pgmap v963: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 887 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-06T03:45:07.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:07.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:07.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:07 smithi050 bash[21467]: cluster 2024-04-06T03:45:06.518801+0000 mgr.y (mgr.24368) 886 : cluster 0 pgmap v963: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 887 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-06T03:45:07.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:07 smithi050 bash[29221]: cluster 2024-04-06T03:45:06.518801+0000 mgr.y (mgr.24368) 886 : cluster 0 pgmap v963: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 887 B/s rd, 52 MiB/s wr, 826 op/s 2024-04-06T03:45:07.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:07.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:07.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:07.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:07.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:07.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:07.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:07.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:07.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:07.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:07.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:08.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:08.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:08.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:08.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:08.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 77 16 94004 93988 76.279 56.125 0.00358899 0.0130998 2024-04-06T03:45:08.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:08 smithi050 bash[21467]: cluster 2024-04-06T03:45:06.948480+0000 osd.2 (osd.2) 457 : cluster 0 2.e scrub starts 2024-04-06T03:45:08.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:08 smithi050 bash[21467]: cluster 2024-04-06T03:45:06.959586+0000 osd.2 (osd.2) 458 : cluster 0 2.e scrub ok 2024-04-06T03:45:08.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:08 smithi050 bash[21467]: cluster 2024-04-06T03:45:07.914988+0000 osd.3 (osd.3) 565 : cluster 0 6.19 scrub starts 2024-04-06T03:45:08.293 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:08 smithi050 bash[29221]: cluster 2024-04-06T03:45:06.948480+0000 osd.2 (osd.2) 457 : cluster 0 2.e scrub starts 2024-04-06T03:45:08.293 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:08 smithi050 bash[29221]: cluster 2024-04-06T03:45:06.959586+0000 osd.2 (osd.2) 458 : cluster 0 2.e scrub ok 2024-04-06T03:45:08.293 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:08 smithi050 bash[29221]: cluster 2024-04-06T03:45:07.914988+0000 osd.3 (osd.3) 565 : cluster 0 6.19 scrub starts 2024-04-06T03:45:08.320 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:08 smithi153 bash[23852]: cluster 2024-04-06T03:45:06.948480+0000 osd.2 (osd.2) 457 : cluster 0 2.e scrub starts 2024-04-06T03:45:08.320 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:08 smithi153 bash[23852]: cluster 2024-04-06T03:45:06.959586+0000 osd.2 (osd.2) 458 : cluster 0 2.e scrub ok 2024-04-06T03:45:08.320 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:08 smithi153 bash[23852]: cluster 2024-04-06T03:45:07.914988+0000 osd.3 (osd.3) 565 : cluster 0 6.19 scrub starts 2024-04-06T03:45:08.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:08.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:08.472 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:08.500 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166296 10.e 10:70aa3018:::benchmark_data_smithi050_121293_object67479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.552144+0000", 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 188.99801117600001, 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.006335285, 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.555 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166296 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552144+0000", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552144+0000", 2024-04-06T03:45:08.556 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552145+0000", 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.161e-06 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552151+0000", 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4280000000000004e-06 2024-04-06T03:45:08.557 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552166+0000", 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.47e-05 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552174+0000", 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0099999999999995e-06 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.558 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552227+0000", 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3377000000000001e-05 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552235+0000", 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4230000000000005e-06 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.552020+0000", 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.99978509699999996 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.559 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.552127+0000", 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000106659 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.552202+0000", 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4739000000000004e-05 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554697+0000", 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024956169999999999 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.560 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558325+0000", 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036280129999999998 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558379+0000", 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3653999999999998e-05 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558436+0000", 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6956000000000002e-05 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.561 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558480+0000", 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3451000000000001e-05 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166306 10.e 10:7a0341ac:::benchmark_data_smithi050_121293_object67489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.563338+0000", 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 188.98681726999999, 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0000557219999999, 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.562 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166306 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563338+0000", 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563338+0000", 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.563 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563339+0000", 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1009999999999999e-06 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563345+0000", 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6269999999999997e-06 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563355+0000", 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9699999999999994e-06 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.564 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563360+0000", 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5010000000000004e-06 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563403+0000", 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2561000000000001e-05 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563409+0000", 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7259999999999999e-06 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.565 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554736+0000", 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.99132699700000004 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554814+0000", 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8863999999999996e-05 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554886+0000", 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1253000000000004e-05 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.566 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558532+0000", 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036460559999999999 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558595+0000", 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3361000000000005e-05 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.567 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563293+0000", 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0046974119999999998 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563351+0000", 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8588000000000002e-05 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563394+0000", 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2704999999999999e-05 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.568 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166355 10.e 10:7bc3e0da:::benchmark_data_smithi050_121293_object67538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.617833+0000", 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 188.932322508, 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94572550600000005, 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166355 2024-04-06T03:45:08.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617833+0000", 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617833+0000", 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617834+0000", 2024-04-06T03:45:08.570 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0780000000000001e-06 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617840+0000", 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8490000000000002e-06 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617852+0000", 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2264e-05 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617861+0000", 2024-04-06T03:45:08.571 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.85e-06 2024-04-06T03:45:08.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.573 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617907+0000", 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6387000000000002e-05 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617917+0000", 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2900000000000008e-06 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.556421+0000", 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93850430200000001 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.574 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.556533+0000", 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112378 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.556666+0000", 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013266600000000001 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558635+0000", 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019686730000000002 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.575 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558648+0000", 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2917e-05 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563449+0000", 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0048014570000000003 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.576 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563505+0000", 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5788999999999997e-05 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563558+0000", 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3606000000000001e-05 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166405 10.e 10:73562072:::benchmark_data_smithi050_121293_object67588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.665681+0000", 2024-04-06T03:45:08.577 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 188.884473958, 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89810087199999999, 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166405 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665681+0000", 2024-04-06T03:45:08.578 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665681+0000", 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665682+0000", 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5599999999999995e-07 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665685+0000", 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7389999999999999e-06 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665690+0000", 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4310000000000002e-06 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665694+0000", 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0389999999999998e-06 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665748+0000", 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3936e-05 2024-04-06T03:45:08.580 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665757+0000", 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4500000000000004e-06 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558697+0000", 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89294079199999998 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558802+0000", 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010429000000000001 2024-04-06T03:45:08.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558925+0000", 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012352700000000001 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563572+0000", 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0046464970000000003 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563634+0000", 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2106999999999993e-05 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.582 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563690+0000", 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6029000000000002e-05 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563741+0000", 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0942000000000001e-05 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563782+0000", 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1536999999999999e-05 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.583 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166462 10.e 10:7a1da6ec:::benchmark_data_smithi050_121293_object67645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.739365+0000", 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 188.810790634, 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.82941186099999997, 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166462 2024-04-06T03:45:08.584 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739365+0000", 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739365+0000", 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739366+0000", 2024-04-06T03:45:08.585 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0240000000000001e-06 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739370+0000", 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8099999999999999e-06 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739382+0000", 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1929999999999999e-05 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.586 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739390+0000", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4160000000000004e-06 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739453+0000", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3379000000000004e-05 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739462+0000", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1239999999999994e-06 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.561006+0000", 2024-04-06T03:45:08.587 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.82154333899999998 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.561132+0000", 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000126327 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.561263+0000", 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013106200000000001 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563794+0000", 2024-04-06T03:45:08.588 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002530568 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563850+0000", 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6280000000000003e-05 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.568671+0000", 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.004821377 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.568727+0000", 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5729999999999997e-05 2024-04-06T03:45:08.589 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.568777+0000", 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9495e-05 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166657 10.0 10:0c5fca79:::benchmark_data_smithi050_121293_object67840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 187.97051871299999, 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.97418345500000003, 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.590 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166657 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.591 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0999999999999998e-07 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579640+0000", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1590000000000002e-06 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579649+0000", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1049999999999995e-06 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.592 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579655+0000", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5769999999999999e-06 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579785+0000", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013009499999999999 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579794+0000", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8040000000000007e-06 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.546805+0000", 2024-04-06T03:45:08.593 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96701049500000003 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.547033+0000", 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022838900000000001 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.547269+0000", 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00023642800000000001 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550576+0000", 2024-04-06T03:45:08.594 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033069089999999998 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550589+0000", 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2306999999999999e-05 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553711+0000", 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031220969999999999 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553768+0000", 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7395000000000002e-05 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.595 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553820+0000", 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2085e-05 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166669 10.0 10:0ee64f06:::benchmark_data_smithi050_121293_object67852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.596319+0000", 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 187.95383616699999, 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96643771099999998, 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.596 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166669 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596319+0000", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596319+0000", 2024-04-06T03:45:08.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596320+0000", 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9500000000000002e-07 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596323+0000", 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0089999999999999e-06 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596328+0000", 2024-04-06T03:45:08.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5400000000000003e-06 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596333+0000", 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3229999999999999e-06 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596372+0000", 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9202999999999997e-05 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596377+0000", 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4909999999999996e-06 2024-04-06T03:45:08.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550616+0000", 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.95423870799999999 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550730+0000", 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000113606 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550853+0000", 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000123683 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558454+0000", 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0076005459999999997 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558518+0000", 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3930000000000006e-05 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562684+0000", 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0041664179999999999 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562726+0000", 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1202999999999998e-05 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562757+0000", 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1455999999999998e-05 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166688 10.0 10:0f46cca1:::benchmark_data_smithi050_121293_object67871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:45:08.602 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.613205+0000", 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 187.93695041000001, 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94968637499999997, 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166688 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613205+0000", 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613205+0000", 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613206+0000", 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9999999999999996e-07 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613211+0000", 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8300000000000003e-06 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613219+0000", 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4489999999999999e-06 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613226+0000", 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.63e-06 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613284+0000", 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7806999999999997e-05 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613292+0000", 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5159999999999994e-06 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.552513+0000", 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93922135100000004 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.552574+0000", 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0516000000000001e-05 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.552638+0000", 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4424999999999994e-05 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558467+0000", 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0058283010000000001 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562800+0000", 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0043330269999999997 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562839+0000", 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8983999999999998e-05 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562868+0000", 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9204999999999999e-05 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562891+0000", 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3533999999999999e-05 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.608 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166694 10.0 10:0549138b:::benchmark_data_smithi050_121293_object67877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.617633+0000", 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 187.932522057, 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.95598773000000004, 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166694 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.609 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617633+0000", 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617633+0000", 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617634+0000", 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4499999999999995e-07 2024-04-06T03:45:08.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617639+0000", 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6349999999999997e-06 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617648+0000", 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9390000000000003e-06 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617654+0000", 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7660000000000004e-06 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617699+0000", 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5386000000000002e-05 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617705+0000", 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1229999999999997e-06 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553859+0000", 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93615364999999995 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553926+0000", 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6758e-05 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553991+0000", 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5053000000000002e-05 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558533+0000", 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0045419589999999999 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562954+0000", 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044217409999999999 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573507+0000", 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010552601 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573570+0000", 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2898000000000007e-05 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.614 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573621+0000", 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1276000000000002e-05 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166721 10.0 10:046565d5:::benchmark_data_smithi050_121293_object67904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 187.898903563, 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.92257131400000003, 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.615 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166721 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7899999999999999e-07 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651255+0000", 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6790000000000001e-06 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651261+0000", 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0499999999999997e-06 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651267+0000", 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5949999999999998e-06 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651320+0000", 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3390000000000002e-05 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651329+0000", 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9809999999999992e-06 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.555249+0000", 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.90391947399999995 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.555365+0000", 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011683700000000001 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.555438+0000", 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.224e-05 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558539+0000", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031015090000000001 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573446+0000", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.014907388000000001 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573720+0000", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00027335100000000002 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573781+0000", 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0846e-05 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573823+0000", 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2494999999999999e-05 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167251 10.d 10:b458ba05:::benchmark_data_smithi050_121293_object68434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.632106+0000", 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.918049817, 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96111719799999995, 2024-04-06T03:45:08.621 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167251 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632106+0000", 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.622 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632106+0000", 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632110+0000", 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0330000000000002e-06 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632125+0000", 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5639999999999999e-05 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632166+0000", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0365999999999999e-05 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632190+0000", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4083e-05 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632315+0000", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000125217 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632331+0000", 2024-04-06T03:45:08.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5639999999999999e-05 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.573490+0000", 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94115940200000003 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.573684+0000", 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000193582 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.573881+0000", 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019781599999999999 2024-04-06T03:45:08.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.579581+0000", 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0056996119999999997 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591195+0000", 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.011613627 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593068+0000", 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001873535 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593159+0000", 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0503999999999999e-05 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593223+0000", 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4140999999999995e-05 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167253 10.d 10:b8271380:::benchmark_data_smithi050_121293_object68436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.634581+0000", 2024-04-06T03:45:08.627 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.91557473399999, 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96146916199999999, 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167253 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634581+0000", 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634581+0000", 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634583+0000", 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0930000000000001e-06 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634596+0000", 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3635000000000001e-05 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634631+0000", 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4479e-05 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634651+0000", 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9927000000000001e-05 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634783+0000", 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013174000000000001 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.630 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634802+0000", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9519000000000001e-05 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.576782+0000", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94197949299999995 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.576874+0000", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2299999999999994e-05 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.576949+0000", 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5228999999999994e-05 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584102+0000", 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0071525440000000003 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593342+0000", 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0092403950000000002 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.595806+0000", 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024639929999999998 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.595946+0000", 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014025499999999999 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.596050+0000", 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010356 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167259 10.d 10:ba4c23a1:::benchmark_data_smithi050_121293_object68442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.660890+0000", 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.889265749, 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93821357100000002, 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167259 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660890+0000", 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660890+0000", 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660891+0000", 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5829999999999999e-06 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660900+0000", 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8100000000000004e-06 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660918+0000", 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8215000000000001e-05 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660931+0000", 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3030999999999999e-05 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.661043+0000", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111331 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.661065+0000", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2019999999999999e-05 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.578359+0000", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.91729432499999997 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.578426+0000", 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7316000000000004e-05 2024-04-06T03:45:08.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.578496+0000", 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0028000000000001e-05 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585367+0000", 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006871116 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.596108+0000", 2024-04-06T03:45:08.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010740201 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.598848+0000", 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027407600000000001 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.599002+0000", 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000153492 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.599103+0000", 2024-04-06T03:45:08.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000101343 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167275 10.d 10:b830d4fc:::benchmark_data_smithi050_121293_object68458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.677093+0000", 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.873062742, 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93899825599999998, 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.640 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167275 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677093+0000", 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677093+0000", 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677094+0000", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4559999999999999e-06 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677102+0000", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4800000000000004e-06 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677115+0000", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3818000000000001e-05 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677127+0000", 2024-04-06T03:45:08.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1504e-05 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677215+0000", 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8467999999999994e-05 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677228+0000", 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2238999999999999e-05 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.579719+0000", 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.90249140900000002 2024-04-06T03:45:08.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.579947+0000", 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000227813 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.580172+0000", 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022501800000000001 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585379+0000", 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0052066550000000001 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.603365+0000", 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017986565999999999 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.615933+0000", 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012568119000000001 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.615997+0000", 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3530999999999999e-05 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616091+0000", 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4179999999999996e-05 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167283 10.d 10:b6ba4186:::benchmark_data_smithi050_121293_object68466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.684070+0000", 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.86608520999999, 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93226327499999995, 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.646 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167283 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684070+0000", 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684070+0000", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684071+0000", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.235e-06 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684078+0000", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8600000000000004e-06 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684095+0000", 2024-04-06T03:45:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6354e-05 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684105+0000", 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0496000000000001e-05 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684187+0000", 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1683999999999999e-05 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684196+0000", 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7679999999999997e-06 2024-04-06T03:45:08.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.582837+0000", 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89864155400000001 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.582938+0000", 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000100642 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.583033+0000", 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5432000000000004e-05 2024-04-06T03:45:08.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.589783+0000", 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006750196 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.603462+0000", 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.013678350000000001 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616167+0000", 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012705363000000001 2024-04-06T03:45:08.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616263+0000", 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5742000000000006e-05 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616333+0000", 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0599000000000006e-05 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167293 10.d 10:bee55742:::benchmark_data_smithi050_121293_object68476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.652 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.696438+0000", 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.853717577, 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.92009158000000002, 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167293 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696438+0000", 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696438+0000", 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696439+0000", 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.226e-06 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696445+0000", 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8769999999999998e-06 2024-04-06T03:45:08.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696458+0000", 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3541e-05 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696472+0000", 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3951000000000001e-05 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697727+0000", 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0012547299999999999 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697740+0000", 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2476999999999999e-05 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584136+0000", 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.88639684500000004 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584233+0000", 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6334000000000003e-05 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584316+0000", 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3487999999999998e-05 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591066+0000", 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0067497080000000001 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.607549+0000", 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.016483003999999999 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616409+0000", 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0088596300000000003 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616465+0000", 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6335000000000002e-05 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616529+0000", 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4434e-05 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.658 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167294 10.d 10:b609007b:::benchmark_data_smithi050_121293_object68477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.696514+0000", 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.85364124700001, 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.92033769799999998, 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167294 2024-04-06T03:45:08.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696514+0000", 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696514+0000", 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696515+0000", 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9400000000000004e-07 2024-04-06T03:45:08.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696520+0000", 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0250000000000002e-06 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696528+0000", 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8609999999999998e-06 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696535+0000", 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4769999999999997e-06 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697771+0000", 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001235547 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697779+0000", 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9130000000000003e-06 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585469+0000", 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.88769055399999997 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585697+0000", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022735 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585918+0000", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022151199999999999 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591072+0000", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.005153393 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.607634+0000", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.01656241 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616606+0000", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0089722410000000006 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616727+0000", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012099000000000001 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616852+0000", 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012463099999999999 2024-04-06T03:45:08.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167314 10.d 10:be2d3a0d:::benchmark_data_smithi050_121293_object68497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.720937+0000", 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.82921808399999, 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89619347000000005, 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167314 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.665 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720937+0000", 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720937+0000", 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720938+0000", 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.54e-07 2024-04-06T03:45:08.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720943+0000", 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3460000000000001e-06 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720955+0000", 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1399999999999999e-05 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720966+0000", 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1372000000000001e-05 2024-04-06T03:45:08.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.721036+0000", 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9919999999999995e-05 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.721046+0000", 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3940000000000002e-06 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.588520+0000", 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.867474262 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.588618+0000", 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8269000000000004e-05 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.588706+0000", 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8362999999999994e-05 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593009+0000", 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0043024109999999999 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.669 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.607684+0000", 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.014675308 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616940+0000", 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0092559180000000001 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.617046+0000", 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000105504 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.617131+0000", 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5248999999999996e-05 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167321 10.d 10:b0b421cc:::benchmark_data_smithi050_121293_object68504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.736250+0000", 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.813904947, 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89355279600000004, 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.671 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167321 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736250+0000", 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736250+0000", 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736252+0000", 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5570000000000001e-06 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736258+0000", 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4400000000000002e-06 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736273+0000", 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4924e-05 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736284+0000", 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0149999999999999e-05 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736353+0000", 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9795000000000005e-05 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736364+0000", 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0353000000000001e-05 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.589820+0000", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.85345636800000002 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.589917+0000", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7200000000000004e-05 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.590004+0000", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6927000000000003e-05 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593239+0000", 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0032349380000000001 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.612346+0000", 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.019107049000000001 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629497+0000", 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017150971000000001 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629665+0000", 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016773600000000001 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629803+0000", 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000138388 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167328 10.d 10:bc0aacba:::benchmark_data_smithi050_121293_object68511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.777788+0000", 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 184.772367312, 2024-04-06T03:45:08.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.85257628699999999, 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167328 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777788+0000", 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777788+0000", 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777791+0000", 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.683e-06 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777807+0000", 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6599000000000002e-05 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777844+0000", 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6470000000000001e-05 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777863+0000", 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9606000000000002e-05 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.778040+0000", 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000176217 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.778067+0000", 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7452000000000001e-05 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591245+0000", 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.81317800600000001 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591320+0000", 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4659999999999993e-05 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591427+0000", 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010718299999999999 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593247+0000", 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001819747 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.612617+0000", 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.019370396000000002 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629941+0000", 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017323966 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.630188+0000", 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024702099999999999 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.630364+0000", 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000176281 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.683 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:08.684 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:08.684 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:45:08.684 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:08.717 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:08.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:08.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:08.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:08.984 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:08.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:09.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:09.107 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:09.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:09.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:09 smithi153 bash[23852]: cluster 2024-04-06T03:45:07.973941+0000 osd.0 (osd.0) 565 : cluster 0 5.2 scrub starts 2024-04-06T03:45:09.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:09 smithi153 bash[23852]: cluster 2024-04-06T03:45:07.980772+0000 osd.0 (osd.0) 566 : cluster 0 5.2 scrub ok 2024-04-06T03:45:09.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:09 smithi153 bash[23852]: cluster 2024-04-06T03:45:08.003821+0000 osd.3 (osd.3) 566 : cluster 0 6.19 scrub ok 2024-04-06T03:45:09.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:09 smithi153 bash[23852]: cluster 2024-04-06T03:45:08.342805+0000 osd.7 (osd.7) 571 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:09.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:09 smithi153 bash[23852]: cluster 2024-04-06T03:45:08.342879+0000 osd.7 (osd.7) 572 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:09.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:09 smithi153 bash[23852]: cluster 2024-04-06T03:45:08.519435+0000 mgr.y (mgr.24368) 887 : cluster 0 pgmap v964: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s 2024-04-06T03:45:09.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:09.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 78 16 94766 94750 75.9116 47.625 0.0032707 0.0131407 2024-04-06T03:45:09.295 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21467]: cluster 2024-04-06T03:45:07.973941+0000 osd.0 (osd.0) 565 : cluster 0 5.2 scrub starts 2024-04-06T03:45:09.295 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21467]: cluster 2024-04-06T03:45:07.980772+0000 osd.0 (osd.0) 566 : cluster 0 5.2 scrub ok 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21467]: cluster 2024-04-06T03:45:08.003821+0000 osd.3 (osd.3) 566 : cluster 0 6.19 scrub ok 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21467]: cluster 2024-04-06T03:45:08.342805+0000 osd.7 (osd.7) 571 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21467]: cluster 2024-04-06T03:45:08.342879+0000 osd.7 (osd.7) 572 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21467]: cluster 2024-04-06T03:45:08.519435+0000 mgr.y (mgr.24368) 887 : cluster 0 pgmap v964: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[29221]: cluster 2024-04-06T03:45:07.973941+0000 osd.0 (osd.0) 565 : cluster 0 5.2 scrub starts 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[29221]: cluster 2024-04-06T03:45:07.980772+0000 osd.0 (osd.0) 566 : cluster 0 5.2 scrub ok 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[29221]: cluster 2024-04-06T03:45:08.003821+0000 osd.3 (osd.3) 566 : cluster 0 6.19 scrub ok 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[29221]: cluster 2024-04-06T03:45:08.342805+0000 osd.7 (osd.7) 571 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[29221]: cluster 2024-04-06T03:45:08.342879+0000 osd.7 (osd.7) 572 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:09.296 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[29221]: cluster 2024-04-06T03:45:08.519435+0000 mgr.y (mgr.24368) 887 : cluster 0 pgmap v964: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 870 op/s 2024-04-06T03:45:09.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:09.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:09.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:09.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:09.491 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:45:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:45:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:45:09.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:09.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:09.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:09.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-06T03:45:09.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:09.915 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:09.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:10.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:10.079 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:10 smithi153 bash[23852]: cluster 2024-04-06T03:45:09.064331+0000 mon.a (mon.0) 1705 : cluster 0 osdmap e360: 8 total, 8 up, 7 in 2024-04-06T03:45:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:10.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:10.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:10.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 79 16 95451 95435 75.4926 42.8125 0.0119942 0.0132418 2024-04-06T03:45:10.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:10.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:10 smithi050 bash[21467]: cluster 2024-04-06T03:45:09.064331+0000 mon.a (mon.0) 1705 : cluster 0 osdmap e360: 8 total, 8 up, 7 in 2024-04-06T03:45:10.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:10 smithi050 bash[29221]: cluster 2024-04-06T03:45:09.064331+0000 mon.a (mon.0) 1705 : cluster 0 osdmap e360: 8 total, 8 up, 7 in 2024-04-06T03:45:10.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:10.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:10.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:10.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:45:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:45:10.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:10.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:10.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:10.745 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:10.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:10.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:10.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:10.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:11.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:11.135 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:11.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:11 smithi050 bash[21467]: audit 2024-04-06T03:45:10.233182+0000 mgr.y (mgr.24368) 888 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:11.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:11 smithi050 bash[21467]: cluster 2024-04-06T03:45:10.520867+0000 mgr.y (mgr.24368) 889 : cluster 0 pgmap v966: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 833 op/s 2024-04-06T03:45:11.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:11 smithi050 bash[29221]: audit 2024-04-06T03:45:10.233182+0000 mgr.y (mgr.24368) 888 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:11.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:11 smithi050 bash[29221]: cluster 2024-04-06T03:45:10.520867+0000 mgr.y (mgr.24368) 889 : cluster 0 pgmap v966: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 833 op/s 2024-04-06T03:45:11.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:11.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:2024-04-06T03:45:11.255570+0000 min lat: 0.00125308 max lat: 0.577121 avg lat: 0.0132589 2024-04-06T03:45:11.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-06T03:45:11.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 80 16 96529 96513 75.391 67.375 0.0266585 0.0132589 2024-04-06T03:45:11.265 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:11.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:11.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:11.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:11 smithi153 bash[23852]: audit 2024-04-06T03:45:10.233182+0000 mgr.y (mgr.24368) 888 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:11.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:11 smithi153 bash[23852]: cluster 2024-04-06T03:45:10.520867+0000 mgr.y (mgr.24368) 889 : cluster 0 pgmap v966: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 52 MiB/s wr, 833 op/s 2024-04-06T03:45:11.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.662 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:45:11.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:11.697 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:45:11.748 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:45:11.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:11.797 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:45:11.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:11.840 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:45:11.889 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:45:11.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:11.934 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:45:11.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:11.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:11.982 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:45:11.992 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:45:11.992 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:45:11.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:45:11.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:45:11.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:45:11.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:45:11.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:45:11.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:45:12.057 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:12.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.087 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:12.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:12.094 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-04-06T03:45:12.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.214 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:12.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:10.890935+0000 osd.2 (osd.2) 459 : cluster 0 2.c scrub starts 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:10.895200+0000 osd.2 (osd.2) 460 : cluster 0 2.c scrub ok 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:10.951534+0000 osd.3 (osd.3) 567 : cluster 0 3.b scrub starts 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:10.955158+0000 osd.3 (osd.3) 568 : cluster 0 3.b scrub ok 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:11.075749+0000 osd.0 (osd.0) 567 : cluster 0 6.a scrub starts 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:11.156220+0000 osd.0 (osd.0) 568 : cluster 0 6.a scrub ok 2024-04-06T03:45:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:11.381744+0000 osd.7 (osd.7) 573 : cluster 0 6.1c scrub starts 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: audit 2024-04-06T03:45:11.581651+0000 mon.a (mon.0) 1706 : audit 0 from='client.? 172.21.15.50:0/2573325762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:11.919882+0000 osd.3 (osd.3) 569 : cluster 0 5.16 scrub starts 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: cluster 2024-04-06T03:45:11.925413+0000 osd.3 (osd.3) 570 : cluster 0 5.16 scrub ok 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[21467]: audit 2024-04-06T03:45:12.029887+0000 mon.a (mon.0) 1707 : audit 1 from='client.? 172.21.15.50:0/2808226783' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:10.890935+0000 osd.2 (osd.2) 459 : cluster 0 2.c scrub starts 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:10.895200+0000 osd.2 (osd.2) 460 : cluster 0 2.c scrub ok 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:10.951534+0000 osd.3 (osd.3) 567 : cluster 0 3.b scrub starts 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:10.955158+0000 osd.3 (osd.3) 568 : cluster 0 3.b scrub ok 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:11.075749+0000 osd.0 (osd.0) 567 : cluster 0 6.a scrub starts 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:11.156220+0000 osd.0 (osd.0) 568 : cluster 0 6.a scrub ok 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:11.381744+0000 osd.7 (osd.7) 573 : cluster 0 6.1c scrub starts 2024-04-06T03:45:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: audit 2024-04-06T03:45:11.581651+0000 mon.a (mon.0) 1706 : audit 0 from='client.? 172.21.15.50:0/2573325762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:45:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:11.919882+0000 osd.3 (osd.3) 569 : cluster 0 5.16 scrub starts 2024-04-06T03:45:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: cluster 2024-04-06T03:45:11.925413+0000 osd.3 (osd.3) 570 : cluster 0 5.16 scrub ok 2024-04-06T03:45:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:12 smithi050 bash[29221]: audit 2024-04-06T03:45:12.029887+0000 mon.a (mon.0) 1707 : audit 1 from='client.? 172.21.15.50:0/2808226783' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:45:12.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 81 16 97537 97521 75.238 63 0.00751275 0.013286 2024-04-06T03:45:12.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:10.890935+0000 osd.2 (osd.2) 459 : cluster 0 2.c scrub starts 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:10.895200+0000 osd.2 (osd.2) 460 : cluster 0 2.c scrub ok 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:10.951534+0000 osd.3 (osd.3) 567 : cluster 0 3.b scrub starts 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:10.955158+0000 osd.3 (osd.3) 568 : cluster 0 3.b scrub ok 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:11.075749+0000 osd.0 (osd.0) 567 : cluster 0 6.a scrub starts 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:11.156220+0000 osd.0 (osd.0) 568 : cluster 0 6.a scrub ok 2024-04-06T03:45:12.327 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:11.381744+0000 osd.7 (osd.7) 573 : cluster 0 6.1c scrub starts 2024-04-06T03:45:12.328 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: audit 2024-04-06T03:45:11.581651+0000 mon.a (mon.0) 1706 : audit 0 from='client.? 172.21.15.50:0/2573325762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:45:12.328 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:11.919882+0000 osd.3 (osd.3) 569 : cluster 0 5.16 scrub starts 2024-04-06T03:45:12.328 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: cluster 2024-04-06T03:45:11.925413+0000 osd.3 (osd.3) 570 : cluster 0 5.16 scrub ok 2024-04-06T03:45:12.328 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:12 smithi153 bash[23852]: audit 2024-04-06T03:45:12.029887+0000 mon.a (mon.0) 1707 : audit 1 from='client.? 172.21.15.50:0/2808226783' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:45:12.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:12.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:12.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:12.611 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:12.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:12.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:12.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:12.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:12.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:12.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:13.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:13.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:13.142 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:13.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:11.392159+0000 osd.7 (osd.7) 574 : cluster 0 6.1c scrub ok 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: audit 2024-04-06T03:45:12.081704+0000 mon.a (mon.0) 1708 : audit 1 from='client.? 172.21.15.50:0/2808226783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:12.084898+0000 mon.a (mon.0) 1709 : cluster 0 osdmap e361: 8 total, 8 up, 7 in 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:12.377186+0000 osd.7 (osd.7) 575 : cluster 0 5.4 scrub starts 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:12.385085+0000 osd.7 (osd.7) 576 : cluster 0 5.4 scrub ok 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:12.524007+0000 mgr.y (mgr.24368) 890 : cluster 0 pgmap v968: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s 2024-04-06T03:45:13.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:12.796331+0000 osd.2 (osd.2) 461 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[21467]: cluster 2024-04-06T03:45:12.796397+0000 osd.2 (osd.2) 462 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:11.392159+0000 osd.7 (osd.7) 574 : cluster 0 6.1c scrub ok 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: audit 2024-04-06T03:45:12.081704+0000 mon.a (mon.0) 1708 : audit 1 from='client.? 172.21.15.50:0/2808226783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:12.084898+0000 mon.a (mon.0) 1709 : cluster 0 osdmap e361: 8 total, 8 up, 7 in 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:12.377186+0000 osd.7 (osd.7) 575 : cluster 0 5.4 scrub starts 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:12.385085+0000 osd.7 (osd.7) 576 : cluster 0 5.4 scrub ok 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:12.524007+0000 mgr.y (mgr.24368) 890 : cluster 0 pgmap v968: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:12.796331+0000 osd.2 (osd.2) 461 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:13.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:13 smithi050 bash[29221]: cluster 2024-04-06T03:45:12.796397+0000 osd.2 (osd.2) 462 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:13.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:13.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 82 16 98637 98621 75.1588 68.75 0.0113493 0.0133002 2024-04-06T03:45:13.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:13.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:13.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:13.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:13.544 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:11.392159+0000 osd.7 (osd.7) 574 : cluster 0 6.1c scrub ok 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: audit 2024-04-06T03:45:12.081704+0000 mon.a (mon.0) 1708 : audit 1 from='client.? 172.21.15.50:0/2808226783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:12.084898+0000 mon.a (mon.0) 1709 : cluster 0 osdmap e361: 8 total, 8 up, 7 in 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:12.377186+0000 osd.7 (osd.7) 575 : cluster 0 5.4 scrub starts 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:12.385085+0000 osd.7 (osd.7) 576 : cluster 0 5.4 scrub ok 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:12.524007+0000 mgr.y (mgr.24368) 890 : cluster 0 pgmap v968: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 859 op/s 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:12.796331+0000 osd.2 (osd.2) 461 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:13.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:13 smithi153 bash[23852]: cluster 2024-04-06T03:45:12.796397+0000 osd.2 (osd.2) 462 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:13.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:13.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:13.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:13.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:13.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:13.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:13.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:14.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:14.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:14.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:14.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:14.254 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 83 16 99407 99391 74.833 48.125 0.0201812 0.0133214 2024-04-06T03:45:14.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:14 smithi153 bash[23852]: cluster 2024-04-06T03:45:13.093695+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e362: 8 total, 8 up, 7 in 2024-04-06T03:45:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:14 smithi153 bash[23852]: cluster 2024-04-06T03:45:13.920684+0000 osd.3 (osd.3) 571 : cluster 0 3.19 scrub starts 2024-04-06T03:45:14.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:14 smithi153 bash[23852]: cluster 2024-04-06T03:45:13.927871+0000 osd.3 (osd.3) 572 : cluster 0 3.19 scrub ok 2024-04-06T03:45:14.314 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:14.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:14.443 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:14.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:14.568 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:14.670 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:45:14.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:14.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:14 smithi050 bash[21467]: cluster 2024-04-06T03:45:13.093695+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e362: 8 total, 8 up, 7 in 2024-04-06T03:45:14.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:14 smithi050 bash[21467]: cluster 2024-04-06T03:45:13.920684+0000 osd.3 (osd.3) 571 : cluster 0 3.19 scrub starts 2024-04-06T03:45:14.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:14 smithi050 bash[21467]: cluster 2024-04-06T03:45:13.927871+0000 osd.3 (osd.3) 572 : cluster 0 3.19 scrub ok 2024-04-06T03:45:14.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:14 smithi050 bash[29221]: cluster 2024-04-06T03:45:13.093695+0000 mon.a (mon.0) 1710 : cluster 0 osdmap e362: 8 total, 8 up, 7 in 2024-04-06T03:45:14.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:14 smithi050 bash[29221]: cluster 2024-04-06T03:45:13.920684+0000 osd.3 (osd.3) 571 : cluster 0 3.19 scrub starts 2024-04-06T03:45:14.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:14 smithi050 bash[29221]: cluster 2024-04-06T03:45:13.927871+0000 osd.3 (osd.3) 572 : cluster 0 3.19 scrub ok 2024-04-06T03:45:14.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:14.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:14.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:14.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:14.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:14.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:15.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:15.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:15.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:15.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:15 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.265457+0000 osd.0 (osd.0) 569 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:15.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:15 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.265556+0000 osd.0 (osd.0) 570 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:15.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:15 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.351783+0000 osd.7 (osd.7) 577 : cluster 0 6.8 scrub starts 2024-04-06T03:45:15.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:15 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.363459+0000 osd.7 (osd.7) 578 : cluster 0 6.8 scrub ok 2024-04-06T03:45:15.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:15 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.524456+0000 mgr.y (mgr.24368) 891 : cluster 0 pgmap v970: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 682 B/s rd, 45 MiB/s wr, 720 op/s 2024-04-06T03:45:15.224 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:15.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 84 16 100383 100367 74.6683 61 0.0135382 0.0133878 2024-04-06T03:45:15.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:15.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.490 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:15.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.615 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:15.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.265457+0000 osd.0 (osd.0) 569 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.265556+0000 osd.0 (osd.0) 570 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.351783+0000 osd.7 (osd.7) 577 : cluster 0 6.8 scrub starts 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.363459+0000 osd.7 (osd.7) 578 : cluster 0 6.8 scrub ok 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.524456+0000 mgr.y (mgr.24368) 891 : cluster 0 pgmap v970: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 682 B/s rd, 45 MiB/s wr, 720 op/s 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.265457+0000 osd.0 (osd.0) 569 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.265556+0000 osd.0 (osd.0) 570 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:15.735 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.351783+0000 osd.7 (osd.7) 577 : cluster 0 6.8 scrub starts 2024-04-06T03:45:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.363459+0000 osd.7 (osd.7) 578 : cluster 0 6.8 scrub ok 2024-04-06T03:45:15.736 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:15 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.524456+0000 mgr.y (mgr.24368) 891 : cluster 0 pgmap v970: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 682 B/s rd, 45 MiB/s wr, 720 op/s 2024-04-06T03:45:15.750 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:15.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:15.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:15.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:15.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:16.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:16.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:16.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:16.128 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:16.231 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.939956+0000 osd.2 (osd.2) 463 : cluster 0 3.0 scrub starts 2024-04-06T03:45:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[21467]: cluster 2024-04-06T03:45:14.950148+0000 osd.2 (osd.2) 464 : cluster 0 3.0 scrub ok 2024-04-06T03:45:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[21467]: cluster 2024-04-06T03:45:15.405015+0000 mon.a (mon.0) 1711 : cluster 0 osdmap e363: 8 total, 8 up, 7 in 2024-04-06T03:45:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[21467]: cluster 2024-04-06T03:45:15.939841+0000 osd.3 (osd.3) 573 : cluster 0 5.12 scrub starts 2024-04-06T03:45:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[21467]: cluster 2024-04-06T03:45:15.944254+0000 osd.3 (osd.3) 574 : cluster 0 5.12 scrub ok 2024-04-06T03:45:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.939956+0000 osd.2 (osd.2) 463 : cluster 0 3.0 scrub starts 2024-04-06T03:45:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[29221]: cluster 2024-04-06T03:45:14.950148+0000 osd.2 (osd.2) 464 : cluster 0 3.0 scrub ok 2024-04-06T03:45:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[29221]: cluster 2024-04-06T03:45:15.405015+0000 mon.a (mon.0) 1711 : cluster 0 osdmap e363: 8 total, 8 up, 7 in 2024-04-06T03:45:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[29221]: cluster 2024-04-06T03:45:15.939841+0000 osd.3 (osd.3) 573 : cluster 0 5.12 scrub starts 2024-04-06T03:45:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:16 smithi050 bash[29221]: cluster 2024-04-06T03:45:15.944254+0000 osd.3 (osd.3) 574 : cluster 0 5.12 scrub ok 2024-04-06T03:45:16.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 85 16 101042 101026 74.2743 41.1875 0.00374233 0.0134202 2024-04-06T03:45:16.255 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:16.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:16 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.939956+0000 osd.2 (osd.2) 463 : cluster 0 3.0 scrub starts 2024-04-06T03:45:16.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:16 smithi153 bash[23852]: cluster 2024-04-06T03:45:14.950148+0000 osd.2 (osd.2) 464 : cluster 0 3.0 scrub ok 2024-04-06T03:45:16.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:16 smithi153 bash[23852]: cluster 2024-04-06T03:45:15.405015+0000 mon.a (mon.0) 1711 : cluster 0 osdmap e363: 8 total, 8 up, 7 in 2024-04-06T03:45:16.267 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:16 smithi153 bash[23852]: cluster 2024-04-06T03:45:15.939841+0000 osd.3 (osd.3) 573 : cluster 0 5.12 scrub starts 2024-04-06T03:45:16.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:16.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:16.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:16.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:16 smithi153 bash[23852]: cluster 2024-04-06T03:45:15.944254+0000 osd.3 (osd.3) 574 : cluster 0 5.12 scrub ok 2024-04-06T03:45:16.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:16.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:16.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:16.778 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:16.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:16.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:16.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:16.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 7 1 2024-04-06T03:45:17.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:17.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:17.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:17.192 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:17 smithi050 bash[21467]: cluster 2024-04-06T03:45:16.524944+0000 mgr.y (mgr.24368) 892 : cluster 0 pgmap v972: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 682 B/s rd, 49 MiB/s wr, 791 op/s 2024-04-06T03:45:17.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:17 smithi050 bash[29221]: cluster 2024-04-06T03:45:16.524944+0000 mgr.y (mgr.24368) 892 : cluster 0 pgmap v972: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 682 B/s rd, 49 MiB/s wr, 791 op/s 2024-04-06T03:45:17.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:17.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 86 16 102027 102011 74.1264 61.5625 0.00843972 0.0134855 2024-04-06T03:45:17.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:17.363 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:17.365 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:17 smithi153 bash[23852]: cluster 2024-04-06T03:45:16.524944+0000 mgr.y (mgr.24368) 892 : cluster 0 pgmap v972: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 682 B/s rd, 49 MiB/s wr, 791 op/s 2024-04-06T03:45:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:17.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:17.497 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:17.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:17.637 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:17.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:17.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:17.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:17.918 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:17.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:18.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:18.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:18.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:18.128 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:45:18.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:18.173 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: cluster 2024-04-06T03:45:16.995150+0000 osd.2 (osd.2) 465 : cluster 0 6.f scrub starts 2024-04-06T03:45:18.173 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.008706+0000 osd.0 (osd.0) 571 : cluster 0 3.1 scrub starts 2024-04-06T03:45:18.173 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.013150+0000 osd.0 (osd.0) 572 : cluster 0 3.1 scrub ok 2024-04-06T03:45:18.173 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.269758+0000 osd.2 (osd.2) 466 : cluster 0 6.f scrub ok 2024-04-06T03:45:18.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.919778+0000 osd.3 (osd.3) 575 : cluster 0 3.18 scrub starts 2024-04-06T03:45:18.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.921593+0000 osd.3 (osd.3) 576 : cluster 0 3.18 scrub ok 2024-04-06T03:45:18.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: audit 2024-04-06T03:45:18.019535+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.50:0/4182027859' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:45:18.174 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:18 smithi153 bash[23852]: audit 2024-04-06T03:45:18.020044+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:45:18.191 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:18.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 87 16 103026 103010 73.992 62.4375 0.0108964 0.0135099 2024-04-06T03:45:18.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:18.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:18.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:18.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: cluster 2024-04-06T03:45:16.995150+0000 osd.2 (osd.2) 465 : cluster 0 6.f scrub starts 2024-04-06T03:45:18.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.008706+0000 osd.0 (osd.0) 571 : cluster 0 3.1 scrub starts 2024-04-06T03:45:18.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.013150+0000 osd.0 (osd.0) 572 : cluster 0 3.1 scrub ok 2024-04-06T03:45:18.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.269758+0000 osd.2 (osd.2) 466 : cluster 0 6.f scrub ok 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.919778+0000 osd.3 (osd.3) 575 : cluster 0 3.18 scrub starts 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.921593+0000 osd.3 (osd.3) 576 : cluster 0 3.18 scrub ok 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: audit 2024-04-06T03:45:18.019535+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.50:0/4182027859' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[21467]: audit 2024-04-06T03:45:18.020044+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: cluster 2024-04-06T03:45:16.995150+0000 osd.2 (osd.2) 465 : cluster 0 6.f scrub starts 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.008706+0000 osd.0 (osd.0) 571 : cluster 0 3.1 scrub starts 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.013150+0000 osd.0 (osd.0) 572 : cluster 0 3.1 scrub ok 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.269758+0000 osd.2 (osd.2) 466 : cluster 0 6.f scrub ok 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.919778+0000 osd.3 (osd.3) 575 : cluster 0 3.18 scrub starts 2024-04-06T03:45:18.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.921593+0000 osd.3 (osd.3) 576 : cluster 0 3.18 scrub ok 2024-04-06T03:45:18.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: audit 2024-04-06T03:45:18.019535+0000 mon.c (mon.2) 109 : audit 1 from='client.? 172.21.15.50:0/4182027859' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:45:18.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:18 smithi050 bash[29221]: audit 2024-04-06T03:45:18.020044+0000 mon.a (mon.0) 1712 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:45:18.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:18.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:18.592 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:18.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:18.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:18.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:18.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:18.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:18.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:45:18.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:19.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:19.161 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:19.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 88 16 103777 103761 73.6845 46.9375 0.022732 0.0135665 2024-04-06T03:45:19.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.321404+0000 osd.4 (osd.4) 553 : cluster 0 5.18 scrub starts 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: cluster 2024-04-06T03:45:17.323804+0000 osd.4 (osd.4) 554 : cluster 0 5.18 scrub ok 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: cluster 2024-04-06T03:45:18.107335+0000 mon.a (mon.0) 1713 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: cluster 2024-04-06T03:45:18.107373+0000 mon.a (mon.0) 1714 : cluster 1 Cluster is now healthy 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: audit 2024-04-06T03:45:18.109452+0000 mon.a (mon.0) 1715 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: cluster 2024-04-06T03:45:18.113859+0000 mon.a (mon.0) 1716 : cluster 0 osdmap e364: 8 total, 8 up, 7 in 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21467]: cluster 2024-04-06T03:45:18.525445+0000 mgr.y (mgr.24368) 893 : cluster 0 pgmap v974: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.321404+0000 osd.4 (osd.4) 553 : cluster 0 5.18 scrub starts 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: cluster 2024-04-06T03:45:17.323804+0000 osd.4 (osd.4) 554 : cluster 0 5.18 scrub ok 2024-04-06T03:45:19.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: cluster 2024-04-06T03:45:18.107335+0000 mon.a (mon.0) 1713 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:45:19.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: cluster 2024-04-06T03:45:18.107373+0000 mon.a (mon.0) 1714 : cluster 1 Cluster is now healthy 2024-04-06T03:45:19.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: audit 2024-04-06T03:45:18.109452+0000 mon.a (mon.0) 1715 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:45:19.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: cluster 2024-04-06T03:45:18.113859+0000 mon.a (mon.0) 1716 : cluster 0 osdmap e364: 8 total, 8 up, 7 in 2024-04-06T03:45:19.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[29221]: cluster 2024-04-06T03:45:18.525445+0000 mgr.y (mgr.24368) 893 : cluster 0 pgmap v974: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-06T03:45:19.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:19.397 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.321404+0000 osd.4 (osd.4) 553 : cluster 0 5.18 scrub starts 2024-04-06T03:45:19.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: cluster 2024-04-06T03:45:17.323804+0000 osd.4 (osd.4) 554 : cluster 0 5.18 scrub ok 2024-04-06T03:45:19.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: cluster 2024-04-06T03:45:18.107335+0000 mon.a (mon.0) 1713 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:45:19.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: cluster 2024-04-06T03:45:18.107373+0000 mon.a (mon.0) 1714 : cluster 1 Cluster is now healthy 2024-04-06T03:45:19.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: audit 2024-04-06T03:45:18.109452+0000 mon.a (mon.0) 1715 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:45:19.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: cluster 2024-04-06T03:45:18.113859+0000 mon.a (mon.0) 1716 : cluster 0 osdmap e364: 8 total, 8 up, 7 in 2024-04-06T03:45:19.398 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:19 smithi153 bash[23852]: cluster 2024-04-06T03:45:18.525445+0000 mgr.y (mgr.24368) 893 : cluster 0 pgmap v974: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-04-06T03:45:19.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:19.431 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:45:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:45:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:45:19.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:19.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:19.576 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:19.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:19.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:19.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:19.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:19.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:20.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:20.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:20.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:20.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:20 smithi050 bash[21467]: cluster 2024-04-06T03:45:18.352083+0000 osd.4 (osd.4) 555 : cluster 0 3.1d scrub starts 2024-04-06T03:45:20.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:20 smithi050 bash[21467]: cluster 2024-04-06T03:45:18.356008+0000 osd.4 (osd.4) 556 : cluster 0 3.1d scrub ok 2024-04-06T03:45:20.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:20 smithi050 bash[29221]: cluster 2024-04-06T03:45:18.352083+0000 osd.4 (osd.4) 555 : cluster 0 3.1d scrub starts 2024-04-06T03:45:20.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:20 smithi050 bash[29221]: cluster 2024-04-06T03:45:18.356008+0000 osd.4 (osd.4) 556 : cluster 0 3.1d scrub ok 2024-04-06T03:45:20.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:20.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout: 89 15 104901 104886 73.6466 70.3125 0.0222168 0.0135739 2024-04-06T03:45:20.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:20.355 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:20.461 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:20.471 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:45:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:45:20.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:20 smithi153 bash[23852]: cluster 2024-04-06T03:45:18.352083+0000 osd.4 (osd.4) 555 : cluster 0 3.1d scrub starts 2024-04-06T03:45:20.471 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:20 smithi153 bash[23852]: cluster 2024-04-06T03:45:18.356008+0000 osd.4 (osd.4) 556 : cluster 0 3.1d scrub ok 2024-04-06T03:45:20.489 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:20.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:20.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:20.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:20.753 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:20.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:20.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:20.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:20.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:21.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:21.112 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:45:21.112 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:45:21.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:21.164 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:21.255 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total time run: 90.0046 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Total writes made: 105971 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Write size: 65536 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Object size: 65536 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Bandwidth (MB/sec): 73.5872 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Bandwidth: 29.5299 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max bandwidth (MB/sec): 175 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min bandwidth (MB/sec): 24.5 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average IOPS: 1177 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev IOPS: 472.478 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max IOPS: 2800 2024-04-06T03:45:21.256 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min IOPS: 392 2024-04-06T03:45:21.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Average Latency(s): 0.0135857 2024-04-06T03:45:21.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Stddev Latency(s): 0.0246306 2024-04-06T03:45:21.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Max latency(s): 0.577121 2024-04-06T03:45:21.257 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Min latency(s): 0.00125308 2024-04-06T03:45:21.259 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Cleaning up (deleting benchmark objects) 2024-04-06T03:45:21.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:21.402 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:21.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:21.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:45:21.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[21467]: audit 2024-04-06T03:45:20.243936+0000 mgr.y (mgr.24368) 894 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:21.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[21467]: audit 2024-04-06T03:45:20.474972+0000 mon.a (mon.0) 1717 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:21.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[21467]: audit 2024-04-06T03:45:20.476651+0000 mon.a (mon.0) 1718 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:21.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[21467]: cluster 2024-04-06T03:45:20.526214+0000 mgr.y (mgr.24368) 895 : cluster 0 pgmap v975: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 826 B/s rd, 61 MiB/s wr, 974 op/s 2024-04-06T03:45:21.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[21467]: cluster 2024-04-06T03:45:20.918511+0000 osd.3 (osd.3) 577 : cluster 0 5.0 scrub starts 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[21467]: cluster 2024-04-06T03:45:20.921237+0000 osd.3 (osd.3) 578 : cluster 0 5.0 scrub ok 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[29221]: audit 2024-04-06T03:45:20.243936+0000 mgr.y (mgr.24368) 894 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[29221]: audit 2024-04-06T03:45:20.474972+0000 mon.a (mon.0) 1717 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[29221]: audit 2024-04-06T03:45:20.476651+0000 mon.a (mon.0) 1718 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[29221]: cluster 2024-04-06T03:45:20.526214+0000 mgr.y (mgr.24368) 895 : cluster 0 pgmap v975: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 826 B/s rd, 61 MiB/s wr, 974 op/s 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[29221]: cluster 2024-04-06T03:45:20.918511+0000 osd.3 (osd.3) 577 : cluster 0 5.0 scrub starts 2024-04-06T03:45:21.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:21 smithi050 bash[29221]: cluster 2024-04-06T03:45:20.921237+0000 osd.3 (osd.3) 578 : cluster 0 5.0 scrub ok 2024-04-06T03:45:21.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:21.574 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:21.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:21.687 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:21 smithi153 bash[23852]: audit 2024-04-06T03:45:20.243936+0000 mgr.y (mgr.24368) 894 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:21.687 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:21 smithi153 bash[23852]: audit 2024-04-06T03:45:20.474972+0000 mon.a (mon.0) 1717 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:21.687 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:21 smithi153 bash[23852]: audit 2024-04-06T03:45:20.476651+0000 mon.a (mon.0) 1718 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:21.687 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:21 smithi153 bash[23852]: cluster 2024-04-06T03:45:20.526214+0000 mgr.y (mgr.24368) 895 : cluster 0 pgmap v975: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 826 B/s rd, 61 MiB/s wr, 974 op/s 2024-04-06T03:45:21.687 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:21 smithi153 bash[23852]: cluster 2024-04-06T03:45:20.918511+0000 osd.3 (osd.3) 577 : cluster 0 5.0 scrub starts 2024-04-06T03:45:21.688 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:21 smithi153 bash[23852]: cluster 2024-04-06T03:45:20.921237+0000 osd.3 (osd.3) 578 : cluster 0 5.0 scrub ok 2024-04-06T03:45:21.702 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:21.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.836 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:21.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:21.937 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:21.962 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:21.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:22.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:22.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:22.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:22.232 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:22.339 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:22.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:22.433 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-06T03:45:22.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:22.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:22.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:22.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:22.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:22.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:22.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:22.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:22.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:22.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:23.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:23.052 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:23.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:23.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:23.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:23.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:23.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:23.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:23.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:23.548 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:23.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:23.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:23.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[21467]: cluster 2024-04-06T03:45:21.977255+0000 osd.0 (osd.0) 573 : cluster 0 2.1f deep-scrub starts 2024-04-06T03:45:23.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[21467]: cluster 2024-04-06T03:45:21.990958+0000 osd.0 (osd.0) 574 : cluster 0 2.1f deep-scrub ok 2024-04-06T03:45:23.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[21467]: cluster 2024-04-06T03:45:22.526754+0000 mgr.y (mgr.24368) 896 : cluster 0 pgmap v976: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 58 MiB/s wr, 941 op/s 2024-04-06T03:45:23.789 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:23.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:23 smithi153 bash[23852]: cluster 2024-04-06T03:45:21.977255+0000 osd.0 (osd.0) 573 : cluster 0 2.1f deep-scrub starts 2024-04-06T03:45:23.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:23 smithi153 bash[23852]: cluster 2024-04-06T03:45:21.990958+0000 osd.0 (osd.0) 574 : cluster 0 2.1f deep-scrub ok 2024-04-06T03:45:23.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:23 smithi153 bash[23852]: cluster 2024-04-06T03:45:22.526754+0000 mgr.y (mgr.24368) 896 : cluster 0 pgmap v976: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 58 MiB/s wr, 941 op/s 2024-04-06T03:45:23.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:23 smithi153 bash[23852]: cluster 2024-04-06T03:45:23.354138+0000 osd.7 (osd.7) 579 : cluster 0 6.5 scrub starts 2024-04-06T03:45:23.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:23.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:23.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[21467]: cluster 2024-04-06T03:45:23.354138+0000 osd.7 (osd.7) 579 : cluster 0 6.5 scrub starts 2024-04-06T03:45:23.930 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[29221]: cluster 2024-04-06T03:45:21.977255+0000 osd.0 (osd.0) 573 : cluster 0 2.1f deep-scrub starts 2024-04-06T03:45:23.930 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[29221]: cluster 2024-04-06T03:45:21.990958+0000 osd.0 (osd.0) 574 : cluster 0 2.1f deep-scrub ok 2024-04-06T03:45:23.930 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[29221]: cluster 2024-04-06T03:45:22.526754+0000 mgr.y (mgr.24368) 896 : cluster 0 pgmap v976: 148 pgs: 148 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 767 B/s rd, 58 MiB/s wr, 941 op/s 2024-04-06T03:45:23.930 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:23 smithi050 bash[29221]: cluster 2024-04-06T03:45:23.354138+0000 osd.7 (osd.7) 579 : cluster 0 6.5 scrub starts 2024-04-06T03:45:23.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:23.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:23.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:24.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:24.076 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:24.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:24.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:24.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:24.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:24.349 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:24.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:24.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:24.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:24.617 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:24.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:24.758 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:24.863 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:24.873 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:24 smithi153 bash[23852]: cluster 2024-04-06T03:45:23.445288+0000 osd.7 (osd.7) 580 : cluster 0 6.5 scrub ok 2024-04-06T03:45:24.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:24 smithi153 bash[23852]: cluster 2024-04-06T03:45:24.304575+0000 osd.7 (osd.7) 581 : cluster 0 2.b scrub starts 2024-04-06T03:45:24.874 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:24 smithi153 bash[23852]: cluster 2024-04-06T03:45:24.308108+0000 osd.7 (osd.7) 582 : cluster 0 2.b scrub ok 2024-04-06T03:45:24.892 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:24.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:24.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:24 smithi050 bash[21467]: cluster 2024-04-06T03:45:23.445288+0000 osd.7 (osd.7) 580 : cluster 0 6.5 scrub ok 2024-04-06T03:45:24.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:24 smithi050 bash[21467]: cluster 2024-04-06T03:45:24.304575+0000 osd.7 (osd.7) 581 : cluster 0 2.b scrub starts 2024-04-06T03:45:24.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:24 smithi050 bash[21467]: cluster 2024-04-06T03:45:24.308108+0000 osd.7 (osd.7) 582 : cluster 0 2.b scrub ok 2024-04-06T03:45:24.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:24 smithi050 bash[29221]: cluster 2024-04-06T03:45:23.445288+0000 osd.7 (osd.7) 580 : cluster 0 6.5 scrub ok 2024-04-06T03:45:24.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:24 smithi050 bash[29221]: cluster 2024-04-06T03:45:24.304575+0000 osd.7 (osd.7) 581 : cluster 0 2.b scrub starts 2024-04-06T03:45:24.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:24 smithi050 bash[29221]: cluster 2024-04-06T03:45:24.308108+0000 osd.7 (osd.7) 582 : cluster 0 2.b scrub ok 2024-04-06T03:45:24.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:25.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:25.163 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:25.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:25.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:25.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:25.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:25.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:25.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:25.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:25 smithi050 bash[29221]: cluster 2024-04-06T03:45:24.527334+0000 mgr.y (mgr.24368) 897 : cluster 0 pgmap v977: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 810 op/s 2024-04-06T03:45:25.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:25 smithi050 bash[29221]: cluster 2024-04-06T03:45:25.264242+0000 osd.7 (osd.7) 583 : cluster 0 5.1a deep-scrub starts 2024-04-06T03:45:25.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:25 smithi050 bash[29221]: cluster 2024-04-06T03:45:25.274774+0000 osd.7 (osd.7) 584 : cluster 0 5.1a deep-scrub ok 2024-04-06T03:45:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:25.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:25 smithi050 bash[21467]: cluster 2024-04-06T03:45:24.527334+0000 mgr.y (mgr.24368) 897 : cluster 0 pgmap v977: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 810 op/s 2024-04-06T03:45:25.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:25 smithi050 bash[21467]: cluster 2024-04-06T03:45:25.264242+0000 osd.7 (osd.7) 583 : cluster 0 5.1a deep-scrub starts 2024-04-06T03:45:25.691 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:25 smithi050 bash[21467]: cluster 2024-04-06T03:45:25.274774+0000 osd.7 (osd.7) 584 : cluster 0 5.1a deep-scrub ok 2024-04-06T03:45:25.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:25 smithi153 bash[23852]: cluster 2024-04-06T03:45:24.527334+0000 mgr.y (mgr.24368) 897 : cluster 0 pgmap v977: 148 pgs: 148 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 810 op/s 2024-04-06T03:45:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:25 smithi153 bash[23852]: cluster 2024-04-06T03:45:25.264242+0000 osd.7 (osd.7) 583 : cluster 0 5.1a deep-scrub starts 2024-04-06T03:45:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:25 smithi153 bash[23852]: cluster 2024-04-06T03:45:25.274774+0000 osd.7 (osd.7) 584 : cluster 0 5.1a deep-scrub ok 2024-04-06T03:45:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:25.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:25.970 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:25.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:26.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:26.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:26.109 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:26.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:26.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:26.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:26.369 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:26.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:26.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:26.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:26.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:26.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:26.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:26.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:26 smithi153 bash[23852]: cluster 2024-04-06T03:45:24.961824+0000 osd.2 (osd.2) 467 : cluster 0 11.d scrub starts 2024-04-06T03:45:26.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:26.921 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:26.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:26.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:26 smithi050 bash[21467]: cluster 2024-04-06T03:45:24.961824+0000 osd.2 (osd.2) 467 : cluster 0 11.d scrub starts 2024-04-06T03:45:26.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:26 smithi050 bash[29221]: cluster 2024-04-06T03:45:24.961824+0000 osd.2 (osd.2) 467 : cluster 0 11.d scrub starts 2024-04-06T03:45:27.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:27.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:27.190 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:27.257 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:45:27.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:27.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:27.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:27.447 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:27.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:27.567 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:27.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:27.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:27.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:27.839 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:27.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:27.975 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:27.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:27.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:27 smithi050 bash[21467]: cluster 2024-04-06T03:45:26.395136+0000 osd.4 (osd.4) 557 : cluster 0 2.1a scrub starts 2024-04-06T03:45:27.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:27 smithi050 bash[21467]: cluster 2024-04-06T03:45:26.399016+0000 osd.4 (osd.4) 558 : cluster 0 2.1a scrub ok 2024-04-06T03:45:27.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:27 smithi050 bash[21467]: cluster 2024-04-06T03:45:26.528056+0000 mgr.y (mgr.24368) 898 : cluster 0 pgmap v978: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 996 op/s 2024-04-06T03:45:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:27 smithi050 bash[29221]: cluster 2024-04-06T03:45:26.395136+0000 osd.4 (osd.4) 557 : cluster 0 2.1a scrub starts 2024-04-06T03:45:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:27 smithi050 bash[29221]: cluster 2024-04-06T03:45:26.399016+0000 osd.4 (osd.4) 558 : cluster 0 2.1a scrub ok 2024-04-06T03:45:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:27 smithi050 bash[29221]: cluster 2024-04-06T03:45:26.528056+0000 mgr.y (mgr.24368) 898 : cluster 0 pgmap v978: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 996 op/s 2024-04-06T03:45:28.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:28.089 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:27 smithi153 bash[23852]: cluster 2024-04-06T03:45:26.395136+0000 osd.4 (osd.4) 557 : cluster 0 2.1a scrub starts 2024-04-06T03:45:28.090 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:27 smithi153 bash[23852]: cluster 2024-04-06T03:45:26.399016+0000 osd.4 (osd.4) 558 : cluster 0 2.1a scrub ok 2024-04-06T03:45:28.090 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:27 smithi153 bash[23852]: cluster 2024-04-06T03:45:26.528056+0000 mgr.y (mgr.24368) 898 : cluster 0 pgmap v978: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 996 op/s 2024-04-06T03:45:28.114 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:28.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:28.239 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:28.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:28.377 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:28.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:28.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:28.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:28.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:28.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:28.756 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:28.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:28.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:28.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:28.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:28 smithi153 bash[23852]: cluster 2024-04-06T03:45:27.058107+0000 osd.2 (osd.2) 468 : cluster 0 11.d scrub ok 2024-04-06T03:45:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:28 smithi153 bash[23852]: cluster 2024-04-06T03:45:27.117460+0000 osd.0 (osd.0) 575 : cluster 0 3.17 scrub starts 2024-04-06T03:45:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:28 smithi153 bash[23852]: cluster 2024-04-06T03:45:27.123643+0000 osd.0 (osd.0) 576 : cluster 0 3.17 scrub ok 2024-04-06T03:45:29.112 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:29.142 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:29.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:29.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:28 smithi050 bash[21467]: cluster 2024-04-06T03:45:27.058107+0000 osd.2 (osd.2) 468 : cluster 0 11.d scrub ok 2024-04-06T03:45:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:28 smithi050 bash[21467]: cluster 2024-04-06T03:45:27.117460+0000 osd.0 (osd.0) 575 : cluster 0 3.17 scrub starts 2024-04-06T03:45:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:28 smithi050 bash[21467]: cluster 2024-04-06T03:45:27.123643+0000 osd.0 (osd.0) 576 : cluster 0 3.17 scrub ok 2024-04-06T03:45:29.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:28 smithi050 bash[29221]: cluster 2024-04-06T03:45:27.058107+0000 osd.2 (osd.2) 468 : cluster 0 11.d scrub ok 2024-04-06T03:45:29.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:28 smithi050 bash[29221]: cluster 2024-04-06T03:45:27.117460+0000 osd.0 (osd.0) 575 : cluster 0 3.17 scrub starts 2024-04-06T03:45:29.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:28 smithi050 bash[29221]: cluster 2024-04-06T03:45:27.123643+0000 osd.0 (osd.0) 576 : cluster 0 3.17 scrub ok 2024-04-06T03:45:29.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:29.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:29.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:29.652 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:29.661 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:29 smithi153 bash[23852]: cluster 2024-04-06T03:45:28.146069+0000 osd.0 (osd.0) 577 : cluster 0 6.0 scrub starts 2024-04-06T03:45:29.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:29 smithi153 bash[23852]: cluster 2024-04-06T03:45:28.216719+0000 osd.0 (osd.0) 578 : cluster 0 6.0 scrub ok 2024-04-06T03:45:29.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:29 smithi153 bash[23852]: cluster 2024-04-06T03:45:28.453815+0000 osd.4 (osd.4) 559 : cluster 0 5.11 scrub starts 2024-04-06T03:45:29.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:29 smithi153 bash[23852]: cluster 2024-04-06T03:45:28.454726+0000 osd.4 (osd.4) 560 : cluster 0 5.11 scrub ok 2024-04-06T03:45:29.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:29 smithi153 bash[23852]: cluster 2024-04-06T03:45:28.528751+0000 mgr.y (mgr.24368) 899 : cluster 0 pgmap v979: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 982 B/s rd, 29 MiB/s wr, 996 op/s 2024-04-06T03:45:29.675 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[21467]: cluster 2024-04-06T03:45:28.146069+0000 osd.0 (osd.0) 577 : cluster 0 6.0 scrub starts 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[21467]: cluster 2024-04-06T03:45:28.216719+0000 osd.0 (osd.0) 578 : cluster 0 6.0 scrub ok 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[21467]: cluster 2024-04-06T03:45:28.453815+0000 osd.4 (osd.4) 559 : cluster 0 5.11 scrub starts 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[21467]: cluster 2024-04-06T03:45:28.454726+0000 osd.4 (osd.4) 560 : cluster 0 5.11 scrub ok 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[21467]: cluster 2024-04-06T03:45:28.528751+0000 mgr.y (mgr.24368) 899 : cluster 0 pgmap v979: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 982 B/s rd, 29 MiB/s wr, 996 op/s 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:45:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:45:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[29221]: cluster 2024-04-06T03:45:28.146069+0000 osd.0 (osd.0) 577 : cluster 0 6.0 scrub starts 2024-04-06T03:45:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[29221]: cluster 2024-04-06T03:45:28.216719+0000 osd.0 (osd.0) 578 : cluster 0 6.0 scrub ok 2024-04-06T03:45:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[29221]: cluster 2024-04-06T03:45:28.453815+0000 osd.4 (osd.4) 559 : cluster 0 5.11 scrub starts 2024-04-06T03:45:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[29221]: cluster 2024-04-06T03:45:28.454726+0000 osd.4 (osd.4) 560 : cluster 0 5.11 scrub ok 2024-04-06T03:45:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:29 smithi050 bash[29221]: cluster 2024-04-06T03:45:28.528751+0000 mgr.y (mgr.24368) 899 : cluster 0 pgmap v979: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 982 B/s rd, 29 MiB/s wr, 996 op/s 2024-04-06T03:45:29.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:29.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:29.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:30.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:30.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:30.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:30.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:30.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:30.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:30.434 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:45:30.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:30.469 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:30.572 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:30.599 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:30.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.005961+0000 osd.3 (osd.3) 579 : cluster 0 6.6 scrub starts 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.035479+0000 osd.3 (osd.3) 580 : cluster 0 6.6 scrub ok 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.041885+0000 osd.2 (osd.2) 469 : cluster 0 2.10 scrub starts 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.043083+0000 osd.2 (osd.2) 470 : cluster 0 2.10 scrub ok 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.463716+0000 osd.4 (osd.4) 561 : cluster 0 2.11 scrub starts 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.469247+0000 osd.4 (osd.4) 562 : cluster 0 2.11 scrub ok 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:29.971969+0000 osd.3 (osd.3) 581 : cluster 0 6.b scrub starts 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:30.224638+0000 osd.7 (osd.7) 585 : cluster 0 2.8 scrub starts 2024-04-06T03:45:30.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:30 smithi153 bash[23852]: cluster 2024-04-06T03:45:30.228885+0000 osd.7 (osd.7) 586 : cluster 0 2.8 scrub ok 2024-04-06T03:45:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.005961+0000 osd.3 (osd.3) 579 : cluster 0 6.6 scrub starts 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.035479+0000 osd.3 (osd.3) 580 : cluster 0 6.6 scrub ok 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.041885+0000 osd.2 (osd.2) 469 : cluster 0 2.10 scrub starts 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.043083+0000 osd.2 (osd.2) 470 : cluster 0 2.10 scrub ok 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.463716+0000 osd.4 (osd.4) 561 : cluster 0 2.11 scrub starts 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.469247+0000 osd.4 (osd.4) 562 : cluster 0 2.11 scrub ok 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:29.971969+0000 osd.3 (osd.3) 581 : cluster 0 6.b scrub starts 2024-04-06T03:45:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:30.224638+0000 osd.7 (osd.7) 585 : cluster 0 2.8 scrub starts 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[21467]: cluster 2024-04-06T03:45:30.228885+0000 osd.7 (osd.7) 586 : cluster 0 2.8 scrub ok 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.005961+0000 osd.3 (osd.3) 579 : cluster 0 6.6 scrub starts 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.035479+0000 osd.3 (osd.3) 580 : cluster 0 6.6 scrub ok 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.041885+0000 osd.2 (osd.2) 469 : cluster 0 2.10 scrub starts 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.043083+0000 osd.2 (osd.2) 470 : cluster 0 2.10 scrub ok 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.463716+0000 osd.4 (osd.4) 561 : cluster 0 2.11 scrub starts 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.469247+0000 osd.4 (osd.4) 562 : cluster 0 2.11 scrub ok 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:29.971969+0000 osd.3 (osd.3) 581 : cluster 0 6.b scrub starts 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:30.224638+0000 osd.7 (osd.7) 585 : cluster 0 2.8 scrub starts 2024-04-06T03:45:30.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:30 smithi050 bash[29221]: cluster 2024-04-06T03:45:30.228885+0000 osd.7 (osd.7) 586 : cluster 0 2.8 scrub ok 2024-04-06T03:45:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:30.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:30.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:30.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:30.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:30.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:31.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:31.110 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:31.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:31.240 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:31.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.379 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:31.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:31.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:31.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:31.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:31.631 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:31.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:31.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:31 smithi153 bash[23852]: cluster 2024-04-06T03:45:30.067618+0000 osd.2 (osd.2) 471 : cluster 0 2.3 scrub starts 2024-04-06T03:45:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:31 smithi153 bash[23852]: cluster 2024-04-06T03:45:30.075877+0000 osd.2 (osd.2) 472 : cluster 0 2.3 scrub ok 2024-04-06T03:45:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:31 smithi153 bash[23852]: cluster 2024-04-06T03:45:30.126238+0000 osd.3 (osd.3) 582 : cluster 0 6.b scrub ok 2024-04-06T03:45:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:31 smithi153 bash[23852]: audit 2024-04-06T03:45:30.251885+0000 mgr.y (mgr.24368) 900 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:31 smithi153 bash[23852]: cluster 2024-04-06T03:45:30.529357+0000 mgr.y (mgr.24368) 901 : cluster 0 pgmap v980: 148 pgs: 148 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-06T03:45:31.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:31.892 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:31.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[21467]: cluster 2024-04-06T03:45:30.067618+0000 osd.2 (osd.2) 471 : cluster 0 2.3 scrub starts 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[21467]: cluster 2024-04-06T03:45:30.075877+0000 osd.2 (osd.2) 472 : cluster 0 2.3 scrub ok 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[21467]: cluster 2024-04-06T03:45:30.126238+0000 osd.3 (osd.3) 582 : cluster 0 6.b scrub ok 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[21467]: audit 2024-04-06T03:45:30.251885+0000 mgr.y (mgr.24368) 900 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[21467]: cluster 2024-04-06T03:45:30.529357+0000 mgr.y (mgr.24368) 901 : cluster 0 pgmap v980: 148 pgs: 148 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[29221]: cluster 2024-04-06T03:45:30.067618+0000 osd.2 (osd.2) 471 : cluster 0 2.3 scrub starts 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[29221]: cluster 2024-04-06T03:45:30.075877+0000 osd.2 (osd.2) 472 : cluster 0 2.3 scrub ok 2024-04-06T03:45:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[29221]: cluster 2024-04-06T03:45:30.126238+0000 osd.3 (osd.3) 582 : cluster 0 6.b scrub ok 2024-04-06T03:45:31.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[29221]: audit 2024-04-06T03:45:30.251885+0000 mgr.y (mgr.24368) 900 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:31.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:31 smithi050 bash[29221]: cluster 2024-04-06T03:45:30.529357+0000 mgr.y (mgr.24368) 901 : cluster 0 pgmap v980: 148 pgs: 148 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 25 MiB/s wr, 1.07k op/s 2024-04-06T03:45:31.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:32.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:32.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:32.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:32.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:32.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:32.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:32.414 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:32.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:32.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:32.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:32.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:32 smithi050 bash[29221]: cluster 2024-04-06T03:45:31.505157+0000 osd.4 (osd.4) 563 : cluster 0 2.18 scrub starts 2024-04-06T03:45:32.666 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:32 smithi050 bash[29221]: cluster 2024-04-06T03:45:31.513711+0000 osd.4 (osd.4) 564 : cluster 0 2.18 scrub ok 2024-04-06T03:45:32.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:32.783 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:32.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:32.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:32.938 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:32 smithi050 bash[21467]: cluster 2024-04-06T03:45:31.505157+0000 osd.4 (osd.4) 563 : cluster 0 2.18 scrub starts 2024-04-06T03:45:32.938 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:32 smithi050 bash[21467]: cluster 2024-04-06T03:45:31.513711+0000 osd.4 (osd.4) 564 : cluster 0 2.18 scrub ok 2024-04-06T03:45:32.958 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:32.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:33.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:32 smithi153 bash[23852]: cluster 2024-04-06T03:45:31.505157+0000 osd.4 (osd.4) 563 : cluster 0 2.18 scrub starts 2024-04-06T03:45:33.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:32 smithi153 bash[23852]: cluster 2024-04-06T03:45:31.513711+0000 osd.4 (osd.4) 564 : cluster 0 2.18 scrub ok 2024-04-06T03:45:33.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:33.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:33.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:33.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:33.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:33.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:33.481 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:33.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:33.616 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:33.662 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:45:33.662 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:45:33.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:33.730 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:33 smithi153 bash[23852]: cluster 2024-04-06T03:45:32.116323+0000 osd.0 (osd.0) 579 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:45:33.730 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:33 smithi153 bash[23852]: cluster 2024-04-06T03:45:32.117760+0000 osd.0 (osd.0) 580 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:45:33.730 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:33 smithi153 bash[23852]: cluster 2024-04-06T03:45:32.529953+0000 mgr.y (mgr.24368) 902 : cluster 0 pgmap v981: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 12 MiB/s wr, 884 op/s 2024-04-06T03:45:33.743 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:33.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:33.876 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:33.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:33.979 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:33 smithi050 bash[21467]: cluster 2024-04-06T03:45:32.116323+0000 osd.0 (osd.0) 579 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:45:34.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:33 smithi050 bash[21467]: cluster 2024-04-06T03:45:32.117760+0000 osd.0 (osd.0) 580 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:45:34.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:33 smithi050 bash[21467]: cluster 2024-04-06T03:45:32.529953+0000 mgr.y (mgr.24368) 902 : cluster 0 pgmap v981: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 12 MiB/s wr, 884 op/s 2024-04-06T03:45:34.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:33 smithi050 bash[29221]: cluster 2024-04-06T03:45:32.116323+0000 osd.0 (osd.0) 579 : cluster 0 5.7 deep-scrub starts 2024-04-06T03:45:34.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:33 smithi050 bash[29221]: cluster 2024-04-06T03:45:32.117760+0000 osd.0 (osd.0) 580 : cluster 0 5.7 deep-scrub ok 2024-04-06T03:45:34.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:33 smithi050 bash[29221]: cluster 2024-04-06T03:45:32.529953+0000 mgr.y (mgr.24368) 902 : cluster 0 pgmap v981: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 12 MiB/s wr, 884 op/s 2024-04-06T03:45:34.014 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:34.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.154 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:34.257 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:34.288 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:34.391 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.426 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:34.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.566 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:34.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:34.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:34.696 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:34.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:34.826 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:34.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-06T03:45:34.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:34.965 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:34.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:35.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:35.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:33.950796+0000 osd.3 (osd.3) 583 : cluster 0 3.2 scrub starts 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:33.956925+0000 osd.3 (osd.3) 584 : cluster 0 3.2 scrub ok 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.004661+0000 osd.2 (osd.2) 473 : cluster 0 5.b scrub starts 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.011278+0000 osd.2 (osd.2) 474 : cluster 0 5.b scrub ok 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.162928+0000 osd.7 (osd.7) 587 : cluster 0 2.5 scrub starts 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.165074+0000 osd.7 (osd.7) 588 : cluster 0 2.5 scrub ok 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.530549+0000 mgr.y (mgr.24368) 903 : cluster 0 pgmap v982: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:33.950796+0000 osd.3 (osd.3) 583 : cluster 0 3.2 scrub starts 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:33.956925+0000 osd.3 (osd.3) 584 : cluster 0 3.2 scrub ok 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.004661+0000 osd.2 (osd.2) 473 : cluster 0 5.b scrub starts 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.011278+0000 osd.2 (osd.2) 474 : cluster 0 5.b scrub ok 2024-04-06T03:45:35.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.162928+0000 osd.7 (osd.7) 587 : cluster 0 2.5 scrub starts 2024-04-06T03:45:35.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.165074+0000 osd.7 (osd.7) 588 : cluster 0 2.5 scrub ok 2024-04-06T03:45:35.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:35 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.530549+0000 mgr.y (mgr.24368) 903 : cluster 0 pgmap v982: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-06T03:45:35.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:35.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:35.215 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:33.950796+0000 osd.3 (osd.3) 583 : cluster 0 3.2 scrub starts 2024-04-06T03:45:35.215 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:33.956925+0000 osd.3 (osd.3) 584 : cluster 0 3.2 scrub ok 2024-04-06T03:45:35.215 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.004661+0000 osd.2 (osd.2) 473 : cluster 0 5.b scrub starts 2024-04-06T03:45:35.215 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.011278+0000 osd.2 (osd.2) 474 : cluster 0 5.b scrub ok 2024-04-06T03:45:35.215 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.162928+0000 osd.7 (osd.7) 587 : cluster 0 2.5 scrub starts 2024-04-06T03:45:35.215 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.165074+0000 osd.7 (osd.7) 588 : cluster 0 2.5 scrub ok 2024-04-06T03:45:35.216 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:35 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.530549+0000 mgr.y (mgr.24368) 903 : cluster 0 pgmap v982: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-06T03:45:35.228 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:35.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:35.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:35.467 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:35.505 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:35.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:35.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:35.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:35.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:35.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:35.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:35.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:36.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:36.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:36.155 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.132043+0000 osd.0 (osd.0) 581 : cluster 0 2.13 scrub starts 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[21467]: cluster 2024-04-06T03:45:34.135470+0000 osd.0 (osd.0) 582 : cluster 0 2.13 scrub ok 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[21467]: cluster 2024-04-06T03:45:35.158619+0000 osd.0 (osd.0) 583 : cluster 0 3.10 scrub starts 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[21467]: cluster 2024-04-06T03:45:35.162105+0000 osd.0 (osd.0) 584 : cluster 0 3.10 scrub ok 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[21467]: audit 2024-04-06T03:45:35.500109+0000 mon.a (mon.0) 1719 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.132043+0000 osd.0 (osd.0) 581 : cluster 0 2.13 scrub starts 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[29221]: cluster 2024-04-06T03:45:34.135470+0000 osd.0 (osd.0) 582 : cluster 0 2.13 scrub ok 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[29221]: cluster 2024-04-06T03:45:35.158619+0000 osd.0 (osd.0) 583 : cluster 0 3.10 scrub starts 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[29221]: cluster 2024-04-06T03:45:35.162105+0000 osd.0 (osd.0) 584 : cluster 0 3.10 scrub ok 2024-04-06T03:45:36.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:36 smithi050 bash[29221]: audit 2024-04-06T03:45:35.500109+0000 mon.a (mon.0) 1719 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:36.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:36 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.132043+0000 osd.0 (osd.0) 581 : cluster 0 2.13 scrub starts 2024-04-06T03:45:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:36 smithi153 bash[23852]: cluster 2024-04-06T03:45:34.135470+0000 osd.0 (osd.0) 582 : cluster 0 2.13 scrub ok 2024-04-06T03:45:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:36 smithi153 bash[23852]: cluster 2024-04-06T03:45:35.158619+0000 osd.0 (osd.0) 583 : cluster 0 3.10 scrub starts 2024-04-06T03:45:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:36 smithi153 bash[23852]: cluster 2024-04-06T03:45:35.162105+0000 osd.0 (osd.0) 584 : cluster 0 3.10 scrub ok 2024-04-06T03:45:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:36 smithi153 bash[23852]: audit 2024-04-06T03:45:35.500109+0000 mon.a (mon.0) 1719 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:36.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:36.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:36.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:36.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:36.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:36.808 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:36.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:36.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:37.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:35.464158+0000 osd.4 (osd.4) 565 : cluster 0 5.f scrub starts 2024-04-06T03:45:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:35.465464+0000 osd.4 (osd.4) 566 : cluster 0 5.f scrub ok 2024-04-06T03:45:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:35.930748+0000 osd.3 (osd.3) 585 : cluster 0 2.12 scrub starts 2024-04-06T03:45:37.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:35.931716+0000 osd.3 (osd.3) 586 : cluster 0 2.12 scrub ok 2024-04-06T03:45:37.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.531185+0000 mgr.y (mgr.24368) 904 : cluster 0 pgmap v983: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 938 B/s rd, 4.7 MiB/s wr, 1.12k op/s 2024-04-06T03:45:37.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.913981+0000 osd.3 (osd.3) 587 : cluster 0 3.7 scrub starts 2024-04-06T03:45:37.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:37 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.917314+0000 osd.3 (osd.3) 588 : cluster 0 3.7 scrub ok 2024-04-06T03:45:37.072 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:37.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:37.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:37.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:37.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:35.464158+0000 osd.4 (osd.4) 565 : cluster 0 5.f scrub starts 2024-04-06T03:45:37.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:35.465464+0000 osd.4 (osd.4) 566 : cluster 0 5.f scrub ok 2024-04-06T03:45:37.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:35.930748+0000 osd.3 (osd.3) 585 : cluster 0 2.12 scrub starts 2024-04-06T03:45:37.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:35.931716+0000 osd.3 (osd.3) 586 : cluster 0 2.12 scrub ok 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.531185+0000 mgr.y (mgr.24368) 904 : cluster 0 pgmap v983: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 938 B/s rd, 4.7 MiB/s wr, 1.12k op/s 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.913981+0000 osd.3 (osd.3) 587 : cluster 0 3.7 scrub starts 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.917314+0000 osd.3 (osd.3) 588 : cluster 0 3.7 scrub ok 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:35.464158+0000 osd.4 (osd.4) 565 : cluster 0 5.f scrub starts 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:35.465464+0000 osd.4 (osd.4) 566 : cluster 0 5.f scrub ok 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:35.930748+0000 osd.3 (osd.3) 585 : cluster 0 2.12 scrub starts 2024-04-06T03:45:37.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:35.931716+0000 osd.3 (osd.3) 586 : cluster 0 2.12 scrub ok 2024-04-06T03:45:37.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.531185+0000 mgr.y (mgr.24368) 904 : cluster 0 pgmap v983: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 938 B/s rd, 4.7 MiB/s wr, 1.12k op/s 2024-04-06T03:45:37.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.913981+0000 osd.3 (osd.3) 587 : cluster 0 3.7 scrub starts 2024-04-06T03:45:37.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:37 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.917314+0000 osd.3 (osd.3) 588 : cluster 0 3.7 scrub ok 2024-04-06T03:45:37.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:37.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:37.483 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:37.589 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:37.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:37.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:37.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:37.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:37.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:37.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:38.002 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:38.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:38.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.495172+0000 osd.4 (osd.4) 567 : cluster 0 5.1f scrub starts 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.499439+0000 osd.4 (osd.4) 568 : cluster 0 5.1f scrub ok 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.975633+0000 osd.6 (osd.6) 109 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:36.975701+0000 osd.6 (osd.6) 110 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.023237+0000 mon.a (mon.0) 1720 : cluster 0 osdmap e365: 8 total, 8 up, 7 in 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.139360+0000 osd.7 (osd.7) 589 : cluster 0 5.19 deep-scrub starts 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.144300+0000 osd.7 (osd.7) 590 : cluster 0 5.19 deep-scrub ok 2024-04-06T03:45:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.202934+0000 osd.0 (osd.0) 585 : cluster 0 3.1b scrub starts 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.208669+0000 osd.0 (osd.0) 586 : cluster 0 3.1b scrub ok 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.495172+0000 osd.4 (osd.4) 567 : cluster 0 5.1f scrub starts 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.499439+0000 osd.4 (osd.4) 568 : cluster 0 5.1f scrub ok 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.975633+0000 osd.6 (osd.6) 109 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:36.975701+0000 osd.6 (osd.6) 110 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.023237+0000 mon.a (mon.0) 1720 : cluster 0 osdmap e365: 8 total, 8 up, 7 in 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.139360+0000 osd.7 (osd.7) 589 : cluster 0 5.19 deep-scrub starts 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.144300+0000 osd.7 (osd.7) 590 : cluster 0 5.19 deep-scrub ok 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.202934+0000 osd.0 (osd.0) 585 : cluster 0 3.1b scrub starts 2024-04-06T03:45:38.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:38 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.208669+0000 osd.0 (osd.0) 586 : cluster 0 3.1b scrub ok 2024-04-06T03:45:38.283 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:38.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.495172+0000 osd.4 (osd.4) 567 : cluster 0 5.1f scrub starts 2024-04-06T03:45:38.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.499439+0000 osd.4 (osd.4) 568 : cluster 0 5.1f scrub ok 2024-04-06T03:45:38.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.975633+0000 osd.6 (osd.6) 109 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:38.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:36.975701+0000 osd.6 (osd.6) 110 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:38.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.023237+0000 mon.a (mon.0) 1720 : cluster 0 osdmap e365: 8 total, 8 up, 7 in 2024-04-06T03:45:38.287 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.139360+0000 osd.7 (osd.7) 589 : cluster 0 5.19 deep-scrub starts 2024-04-06T03:45:38.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.144300+0000 osd.7 (osd.7) 590 : cluster 0 5.19 deep-scrub ok 2024-04-06T03:45:38.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.202934+0000 osd.0 (osd.0) 585 : cluster 0 3.1b scrub starts 2024-04-06T03:45:38.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:38 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.208669+0000 osd.0 (osd.0) 586 : cluster 0 3.1b scrub ok 2024-04-06T03:45:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:38.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:38.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:38.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:38.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:38.691 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:38.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:38.819 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:38.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:38.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:38.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:39.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:39.081 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:39.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:39.203 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.473694+0000 osd.4 (osd.4) 569 : cluster 0 6.e scrub starts 2024-04-06T03:45:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[21467]: cluster 2024-04-06T03:45:37.525978+0000 osd.4 (osd.4) 570 : cluster 0 6.e scrub ok 2024-04-06T03:45:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.158019+0000 osd.7 (osd.7) 591 : cluster 0 3.e scrub starts 2024-04-06T03:45:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.163671+0000 osd.7 (osd.7) 592 : cluster 0 3.e scrub ok 2024-04-06T03:45:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.531996+0000 mgr.y (mgr.24368) 905 : cluster 0 pgmap v985: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:45:39.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.473694+0000 osd.4 (osd.4) 569 : cluster 0 6.e scrub starts 2024-04-06T03:45:39.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[29221]: cluster 2024-04-06T03:45:37.525978+0000 osd.4 (osd.4) 570 : cluster 0 6.e scrub ok 2024-04-06T03:45:39.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.158019+0000 osd.7 (osd.7) 591 : cluster 0 3.e scrub starts 2024-04-06T03:45:39.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.163671+0000 osd.7 (osd.7) 592 : cluster 0 3.e scrub ok 2024-04-06T03:45:39.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.531996+0000 mgr.y (mgr.24368) 905 : cluster 0 pgmap v985: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:45:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:39 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.473694+0000 osd.4 (osd.4) 569 : cluster 0 6.e scrub starts 2024-04-06T03:45:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:39 smithi153 bash[23852]: cluster 2024-04-06T03:45:37.525978+0000 osd.4 (osd.4) 570 : cluster 0 6.e scrub ok 2024-04-06T03:45:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:39 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.158019+0000 osd.7 (osd.7) 591 : cluster 0 3.e scrub starts 2024-04-06T03:45:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:39 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.163671+0000 osd.7 (osd.7) 592 : cluster 0 3.e scrub ok 2024-04-06T03:45:39.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:39 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.531996+0000 mgr.y (mgr.24368) 905 : cluster 0 pgmap v985: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:45:39.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:39.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:39.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:39.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:39.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:39.605 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:39.669 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:45:39.698 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:45:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:45:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:45:39.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:39.732 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:39.839 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:39.865 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:39.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:39.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:40.000 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:40.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:40.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.320076+0000 osd.5 (osd.5) 151 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:40.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.320141+0000 osd.5 (osd.5) 152 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:40.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.467652+0000 osd.4 (osd.4) 571 : cluster 0 6.1a scrub starts 2024-04-06T03:45:40.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[23852]: cluster 2024-04-06T03:45:38.544482+0000 osd.4 (osd.4) 572 : cluster 0 6.1a scrub ok 2024-04-06T03:45:40.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[23852]: cluster 2024-04-06T03:45:39.035806+0000 mon.a (mon.0) 1721 : cluster 0 osdmap e366: 8 total, 8 up, 7 in 2024-04-06T03:45:40.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[23852]: cluster 2024-04-06T03:45:39.158524+0000 osd.0 (osd.0) 587 : cluster 0 11.6 scrub starts 2024-04-06T03:45:40.130 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:40.234 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:40.265 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:40.298 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:45:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:45:40.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:40.398 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:40.430 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.320076+0000 osd.5 (osd.5) 151 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:40.430 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.320141+0000 osd.5 (osd.5) 152 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:40.430 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.467652+0000 osd.4 (osd.4) 571 : cluster 0 6.1a scrub starts 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[21467]: cluster 2024-04-06T03:45:38.544482+0000 osd.4 (osd.4) 572 : cluster 0 6.1a scrub ok 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[21467]: cluster 2024-04-06T03:45:39.035806+0000 mon.a (mon.0) 1721 : cluster 0 osdmap e366: 8 total, 8 up, 7 in 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[21467]: cluster 2024-04-06T03:45:39.158524+0000 osd.0 (osd.0) 587 : cluster 0 11.6 scrub starts 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.320076+0000 osd.5 (osd.5) 151 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.320141+0000 osd.5 (osd.5) 152 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.467652+0000 osd.4 (osd.4) 571 : cluster 0 6.1a scrub starts 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[29221]: cluster 2024-04-06T03:45:38.544482+0000 osd.4 (osd.4) 572 : cluster 0 6.1a scrub ok 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[29221]: cluster 2024-04-06T03:45:39.035806+0000 mon.a (mon.0) 1721 : cluster 0 osdmap e366: 8 total, 8 up, 7 in 2024-04-06T03:45:40.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:40 smithi050 bash[29221]: cluster 2024-04-06T03:45:39.158524+0000 osd.0 (osd.0) 587 : cluster 0 11.6 scrub starts 2024-04-06T03:45:40.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:40.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:40.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:40.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:40.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:40.779 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:40.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:40.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:40.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:41.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:41.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:41.071 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-06T03:45:41.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:41.172 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:41.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:41.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:41.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:41.413 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: cluster 2024-04-06T03:45:39.288306+0000 osd.1 (osd.1) 399 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:41.413 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: cluster 2024-04-06T03:45:39.288386+0000 osd.1 (osd.1) 400 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:41.413 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: cluster 2024-04-06T03:45:40.042690+0000 mon.a (mon.0) 1722 : cluster 0 osdmap e367: 8 total, 8 up, 7 in 2024-04-06T03:45:41.413 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: audit 2024-04-06T03:45:40.259845+0000 mgr.y (mgr.24368) 906 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:41.413 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: cluster 2024-04-06T03:45:40.532484+0000 mgr.y (mgr.24368) 907 : cluster 0 pgmap v988: 148 pgs: 148 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.43k op/s 2024-04-06T03:45:41.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: audit 2024-04-06T03:45:40.976004+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.50:0/2014965167' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:45:41.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: cluster 2024-04-06T03:45:40.993947+0000 osd.3 (osd.3) 589 : cluster 0 5.14 scrub starts 2024-04-06T03:45:41.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:41 smithi153 bash[23852]: cluster 2024-04-06T03:45:41.030583+0000 osd.3 (osd.3) 590 : cluster 0 5.14 scrub ok 2024-04-06T03:45:41.428 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: cluster 2024-04-06T03:45:39.288306+0000 osd.1 (osd.1) 399 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: cluster 2024-04-06T03:45:39.288386+0000 osd.1 (osd.1) 400 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: cluster 2024-04-06T03:45:40.042690+0000 mon.a (mon.0) 1722 : cluster 0 osdmap e367: 8 total, 8 up, 7 in 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: audit 2024-04-06T03:45:40.259845+0000 mgr.y (mgr.24368) 906 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: cluster 2024-04-06T03:45:40.532484+0000 mgr.y (mgr.24368) 907 : cluster 0 pgmap v988: 148 pgs: 148 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.43k op/s 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: audit 2024-04-06T03:45:40.976004+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.50:0/2014965167' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: cluster 2024-04-06T03:45:40.993947+0000 osd.3 (osd.3) 589 : cluster 0 5.14 scrub starts 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[21467]: cluster 2024-04-06T03:45:41.030583+0000 osd.3 (osd.3) 590 : cluster 0 5.14 scrub ok 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: cluster 2024-04-06T03:45:39.288306+0000 osd.1 (osd.1) 399 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: cluster 2024-04-06T03:45:39.288386+0000 osd.1 (osd.1) 400 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:41.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: cluster 2024-04-06T03:45:40.042690+0000 mon.a (mon.0) 1722 : cluster 0 osdmap e367: 8 total, 8 up, 7 in 2024-04-06T03:45:41.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: audit 2024-04-06T03:45:40.259845+0000 mgr.y (mgr.24368) 906 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:41.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: cluster 2024-04-06T03:45:40.532484+0000 mgr.y (mgr.24368) 907 : cluster 0 pgmap v988: 148 pgs: 148 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.43k op/s 2024-04-06T03:45:41.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: audit 2024-04-06T03:45:40.976004+0000 mon.a (mon.0) 1723 : audit 1 from='client.? 172.21.15.50:0/2014965167' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:45:41.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: cluster 2024-04-06T03:45:40.993947+0000 osd.3 (osd.3) 589 : cluster 0 5.14 scrub starts 2024-04-06T03:45:41.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:41 smithi050 bash[29221]: cluster 2024-04-06T03:45:41.030583+0000 osd.3 (osd.3) 590 : cluster 0 5.14 scrub ok 2024-04-06T03:45:41.526 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:45:41.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:41.557 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:45:41.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:41.603 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:45:41.648 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:45:41.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:41.687 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:41.702 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:45:41.735 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:45:41.770 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:45:41.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:41.794 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:45:41.795 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:45:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:45:41.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:41.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:41.949 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:41.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:42.054 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:45:42.099 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:42.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: audit 2024-04-06T03:45:41.056604+0000 mon.a (mon.0) 1724 : audit 1 from='client.? 172.21.15.50:0/2014965167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: cluster 2024-04-06T03:45:41.061207+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e368: 8 total, 8 up, 7 in 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: cluster 2024-04-06T03:45:41.170731+0000 osd.0 (osd.0) 588 : cluster 0 11.6 scrub ok 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: audit 2024-04-06T03:45:41.498001+0000 mon.c (mon.2) 110 : audit 0 from='client.? 172.21.15.50:0/1115332611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: audit 2024-04-06T03:45:41.834208+0000 mon.a (mon.0) 1726 : audit 1 from='client.? 172.21.15.50:0/2679033508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: cluster 2024-04-06T03:45:42.010887+0000 osd.3 (osd.3) 591 : cluster 0 6.c scrub starts 2024-04-06T03:45:42.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:42 smithi153 bash[23852]: cluster 2024-04-06T03:45:42.012034+0000 osd.3 (osd.3) 592 : cluster 0 6.c scrub ok 2024-04-06T03:45:42.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:42.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: audit 2024-04-06T03:45:41.056604+0000 mon.a (mon.0) 1724 : audit 1 from='client.? 172.21.15.50:0/2014965167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:45:42.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: cluster 2024-04-06T03:45:41.061207+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e368: 8 total, 8 up, 7 in 2024-04-06T03:45:42.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: cluster 2024-04-06T03:45:41.170731+0000 osd.0 (osd.0) 588 : cluster 0 11.6 scrub ok 2024-04-06T03:45:42.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: audit 2024-04-06T03:45:41.498001+0000 mon.c (mon.2) 110 : audit 0 from='client.? 172.21.15.50:0/1115332611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: audit 2024-04-06T03:45:41.834208+0000 mon.a (mon.0) 1726 : audit 1 from='client.? 172.21.15.50:0/2679033508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: cluster 2024-04-06T03:45:42.010887+0000 osd.3 (osd.3) 591 : cluster 0 6.c scrub starts 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[21467]: cluster 2024-04-06T03:45:42.012034+0000 osd.3 (osd.3) 592 : cluster 0 6.c scrub ok 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: audit 2024-04-06T03:45:41.056604+0000 mon.a (mon.0) 1724 : audit 1 from='client.? 172.21.15.50:0/2014965167' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: cluster 2024-04-06T03:45:41.061207+0000 mon.a (mon.0) 1725 : cluster 0 osdmap e368: 8 total, 8 up, 7 in 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: cluster 2024-04-06T03:45:41.170731+0000 osd.0 (osd.0) 588 : cluster 0 11.6 scrub ok 2024-04-06T03:45:42.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: audit 2024-04-06T03:45:41.498001+0000 mon.c (mon.2) 110 : audit 0 from='client.? 172.21.15.50:0/1115332611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:45:42.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: audit 2024-04-06T03:45:41.834208+0000 mon.a (mon.0) 1726 : audit 1 from='client.? 172.21.15.50:0/2679033508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:45:42.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: cluster 2024-04-06T03:45:42.010887+0000 osd.3 (osd.3) 591 : cluster 0 6.c scrub starts 2024-04-06T03:45:42.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:42 smithi050 bash[29221]: cluster 2024-04-06T03:45:42.012034+0000 osd.3 (osd.3) 592 : cluster 0 6.c scrub ok 2024-04-06T03:45:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:42.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:42.497 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:42.601 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:42.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:42.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:42.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:42.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:42.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:42.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:42.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:43.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:43.129 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:43.161 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:43.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[29221]: cluster 2024-04-06T03:45:41.424260+0000 osd.4 (osd.4) 573 : cluster 0 6.2 scrub starts 2024-04-06T03:45:43.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[29221]: cluster 2024-04-06T03:45:41.425541+0000 osd.4 (osd.4) 574 : cluster 0 6.2 scrub ok 2024-04-06T03:45:43.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[29221]: cluster 2024-04-06T03:45:42.056866+0000 mon.a (mon.0) 1727 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:45:43.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[29221]: audit 2024-04-06T03:45:42.060598+0000 mon.a (mon.0) 1728 : audit 1 from='client.? 172.21.15.50:0/2679033508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:45:43.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[29221]: cluster 2024-04-06T03:45:42.064098+0000 mon.a (mon.0) 1729 : cluster 0 osdmap e369: 8 total, 8 up, 7 in 2024-04-06T03:45:43.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[29221]: cluster 2024-04-06T03:45:42.533033+0000 mgr.y (mgr.24368) 908 : cluster 0 pgmap v991: 148 pgs: 148 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-04-06T03:45:43.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[21467]: cluster 2024-04-06T03:45:41.424260+0000 osd.4 (osd.4) 573 : cluster 0 6.2 scrub starts 2024-04-06T03:45:43.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[21467]: cluster 2024-04-06T03:45:41.425541+0000 osd.4 (osd.4) 574 : cluster 0 6.2 scrub ok 2024-04-06T03:45:43.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[21467]: cluster 2024-04-06T03:45:42.056866+0000 mon.a (mon.0) 1727 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:45:43.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[21467]: audit 2024-04-06T03:45:42.060598+0000 mon.a (mon.0) 1728 : audit 1 from='client.? 172.21.15.50:0/2679033508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:45:43.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[21467]: cluster 2024-04-06T03:45:42.064098+0000 mon.a (mon.0) 1729 : cluster 0 osdmap e369: 8 total, 8 up, 7 in 2024-04-06T03:45:43.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:43 smithi050 bash[21467]: cluster 2024-04-06T03:45:42.533033+0000 mgr.y (mgr.24368) 908 : cluster 0 pgmap v991: 148 pgs: 148 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-04-06T03:45:43.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:43.288 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:43.330 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:43 smithi153 bash[23852]: cluster 2024-04-06T03:45:41.424260+0000 osd.4 (osd.4) 573 : cluster 0 6.2 scrub starts 2024-04-06T03:45:43.330 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:43 smithi153 bash[23852]: cluster 2024-04-06T03:45:41.425541+0000 osd.4 (osd.4) 574 : cluster 0 6.2 scrub ok 2024-04-06T03:45:43.330 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:43 smithi153 bash[23852]: cluster 2024-04-06T03:45:42.056866+0000 mon.a (mon.0) 1727 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:45:43.330 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:43 smithi153 bash[23852]: audit 2024-04-06T03:45:42.060598+0000 mon.a (mon.0) 1728 : audit 1 from='client.? 172.21.15.50:0/2679033508' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:45:43.331 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:43 smithi153 bash[23852]: cluster 2024-04-06T03:45:42.064098+0000 mon.a (mon.0) 1729 : cluster 0 osdmap e369: 8 total, 8 up, 7 in 2024-04-06T03:45:43.331 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:43 smithi153 bash[23852]: cluster 2024-04-06T03:45:42.533033+0000 mgr.y (mgr.24368) 908 : cluster 0 pgmap v991: 148 pgs: 148 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 778 op/s 2024-04-06T03:45:43.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:43.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:43.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:43.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:43.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:43.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:43.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:43.822 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:43.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:43.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:43.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:44.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:44.076 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:44.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:44.205 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:44.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:44.332 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:44.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:44.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:44.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:44.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:44.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:44.713 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:44.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:44.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:44.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:44.982 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:44.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:45.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:45.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:45 smithi050 bash[21467]: cluster 2024-04-06T03:45:44.533625+0000 mgr.y (mgr.24368) 909 : cluster 0 pgmap v992: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 930 B/s rd, 0 B/s wr, 858 op/s 2024-04-06T03:45:45.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:45 smithi050 bash[29221]: cluster 2024-04-06T03:45:44.533625+0000 mgr.y (mgr.24368) 909 : cluster 0 pgmap v992: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 930 B/s rd, 0 B/s wr, 858 op/s 2024-04-06T03:45:45.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:45.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:45.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:45.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:44 smithi153 bash[23852]: cluster 2024-04-06T03:45:44.533625+0000 mgr.y (mgr.24368) 909 : cluster 0 pgmap v992: 148 pgs: 148 active+clean; 5.2 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 930 B/s rd, 0 B/s wr, 858 op/s 2024-04-06T03:45:45.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:45.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:45.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:45.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:45.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.635 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:45.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:45.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:45.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:45.868 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:45.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:45.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:46.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:46.040 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:46.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:46.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:46.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 6 1 2024-04-06T03:45:46.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:126610 9.7 e260/231 9:e82ff979:::benchmark_data_smithi050_105098_object29700:head v 260'7771, mlcod=260'7767)", 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:52.144462+0000", 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 534.20776548000003, 2024-04-06T03:45:46.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045288175, 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144462+0000", 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144462+0000", 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144463+0000", 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6300000000000005e-07 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144468+0000", 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7339999999999996e-06 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144478+0000", 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8400000000000007e-06 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144486+0000", 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.272e-06 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144536+0000", 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0339e-05 2024-04-06T03:45:46.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.144556+0000", 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9763e-05 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.189665+0000", 2024-04-06T03:45:46.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045109690000000001 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.189750+0000", 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4874000000000001e-05 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.361 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:126609 9.b e260/231 9:d63f7ecc:::benchmark_data_smithi050_105098_object29699:head v 260'7942, mlcod=260'7940)", 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 534.20627273399998, 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.047109446999999999, 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.362 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145955+0000", 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.06e-07 2024-04-06T03:45:46.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145960+0000", 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9479999999999999e-06 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145968+0000", 2024-04-06T03:45:46.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.028e-06 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.145973+0000", 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0000000000000004e-06 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184892+0000", 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.038918992999999999 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184922+0000", 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9802999999999999e-05 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.192971+0000", 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0080485069999999999 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.193064+0000", 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3561999999999996e-05 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:126611 9.5 e260/231 9:a229340f:::benchmark_data_smithi050_105098_object29701:head v 260'8046, mlcod=260'8044)", 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 534.20575177700005, 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.047985633999999999, 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.369 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146476+0000", 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3200000000000005e-07 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.146482+0000", 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.434e-06 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184884+0000", 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.038401804999999997 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184897+0000", 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3607e-05 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.184969+0000", 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1616e-05 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.185238+0000", 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026927299999999999 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.192198+0000", 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006959659 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:36:52.194461+0000", 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0022636079999999999 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158006 9.b e266/231 9:d95ff99c:::benchmark_data_smithi050_105098_object61096:head v 266'9938, mlcod=266'9935)", 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.269568+0000", 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 500.08265956899999, 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38518448100000002, 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.374 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269568+0000", 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269568+0000", 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269569+0000", 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5099999999999999e-07 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269576+0000", 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3690000000000001e-06 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269585+0000", 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2380000000000003e-06 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269592+0000", 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9299999999999997e-06 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269645+0000", 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2774e-05 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.269666+0000", 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0922999999999999e-05 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654688+0000", 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.385021944 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654752+0000", 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4652000000000004e-05 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158007 9.b e266/231 9:d021809f:::benchmark_data_smithi050_105098_object61097:head v 266'9939, mlcod=266'9936)", 2024-04-06T03:45:46.378 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.270463+0000", 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 500.08176428000002, 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.39470748900000002, 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270463+0000", 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270463+0000", 2024-04-06T03:45:46.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270464+0000", 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.37e-07 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270469+0000", 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8409999999999999e-06 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270478+0000", 2024-04-06T03:45:46.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6390000000000004e-06 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270485+0000", 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8369999999999999e-06 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654855+0000", 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38436940200000003 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654898+0000", 2024-04-06T03:45:46.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3735000000000001e-05 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.665106+0000", 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010207941999999999 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.665171+0000", 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4455999999999994e-05 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158010 9.7 e266/231 9:eb52c8f4:::benchmark_data_smithi050_105098_object61100:head v 266'9750, mlcod=266'9748)", 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.270494+0000", 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 500.08173365599998, 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38428609699999999, 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270494+0000", 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270494+0000", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270495+0000", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3200000000000004e-07 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270502+0000", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0119999999999999e-06 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270512+0000", 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0539e-05 2024-04-06T03:45:46.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270520+0000", 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8769999999999993e-06 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270566+0000", 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5643999999999999e-05 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270584+0000", 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8638999999999999e-05 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654721+0000", 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.384136431 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654780+0000", 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9123000000000003e-05 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158012 9.9 e266/231 9:9f35b491:::benchmark_data_smithi050_105098_object61102:head v 266'9951, mlcod=266'9949)", 2024-04-06T03:45:46.386 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.270818+0000", 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 500.08140922500002, 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.39316358299999998, 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270818+0000", 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270818+0000", 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270819+0000", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2399999999999998e-07 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270826+0000", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3390000000000004e-06 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270836+0000", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0135e-05 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.270843+0000", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5509999999999996e-06 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654797+0000", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.383953716 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654874+0000", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7322000000000003e-05 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.663935+0000", 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0090613699999999991 2024-04-06T03:45:46.389 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.663982+0000", 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6526000000000001e-05 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158016 9.a e266/231 9:5a69f0d8:::benchmark_data_smithi050_105098_object61106:head v 266'9932, mlcod=266'9929)", 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.272633+0000", 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 500.07959492800001, 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.396393147, 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.390 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272633+0000", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272633+0000", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272634+0000", 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5499999999999996e-07 2024-04-06T03:45:46.391 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.272640+0000", 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8689999999999998e-06 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654752+0000", 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.382111068 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654773+0000", 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1979000000000001e-05 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.392 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.655063+0000", 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00028948599999999999 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.655122+0000", 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8545999999999998e-05 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.668903+0000", 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.013781513 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.393 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.669026+0000", 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000122731 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15135.0:158023 9.8 e266/231 9:12f66281:::benchmark_data_smithi050_105098_object61113:head v 266'9691, mlcod=266'9689)", 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:37:26.283140+0000", 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 500.06908757600002, 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.38472973399999999, 2024-04-06T03:45:46.394 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283140+0000", 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283140+0000", 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.395 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283141+0000", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1969999999999999e-06 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.283150+0000", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3149999999999998e-06 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654662+0000", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.37151177299999999 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654694+0000", 2024-04-06T03:45:46.396 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2109e-05 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654862+0000", 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000167388 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.654891+0000", 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9600000000000001e-05 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.667814+0000", 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012922849 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:37:26.667870+0000", 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5503e-05 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:121281 10.7 e307/280 10:e86db8b7:::benchmark_data_smithi050_121293_object22464:head v 307'7621, mlcod=307'7619)", 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:01.238145+0000", 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.114082139, 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.059678084999999999, 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.398 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238145+0000", 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238145+0000", 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238146+0000", 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7300000000000005e-07 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.399 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238158+0000", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1812e-05 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238170+0000", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2108e-05 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238179+0000", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9500000000000007e-06 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238254+0000", 2024-04-06T03:45:46.400 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4880000000000001e-05 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238274+0000", 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9653e-05 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297782+0000", 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.059508013999999998 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297823+0000", 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1795000000000002e-05 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.401 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:121285 10.7 e307/280 10:e18fdbd2:::benchmark_data_smithi050_121293_object22468:head v 307'7622, mlcod=307'7619)", 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.10796997, 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053663272999999997, 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:45:46.402 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9199999999999999e-07 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244268+0000", 2024-04-06T03:45:46.403 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7389999999999994e-06 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244281+0000", 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3088000000000001e-05 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244289+0000", 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9149999999999996e-06 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244345+0000", 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5809e-05 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244365+0000", 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0466e-05 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297828+0000", 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053462618000000003 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297921+0000", 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2745999999999996e-05 2024-04-06T03:45:46.405 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:121288 10.7 e307/280 10:e6b948e9:::benchmark_data_smithi050_121293_object22471:head v 307'7623, mlcod=307'7619)", 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:01.247618+0000", 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.10460919299999, 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.050333576999999997, 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.406 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247618+0000", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247618+0000", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247619+0000", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5499999999999997e-07 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247629+0000", 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0178e-05 2024-04-06T03:45:46.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247643+0000", 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3675999999999999e-05 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247651+0000", 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2090000000000005e-06 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.408 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247731+0000", 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9338999999999994e-05 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247753+0000", 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2084000000000001e-05 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297923+0000", 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.050170807999999997 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297952+0000", 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8428e-05 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167168 10.a e323/322 10:5d6f7159:::benchmark_data_smithi050_121293_object68351:head v 323'10421, mlcod=0'0)", 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.603535+0000", 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 222.74869247199999, 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.056436527, 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.410 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603535+0000", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603535+0000", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603537+0000", 2024-04-06T03:45:46.411 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7400000000000001e-06 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603550+0000", 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2877000000000001e-05 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603576+0000", 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6196000000000001e-05 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.412 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603591+0000", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5123000000000001e-05 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605536+0000", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001944999 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605585+0000", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9082999999999997e-05 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.659866+0000", 2024-04-06T03:45:46.413 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.054280796999999999 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.659972+0000", 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010571199999999999 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167177 10.a e323/322 10:53f21d76:::benchmark_data_smithi050_121293_object68360:head v 323'10422, mlcod=0'0)", 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.605736+0000", 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 222.74649115099999, 2024-04-06T03:45:46.414 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.054311015999999997, 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605736+0000", 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605736+0000", 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.415 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605738+0000", 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7439999999999999e-06 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605753+0000", 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5233e-05 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605784+0000", 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0306000000000001e-05 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.416 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605805+0000", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1416e-05 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.609992+0000", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0041874030000000001 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.610042+0000", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9784999999999999e-05 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.417 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.659984+0000", 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.049941343999999999 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660047+0000", 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3785e-05 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167203 10.a e323/322 10:5cd24c27:::benchmark_data_smithi050_121293_object68386:head v 323'10423, mlcod=321'10417)", 2024-04-06T03:45:46.418 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.608662+0000", 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 222.74356504299999, 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.051455097999999998, 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608662+0000", 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608662+0000", 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.419 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608664+0000", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5629999999999999e-06 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608676+0000", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2043e-05 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608702+0000", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5629000000000001e-05 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.420 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608714+0000", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2697000000000001e-05 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614826+0000", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0061117819999999996 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614881+0000", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4412e-05 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660053+0000", 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045172333000000002 2024-04-06T03:45:46.421 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660118+0000", 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4639000000000002e-05 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167209 10.a e323/322 10:5c8a3898:::benchmark_data_smithi050_121293_object68392:head v 323'10424, mlcod=321'10417)", 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.611549+0000", 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 222.740678489, 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.048630194000000002, 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.422 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611549+0000", 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611549+0000", 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.423 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611552+0000", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6960000000000001e-06 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611568+0000", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6537000000000001e-05 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611598+0000", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0086e-05 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.424 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611621+0000", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2909999999999999e-05 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.619572+0000", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0079503170000000001 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.619624+0000", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2534999999999998e-05 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660121+0000", 2024-04-06T03:45:46.425 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.040496826 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660179+0000", 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8286999999999999e-05 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167211 10.a e323/322 10:57f351c1:::benchmark_data_smithi050_121293_object68394:head v 323'10425, mlcod=323'10418)", 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.614381+0000", 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 222.73784627000001, 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045874677000000003, 2024-04-06T03:45:46.426 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614381+0000", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614381+0000", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614383+0000", 2024-04-06T03:45:46.427 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1600000000000001e-06 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614400+0000", 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6192e-05 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614429+0000", 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9447000000000001e-05 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614449+0000", 2024-04-06T03:45:46.428 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9732e-05 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634663+0000", 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.020214075000000001 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634720+0000", 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6839000000000002e-05 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660183+0000", 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.025463362 2024-04-06T03:45:46.429 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660256+0000", 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2869999999999999e-05 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:180408 10.e e325/319 10:772c70ba:::benchmark_data_smithi050_121293_object81591:head v 325'11002, mlcod=324'11000)", 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.512830+0000", 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 205.83939712599999, 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.046811765999999998, 2024-04-06T03:45:46.430 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512830+0000", 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512830+0000", 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.431 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512831+0000", 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7399999999999999e-07 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512835+0000", 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6700000000000002e-06 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512856+0000", 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0914e-05 2024-04-06T03:45:46.432 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512865+0000", 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3019999999999995e-06 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512952+0000", 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7357999999999999e-05 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.512972+0000", 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0171999999999999e-05 2024-04-06T03:45:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559595+0000", 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.046622780000000003 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559642+0000", 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7196e-05 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:180406 10.9 e325/280 10:9e5a89f0:::benchmark_data_smithi050_121293_object81589:head v 325'11334, mlcod=324'11332)", 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.515635+0000", 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 205.836592997, 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.044290147000000002, 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515635+0000", 2024-04-06T03:45:46.435 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515635+0000", 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515635+0000", 2024-04-06T03:45:46.436 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1300000000000001e-07 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515641+0000", 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6470000000000003e-06 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515651+0000", 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0214000000000001e-05 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.437 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515658+0000", 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7810000000000005e-06 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515713+0000", 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4475000000000003e-05 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.515737+0000", 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3830000000000001e-05 2024-04-06T03:45:46.438 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559855+0000", 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.044118652000000001 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559925+0000", 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9234999999999997e-05 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:180404 10.a e324/322 10:5f8d1ba9:::benchmark_data_smithi050_121293_object81587:head v 324'11261, mlcod=324'11255)", 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.518578+0000", 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 205.83364913400001, 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.049958333000000001, 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518578+0000", 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:45:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518578+0000", 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518579+0000", 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5899999999999995e-07 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518586+0000", 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0450000000000003e-06 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:45:46.441 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559643+0000", 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.041056895000000003 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559653+0000", 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5920000000000007e-06 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559742+0000", 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9581999999999997e-05 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.442 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559764+0000", 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1433e-05 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.568500+0000", 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0087363819999999991 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.568537+0000", 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6545000000000003e-05 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:45:46.443 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:45:46.444 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:45:46.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:46.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:46.573 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:46.598 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:46.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:46.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:46.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:46.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:46.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:46.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:47.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:47.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:47.219 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:47.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:47.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:47.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:47.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:47.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:47.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:47 smithi050 bash[21467]: cluster 2024-04-06T03:45:46.534352+0000 mgr.y (mgr.24368) 910 : cluster 0 pgmap v993: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 788 B/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:45:47.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:47 smithi050 bash[29221]: cluster 2024-04-06T03:45:46.534352+0000 mgr.y (mgr.24368) 910 : cluster 0 pgmap v993: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 788 B/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:45:47.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:47.710 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-06T03:45:47.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:47.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:47.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:47.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:47.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:48.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.014 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:47 smithi153 bash[23852]: cluster 2024-04-06T03:45:46.534352+0000 mgr.y (mgr.24368) 910 : cluster 0 pgmap v993: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 788 B/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:45:48.030 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:48.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:48.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:48.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:48.286 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:48.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:48.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:48.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:48.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.680 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:48.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:45:48.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:45:48.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:48.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:48.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:48.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:49.080 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:49.104 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:49.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:49.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:49.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:49.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:49.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:49.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:49.478 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:45:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:45:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:45:49.488 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:49.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:49.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:49.727 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:49.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:49 smithi050 bash[21467]: cluster 2024-04-06T03:45:48.534957+0000 mgr.y (mgr.24368) 911 : cluster 0 pgmap v994: 148 pgs: 148 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:45:49.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:49 smithi050 bash[29221]: cluster 2024-04-06T03:45:48.534957+0000 mgr.y (mgr.24368) 911 : cluster 0 pgmap v994: 148 pgs: 148 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:45:49.767 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:49.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:49.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:49.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:50.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:50.011 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:49 smithi153 bash[23852]: cluster 2024-04-06T03:45:48.534957+0000 mgr.y (mgr.24368) 911 : cluster 0 pgmap v994: 148 pgs: 148 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 825 op/s 2024-04-06T03:45:50.034 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:50.137 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:50.168 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:50.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:50.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:50.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:50.432 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:50.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:50.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:45:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:45:50.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:50.662 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:50.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:50 smithi153 bash[23852]: cluster 2024-04-06T03:45:50.237781+0000 osd.3 (osd.3) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:50.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:50 smithi153 bash[23852]: cluster 2024-04-06T03:45:50.237852+0000 osd.3 (osd.3) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:50.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:50 smithi153 bash[23852]: audit 2024-04-06T03:45:50.527270+0000 mon.a (mon.0) 1730 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:50.672 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:50 smithi153 bash[23852]: audit 2024-04-06T03:45:50.528913+0000 mon.a (mon.0) 1731 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:50.694 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:50.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:50.825 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:50.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:50.959 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:50.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[21467]: cluster 2024-04-06T03:45:50.237781+0000 osd.3 (osd.3) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[21467]: cluster 2024-04-06T03:45:50.237852+0000 osd.3 (osd.3) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[21467]: audit 2024-04-06T03:45:50.527270+0000 mon.a (mon.0) 1730 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[21467]: audit 2024-04-06T03:45:50.528913+0000 mon.a (mon.0) 1731 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[29221]: cluster 2024-04-06T03:45:50.237781+0000 osd.3 (osd.3) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[29221]: cluster 2024-04-06T03:45:50.237852+0000 osd.3 (osd.3) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[29221]: audit 2024-04-06T03:45:50.527270+0000 mon.a (mon.0) 1730 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:45:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:50 smithi050 bash[29221]: audit 2024-04-06T03:45:50.528913+0000 mon.a (mon.0) 1731 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:45:51.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:51.089 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:51.089 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:51.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:51.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:51.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:51.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:51.462 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:51.491 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:51.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:51.620 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:51.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:51.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:51 smithi153 bash[23852]: audit 2024-04-06T03:45:50.267830+0000 mgr.y (mgr.24368) 912 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:51 smithi153 bash[23852]: cluster 2024-04-06T03:45:50.535509+0000 mgr.y (mgr.24368) 913 : cluster 0 pgmap v995: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:45:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:51 smithi153 bash[23852]: cluster 2024-04-06T03:45:50.597857+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e370: 8 total, 8 up, 7 in 2024-04-06T03:45:51.848 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:51.871 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:51.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:51.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:51 smithi050 bash[21467]: audit 2024-04-06T03:45:50.267830+0000 mgr.y (mgr.24368) 912 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:51.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:51 smithi050 bash[21467]: cluster 2024-04-06T03:45:50.535509+0000 mgr.y (mgr.24368) 913 : cluster 0 pgmap v995: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:45:51.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:51 smithi050 bash[21467]: cluster 2024-04-06T03:45:50.597857+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e370: 8 total, 8 up, 7 in 2024-04-06T03:45:51.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:52.004 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:52.110 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:52.146 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:52.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:51 smithi050 bash[29221]: audit 2024-04-06T03:45:50.267830+0000 mgr.y (mgr.24368) 912 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:45:52.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:51 smithi050 bash[29221]: cluster 2024-04-06T03:45:50.535509+0000 mgr.y (mgr.24368) 913 : cluster 0 pgmap v995: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:45:52.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:51 smithi050 bash[29221]: cluster 2024-04-06T03:45:50.597857+0000 mon.a (mon.0) 1732 : cluster 0 osdmap e370: 8 total, 8 up, 7 in 2024-04-06T03:45:52.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:52.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:52.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:52.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:52.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:52.524 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:45:52.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:52.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:52.659 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:52.763 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:52.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:52.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:52.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:53.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:53.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:53.123 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:53.124 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:45:53.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:53.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:53.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:53.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:53.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:53.433 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:53.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:53.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:53.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:53.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:53 smithi153 bash[23852]: cluster 2024-04-06T03:45:52.536075+0000 mgr.y (mgr.24368) 914 : cluster 0 pgmap v997: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-04-06T03:45:53.837 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:53 smithi050 bash[21467]: cluster 2024-04-06T03:45:52.536075+0000 mgr.y (mgr.24368) 914 : cluster 0 pgmap v997: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-04-06T03:45:53.839 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:53.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:53.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:53 smithi050 bash[29221]: cluster 2024-04-06T03:45:52.536075+0000 mgr.y (mgr.24368) 914 : cluster 0 pgmap v997: 148 pgs: 148 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 960 op/s 2024-04-06T03:45:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:53.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:54.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:54.084 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:54.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:54.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:54.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:54.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:54.344 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:54.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:54.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:54.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:54.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:54.737 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:54.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:54.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:54.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:54.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:55.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:55.108 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:54 smithi153 bash[23852]: cluster 2024-04-06T03:45:54.536997+0000 mgr.y (mgr.24368) 915 : cluster 0 pgmap v998: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:45:55.132 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:55.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:55.257 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:55.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:55.408 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:55.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:55 smithi050 bash[21467]: cluster 2024-04-06T03:45:54.536997+0000 mgr.y (mgr.24368) 915 : cluster 0 pgmap v998: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:45:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:55 smithi050 bash[29221]: cluster 2024-04-06T03:45:54.536997+0000 mgr.y (mgr.24368) 915 : cluster 0 pgmap v998: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:45:55.512 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:55.531 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:55.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:55.657 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:55.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:55.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:55.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:55.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:56.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:56.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:56.167 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:56.184 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:56.287 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:56.427 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:56.457 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:56.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:56.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:56.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:56.721 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:56.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:56.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:56.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:56.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:56.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:57.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:57.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:57.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:57.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:57.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:57.340 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:57.366 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:57.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:57.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:57.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:57.620 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:57 smithi153 bash[23852]: cluster 2024-04-06T03:45:56.537632+0000 mgr.y (mgr.24368) 916 : cluster 0 pgmap v999: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-06T03:45:57.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:57.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:57.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:57.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:57.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:58.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:58.048 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:58.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:58.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:45:58.175 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:57 smithi050 bash[21467]: cluster 2024-04-06T03:45:56.537632+0000 mgr.y (mgr.24368) 916 : cluster 0 pgmap v999: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-06T03:45:58.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:57 smithi050 bash[29221]: cluster 2024-04-06T03:45:56.537632+0000 mgr.y (mgr.24368) 916 : cluster 0 pgmap v999: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-04-06T03:45:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:45:58.279 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:58.308 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:58.414 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:58.438 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:45:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:45:58.546 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:58.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:45:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:45:58.649 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:45:58.649 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:45:58.649 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:45:58.649 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:45:58.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:58.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:58.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:58.816 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:58.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:58.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:45:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:58.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:45:59.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:59.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:59.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:59.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:45:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:45:59.306 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:59.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:59.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:59.389 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:45:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:45:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:45:59.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:59.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:59.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:59.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:45:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:45:59.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-06T03:45:59.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:45:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:45:59.857 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:59.864 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:45:59 smithi153 bash[23852]: cluster 2024-04-06T03:45:58.538164+0000 mgr.y (mgr.24368) 917 : cluster 0 pgmap v1000: 148 pgs: 148 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:45:59.875 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:45:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:45:59.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:45:59.934 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:45:59 smithi050 bash[21467]: cluster 2024-04-06T03:45:58.538164+0000 mgr.y (mgr.24368) 917 : cluster 0 pgmap v1000: 148 pgs: 148 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:45:59.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:45:59 smithi050 bash[29221]: cluster 2024-04-06T03:45:58.538164+0000 mgr.y (mgr.24368) 917 : cluster 0 pgmap v1000: 148 pgs: 148 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-04-06T03:45:59.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:00.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:00.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:00.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:00.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:00.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:00.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:00.538 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:00.538 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:46:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:46:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:00.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:00.659 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:00.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:00.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:00.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:00.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:00.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:01.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:01.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:01.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:01.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:01.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:01.329 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:01.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:01.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:01.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:01.714 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:01.736 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:01.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:01 smithi050 bash[29221]: audit 2024-04-06T03:46:00.277519+0000 mgr.y (mgr.24368) 918 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:01.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:01 smithi050 bash[29221]: cluster 2024-04-06T03:46:00.539240+0000 mgr.y (mgr.24368) 919 : cluster 0 pgmap v1001: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s 2024-04-06T03:46:01.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:01 smithi050 bash[29221]: audit 2024-04-06T03:46:00.886348+0000 mon.a (mon.0) 1733 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:46:01.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:01 smithi050 bash[21467]: audit 2024-04-06T03:46:00.277519+0000 mgr.y (mgr.24368) 918 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:01.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:01 smithi050 bash[21467]: cluster 2024-04-06T03:46:00.539240+0000 mgr.y (mgr.24368) 919 : cluster 0 pgmap v1001: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s 2024-04-06T03:46:01.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:01 smithi050 bash[21467]: audit 2024-04-06T03:46:00.886348+0000 mon.a (mon.0) 1733 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:46:01.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:01.869 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:01.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:01.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:01 smithi153 bash[23852]: audit 2024-04-06T03:46:00.277519+0000 mgr.y (mgr.24368) 918 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:01 smithi153 bash[23852]: cluster 2024-04-06T03:46:00.539240+0000 mgr.y (mgr.24368) 919 : cluster 0 pgmap v1001: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 944 op/s 2024-04-06T03:46:01.982 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:01 smithi153 bash[23852]: audit 2024-04-06T03:46:00.886348+0000 mon.a (mon.0) 1733 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:46:01.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:02.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:02.140 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:02.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:02.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:02.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:02.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:02.533 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:02.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:02.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:02.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:02.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:02.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:02 smithi153 bash[23852]: audit 2024-04-06T03:46:01.728008+0000 mon.a (mon.0) 1734 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:46:02.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:02 smithi153 bash[23852]: audit 2024-04-06T03:46:01.729069+0000 mon.a (mon.0) 1735 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:46:02.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:02 smithi153 bash[23852]: audit 2024-04-06T03:46:01.737175+0000 mon.a (mon.0) 1736 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:46:02.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:02.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:02 smithi050 bash[21467]: audit 2024-04-06T03:46:01.728008+0000 mon.a (mon.0) 1734 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:46:02.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:02 smithi050 bash[21467]: audit 2024-04-06T03:46:01.729069+0000 mon.a (mon.0) 1735 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:46:02.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:02 smithi050 bash[21467]: audit 2024-04-06T03:46:01.737175+0000 mon.a (mon.0) 1736 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:46:02.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:02 smithi050 bash[29221]: audit 2024-04-06T03:46:01.728008+0000 mon.a (mon.0) 1734 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:46:02.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:02 smithi050 bash[29221]: audit 2024-04-06T03:46:01.729069+0000 mon.a (mon.0) 1735 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:46:02.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:02 smithi050 bash[29221]: audit 2024-04-06T03:46:01.737175+0000 mon.a (mon.0) 1736 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:46:02.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:02.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:03.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:03.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:03.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:03.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:03.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:03.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:03.427 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:03.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:03.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:03.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:03.680 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:03.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:03.810 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:03.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:03.924 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:03 smithi153 bash[23852]: cluster 2024-04-06T03:46:02.539846+0000 mgr.y (mgr.24368) 920 : cluster 0 pgmap v1002: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 810 op/s 2024-04-06T03:46:03.942 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:03.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:04.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:04.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:04.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:04.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:04.202 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:03 smithi050 bash[21467]: cluster 2024-04-06T03:46:02.539846+0000 mgr.y (mgr.24368) 920 : cluster 0 pgmap v1002: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 810 op/s 2024-04-06T03:46:04.202 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:03 smithi050 bash[29221]: cluster 2024-04-06T03:46:02.539846+0000 mgr.y (mgr.24368) 920 : cluster 0 pgmap v1002: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 810 op/s 2024-04-06T03:46:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:04.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:04.332 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:04.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:04.449 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:46:04.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:04.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:04.594 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:04.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:04.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:04.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:04.856 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:04.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:04.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:05.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:05.113 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:05.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:05.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:05.299 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:05 smithi153 bash[23852]: cluster 2024-04-06T03:46:04.540587+0000 mgr.y (mgr.24368) 921 : cluster 0 pgmap v1003: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:46:05.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:05.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:05.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:05.498 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:05.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:05.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:05 smithi050 bash[21467]: cluster 2024-04-06T03:46:04.540587+0000 mgr.y (mgr.24368) 921 : cluster 0 pgmap v1003: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:46:05.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:05 smithi050 bash[29221]: cluster 2024-04-06T03:46:04.540587+0000 mgr.y (mgr.24368) 921 : cluster 0 pgmap v1003: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 888 op/s 2024-04-06T03:46:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:05.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:05.753 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:05.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:05.889 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:05.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:05.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:06.018 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:06.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:06.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:06.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:06.269 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[21467]: cluster 2024-04-06T03:46:04.503095+0000 osd.4 (osd.4) 575 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:06.269 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[21467]: cluster 2024-04-06T03:46:04.503160+0000 osd.4 (osd.4) 576 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:06.269 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[21467]: cluster 2024-04-06T03:46:05.003256+0000 mon.a (mon.0) 1737 : cluster 0 osdmap e371: 8 total, 8 up, 7 in 2024-04-06T03:46:06.269 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[21467]: audit 2024-04-06T03:46:05.553989+0000 mon.a (mon.0) 1738 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:06.269 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[29221]: cluster 2024-04-06T03:46:04.503095+0000 osd.4 (osd.4) 575 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:06.269 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[29221]: cluster 2024-04-06T03:46:04.503160+0000 osd.4 (osd.4) 576 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:06.270 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[29221]: cluster 2024-04-06T03:46:05.003256+0000 mon.a (mon.0) 1737 : cluster 0 osdmap e371: 8 total, 8 up, 7 in 2024-04-06T03:46:06.270 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:06 smithi050 bash[29221]: audit 2024-04-06T03:46:05.553989+0000 mon.a (mon.0) 1738 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:06.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:06.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:06.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:06.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:06.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:06.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:06 smithi153 bash[23852]: cluster 2024-04-06T03:46:04.503095+0000 osd.4 (osd.4) 575 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:06.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:06 smithi153 bash[23852]: cluster 2024-04-06T03:46:04.503160+0000 osd.4 (osd.4) 576 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:06.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:06 smithi153 bash[23852]: cluster 2024-04-06T03:46:05.003256+0000 mon.a (mon.0) 1737 : cluster 0 osdmap e371: 8 total, 8 up, 7 in 2024-04-06T03:46:06.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:06 smithi153 bash[23852]: audit 2024-04-06T03:46:05.553989+0000 mon.a (mon.0) 1738 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:06.550 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:06.653 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:06.684 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:06.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:06.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:06.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:06.939 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:06.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:07.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:07 smithi153 bash[23852]: cluster 2024-04-06T03:46:06.541145+0000 mgr.y (mgr.24368) 922 : cluster 0 pgmap v1005: 148 pgs: 148 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 818 op/s 2024-04-06T03:46:07.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:07.177 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:07.211 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:07.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:07.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:07.334 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:07.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:07.473 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:07.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:07 smithi050 bash[21467]: cluster 2024-04-06T03:46:06.541145+0000 mgr.y (mgr.24368) 922 : cluster 0 pgmap v1005: 148 pgs: 148 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 818 op/s 2024-04-06T03:46:07.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:07 smithi050 bash[29221]: cluster 2024-04-06T03:46:06.541145+0000 mgr.y (mgr.24368) 922 : cluster 0 pgmap v1005: 148 pgs: 148 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 818 op/s 2024-04-06T03:46:07.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:07.607 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:07.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:07.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:07.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:07.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:07.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:07.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:08.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:08.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:08.133 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:08.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:08.272 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:08.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:08.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:08.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:08.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:08.670 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:08.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:08.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:08.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:08.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:09.035 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:09.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:09.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:09.206 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:09.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:09.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:09.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:09.459 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:09.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:09.573 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:46:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:46:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:46:09.591 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:09.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:09.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:09.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:09 smithi153 bash[23852]: cluster 2024-04-06T03:46:08.542039+0000 mgr.y (mgr.24368) 923 : cluster 0 pgmap v1006: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:46:09.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:09.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:09 smithi050 bash[21467]: cluster 2024-04-06T03:46:08.542039+0000 mgr.y (mgr.24368) 923 : cluster 0 pgmap v1006: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:46:09.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:09 smithi050 bash[29221]: cluster 2024-04-06T03:46:08.542039+0000 mgr.y (mgr.24368) 923 : cluster 0 pgmap v1006: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s 2024-04-06T03:46:09.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:09.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:09.950 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:46:10.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:10.019 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:46:10.028 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:10.057 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:46:10.110 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:46:10.133 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:10.157 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:10.160 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:46:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:10.193 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:46:10.215 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:46:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:46:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:46:10.215 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:46:10.216 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:46:10.223 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:46:10.265 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:46:10.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:46:10.275 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:46:10.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:10.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:10.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:10.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:10.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:46:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:46:10.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:10.618 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:46:10.625 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-06T03:46:10.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:10.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:10.790 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:10 smithi153 bash[23852]: audit 2024-04-06T03:46:09.885358+0000 mon.c (mon.2) 111 : audit 0 from='client.? 172.21.15.50:0/3869531670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:46:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:10 smithi153 bash[23852]: audit 2024-04-06T03:46:10.094760+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.50:0/3266117874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:46:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:10 smithi153 bash[23852]: audit 2024-04-06T03:46:10.095246+0000 mon.a (mon.0) 1739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:46:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:10 smithi153 bash[23852]: audit 2024-04-06T03:46:10.176659+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.50:0/2282148681' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:46:10.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:10 smithi153 bash[23852]: audit 2024-04-06T03:46:10.177704+0000 mon.a (mon.0) 1740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:46:10.807 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:10.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:10.932 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:10.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[21467]: audit 2024-04-06T03:46:09.885358+0000 mon.c (mon.2) 111 : audit 0 from='client.? 172.21.15.50:0/3869531670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[21467]: audit 2024-04-06T03:46:10.094760+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.50:0/3266117874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[21467]: audit 2024-04-06T03:46:10.095246+0000 mon.a (mon.0) 1739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[21467]: audit 2024-04-06T03:46:10.176659+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.50:0/2282148681' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[21467]: audit 2024-04-06T03:46:10.177704+0000 mon.a (mon.0) 1740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[29221]: audit 2024-04-06T03:46:09.885358+0000 mon.c (mon.2) 111 : audit 0 from='client.? 172.21.15.50:0/3869531670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[29221]: audit 2024-04-06T03:46:10.094760+0000 mon.c (mon.2) 112 : audit 1 from='client.? 172.21.15.50:0/3266117874' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[29221]: audit 2024-04-06T03:46:10.095246+0000 mon.a (mon.0) 1739 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[29221]: audit 2024-04-06T03:46:10.176659+0000 mon.b (mon.1) 42 : audit 1 from='client.? 172.21.15.50:0/2282148681' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:46:10.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:10 smithi050 bash[29221]: audit 2024-04-06T03:46:10.177704+0000 mon.a (mon.0) 1740 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:46:11.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:11.165 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:11.192 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:11.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:11.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:11.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:11.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:11.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-06T03:46:11.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:11.581 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:11.686 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:11.737 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:11.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:11 smithi153 bash[23852]: audit 2024-04-06T03:46:10.287872+0000 mgr.y (mgr.24368) 924 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:11.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:11 smithi153 bash[23852]: cluster 2024-04-06T03:46:10.542762+0000 mgr.y (mgr.24368) 925 : cluster 0 pgmap v1007: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:46:11.740 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:11 smithi153 bash[23852]: cluster 2024-04-06T03:46:10.596707+0000 mon.a (mon.0) 1741 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:46:11.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:11 smithi153 bash[23852]: audit 2024-04-06T03:46:10.602883+0000 mon.a (mon.0) 1742 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:46:11.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:11 smithi153 bash[23852]: audit 2024-04-06T03:46:10.603010+0000 mon.a (mon.0) 1743 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:46:11.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:11 smithi153 bash[23852]: cluster 2024-04-06T03:46:10.607205+0000 mon.a (mon.0) 1744 : cluster 0 osdmap e372: 8 total, 8 up, 7 in 2024-04-06T03:46:11.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:11.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[21467]: audit 2024-04-06T03:46:10.287872+0000 mgr.y (mgr.24368) 924 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[21467]: cluster 2024-04-06T03:46:10.542762+0000 mgr.y (mgr.24368) 925 : cluster 0 pgmap v1007: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[21467]: cluster 2024-04-06T03:46:10.596707+0000 mon.a (mon.0) 1741 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[21467]: audit 2024-04-06T03:46:10.602883+0000 mon.a (mon.0) 1742 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[21467]: audit 2024-04-06T03:46:10.603010+0000 mon.a (mon.0) 1743 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[21467]: cluster 2024-04-06T03:46:10.607205+0000 mon.a (mon.0) 1744 : cluster 0 osdmap e372: 8 total, 8 up, 7 in 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[29221]: audit 2024-04-06T03:46:10.287872+0000 mgr.y (mgr.24368) 924 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[29221]: cluster 2024-04-06T03:46:10.542762+0000 mgr.y (mgr.24368) 925 : cluster 0 pgmap v1007: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[29221]: cluster 2024-04-06T03:46:10.596707+0000 mon.a (mon.0) 1741 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:46:11.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[29221]: audit 2024-04-06T03:46:10.602883+0000 mon.a (mon.0) 1742 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:46:11.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[29221]: audit 2024-04-06T03:46:10.603010+0000 mon.a (mon.0) 1743 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:46:11.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:11 smithi050 bash[29221]: cluster 2024-04-06T03:46:10.607205+0000 mon.a (mon.0) 1744 : cluster 0 osdmap e372: 8 total, 8 up, 7 in 2024-04-06T03:46:11.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:11.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:11.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:11.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:12.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:12.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:12.219 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:12.238 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:12.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:12.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:12.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:12.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:12.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:12.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:12.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:12.788 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:12.893 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:12.915 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:12.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:13.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:13.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:13.152 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:13.173 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:13.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:13.302 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:13.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:13.422 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:13.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:13.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:13.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:13.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:13.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:13.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:13 smithi050 bash[21467]: cluster 2024-04-06T03:46:12.543316+0000 mgr.y (mgr.24368) 926 : cluster 0 pgmap v1009: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 893 op/s 2024-04-06T03:46:13.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:13 smithi050 bash[29221]: cluster 2024-04-06T03:46:12.543316+0000 mgr.y (mgr.24368) 926 : cluster 0 pgmap v1009: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 893 op/s 2024-04-06T03:46:13.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:13 smithi153 bash[23852]: cluster 2024-04-06T03:46:12.543316+0000 mgr.y (mgr.24368) 926 : cluster 0 pgmap v1009: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 893 op/s 2024-04-06T03:46:13.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:13.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:13.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:13.940 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:13.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:14.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:14.068 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:14.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:14.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:14.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:14.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:14.429 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:14.460 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:14.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:14.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:14.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:14.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:46:14.719 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:14.822 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.862 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:14.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:14.884 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-06T03:46:14.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:46:14.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:14.989 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:14.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:14.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:15.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:15.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:15.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:15.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:15 smithi050 bash[21467]: cluster 2024-04-06T03:46:14.543908+0000 mgr.y (mgr.24368) 927 : cluster 0 pgmap v1010: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 737 op/s 2024-04-06T03:46:15.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:15 smithi050 bash[29221]: cluster 2024-04-06T03:46:14.543908+0000 mgr.y (mgr.24368) 927 : cluster 0 pgmap v1010: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 737 op/s 2024-04-06T03:46:15.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:15 smithi153 bash[23852]: cluster 2024-04-06T03:46:14.543908+0000 mgr.y (mgr.24368) 927 : cluster 0 pgmap v1010: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 737 op/s 2024-04-06T03:46:15.261 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:15.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:15.390 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:15.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:15.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:15.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:15.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:15.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:15.793 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:15.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:15.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:15.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:16.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:16.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:16.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:16.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:16.305 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:46:16.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:16.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:16.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:16.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.702 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:16.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:16.827 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:16.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:16.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:17.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:46:17.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:17.086 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:17.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:17.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:17.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:17.347 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:17.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:17.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:17.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:17.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:17.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:17.755 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:17.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:17.868 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:17 smithi050 bash[21467]: cluster 2024-04-06T03:46:16.544550+0000 mgr.y (mgr.24368) 928 : cluster 0 pgmap v1011: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s 2024-04-06T03:46:17.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:17 smithi050 bash[29221]: cluster 2024-04-06T03:46:16.544550+0000 mgr.y (mgr.24368) 928 : cluster 0 pgmap v1011: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s 2024-04-06T03:46:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:17.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:17.988 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:18.024 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:18.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:18.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:18.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:18.266 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:17 smithi153 bash[23852]: cluster 2024-04-06T03:46:16.544550+0000 mgr.y (mgr.24368) 928 : cluster 0 pgmap v1011: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 865 op/s 2024-04-06T03:46:18.277 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:18.380 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:18.407 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:18.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:18.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:18.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:18.671 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:18.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:18.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:18 smithi050 bash[21467]: cluster 2024-04-06T03:46:17.088816+0000 osd.2 (osd.2) 475 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:18.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:18 smithi050 bash[21467]: cluster 2024-04-06T03:46:17.088887+0000 osd.2 (osd.2) 476 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:18.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:18 smithi050 bash[21467]: cluster 2024-04-06T03:46:17.604020+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e373: 8 total, 8 up, 7 in 2024-04-06T03:46:18.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:18 smithi050 bash[29221]: cluster 2024-04-06T03:46:17.088816+0000 osd.2 (osd.2) 475 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:18.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:18 smithi050 bash[29221]: cluster 2024-04-06T03:46:17.088887+0000 osd.2 (osd.2) 476 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:18.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:18 smithi050 bash[29221]: cluster 2024-04-06T03:46:17.604020+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e373: 8 total, 8 up, 7 in 2024-04-06T03:46:18.803 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:18.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:18.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:18.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:19.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:19.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:18 smithi153 bash[23852]: cluster 2024-04-06T03:46:17.088816+0000 osd.2 (osd.2) 475 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:19.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:18 smithi153 bash[23852]: cluster 2024-04-06T03:46:17.088887+0000 osd.2 (osd.2) 476 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:19.046 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:18 smithi153 bash[23852]: cluster 2024-04-06T03:46:17.604020+0000 mon.a (mon.0) 1745 : cluster 0 osdmap e373: 8 total, 8 up, 7 in 2024-04-06T03:46:19.075 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:19.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:19.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:19.314 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:19.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:19.461 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:46:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:46:19.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:19.567 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:46:19.571 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:46:19.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:19.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:19.732 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:19.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[21467]: cluster 2024-04-06T03:46:18.545209+0000 mgr.y (mgr.24368) 929 : cluster 0 pgmap v1013: 148 pgs: 148 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 827 op/s 2024-04-06T03:46:19.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[21467]: cluster 2024-04-06T03:46:18.675469+0000 osd.0 (osd.0) 589 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:19.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[21467]: cluster 2024-04-06T03:46:18.675533+0000 osd.0 (osd.0) 590 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[29221]: cluster 2024-04-06T03:46:18.545209+0000 mgr.y (mgr.24368) 929 : cluster 0 pgmap v1013: 148 pgs: 148 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 827 op/s 2024-04-06T03:46:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[29221]: cluster 2024-04-06T03:46:18.675469+0000 osd.0 (osd.0) 589 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:19 smithi050 bash[29221]: cluster 2024-04-06T03:46:18.675533+0000 osd.0 (osd.0) 590 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:19.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:19 smithi153 bash[23852]: cluster 2024-04-06T03:46:18.545209+0000 mgr.y (mgr.24368) 929 : cluster 0 pgmap v1013: 148 pgs: 148 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 827 op/s 2024-04-06T03:46:19.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:19 smithi153 bash[23852]: cluster 2024-04-06T03:46:18.675469+0000 osd.0 (osd.0) 589 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:19.742 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:19 smithi153 bash[23852]: cluster 2024-04-06T03:46:18.675533+0000 osd.0 (osd.0) 590 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:19.762 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:19.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:19.892 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:19.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:19.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:20.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:20.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:20.146 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:20.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:20.275 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:20.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:20.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:20.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:46:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:46:20.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:20.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:20.675 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:20.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.810 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:20.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:20.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:20.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:20 smithi050 bash[21467]: cluster 2024-04-06T03:46:19.624155+0000 mon.a (mon.0) 1746 : cluster 0 osdmap e374: 8 total, 8 up, 7 in 2024-04-06T03:46:20.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:20 smithi050 bash[21467]: audit 2024-04-06T03:46:20.583769+0000 mon.a (mon.0) 1747 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:20.921 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:20 smithi050 bash[29221]: cluster 2024-04-06T03:46:19.624155+0000 mon.a (mon.0) 1746 : cluster 0 osdmap e374: 8 total, 8 up, 7 in 2024-04-06T03:46:20.921 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:20 smithi050 bash[29221]: audit 2024-04-06T03:46:20.583769+0000 mon.a (mon.0) 1747 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:20.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:20.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:20.968 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:20 smithi153 bash[23852]: cluster 2024-04-06T03:46:19.624155+0000 mon.a (mon.0) 1746 : cluster 0 osdmap e374: 8 total, 8 up, 7 in 2024-04-06T03:46:20.968 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:20 smithi153 bash[23852]: audit 2024-04-06T03:46:20.583769+0000 mon.a (mon.0) 1747 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:21.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:21.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:21.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:21.189 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:21.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:21.321 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:21.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:21.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:21.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:21.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:21.587 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:46:21.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:21.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:21.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:21.809 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:21.819 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:21 smithi153 bash[23852]: audit 2024-04-06T03:46:20.295928+0000 mgr.y (mgr.24368) 930 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:21.819 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:21 smithi153 bash[23852]: cluster 2024-04-06T03:46:20.545833+0000 mgr.y (mgr.24368) 931 : cluster 0 pgmap v1015: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-04-06T03:46:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.834 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:21.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:21.935 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:21.957 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:21.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:21.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:21 smithi050 bash[21467]: audit 2024-04-06T03:46:20.295928+0000 mgr.y (mgr.24368) 930 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:21.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:21 smithi050 bash[21467]: cluster 2024-04-06T03:46:20.545833+0000 mgr.y (mgr.24368) 931 : cluster 0 pgmap v1015: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-04-06T03:46:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:21 smithi050 bash[29221]: audit 2024-04-06T03:46:20.295928+0000 mgr.y (mgr.24368) 930 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:21 smithi050 bash[29221]: cluster 2024-04-06T03:46:20.545833+0000 mgr.y (mgr.24368) 931 : cluster 0 pgmap v1015: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-04-06T03:46:22.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:22.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:22.194 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:22.227 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:22.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:22.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:22.478 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:22.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:22.603 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:22.616 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:46:22.616 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:46:22.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:22.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:22.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:22.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:22.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:22.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:22.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:23.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:23.131 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:23.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:23.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:23.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:23.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:23.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-06T03:46:23.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:23.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:23.783 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:23.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:23.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:23 smithi050 bash[21467]: cluster 2024-04-06T03:46:22.546800+0000 mgr.y (mgr.24368) 932 : cluster 0 pgmap v1016: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:46:23.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:23 smithi050 bash[29221]: cluster 2024-04-06T03:46:22.546800+0000 mgr.y (mgr.24368) 932 : cluster 0 pgmap v1016: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:46:23.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:23.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:24.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:24.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:23 smithi153 bash[23852]: cluster 2024-04-06T03:46:22.546800+0000 mgr.y (mgr.24368) 932 : cluster 0 pgmap v1016: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:46:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:24.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:24.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:24.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:24.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:24.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:24.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:24.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:24.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:24.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:24.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:24.833 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:24.938 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:24.981 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:24.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:25.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:25.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:25.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:25.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:25.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:25.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:25.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.633 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:25.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:25.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:25 smithi050 bash[21467]: cluster 2024-04-06T03:46:24.547359+0000 mgr.y (mgr.24368) 933 : cluster 0 pgmap v1017: 148 pgs: 148 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 947 op/s 2024-04-06T03:46:25.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:25 smithi050 bash[29221]: cluster 2024-04-06T03:46:24.547359+0000 mgr.y (mgr.24368) 933 : cluster 0 pgmap v1017: 148 pgs: 148 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 947 op/s 2024-04-06T03:46:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:25.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:25.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.877 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:25 smithi153 bash[23852]: cluster 2024-04-06T03:46:24.547359+0000 mgr.y (mgr.24368) 933 : cluster 0 pgmap v1017: 148 pgs: 148 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 947 op/s 2024-04-06T03:46:25.894 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:25.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:26.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:26.025 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:26.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:26.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:26.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:26.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:26.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:26.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:26.415 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:26.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:26.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:26.645 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:26.664 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:26.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:26.789 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:26.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:26.922 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:26.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:27.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:27.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.054 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:27.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:27.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:27.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:27.309 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:27.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:27.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:27.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:27.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:27.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:27.691 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:27.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:27.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:27.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:27.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:27 smithi153 bash[23852]: cluster 2024-04-06T03:46:26.547960+0000 mgr.y (mgr.24368) 934 : cluster 0 pgmap v1018: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:46:28.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:28.083 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:28.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:28.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:27 smithi050 bash[21467]: cluster 2024-04-06T03:46:26.547960+0000 mgr.y (mgr.24368) 934 : cluster 0 pgmap v1018: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:46:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:27 smithi050 bash[29221]: cluster 2024-04-06T03:46:26.547960+0000 mgr.y (mgr.24368) 934 : cluster 0 pgmap v1018: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-04-06T03:46:28.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:28.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:28.322 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:28.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:28.375 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:46:28.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:28.466 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:28.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:28.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:28.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:28.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:28.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:28.862 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:28.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:28.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:29.003 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:29.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:29.238 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:29.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:29.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:29.404 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:29.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:46:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:46:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:46:29.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:29.536 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:29.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:29.660 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:29.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:29.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:29.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:29 smithi153 bash[23852]: cluster 2024-04-06T03:46:28.549676+0000 mgr.y (mgr.24368) 935 : cluster 0 pgmap v1019: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 936 op/s 2024-04-06T03:46:29.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:29.918 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:29.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:29.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:29 smithi050 bash[21467]: cluster 2024-04-06T03:46:28.549676+0000 mgr.y (mgr.24368) 935 : cluster 0 pgmap v1019: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 936 op/s 2024-04-06T03:46:29.921 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:29 smithi050 bash[29221]: cluster 2024-04-06T03:46:28.549676+0000 mgr.y (mgr.24368) 935 : cluster 0 pgmap v1019: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 936 op/s 2024-04-06T03:46:30.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:30.059 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:30.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:30.187 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:30.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:30.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:30.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:30.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:30.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:30.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:30.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:46:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:46:30.570 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:30.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:30.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:30.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:30.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:30.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:30.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:30.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:30.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:31.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:31.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:31.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:31.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:31.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:31.346 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:31.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:31.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:31.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:31.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:31.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:31.858 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:31.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:31.906 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:31 smithi153 bash[23852]: audit 2024-04-06T03:46:30.303983+0000 mgr.y (mgr.24368) 936 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:31.906 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:31 smithi153 bash[23852]: cluster 2024-04-06T03:46:30.550528+0000 mgr.y (mgr.24368) 937 : cluster 0 pgmap v1020: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 936 B/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:46:31.966 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.002 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:32.105 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:32.129 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:32.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:32.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:31 smithi050 bash[21467]: audit 2024-04-06T03:46:30.303983+0000 mgr.y (mgr.24368) 936 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:32.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:31 smithi050 bash[21467]: cluster 2024-04-06T03:46:30.550528+0000 mgr.y (mgr.24368) 937 : cluster 0 pgmap v1020: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 936 B/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:46:32.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:31 smithi050 bash[29221]: audit 2024-04-06T03:46:30.303983+0000 mgr.y (mgr.24368) 936 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:32.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:31 smithi050 bash[29221]: cluster 2024-04-06T03:46:30.550528+0000 mgr.y (mgr.24368) 937 : cluster 0 pgmap v1020: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 936 B/s rd, 0 B/s wr, 1.03k op/s 2024-04-06T03:46:32.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:32.364 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.397 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:32.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:32.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:32.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.685 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:32.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:32.813 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:32.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.937 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:32.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:33.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:33.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:33.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:33.194 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:33.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:33.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:33.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:33.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:33.552 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:33.581 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:33.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:33.709 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:33.812 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:33.822 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:33 smithi153 bash[23852]: cluster 2024-04-06T03:46:32.551006+0000 mgr.y (mgr.24368) 938 : cluster 0 pgmap v1021: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s 2024-04-06T03:46:33.822 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:33 smithi153 bash[23852]: cluster 2024-04-06T03:46:32.692510+0000 osd.7 (osd.7) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:33.822 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:33 smithi153 bash[23852]: cluster 2024-04-06T03:46:32.692576+0000 osd.7 (osd.7) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:33.842 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:33.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:33 smithi050 bash[29221]: cluster 2024-04-06T03:46:32.551006+0000 mgr.y (mgr.24368) 938 : cluster 0 pgmap v1021: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s 2024-04-06T03:46:33.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:33 smithi050 bash[29221]: cluster 2024-04-06T03:46:32.692510+0000 osd.7 (osd.7) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:33.942 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:33 smithi050 bash[29221]: cluster 2024-04-06T03:46:32.692576+0000 osd.7 (osd.7) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:33.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:33 smithi050 bash[21467]: cluster 2024-04-06T03:46:32.551006+0000 mgr.y (mgr.24368) 938 : cluster 0 pgmap v1021: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 752 op/s 2024-04-06T03:46:33.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:33 smithi050 bash[21467]: cluster 2024-04-06T03:46:32.692510+0000 osd.7 (osd.7) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:33.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:33 smithi050 bash[21467]: cluster 2024-04-06T03:46:32.692576+0000 osd.7 (osd.7) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:33.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:33.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:33.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:34.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:34.097 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:34.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:34.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:34.332 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:34.368 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:34.466 INFO:teuthology.orchestra.run.smithi153.stdout:ERROR: (22) Invalid argument 2024-04-06T03:46:34.466 INFO:teuthology.orchestra.run.smithi153.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:46:34.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:34.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:34.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:34.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:34.733 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:34.736 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:34 smithi153 bash[23852]: cluster 2024-04-06T03:46:33.669504+0000 mon.a (mon.0) 1748 : cluster 0 osdmap e375: 8 total, 8 up, 7 in 2024-04-06T03:46:34.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:34 smithi050 bash[21467]: cluster 2024-04-06T03:46:33.669504+0000 mon.a (mon.0) 1748 : cluster 0 osdmap e375: 8 total, 8 up, 7 in 2024-04-06T03:46:34.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:34 smithi050 bash[29221]: cluster 2024-04-06T03:46:33.669504+0000 mon.a (mon.0) 1748 : cluster 0 osdmap e375: 8 total, 8 up, 7 in 2024-04-06T03:46:34.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:34.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:34.877 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:34.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:34.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.005 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:35.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:35.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:35.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:35.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:35.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.396 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:35.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:35.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:35.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:35.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:35.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-06T03:46:35.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.796 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:35.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.916 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:35.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:35.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:35 smithi050 bash[21467]: cluster 2024-04-06T03:46:34.551419+0000 mgr.y (mgr.24368) 939 : cluster 0 pgmap v1023: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s 2024-04-06T03:46:35.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:35 smithi050 bash[21467]: audit 2024-04-06T03:46:35.601538+0000 mon.a (mon.0) 1749 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:35 smithi050 bash[29221]: cluster 2024-04-06T03:46:34.551419+0000 mgr.y (mgr.24368) 939 : cluster 0 pgmap v1023: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s 2024-04-06T03:46:35.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:35 smithi050 bash[29221]: audit 2024-04-06T03:46:35.601538+0000 mon.a (mon.0) 1749 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:36.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:36.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:35 smithi153 bash[23852]: cluster 2024-04-06T03:46:34.551419+0000 mgr.y (mgr.24368) 939 : cluster 0 pgmap v1023: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s 2024-04-06T03:46:36.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:35 smithi153 bash[23852]: audit 2024-04-06T03:46:35.601538+0000 mon.a (mon.0) 1749 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:36.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:36.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:36.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:36.282 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:36.299 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:36.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:36.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:36.557 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:36.661 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:36.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:36.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:36.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:36.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:37.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:37.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:37.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:37.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:37.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:37.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:37.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:37.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:37.862 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:37.891 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:37.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:37.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:37 smithi050 bash[21467]: cluster 2024-04-06T03:46:36.552079+0000 mgr.y (mgr.24368) 940 : cluster 0 pgmap v1024: 148 pgs: 148 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 872 op/s 2024-04-06T03:46:37.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:37 smithi050 bash[29221]: cluster 2024-04-06T03:46:36.552079+0000 mgr.y (mgr.24368) 940 : cluster 0 pgmap v1024: 148 pgs: 148 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 872 op/s 2024-04-06T03:46:37.996 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:38.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:37 smithi153 bash[23852]: cluster 2024-04-06T03:46:36.552079+0000 mgr.y (mgr.24368) 940 : cluster 0 pgmap v1024: 148 pgs: 148 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 872 op/s 2024-04-06T03:46:38.046 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:38.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:38.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:38.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:38.296 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:38.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:38.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:38.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:38.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:38.554 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:38.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:38.690 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:38.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:38.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:38.953 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:38.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:38.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:39.056 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:39.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:46:39.125 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:46:39.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:39.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:39.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:39.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:39.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:39.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:39.597 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:39.604 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:46:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:46:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:46:39.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:39.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:39.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:39.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:39.966 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:39 smithi153 bash[23852]: cluster 2024-04-06T03:46:38.553460+0000 mgr.y (mgr.24368) 941 : cluster 0 pgmap v1025: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 868 op/s 2024-04-06T03:46:39.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:39 smithi050 bash[21467]: cluster 2024-04-06T03:46:38.553460+0000 mgr.y (mgr.24368) 941 : cluster 0 pgmap v1025: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 868 op/s 2024-04-06T03:46:39.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:39 smithi050 bash[29221]: cluster 2024-04-06T03:46:38.553460+0000 mgr.y (mgr.24368) 941 : cluster 0 pgmap v1025: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 868 op/s 2024-04-06T03:46:39.994 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:39.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:40.004 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:46:40.039 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:46:40.081 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:46:40.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.108 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:46:40.124 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:40.158 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:46:40.195 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:46:40.229 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.247 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:46:40.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:40.297 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:46:40.303 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:46:40.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:40.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:40.471 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:46:40.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.496 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:46:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:46:40.522 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:40.625 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.650 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:40.693 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:46:40.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:40.774 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:40.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:40.907 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:40.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:40.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[21467]: audit 2024-04-06T03:46:39.069000+0000 mgr.y (mgr.24368) 942 : audit 0 from='client.25481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:46:40.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[21467]: audit 2024-04-06T03:46:39.968052+0000 mon.a (mon.0) 1750 : audit 0 from='client.? 172.21.15.50:0/1895352319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:46:40.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[21467]: audit 2024-04-06T03:46:40.371881+0000 mon.c (mon.2) 113 : audit 1 from='client.? 172.21.15.50:0/402463763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:46:40.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[21467]: audit 2024-04-06T03:46:40.372485+0000 mon.a (mon.0) 1751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:46:40.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[29221]: audit 2024-04-06T03:46:39.069000+0000 mgr.y (mgr.24368) 942 : audit 0 from='client.25481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:46:40.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[29221]: audit 2024-04-06T03:46:39.968052+0000 mon.a (mon.0) 1750 : audit 0 from='client.? 172.21.15.50:0/1895352319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:46:40.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[29221]: audit 2024-04-06T03:46:40.371881+0000 mon.c (mon.2) 113 : audit 1 from='client.? 172.21.15.50:0/402463763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:46:40.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:40 smithi050 bash[29221]: audit 2024-04-06T03:46:40.372485+0000 mon.a (mon.0) 1751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:46:41.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:41.044 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:41.145 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1025,"stamp":"2024-04-06T03:46:38.552321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2306087286,"num_objects":35410,"num_object_clones":0,"num_object_copies":106230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35410,"num_whiteouts":0,"num_read":3605,"num_read_kb":3419,"num_write":178143,"num_write_kb":6784606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":2958161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78122,"ondisk_log_size":78122,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10933512,"kb_used_data":6944516,"kb_used_omap":10,"kb_used_meta":3988789,"kb_avail":645282552,"statfs":{"total":671965249536,"available":660769333248,"internally_reserved":0,"allocated":7111184384,"data_stored":7108851905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4084520043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-568786944,"num_objects":-8679,"num_object_clones":0,"num_object_copies":-26037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8679,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8679,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003153"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669246+0000","last_change":"2024-04-06T03:43:33.261450+0000","last_active":"2024-04-06T03:46:33.669246+0000","last_peered":"2024-04-06T03:46:33.669246+0000","last_clean":"2024-04-06T03:46:33.669246+0000","last_became_active":"2024-04-06T03:42:23.588459+0000","last_became_peered":"2024-04-06T03:42:23.588459+0000","last_unstale":"2024-04-06T03:46:33.669246+0000","last_undegraded":"2024-04-06T03:46:33.669246+0000","last_fullsized":"2024-04-06T03:46:33.669246+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_clean_scrub_stamp":"2024-04-06T03:43:33.261299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774754+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":705,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676086+0000","last_change":"2024-04-06T03:45:17.323934+0000","last_active":"2024-04-06T03:46:33.676086+0000","last_peered":"2024-04-06T03:46:33.676086+0000","last_clean":"2024-04-06T03:46:33.676086+0000","last_became_active":"2024-04-06T03:42:57.110401+0000","last_became_peered":"2024-04-06T03:42:57.110401+0000","last_unstale":"2024-04-06T03:46:33.676086+0000","last_undegraded":"2024-04-06T03:46:33.676086+0000","last_fullsized":"2024-04-06T03:46:33.676086+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:17.323812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.323812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:38.295565+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":723,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.683425+0000","last_change":"2024-04-06T03:45:21.991669+0000","last_active":"2024-04-06T03:46:33.683425+0000","last_peered":"2024-04-06T03:46:33.683425+0000","last_clean":"2024-04-06T03:46:33.683425+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:46:33.683425+0000","last_undegraded":"2024-04-06T03:46:33.683425+0000","last_fullsized":"2024-04-06T03:46:33.683425+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_clean_scrub_stamp":"2024-04-06T03:45:21.990975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:43.481763+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"307'42","reported_seq":796,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670196+0000","last_change":"2024-04-06T03:44:58.409103+0000","last_active":"2024-04-06T03:46:33.670196+0000","last_peered":"2024-04-06T03:46:33.670196+0000","last_clean":"2024-04-06T03:46:33.670196+0000","last_became_active":"2024-04-06T03:42:23.588857+0000","last_became_peered":"2024-04-06T03:42:23.588857+0000","last_unstale":"2024-04-06T03:46:33.670196+0000","last_undegraded":"2024-04-06T03:46:33.670196+0000","last_fullsized":"2024-04-06T03:46:33.670196+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'42","last_scrub_stamp":"2024-04-06T03:44:58.408958+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.408958+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":697,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676085+0000","last_change":"2024-04-06T03:45:38.544714+0000","last_active":"2024-04-06T03:46:33.676085+0000","last_peered":"2024-04-06T03:46:33.676085+0000","last_clean":"2024-04-06T03:46:33.676085+0000","last_became_active":"2024-04-06T03:42:57.091902+0000","last_became_peered":"2024-04-06T03:42:57.091902+0000","last_unstale":"2024-04-06T03:46:33.676085+0000","last_undegraded":"2024-04-06T03:46:33.676085+0000","last_fullsized":"2024-04-06T03:46:33.676085+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:38.544490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:38.544490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:57.209066+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":580,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678374+0000","last_change":"2024-04-06T03:45:37.144389+0000","last_active":"2024-04-06T03:46:33.678374+0000","last_peered":"2024-04-06T03:46:33.678374+0000","last_clean":"2024-04-06T03:46:33.678374+0000","last_became_active":"2024-04-06T03:42:57.100816+0000","last_became_peered":"2024-04-06T03:42:57.100816+0000","last_unstale":"2024-04-06T03:46:33.678374+0000","last_undegraded":"2024-04-06T03:46:33.678374+0000","last_fullsized":"2024-04-06T03:46:33.678374+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:37.144305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:37.144305+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.144305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.025433+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":700,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670071+0000","last_change":"2024-04-06T03:44:57.401411+0000","last_active":"2024-04-06T03:46:33.670071+0000","last_peered":"2024-04-06T03:46:33.670071+0000","last_clean":"2024-04-06T03:46:33.670071+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:46:33.670071+0000","last_undegraded":"2024-04-06T03:46:33.670071+0000","last_fullsized":"2024-04-06T03:46:33.670071+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:57.401286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.401286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"307'39","reported_seq":760,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.683188+0000","last_change":"2024-04-06T03:43:18.135428+0000","last_active":"2024-04-06T03:46:33.683188+0000","last_peered":"2024-04-06T03:46:33.683188+0000","last_clean":"2024-04-06T03:46:33.683188+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:46:33.683188+0000","last_undegraded":"2024-04-06T03:46:33.683188+0000","last_fullsized":"2024-04-06T03:46:33.683188+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:43:18.135327+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:43:18.135327+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":99,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.668513+0000","last_change":"2024-04-06T03:45:08.003956+0000","last_active":"2024-04-06T03:46:33.668513+0000","last_peered":"2024-04-06T03:46:33.668513+0000","last_clean":"2024-04-06T03:46:33.668513+0000","last_became_active":"2024-04-06T03:42:57.078887+0000","last_became_peered":"2024-04-06T03:42:57.078887+0000","last_unstale":"2024-04-06T03:46:33.668513+0000","last_undegraded":"2024-04-06T03:46:33.668513+0000","last_fullsized":"2024-04-06T03:46:33.668513+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:08.003828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:45:08.003828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775255+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":691,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678370+0000","last_change":"2024-04-06T03:45:25.274860+0000","last_active":"2024-04-06T03:46:33.678370+0000","last_peered":"2024-04-06T03:46:33.678370+0000","last_clean":"2024-04-06T03:46:33.678370+0000","last_became_active":"2024-04-06T03:42:57.112847+0000","last_became_peered":"2024-04-06T03:42:57.112847+0000","last_unstale":"2024-04-06T03:46:33.678370+0000","last_undegraded":"2024-04-06T03:46:33.678370+0000","last_fullsized":"2024-04-06T03:46:33.678370+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_clean_scrub_stamp":"2024-04-06T03:45:25.274778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:41.589312+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":729,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679769+0000","last_change":"2024-04-06T03:45:04.452779+0000","last_active":"2024-04-06T03:46:33.679769+0000","last_peered":"2024-04-06T03:46:33.679769+0000","last_clean":"2024-04-06T03:46:33.679769+0000","last_became_active":"2024-04-06T03:42:23.570089+0000","last_became_peered":"2024-04-06T03:42:23.570089+0000","last_unstale":"2024-04-06T03:46:33.679769+0000","last_undegraded":"2024-04-06T03:46:33.679769+0000","last_fullsized":"2024-04-06T03:46:33.679769+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:45:04.452660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.452660+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110628+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"307'46","reported_seq":731,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678200+0000","last_change":"2024-04-06T03:44:57.339512+0000","last_active":"2024-04-06T03:46:33.678200+0000","last_peered":"2024-04-06T03:46:33.678200+0000","last_clean":"2024-04-06T03:46:33.678200+0000","last_became_active":"2024-04-06T03:42:57.110510+0000","last_became_peered":"2024-04-06T03:42:57.110510+0000","last_unstale":"2024-04-06T03:46:33.678200+0000","last_undegraded":"2024-04-06T03:46:33.678200+0000","last_fullsized":"2024-04-06T03:46:33.678200+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'46","last_scrub_stamp":"2024-04-06T03:44:57.339353+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.339353+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230526+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677426+0000","last_change":"2024-04-06T03:45:05.101359+0000","last_active":"2024-04-06T03:46:33.677426+0000","last_peered":"2024-04-06T03:46:33.677426+0000","last_clean":"2024-04-06T03:46:33.677426+0000","last_became_active":"2024-04-06T03:42:57.092354+0000","last_became_peered":"2024-04-06T03:42:57.092354+0000","last_unstale":"2024-04-06T03:46:33.677426+0000","last_undegraded":"2024-04-06T03:46:33.677426+0000","last_fullsized":"2024-04-06T03:46:33.677426+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:05.101251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:05.101251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:38.143011+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":570,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678074+0000","last_change":"2024-04-06T03:45:02.133616+0000","last_active":"2024-04-06T03:46:33.678074+0000","last_peered":"2024-04-06T03:46:33.678074+0000","last_clean":"2024-04-06T03:46:33.678074+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:46:33.678074+0000","last_undegraded":"2024-04-06T03:46:33.678074+0000","last_fullsized":"2024-04-06T03:46:33.678074+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:02.132904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.132904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.977106+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":693,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679724+0000","last_change":"2024-04-06T03:43:13.319946+0000","last_active":"2024-04-06T03:46:33.679724+0000","last_peered":"2024-04-06T03:46:33.679724+0000","last_clean":"2024-04-06T03:46:33.679724+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:46:33.679724+0000","last_undegraded":"2024-04-06T03:46:33.679724+0000","last_fullsized":"2024-04-06T03:46:33.679724+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:13.319724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:43:13.319724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.099399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"307'36","reported_seq":718,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677044+0000","last_change":"2024-04-06T03:45:18.356128+0000","last_active":"2024-04-06T03:46:33.677044+0000","last_peered":"2024-04-06T03:46:33.677044+0000","last_clean":"2024-04-06T03:46:33.677044+0000","last_became_active":"2024-04-06T03:42:23.592623+0000","last_became_peered":"2024-04-06T03:42:23.592623+0000","last_unstale":"2024-04-06T03:46:33.677044+0000","last_undegraded":"2024-04-06T03:46:33.677044+0000","last_fullsized":"2024-04-06T03:46:33.677044+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'36","last_scrub_stamp":"2024-04-06T03:45:18.356021+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:45:18.356021+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":697,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669683+0000","last_change":"2024-04-06T03:43:04.325737+0000","last_active":"2024-04-06T03:46:33.669683+0000","last_peered":"2024-04-06T03:46:33.669683+0000","last_clean":"2024-04-06T03:46:33.669683+0000","last_became_active":"2024-04-06T03:42:23.577258+0000","last_became_peered":"2024-04-06T03:42:23.577258+0000","last_unstale":"2024-04-06T03:46:33.669683+0000","last_undegraded":"2024-04-06T03:46:33.669683+0000","last_fullsized":"2024-04-06T03:46:33.669683+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:04.325567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:04.325567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774429+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.681319+0000","last_change":"2024-04-06T03:43:23.225743+0000","last_active":"2024-04-06T03:46:33.681319+0000","last_peered":"2024-04-06T03:46:33.681319+0000","last_clean":"2024-04-06T03:46:33.681319+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:46:33.681319+0000","last_undegraded":"2024-04-06T03:46:33.681319+0000","last_fullsized":"2024-04-06T03:46:33.681319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.225643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.225643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":699,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669413+0000","last_change":"2024-04-06T03:45:01.585396+0000","last_active":"2024-04-06T03:46:33.669413+0000","last_peered":"2024-04-06T03:46:33.669413+0000","last_clean":"2024-04-06T03:46:33.669413+0000","last_became_active":"2024-04-06T03:42:23.588672+0000","last_became_peered":"2024-04-06T03:42:23.588672+0000","last_unstale":"2024-04-06T03:46:33.669413+0000","last_undegraded":"2024-04-06T03:46:33.669413+0000","last_fullsized":"2024-04-06T03:46:33.669413+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.585125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.585125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775216+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"307'39","reported_seq":783,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676449+0000","last_change":"2024-04-06T03:45:04.428891+0000","last_active":"2024-04-06T03:46:33.676449+0000","last_peered":"2024-04-06T03:46:33.676449+0000","last_clean":"2024-04-06T03:46:33.676449+0000","last_became_active":"2024-04-06T03:42:57.096631+0000","last_became_peered":"2024-04-06T03:42:57.096631+0000","last_unstale":"2024-04-06T03:46:33.676449+0000","last_undegraded":"2024-04-06T03:46:33.676449+0000","last_fullsized":"2024-04-06T03:46:33.676449+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:45:04.428726+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.428726+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":697,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.681629+0000","last_change":"2024-04-06T03:43:35.230118+0000","last_active":"2024-04-06T03:46:33.681629+0000","last_peered":"2024-04-06T03:46:33.681629+0000","last_clean":"2024-04-06T03:46:33.681629+0000","last_became_active":"2024-04-06T03:42:23.594112+0000","last_became_peered":"2024-04-06T03:42:23.594112+0000","last_unstale":"2024-04-06T03:46:33.681629+0000","last_undegraded":"2024-04-06T03:46:33.681629+0000","last_fullsized":"2024-04-06T03:46:33.681629+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:35.230016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:07.110775+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.230016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230281+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":96,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677129+0000","last_change":"2024-04-06T03:43:36.185334+0000","last_active":"2024-04-06T03:46:33.677129+0000","last_peered":"2024-04-06T03:46:33.677129+0000","last_clean":"2024-04-06T03:46:33.677129+0000","last_became_active":"2024-04-06T03:42:57.103063+0000","last_became_peered":"2024-04-06T03:42:57.103063+0000","last_unstale":"2024-04-06T03:46:33.677129+0000","last_undegraded":"2024-04-06T03:46:33.677129+0000","last_fullsized":"2024-04-06T03:46:33.677129+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:36.185042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:43:36.185042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":568,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676507+0000","last_change":"2024-04-06T03:45:26.399149+0000","last_active":"2024-04-06T03:46:33.676507+0000","last_peered":"2024-04-06T03:46:33.676507+0000","last_clean":"2024-04-06T03:46:33.676507+0000","last_became_active":"2024-04-06T03:42:23.593721+0000","last_became_peered":"2024-04-06T03:42:23.593721+0000","last_unstale":"2024-04-06T03:46:33.676507+0000","last_undegraded":"2024-04-06T03:46:33.676507+0000","last_fullsized":"2024-04-06T03:46:33.676507+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:26.399024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:26.399024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:52.434811+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"307'18","reported_seq":736,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.682097+0000","last_change":"2024-04-06T03:45:37.208824+0000","last_active":"2024-04-06T03:46:33.682097+0000","last_peered":"2024-04-06T03:46:33.682097+0000","last_clean":"2024-04-06T03:46:33.682097+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:46:33.682097+0000","last_undegraded":"2024-04-06T03:46:33.682097+0000","last_fullsized":"2024-04-06T03:46:33.682097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'18","last_scrub_stamp":"2024-04-06T03:45:37.208678+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.208678+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:45.514382+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":571,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676822+0000","last_change":"2024-04-06T03:43:44.166907+0000","last_active":"2024-04-06T03:46:33.676822+0000","last_peered":"2024-04-06T03:46:33.676822+0000","last_clean":"2024-04-06T03:46:33.676822+0000","last_became_active":"2024-04-06T03:42:57.096442+0000","last_became_peered":"2024-04-06T03:42:57.096442+0000","last_unstale":"2024-04-06T03:46:33.676822+0000","last_undegraded":"2024-04-06T03:46:33.676822+0000","last_fullsized":"2024-04-06T03:46:33.676822+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:44.166672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:44.166672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230336+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.680294+0000","last_change":"2024-04-06T03:43:26.134513+0000","last_active":"2024-04-06T03:46:33.680294+0000","last_peered":"2024-04-06T03:46:33.680294+0000","last_clean":"2024-04-06T03:46:33.680294+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:46:33.680294+0000","last_undegraded":"2024-04-06T03:46:33.680294+0000","last_fullsized":"2024-04-06T03:46:33.680294+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.134284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976908+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":708,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.668823+0000","last_change":"2024-04-06T03:43:28.171388+0000","last_active":"2024-04-06T03:46:33.668823+0000","last_peered":"2024-04-06T03:46:33.668823+0000","last_clean":"2024-04-06T03:46:33.668823+0000","last_became_active":"2024-04-06T03:42:23.577120+0000","last_became_peered":"2024-04-06T03:42:23.577120+0000","last_unstale":"2024-04-06T03:46:33.668823+0000","last_undegraded":"2024-04-06T03:46:33.668823+0000","last_fullsized":"2024-04-06T03:46:33.668823+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:43:28.171183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:28.171183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"307'32","reported_seq":752,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.668551+0000","last_change":"2024-04-06T03:45:17.921718+0000","last_active":"2024-04-06T03:46:33.668551+0000","last_peered":"2024-04-06T03:46:33.668551+0000","last_clean":"2024-04-06T03:46:33.668551+0000","last_became_active":"2024-04-06T03:42:57.070724+0000","last_became_peered":"2024-04-06T03:42:57.070724+0000","last_unstale":"2024-04-06T03:46:33.668551+0000","last_undegraded":"2024-04-06T03:46:33.668551+0000","last_fullsized":"2024-04-06T03:46:33.668551+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'32","last_scrub_stamp":"2024-04-06T03:45:17.921596+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.921596+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.757407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":684,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.680103+0000","last_change":"2024-04-06T03:45:11.392248+0000","last_active":"2024-04-06T03:46:33.680103+0000","last_peered":"2024-04-06T03:46:33.680103+0000","last_clean":"2024-04-06T03:46:33.680103+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:46:33.680103+0000","last_undegraded":"2024-04-06T03:46:33.680103+0000","last_fullsized":"2024-04-06T03:46:33.680103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:45:11.392164+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:41:29.127643+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.392164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110610+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"327'15","reported_seq":905,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677913+0000","last_change":"2024-04-06T03:45:36.499577+0000","last_active":"2024-04-06T03:46:33.677913+0000","last_peered":"2024-04-06T03:46:33.677913+0000","last_clean":"2024-04-06T03:46:33.677913+0000","last_became_active":"2024-04-06T03:42:23.594397+0000","last_became_peered":"2024-04-06T03:42:23.594397+0000","last_unstale":"2024-04-06T03:46:33.677913+0000","last_undegraded":"2024-04-06T03:46:33.677913+0000","last_fullsized":"2024-04-06T03:46:33.677913+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-04-06T03:45:36.499448+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:45:36.499448+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:43.246133+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":570,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676112+0000","last_change":"2024-04-06T03:45:31.513869+0000","last_active":"2024-04-06T03:46:33.676112+0000","last_peered":"2024-04-06T03:46:33.676112+0000","last_clean":"2024-04-06T03:46:33.676112+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:46:33.676112+0000","last_undegraded":"2024-04-06T03:46:33.676112+0000","last_fullsized":"2024-04-06T03:46:33.676112+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:31.513718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:45:31.513718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:57.374026+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"307'74","reported_seq":796,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669217+0000","last_change":"2024-04-06T03:45:13.928026+0000","last_active":"2024-04-06T03:46:33.669217+0000","last_peered":"2024-04-06T03:46:33.669217+0000","last_clean":"2024-04-06T03:46:33.669217+0000","last_became_active":"2024-04-06T03:42:57.093866+0000","last_became_peered":"2024-04-06T03:42:57.093866+0000","last_unstale":"2024-04-06T03:46:33.669217+0000","last_undegraded":"2024-04-06T03:46:33.669217+0000","last_fullsized":"2024-04-06T03:46:33.669217+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'74","last_scrub_stamp":"2024-04-06T03:45:13.927878+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:45:13.927878+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.013606+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":687,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671469+0000","last_change":"2024-04-06T03:43:34.209777+0000","last_active":"2024-04-06T03:46:33.671469+0000","last_peered":"2024-04-06T03:46:33.671469+0000","last_clean":"2024-04-06T03:46:33.671469+0000","last_became_active":"2024-04-06T03:42:23.588998+0000","last_became_peered":"2024-04-06T03:42:23.588998+0000","last_unstale":"2024-04-06T03:46:33.671469+0000","last_undegraded":"2024-04-06T03:46:33.671469+0000","last_fullsized":"2024-04-06T03:46:33.671469+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:34.209617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:43:34.209617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775137+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":697,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679443+0000","last_change":"2024-04-06T03:43:03.187581+0000","last_active":"2024-04-06T03:46:33.679443+0000","last_peered":"2024-04-06T03:46:33.679443+0000","last_clean":"2024-04-06T03:46:33.679443+0000","last_became_active":"2024-04-06T03:42:23.577555+0000","last_became_peered":"2024-04-06T03:42:23.577555+0000","last_unstale":"2024-04-06T03:46:33.679443+0000","last_undegraded":"2024-04-06T03:46:33.679443+0000","last_fullsized":"2024-04-06T03:46:33.679443+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:03.187375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:43:03.187375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.099425+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":584,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671842+0000","last_change":"2024-04-06T03:43:23.988357+0000","last_active":"2024-04-06T03:46:33.671842+0000","last_peered":"2024-04-06T03:46:33.671842+0000","last_clean":"2024-04-06T03:46:33.671842+0000","last_became_active":"2024-04-06T03:42:23.582180+0000","last_became_peered":"2024-04-06T03:42:23.582180+0000","last_unstale":"2024-04-06T03:46:33.671842+0000","last_undegraded":"2024-04-06T03:46:33.671842+0000","last_fullsized":"2024-04-06T03:46:33.671842+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.988141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.988141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"344'39","reported_seq":665,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679615+0000","last_change":"2024-04-06T03:43:22.294700+0000","last_active":"2024-04-06T03:46:33.679615+0000","last_peered":"2024-04-06T03:46:33.679615+0000","last_clean":"2024-04-06T03:46:33.679615+0000","last_became_active":"2024-04-06T03:42:57.093955+0000","last_became_peered":"2024-04-06T03:42:57.093955+0000","last_unstale":"2024-04-06T03:46:33.679615+0000","last_undegraded":"2024-04-06T03:46:33.679615+0000","last_fullsized":"2024-04-06T03:46:33.679615+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'39","last_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_deep_scrub":"344'39","last_deep_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_clean_scrub_stamp":"2024-04-06T03:43:22.294366+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110293+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":681,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678515+0000","last_change":"2024-04-06T03:43:08.294316+0000","last_active":"2024-04-06T03:46:33.678515+0000","last_peered":"2024-04-06T03:46:33.678515+0000","last_clean":"2024-04-06T03:46:33.678515+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:46:33.678515+0000","last_undegraded":"2024-04-06T03:46:33.678515+0000","last_fullsized":"2024-04-06T03:46:33.678515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:08.294075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:43:08.294075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.099371+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":569,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676281+0000","last_change":"2024-04-06T03:45:28.454818+0000","last_active":"2024-04-06T03:46:33.676281+0000","last_peered":"2024-04-06T03:46:33.676281+0000","last_clean":"2024-04-06T03:46:33.676281+0000","last_became_active":"2024-04-06T03:42:23.580807+0000","last_became_peered":"2024-04-06T03:42:23.580807+0000","last_unstale":"2024-04-06T03:46:33.676281+0000","last_undegraded":"2024-04-06T03:46:33.676281+0000","last_fullsized":"2024-04-06T03:46:33.676281+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:28.454731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:28.454731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.196174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":587,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.672074+0000","last_change":"2024-04-06T03:43:32.891066+0000","last_active":"2024-04-06T03:46:33.672074+0000","last_peered":"2024-04-06T03:46:33.672074+0000","last_clean":"2024-04-06T03:46:33.672074+0000","last_became_active":"2024-04-06T03:42:23.578658+0000","last_became_peered":"2024-04-06T03:42:23.578658+0000","last_unstale":"2024-04-06T03:46:33.672074+0000","last_undegraded":"2024-04-06T03:46:33.672074+0000","last_fullsized":"2024-04-06T03:46:33.672074+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:32.890813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.890813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"307'28","reported_seq":756,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.683137+0000","last_change":"2024-04-06T03:45:27.123773+0000","last_active":"2024-04-06T03:46:33.683137+0000","last_peered":"2024-04-06T03:46:33.683137+0000","last_clean":"2024-04-06T03:46:33.683137+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:46:33.683137+0000","last_undegraded":"2024-04-06T03:46:33.683137+0000","last_fullsized":"2024-04-06T03:46:33.683137+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'28","last_scrub_stamp":"2024-04-06T03:45:27.123648+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:45:27.123648+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:43.340104+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.668625+0000","last_change":"2024-04-06T03:43:29.215763+0000","last_active":"2024-04-06T03:46:33.668625+0000","last_peered":"2024-04-06T03:46:33.668625+0000","last_clean":"2024-04-06T03:46:33.668625+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:46:33.668625+0000","last_undegraded":"2024-04-06T03:46:33.668625+0000","last_fullsized":"2024-04-06T03:46:33.668625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:29.215509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:29.215509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774933+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":96,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671990+0000","last_change":"2024-04-06T03:45:15.944386+0000","last_active":"2024-04-06T03:46:33.671990+0000","last_peered":"2024-04-06T03:46:33.671990+0000","last_clean":"2024-04-06T03:46:33.671990+0000","last_became_active":"2024-04-06T03:42:57.085102+0000","last_became_peered":"2024-04-06T03:42:57.085102+0000","last_unstale":"2024-04-06T03:46:33.671990+0000","last_undegraded":"2024-04-06T03:46:33.671990+0000","last_fullsized":"2024-04-06T03:46:33.671990+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:15.944262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:15.944262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775380+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":95,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677719+0000","last_change":"2024-04-06T03:43:29.201914+0000","last_active":"2024-04-06T03:46:33.677719+0000","last_peered":"2024-04-06T03:46:33.677719+0000","last_clean":"2024-04-06T03:46:33.677719+0000","last_became_active":"2024-04-06T03:42:57.091715+0000","last_became_peered":"2024-04-06T03:42:57.091715+0000","last_unstale":"2024-04-06T03:46:33.677719+0000","last_undegraded":"2024-04-06T03:46:33.677719+0000","last_fullsized":"2024-04-06T03:46:33.677719+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:29.201580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:43:29.201580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"307'43","reported_seq":813,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678303+0000","last_change":"2024-04-06T03:43:26.202165+0000","last_active":"2024-04-06T03:46:33.678303+0000","last_peered":"2024-04-06T03:46:33.678303+0000","last_clean":"2024-04-06T03:46:33.678303+0000","last_became_active":"2024-04-06T03:42:23.592079+0000","last_became_peered":"2024-04-06T03:42:23.592079+0000","last_unstale":"2024-04-06T03:46:33.678303+0000","last_undegraded":"2024-04-06T03:46:33.678303+0000","last_fullsized":"2024-04-06T03:46:33.678303+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'43","last_scrub_stamp":"2024-04-06T03:43:26.202029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.202029+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":696,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677254+0000","last_change":"2024-04-06T03:43:45.957138+0000","last_active":"2024-04-06T03:46:33.677254+0000","last_peered":"2024-04-06T03:46:33.677254+0000","last_clean":"2024-04-06T03:46:33.677254+0000","last_became_active":"2024-04-06T03:42:57.089541+0000","last_became_peered":"2024-04-06T03:42:57.089541+0000","last_unstale":"2024-04-06T03:46:33.677254+0000","last_undegraded":"2024-04-06T03:46:33.677254+0000","last_fullsized":"2024-04-06T03:46:33.677254+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_clean_scrub_stamp":"2024-04-06T03:43:45.956977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":685,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671286+0000","last_change":"2024-04-06T03:43:26.163518+0000","last_active":"2024-04-06T03:46:33.671286+0000","last_peered":"2024-04-06T03:46:33.671286+0000","last_clean":"2024-04-06T03:46:33.671286+0000","last_became_active":"2024-04-06T03:42:57.079386+0000","last_became_peered":"2024-04-06T03:42:57.079386+0000","last_unstale":"2024-04-06T03:46:33.671286+0000","last_undegraded":"2024-04-06T03:46:33.671286+0000","last_fullsized":"2024-04-06T03:46:33.671286+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:26.163320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.163320+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.163320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":560,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670315+0000","last_change":"2024-04-06T03:43:41.985445+0000","last_active":"2024-04-06T03:46:33.670315+0000","last_peered":"2024-04-06T03:46:33.670315+0000","last_clean":"2024-04-06T03:46:33.670315+0000","last_became_active":"2024-04-06T03:42:23.589224+0000","last_became_peered":"2024-04-06T03:42:23.589224+0000","last_unstale":"2024-04-06T03:46:33.670315+0000","last_undegraded":"2024-04-06T03:46:33.670315+0000","last_fullsized":"2024-04-06T03:46:33.670315+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:41.985333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:43:41.985333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"307'27","reported_seq":735,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.680094+0000","last_change":"2024-04-06T03:43:38.248493+0000","last_active":"2024-04-06T03:46:33.680094+0000","last_peered":"2024-04-06T03:46:33.680094+0000","last_clean":"2024-04-06T03:46:33.680094+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:46:33.680094+0000","last_undegraded":"2024-04-06T03:46:33.680094+0000","last_fullsized":"2024-04-06T03:46:33.680094+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:43:38.248258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:38.248258+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":691,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678088+0000","last_change":"2024-04-06T03:44:58.475184+0000","last_active":"2024-04-06T03:46:33.678088+0000","last_peered":"2024-04-06T03:46:33.678088+0000","last_clean":"2024-04-06T03:46:33.678088+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:46:33.678088+0000","last_undegraded":"2024-04-06T03:46:33.678088+0000","last_fullsized":"2024-04-06T03:46:33.678088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:58.475083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.475083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230498+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":1008,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670673+0000","last_change":"2024-04-06T03:45:41.030902+0000","last_active":"2024-04-06T03:46:33.670673+0000","last_peered":"2024-04-06T03:46:33.670673+0000","last_clean":"2024-04-06T03:46:33.670673+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:46:33.670673+0000","last_undegraded":"2024-04-06T03:46:33.670673+0000","last_fullsized":"2024-04-06T03:46:33.670673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:45:41.030592+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.030592+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:04.211736+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":710,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.686775+0000","last_change":"2024-04-06T03:45:34.135652+0000","last_active":"2024-04-06T03:46:33.686775+0000","last_peered":"2024-04-06T03:46:33.686775+0000","last_clean":"2024-04-06T03:46:33.686775+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:46:33.686775+0000","last_undegraded":"2024-04-06T03:46:33.686775+0000","last_fullsized":"2024-04-06T03:46:33.686775+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.135479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.135479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:51.604553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"307'21","reported_seq":731,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.682176+0000","last_change":"2024-04-06T03:43:37.054237+0000","last_active":"2024-04-06T03:46:33.682176+0000","last_peered":"2024-04-06T03:46:33.682176+0000","last_clean":"2024-04-06T03:46:33.682176+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:46:33.682176+0000","last_undegraded":"2024-04-06T03:46:33.682176+0000","last_fullsized":"2024-04-06T03:46:33.682176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'21","last_scrub_stamp":"2024-04-06T03:43:37.053855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:37.053855+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":675,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.680523+0000","last_change":"2024-04-06T03:43:15.163263+0000","last_active":"2024-04-06T03:46:33.680523+0000","last_peered":"2024-04-06T03:46:33.680523+0000","last_clean":"2024-04-06T03:46:33.680523+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:46:33.680523+0000","last_undegraded":"2024-04-06T03:46:33.680523+0000","last_fullsized":"2024-04-06T03:46:33.680523+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:15.163100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:43:15.163100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976798+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"338'15","reported_seq":798,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:37.409080+0000","last_change":"2024-04-06T03:43:27.170941+0000","last_active":"2024-04-06T03:46:37.409080+0000","last_peered":"2024-04-06T03:46:37.409080+0000","last_clean":"2024-04-06T03:46:37.409080+0000","last_became_active":"2024-04-06T03:42:57.087361+0000","last_became_peered":"2024-04-06T03:42:57.087361+0000","last_unstale":"2024-04-06T03:46:37.409080+0000","last_undegraded":"2024-04-06T03:46:37.409080+0000","last_fullsized":"2024-04-06T03:46:37.409080+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'15","last_scrub_stamp":"2024-04-06T03:43:27.170713+0000","last_deep_scrub":"309'14","last_deep_scrub_stamp":"2024-04-06T03:41:21.025392+0000","last_clean_scrub_stamp":"2024-04-06T03:43:27.170713+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":560,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671932+0000","last_change":"2024-04-06T03:45:35.931813+0000","last_active":"2024-04-06T03:46:33.671932+0000","last_peered":"2024-04-06T03:46:33.671932+0000","last_clean":"2024-04-06T03:46:33.671932+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:46:33.671932+0000","last_undegraded":"2024-04-06T03:46:33.671932+0000","last_fullsized":"2024-04-06T03:46:33.671932+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:35.931720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:21.406614+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.931720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:55.171073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"307'32","reported_seq":738,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679201+0000","last_change":"2024-04-06T03:43:10.275147+0000","last_active":"2024-04-06T03:46:33.679201+0000","last_peered":"2024-04-06T03:46:33.679201+0000","last_clean":"2024-04-06T03:46:33.679201+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:46:33.679201+0000","last_undegraded":"2024-04-06T03:46:33.679201+0000","last_fullsized":"2024-04-06T03:46:33.679201+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'32","last_scrub_stamp":"2024-04-06T03:43:10.274736+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:43:10.274736+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":701,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678988+0000","last_change":"2024-04-06T03:44:59.520843+0000","last_active":"2024-04-06T03:46:33.678988+0000","last_peered":"2024-04-06T03:46:33.678988+0000","last_clean":"2024-04-06T03:46:33.678988+0000","last_became_active":"2024-04-06T03:42:23.574848+0000","last_became_peered":"2024-04-06T03:42:23.574848+0000","last_unstale":"2024-04-06T03:46:33.678988+0000","last_undegraded":"2024-04-06T03:46:33.678988+0000","last_fullsized":"2024-04-06T03:46:33.678988+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:59.520731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:44:59.520731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110658+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":560,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669976+0000","last_change":"2024-04-06T03:45:11.925553+0000","last_active":"2024-04-06T03:46:33.669976+0000","last_peered":"2024-04-06T03:46:33.669976+0000","last_clean":"2024-04-06T03:46:33.669976+0000","last_became_active":"2024-04-06T03:42:57.092555+0000","last_became_peered":"2024-04-06T03:42:57.092555+0000","last_unstale":"2024-04-06T03:46:33.669976+0000","last_undegraded":"2024-04-06T03:46:33.669976+0000","last_fullsized":"2024-04-06T03:46:33.669976+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:11.925422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.925422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775295+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":574,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677022+0000","last_change":"2024-04-06T03:45:29.469379+0000","last_active":"2024-04-06T03:46:33.677022+0000","last_peered":"2024-04-06T03:46:33.677022+0000","last_clean":"2024-04-06T03:46:33.677022+0000","last_became_active":"2024-04-06T03:42:57.109758+0000","last_became_peered":"2024-04-06T03:42:57.109758+0000","last_unstale":"2024-04-06T03:46:33.677022+0000","last_undegraded":"2024-04-06T03:46:33.677022+0000","last_fullsized":"2024-04-06T03:46:33.677022+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.469254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.469254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:57.458772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"307'14","reported_seq":689,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677720+0000","last_change":"2024-04-06T03:45:35.162259+0000","last_active":"2024-04-06T03:46:33.677720+0000","last_peered":"2024-04-06T03:46:33.677720+0000","last_clean":"2024-04-06T03:46:33.677720+0000","last_became_active":"2024-04-06T03:42:23.581903+0000","last_became_peered":"2024-04-06T03:42:23.581903+0000","last_unstale":"2024-04-06T03:46:33.677720+0000","last_undegraded":"2024-04-06T03:46:33.677720+0000","last_fullsized":"2024-04-06T03:46:33.677720+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'14","last_scrub_stamp":"2024-04-06T03:45:35.162112+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.162112+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:38.722934+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":709,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670842+0000","last_change":"2024-04-06T03:43:17.931745+0000","last_active":"2024-04-06T03:46:33.670842+0000","last_peered":"2024-04-06T03:46:33.670842+0000","last_clean":"2024-04-06T03:46:33.670842+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:46:33.670842+0000","last_undegraded":"2024-04-06T03:46:33.670842+0000","last_fullsized":"2024-04-06T03:46:33.670842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:43:17.931636+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:43:17.931636+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":693,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669775+0000","last_change":"2024-04-06T03:43:30.232514+0000","last_active":"2024-04-06T03:46:33.669775+0000","last_peered":"2024-04-06T03:46:33.669775+0000","last_clean":"2024-04-06T03:46:33.669775+0000","last_became_active":"2024-04-06T03:42:57.084986+0000","last_became_peered":"2024-04-06T03:42:57.084986+0000","last_unstale":"2024-04-06T03:46:33.669775+0000","last_undegraded":"2024-04-06T03:46:33.669775+0000","last_fullsized":"2024-04-06T03:46:33.669775+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:30.232267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:43:30.232267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774385+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":734,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670287+0000","last_change":"2024-04-06T03:45:29.043189+0000","last_active":"2024-04-06T03:46:33.670287+0000","last_peered":"2024-04-06T03:46:33.670287+0000","last_clean":"2024-04-06T03:46:33.670287+0000","last_became_active":"2024-04-06T03:42:57.071688+0000","last_became_peered":"2024-04-06T03:42:57.071688+0000","last_unstale":"2024-04-06T03:46:33.670287+0000","last_undegraded":"2024-04-06T03:46:33.670287+0000","last_fullsized":"2024-04-06T03:46:33.670287+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.043086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.043086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:43.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"307'39","reported_seq":799,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678558+0000","last_change":"2024-04-06T03:43:26.230289+0000","last_active":"2024-04-06T03:46:33.678558+0000","last_peered":"2024-04-06T03:46:33.678558+0000","last_clean":"2024-04-06T03:46:33.678558+0000","last_became_active":"2024-04-06T03:42:23.574886+0000","last_became_peered":"2024-04-06T03:42:23.574886+0000","last_unstale":"2024-04-06T03:46:33.678558+0000","last_undegraded":"2024-04-06T03:46:33.678558+0000","last_fullsized":"2024-04-06T03:46:33.678558+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:43:26.230137+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.230137+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"375'11291","reported_seq":11426,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:37.445987+0000","last_change":"2024-04-06T03:45:41.170863+0000","last_active":"2024-04-06T03:46:37.445987+0000","last_peered":"2024-04-06T03:46:37.445987+0000","last_clean":"2024-04-06T03:46:37.445987+0000","last_became_active":"2024-04-06T03:43:25.018581+0000","last_became_peered":"2024-04-06T03:43:25.018581+0000","last_unstale":"2024-04-06T03:46:37.445987+0000","last_undegraded":"2024-04-06T03:46:37.445987+0000","last_fullsized":"2024-04-06T03:46:37.445987+0000","mapping_epoch":345,"log_start":"364'6800","ondisk_log_start":"364'6800","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'8005","last_scrub_stamp":"2024-04-06T03:45:41.170736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.170736+0000","objects_scrubbed":5473,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:05.311420+0000","scrub_duration":2019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140181504,"num_objects":2139,"num_object_clones":0,"num_object_copies":6417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11291,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":691,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671374+0000","last_change":"2024-04-06T03:45:30.126413+0000","last_active":"2024-04-06T03:46:33.671374+0000","last_peered":"2024-04-06T03:46:33.671374+0000","last_clean":"2024-04-06T03:46:33.671374+0000","last_became_active":"2024-04-06T03:42:57.093752+0000","last_became_peered":"2024-04-06T03:42:57.093752+0000","last_unstale":"2024-04-06T03:46:33.671374+0000","last_undegraded":"2024-04-06T03:46:33.671374+0000","last_fullsized":"2024-04-06T03:46:33.671374+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.126246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.126246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.233350+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":722,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677555+0000","last_change":"2024-04-06T03:44:59.372004+0000","last_active":"2024-04-06T03:46:33.677555+0000","last_peered":"2024-04-06T03:46:33.677555+0000","last_clean":"2024-04-06T03:46:33.677555+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:46:33.677555+0000","last_undegraded":"2024-04-06T03:46:33.677555+0000","last_fullsized":"2024-04-06T03:46:33.677555+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:44:59.371923+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:44:59.371923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"307'39","reported_seq":787,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679409+0000","last_change":"2024-04-06T03:45:38.163802+0000","last_active":"2024-04-06T03:46:33.679409+0000","last_peered":"2024-04-06T03:46:33.679409+0000","last_clean":"2024-04-06T03:46:33.679409+0000","last_became_active":"2024-04-06T03:42:57.113147+0000","last_became_peered":"2024-04-06T03:42:57.113147+0000","last_unstale":"2024-04-06T03:46:33.679409+0000","last_undegraded":"2024-04-06T03:46:33.679409+0000","last_fullsized":"2024-04-06T03:46:33.679409+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:45:38.163676+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:45:38.163676+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:55.642070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":718,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670368+0000","last_change":"2024-04-06T03:43:39.845859+0000","last_active":"2024-04-06T03:46:33.670368+0000","last_peered":"2024-04-06T03:46:33.670368+0000","last_clean":"2024-04-06T03:46:33.670368+0000","last_became_active":"2024-04-06T03:42:57.074804+0000","last_became_peered":"2024-04-06T03:42:57.074804+0000","last_unstale":"2024-04-06T03:46:33.670368+0000","last_undegraded":"2024-04-06T03:46:33.670368+0000","last_fullsized":"2024-04-06T03:46:33.670368+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:39.845625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:43:39.845625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"375'11191","reported_seq":11335,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:35.341207+0000","last_change":"2024-04-06T03:45:27.058446+0000","last_active":"2024-04-06T03:46:35.341207+0000","last_peered":"2024-04-06T03:46:35.341207+0000","last_clean":"2024-04-06T03:46:35.341207+0000","last_became_active":"2024-04-06T03:43:25.130333+0000","last_became_peered":"2024-04-06T03:43:25.130333+0000","last_unstale":"2024-04-06T03:46:35.341207+0000","last_undegraded":"2024-04-06T03:46:35.341207+0000","last_fullsized":"2024-04-06T03:46:35.341207+0000","mapping_epoch":345,"log_start":"359'5700","ondisk_log_start":"359'5700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'7153","last_scrub_stamp":"2024-04-06T03:45:27.058114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:27.058114+0000","objects_scrubbed":6395,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.588818+0000","scrub_duration":2095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149422104,"num_objects":2281,"num_object_clones":0,"num_object_copies":6843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11191,"num_write_kb":431041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":699,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677311+0000","last_change":"2024-04-06T03:45:28.217075+0000","last_active":"2024-04-06T03:46:33.677311+0000","last_peered":"2024-04-06T03:46:33.677311+0000","last_clean":"2024-04-06T03:46:33.677311+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:46:33.677311+0000","last_undegraded":"2024-04-06T03:46:33.677311+0000","last_fullsized":"2024-04-06T03:46:33.677311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:28.216763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:45:28.216763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:38.143802+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":661,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677314+0000","last_change":"2024-04-06T03:43:34.098016+0000","last_active":"2024-04-06T03:46:33.677314+0000","last_peered":"2024-04-06T03:46:33.677314+0000","last_clean":"2024-04-06T03:46:33.677314+0000","last_became_active":"2024-04-06T03:42:57.101420+0000","last_became_peered":"2024-04-06T03:42:57.101420+0000","last_unstale":"2024-04-06T03:46:33.677314+0000","last_undegraded":"2024-04-06T03:46:33.677314+0000","last_fullsized":"2024-04-06T03:46:33.677314+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:43:34.097807+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:43:34.097807+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"307'49","reported_seq":891,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669967+0000","last_change":"2024-04-06T03:43:10.099251+0000","last_active":"2024-04-06T03:46:33.669967+0000","last_peered":"2024-04-06T03:46:33.669967+0000","last_clean":"2024-04-06T03:46:33.669967+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:46:33.669967+0000","last_undegraded":"2024-04-06T03:46:33.669967+0000","last_fullsized":"2024-04-06T03:46:33.669967+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'49","last_scrub_stamp":"2024-04-06T03:43:10.099011+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:43:10.099011+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":169,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677289+0000","last_change":"2024-04-06T03:43:32.211974+0000","last_active":"2024-04-06T03:46:33.677289+0000","last_peered":"2024-04-06T03:46:33.677289+0000","last_clean":"2024-04-06T03:46:33.677289+0000","last_became_active":"2024-04-06T03:42:57.119712+0000","last_became_peered":"2024-04-06T03:42:57.119712+0000","last_unstale":"2024-04-06T03:46:33.677289+0000","last_undegraded":"2024-04-06T03:46:33.677289+0000","last_fullsized":"2024-04-06T03:46:33.677289+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:43:32.211738+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.211738+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"327'13","reported_seq":1037,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.680147+0000","last_change":"2024-04-06T03:43:33.097564+0000","last_active":"2024-04-06T03:46:33.680147+0000","last_peered":"2024-04-06T03:46:33.680147+0000","last_clean":"2024-04-06T03:46:33.680147+0000","last_became_active":"2024-04-06T03:42:23.582156+0000","last_became_peered":"2024-04-06T03:42:23.582156+0000","last_unstale":"2024-04-06T03:46:33.680147+0000","last_undegraded":"2024-04-06T03:46:33.680147+0000","last_fullsized":"2024-04-06T03:46:33.680147+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'13","last_scrub_stamp":"2024-04-06T03:43:33.096089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:43:33.096089+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"375'10986","reported_seq":11056,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:38.270456+0000","last_change":"2024-04-06T03:43:25.029317+0000","last_active":"2024-04-06T03:46:38.270456+0000","last_peered":"2024-04-06T03:46:38.270456+0000","last_clean":"2024-04-06T03:46:38.270456+0000","last_became_active":"2024-04-06T03:43:25.028550+0000","last_became_peered":"2024-04-06T03:43:25.028550+0000","last_unstale":"2024-04-06T03:46:38.270456+0000","last_undegraded":"2024-04-06T03:46:38.270456+0000","last_fullsized":"2024-04-06T03:46:38.270456+0000","mapping_epoch":345,"log_start":"362'6100","ondisk_log_start":"362'6100","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.988294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136839168,"num_objects":2088,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":418368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":684,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670095+0000","last_change":"2024-04-06T03:45:29.035614+0000","last_active":"2024-04-06T03:46:33.670095+0000","last_peered":"2024-04-06T03:46:33.670095+0000","last_clean":"2024-04-06T03:46:33.670095+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:46:33.670095+0000","last_undegraded":"2024-04-06T03:46:33.670095+0000","last_fullsized":"2024-04-06T03:46:33.670095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.035484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.035484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:53.348498+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":94,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.682077+0000","last_change":"2024-04-06T03:43:25.155768+0000","last_active":"2024-04-06T03:46:33.682077+0000","last_peered":"2024-04-06T03:46:33.682077+0000","last_clean":"2024-04-06T03:46:33.682077+0000","last_became_active":"2024-04-06T03:42:57.089778+0000","last_became_peered":"2024-04-06T03:42:57.089778+0000","last_unstale":"2024-04-06T03:46:33.682077+0000","last_undegraded":"2024-04-06T03:46:33.682077+0000","last_fullsized":"2024-04-06T03:46:33.682077+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:25.155453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:25.155453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976771+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"307'54","reported_seq":832,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678786+0000","last_change":"2024-04-06T03:43:39.143819+0000","last_active":"2024-04-06T03:46:33.678786+0000","last_peered":"2024-04-06T03:46:33.678786+0000","last_clean":"2024-04-06T03:46:33.678786+0000","last_became_active":"2024-04-06T03:42:23.591518+0000","last_became_peered":"2024-04-06T03:42:23.591518+0000","last_unstale":"2024-04-06T03:46:33.678786+0000","last_undegraded":"2024-04-06T03:46:33.678786+0000","last_fullsized":"2024-04-06T03:46:33.678786+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'54","last_scrub_stamp":"2024-04-06T03:43:39.143539+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:43:39.143539+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230255+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":701,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677437+0000","last_change":"2024-04-06T03:43:38.985859+0000","last_active":"2024-04-06T03:46:33.677437+0000","last_peered":"2024-04-06T03:46:33.677437+0000","last_clean":"2024-04-06T03:46:33.677437+0000","last_became_active":"2024-04-06T03:42:57.097491+0000","last_became_peered":"2024-04-06T03:42:57.097491+0000","last_unstale":"2024-04-06T03:46:33.677437+0000","last_undegraded":"2024-04-06T03:46:33.677437+0000","last_fullsized":"2024-04-06T03:46:33.677437+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:38.985733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:43:38.985733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.977024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"375'11429","reported_seq":11500,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:38.252014+0000","last_change":"2024-04-06T03:43:25.131438+0000","last_active":"2024-04-06T03:46:38.252014+0000","last_peered":"2024-04-06T03:46:38.252014+0000","last_clean":"2024-04-06T03:46:38.252014+0000","last_became_active":"2024-04-06T03:43:25.130484+0000","last_became_peered":"2024-04-06T03:43:25.130484+0000","last_unstale":"2024-04-06T03:46:38.252014+0000","last_undegraded":"2024-04-06T03:46:38.252014+0000","last_fullsized":"2024-04-06T03:46:38.252014+0000","mapping_epoch":345,"log_start":"363'6500","ondisk_log_start":"363'6500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.989547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136904704,"num_objects":2089,"num_object_clones":0,"num_object_copies":6267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11429,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":703,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679652+0000","last_change":"2024-04-06T03:45:23.445396+0000","last_active":"2024-04-06T03:46:33.679652+0000","last_peered":"2024-04-06T03:46:33.679652+0000","last_clean":"2024-04-06T03:46:33.679652+0000","last_became_active":"2024-04-06T03:42:23.569974+0000","last_became_peered":"2024-04-06T03:42:23.569974+0000","last_unstale":"2024-04-06T03:46:33.679652+0000","last_undegraded":"2024-04-06T03:46:33.679652+0000","last_fullsized":"2024-04-06T03:46:33.679652+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:23.445295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:45:23.445295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110643+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"307'54","reported_seq":738,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670176+0000","last_change":"2024-04-06T03:45:14.950304+0000","last_active":"2024-04-06T03:46:33.670176+0000","last_peered":"2024-04-06T03:46:33.670176+0000","last_clean":"2024-04-06T03:46:33.670176+0000","last_became_active":"2024-04-06T03:42:57.096119+0000","last_became_peered":"2024-04-06T03:42:57.096119+0000","last_unstale":"2024-04-06T03:46:33.670176+0000","last_undegraded":"2024-04-06T03:46:33.670176+0000","last_fullsized":"2024-04-06T03:46:33.670176+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'54","last_scrub_stamp":"2024-04-06T03:45:14.950155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:14.950155+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294699+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":721,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670837+0000","last_change":"2024-04-06T03:43:38.853745+0000","last_active":"2024-04-06T03:46:33.670837+0000","last_peered":"2024-04-06T03:46:33.670837+0000","last_clean":"2024-04-06T03:46:33.670837+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:46:33.670837+0000","last_undegraded":"2024-04-06T03:46:33.670837+0000","last_fullsized":"2024-04-06T03:46:33.670837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:38.853463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:43:38.853463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":720,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.672160+0000","last_change":"2024-04-06T03:44:58.598242+0000","last_active":"2024-04-06T03:46:33.672160+0000","last_peered":"2024-04-06T03:46:33.672160+0000","last_clean":"2024-04-06T03:46:33.672160+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:46:33.672160+0000","last_undegraded":"2024-04-06T03:46:33.672160+0000","last_fullsized":"2024-04-06T03:46:33.672160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:58.598017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.598017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"375'10990","reported_seq":11061,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:38.131437+0000","last_change":"2024-04-06T03:43:25.131247+0000","last_active":"2024-04-06T03:46:38.131437+0000","last_peered":"2024-04-06T03:46:38.131437+0000","last_clean":"2024-04-06T03:46:38.131437+0000","last_became_active":"2024-04-06T03:43:25.130333+0000","last_became_peered":"2024-04-06T03:43:25.130333+0000","last_unstale":"2024-04-06T03:46:38.131437+0000","last_undegraded":"2024-04-06T03:46:38.131437+0000","last_fullsized":"2024-04-06T03:46:38.131437+0000","mapping_epoch":345,"log_start":"362'6100","ondisk_log_start":"362'6100","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4890,"log_dups_size":0,"ondisk_log_size":4890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.989035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10990,"num_write_kb":419072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":548,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669860+0000","last_change":"2024-04-06T03:43:21.171626+0000","last_active":"2024-04-06T03:46:33.669860+0000","last_peered":"2024-04-06T03:46:33.669860+0000","last_clean":"2024-04-06T03:46:33.669860+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:46:33.669860+0000","last_undegraded":"2024-04-06T03:46:33.669860+0000","last_fullsized":"2024-04-06T03:46:33.669860+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:21.171511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:21.171511+0000","last_clean_scrub_stamp":"2024-04-06T03:43:21.171511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774470+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"307'42","reported_seq":796,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671873+0000","last_change":"2024-04-06T03:45:33.957089+0000","last_active":"2024-04-06T03:46:33.671873+0000","last_peered":"2024-04-06T03:46:33.671873+0000","last_clean":"2024-04-06T03:46:33.671873+0000","last_became_active":"2024-04-06T03:42:23.588643+0000","last_became_peered":"2024-04-06T03:42:23.588643+0000","last_unstale":"2024-04-06T03:46:33.671873+0000","last_undegraded":"2024-04-06T03:46:33.671873+0000","last_fullsized":"2024-04-06T03:46:33.671873+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'42","last_scrub_stamp":"2024-04-06T03:45:33.956933+0000","last_deep_scrub":"307'42","last_deep_scrub_stamp":"2024-04-06T03:41:26.470096+0000","last_clean_scrub_stamp":"2024-04-06T03:45:33.956933+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:44.956289+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":587,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670567+0000","last_change":"2024-04-06T03:45:30.075970+0000","last_active":"2024-04-06T03:46:33.670567+0000","last_peered":"2024-04-06T03:46:33.670567+0000","last_clean":"2024-04-06T03:46:33.670567+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:46:33.670567+0000","last_undegraded":"2024-04-06T03:46:33.670567+0000","last_fullsized":"2024-04-06T03:46:33.670567+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.075882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.075882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:44.406473+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":985,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679053+0000","last_change":"2024-04-06T03:43:11.279746+0000","last_active":"2024-04-06T03:46:33.679053+0000","last_peered":"2024-04-06T03:46:33.679053+0000","last_clean":"2024-04-06T03:46:33.679053+0000","last_became_active":"2024-04-06T03:42:26.582173+0000","last_became_peered":"2024-04-06T03:42:26.582173+0000","last_unstale":"2024-04-06T03:46:33.679053+0000","last_undegraded":"2024-04-06T03:46:33.679053+0000","last_fullsized":"2024-04-06T03:46:33.679053+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:43:11.279588+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:43:11.279588+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110235+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679951+0000","last_change":"2024-04-06T03:45:12.385460+0000","last_active":"2024-04-06T03:46:33.679951+0000","last_peered":"2024-04-06T03:46:33.679951+0000","last_clean":"2024-04-06T03:46:33.679951+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:46:33.679951+0000","last_undegraded":"2024-04-06T03:46:33.679951+0000","last_fullsized":"2024-04-06T03:46:33.679951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:12.385092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:45:12.385092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110673+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"375'11357","reported_seq":11428,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:38.251796+0000","last_change":"2024-04-06T03:43:25.133000+0000","last_active":"2024-04-06T03:46:38.251796+0000","last_peered":"2024-04-06T03:46:38.251796+0000","last_clean":"2024-04-06T03:46:38.251796+0000","last_became_active":"2024-04-06T03:43:25.130714+0000","last_became_peered":"2024-04-06T03:43:25.130714+0000","last_unstale":"2024-04-06T03:46:38.251796+0000","last_undegraded":"2024-04-06T03:46:38.251796+0000","last_fullsized":"2024-04-06T03:46:38.251796+0000","mapping_epoch":345,"log_start":"363'6500","ondisk_log_start":"363'6500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.986196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":6429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11357,"num_write_kb":432000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":95,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670857+0000","last_change":"2024-04-06T03:43:42.965111+0000","last_active":"2024-04-06T03:46:33.670857+0000","last_peered":"2024-04-06T03:46:33.670857+0000","last_clean":"2024-04-06T03:46:33.670857+0000","last_became_active":"2024-04-06T03:42:57.093548+0000","last_became_peered":"2024-04-06T03:42:57.093548+0000","last_unstale":"2024-04-06T03:46:33.670857+0000","last_undegraded":"2024-04-06T03:46:33.670857+0000","last_fullsized":"2024-04-06T03:46:33.670857+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.964733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:37.513299+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.964733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775021+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"307'49","reported_seq":783,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.683608+0000","last_change":"2024-04-06T03:45:17.013250+0000","last_active":"2024-04-06T03:46:33.683608+0000","last_peered":"2024-04-06T03:46:33.683608+0000","last_clean":"2024-04-06T03:46:33.683608+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:46:33.683608+0000","last_undegraded":"2024-04-06T03:46:33.683608+0000","last_fullsized":"2024-04-06T03:46:33.683608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'49","last_scrub_stamp":"2024-04-06T03:45:17.013155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.013155+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:38.143862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":701,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679007+0000","last_change":"2024-04-06T03:43:42.231771+0000","last_active":"2024-04-06T03:46:33.679007+0000","last_peered":"2024-04-06T03:46:33.679007+0000","last_clean":"2024-04-06T03:46:33.679007+0000","last_became_active":"2024-04-06T03:42:57.093654+0000","last_became_peered":"2024-04-06T03:42:57.093654+0000","last_unstale":"2024-04-06T03:46:33.679007+0000","last_undegraded":"2024-04-06T03:46:33.679007+0000","last_fullsized":"2024-04-06T03:46:33.679007+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.231565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.231565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110487+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":106,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.683035+0000","last_change":"2024-04-06T03:45:32.117873+0000","last_active":"2024-04-06T03:46:33.683035+0000","last_peered":"2024-04-06T03:46:33.683035+0000","last_clean":"2024-04-06T03:46:33.683035+0000","last_became_active":"2024-04-06T03:42:57.098171+0000","last_became_peered":"2024-04-06T03:42:57.098171+0000","last_unstale":"2024-04-06T03:46:33.683035+0000","last_undegraded":"2024-04-06T03:46:33.683035+0000","last_fullsized":"2024-04-06T03:46:33.683035+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_clean_scrub_stamp":"2024-04-06T03:45:32.117767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:54.670260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"375'10888","reported_seq":11022,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:35.354586+0000","last_change":"2024-04-06T03:45:03.993018+0000","last_active":"2024-04-06T03:46:35.354586+0000","last_peered":"2024-04-06T03:46:35.354586+0000","last_clean":"2024-04-06T03:46:35.354586+0000","last_became_active":"2024-04-06T03:43:25.027401+0000","last_became_peered":"2024-04-06T03:43:25.027401+0000","last_unstale":"2024-04-06T03:46:35.354586+0000","last_undegraded":"2024-04-06T03:46:35.354586+0000","last_fullsized":"2024-04-06T03:46:35.354586+0000","mapping_epoch":345,"log_start":"359'5400","ondisk_log_start":"359'5400","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'5530","last_scrub_stamp":"2024-04-06T03:45:03.992912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:03.992912+0000","objects_scrubbed":5518,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.295456+0000","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149291008,"num_objects":2278,"num_object_clones":0,"num_object_copies":6834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":699,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670990+0000","last_change":"2024-04-06T03:45:01.597564+0000","last_active":"2024-04-06T03:46:33.670990+0000","last_peered":"2024-04-06T03:46:33.670990+0000","last_clean":"2024-04-06T03:46:33.670990+0000","last_became_active":"2024-04-06T03:42:57.095020+0000","last_became_peered":"2024-04-06T03:42:57.095020+0000","last_unstale":"2024-04-06T03:46:33.670990+0000","last_undegraded":"2024-04-06T03:46:33.670990+0000","last_fullsized":"2024-04-06T03:46:33.670990+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.597125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.597125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294714+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":703,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679630+0000","last_change":"2024-04-06T03:45:34.168883+0000","last_active":"2024-04-06T03:46:33.679630+0000","last_peered":"2024-04-06T03:46:33.679630+0000","last_clean":"2024-04-06T03:46:33.679630+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:46:33.679630+0000","last_undegraded":"2024-04-06T03:46:33.679630+0000","last_fullsized":"2024-04-06T03:46:33.679630+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.165081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.165081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.980911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"338'64","reported_seq":1034,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670775+0000","last_change":"2024-04-06T03:43:27.916265+0000","last_active":"2024-04-06T03:46:33.670775+0000","last_peered":"2024-04-06T03:46:33.670775+0000","last_clean":"2024-04-06T03:46:33.670775+0000","last_became_active":"2024-04-06T03:42:57.101642+0000","last_became_peered":"2024-04-06T03:42:57.101642+0000","last_unstale":"2024-04-06T03:46:33.670775+0000","last_undegraded":"2024-04-06T03:46:33.670775+0000","last_fullsized":"2024-04-06T03:46:33.670775+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'64","last_scrub_stamp":"2024-04-06T03:43:27.916059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:27.916059+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":578,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.681976+0000","last_change":"2024-04-06T03:45:07.981010+0000","last_active":"2024-04-06T03:46:33.681976+0000","last_peered":"2024-04-06T03:46:33.681976+0000","last_clean":"2024-04-06T03:46:33.681976+0000","last_became_active":"2024-04-06T03:42:23.569142+0000","last_became_peered":"2024-04-06T03:42:23.569142+0000","last_unstale":"2024-04-06T03:46:33.681976+0000","last_undegraded":"2024-04-06T03:46:33.681976+0000","last_fullsized":"2024-04-06T03:46:33.681976+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:07.980779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:07.980779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.977132+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"375'10864","reported_seq":10935,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:34.605743+0000","last_change":"2024-04-06T03:43:25.029080+0000","last_active":"2024-04-06T03:46:34.605743+0000","last_peered":"2024-04-06T03:46:34.605743+0000","last_clean":"2024-04-06T03:46:34.605743+0000","last_became_active":"2024-04-06T03:43:25.028236+0000","last_became_peered":"2024-04-06T03:43:25.028236+0000","last_unstale":"2024-04-06T03:46:34.605743+0000","last_undegraded":"2024-04-06T03:46:34.605743+0000","last_fullsized":"2024-04-06T03:46:34.605743+0000","mapping_epoch":345,"log_start":"364'6600","ondisk_log_start":"364'6600","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149159936,"num_objects":2276,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10864,"num_write_kb":420480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":693,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676499+0000","last_change":"2024-04-06T03:45:41.425674+0000","last_active":"2024-04-06T03:46:33.676499+0000","last_peered":"2024-04-06T03:46:33.676499+0000","last_clean":"2024-04-06T03:46:33.676499+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:46:33.676499+0000","last_undegraded":"2024-04-06T03:46:33.676499+0000","last_fullsized":"2024-04-06T03:46:33.676499+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:41.425546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.425546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:00.450786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":107,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678266+0000","last_change":"2024-04-06T03:43:29.179137+0000","last_active":"2024-04-06T03:46:33.678266+0000","last_peered":"2024-04-06T03:46:33.678266+0000","last_clean":"2024-04-06T03:46:33.678266+0000","last_became_active":"2024-04-06T03:42:57.116773+0000","last_became_peered":"2024-04-06T03:42:57.116773+0000","last_unstale":"2024-04-06T03:46:33.678266+0000","last_undegraded":"2024-04-06T03:46:33.678266+0000","last_fullsized":"2024-04-06T03:46:33.678266+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:43:29.178882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:29.178882+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"307'27","reported_seq":743,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669464+0000","last_change":"2024-04-06T03:45:36.917456+0000","last_active":"2024-04-06T03:46:33.669464+0000","last_peered":"2024-04-06T03:46:33.669464+0000","last_clean":"2024-04-06T03:46:33.669464+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:46:33.669464+0000","last_undegraded":"2024-04-06T03:46:33.669464+0000","last_fullsized":"2024-04-06T03:46:33.669464+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:45:36.917321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:36.917321+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:02.293191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2206,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:36.950206+0000","last_change":"2024-04-06T03:43:02.082381+0000","last_active":"2024-04-06T03:46:36.950206+0000","last_peered":"2024-04-06T03:46:36.950206+0000","last_clean":"2024-04-06T03:46:36.950206+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:46:36.950206+0000","last_undegraded":"2024-04-06T03:46:36.950206+0000","last_fullsized":"2024-04-06T03:46:36.950206+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:43:02.082118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:43:02.082118+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1513,"num_read_kb":1508,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":689,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676376+0000","last_change":"2024-04-06T03:43:18.143508+0000","last_active":"2024-04-06T03:46:33.676376+0000","last_peered":"2024-04-06T03:46:33.676376+0000","last_clean":"2024-04-06T03:46:33.676376+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:46:33.676376+0000","last_undegraded":"2024-04-06T03:46:33.676376+0000","last_fullsized":"2024-04-06T03:46:33.676376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:18.143371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_clean_scrub_stamp":"2024-04-06T03:43:18.143371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"375'10999","reported_seq":11070,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:35.014853+0000","last_change":"2024-04-06T03:43:25.025446+0000","last_active":"2024-04-06T03:46:35.014853+0000","last_peered":"2024-04-06T03:46:35.014853+0000","last_clean":"2024-04-06T03:46:35.014853+0000","last_became_active":"2024-04-06T03:43:25.024471+0000","last_became_peered":"2024-04-06T03:43:25.024471+0000","last_unstale":"2024-04-06T03:46:35.014853+0000","last_undegraded":"2024-04-06T03:46:35.014853+0000","last_fullsized":"2024-04-06T03:46:35.014853+0000","mapping_epoch":345,"log_start":"363'6400","ondisk_log_start":"363'6400","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144900096,"num_objects":2211,"num_object_clones":0,"num_object_copies":6633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":695,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678320+0000","last_change":"2024-04-06T03:43:23.337277+0000","last_active":"2024-04-06T03:46:33.678320+0000","last_peered":"2024-04-06T03:46:33.678320+0000","last_clean":"2024-04-06T03:46:33.678320+0000","last_became_active":"2024-04-06T03:42:23.562715+0000","last_became_peered":"2024-04-06T03:42:23.562715+0000","last_unstale":"2024-04-06T03:46:33.678320+0000","last_undegraded":"2024-04-06T03:46:33.678320+0000","last_fullsized":"2024-04-06T03:46:33.678320+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.337146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.337146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110321+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":569,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679324+0000","last_change":"2024-04-06T03:43:14.348107+0000","last_active":"2024-04-06T03:46:33.679324+0000","last_peered":"2024-04-06T03:46:33.679324+0000","last_clean":"2024-04-06T03:46:33.679324+0000","last_became_active":"2024-04-06T03:42:23.567776+0000","last_became_peered":"2024-04-06T03:42:23.567776+0000","last_unstale":"2024-04-06T03:46:33.679324+0000","last_undegraded":"2024-04-06T03:46:33.679324+0000","last_fullsized":"2024-04-06T03:46:33.679324+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:14.347877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:43:14.347877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.099479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"312'42","reported_seq":786,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678102+0000","last_change":"2024-04-06T03:45:04.006917+0000","last_active":"2024-04-06T03:46:33.678102+0000","last_peered":"2024-04-06T03:46:33.678102+0000","last_clean":"2024-04-06T03:46:33.678102+0000","last_became_active":"2024-04-06T03:42:57.102077+0000","last_became_peered":"2024-04-06T03:42:57.102077+0000","last_unstale":"2024-04-06T03:46:33.678102+0000","last_undegraded":"2024-04-06T03:46:33.678102+0000","last_fullsized":"2024-04-06T03:46:33.678102+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'42","last_scrub_stamp":"2024-04-06T03:45:04.006799+0000","last_deep_scrub":"312'42","last_deep_scrub_stamp":"2024-04-06T03:41:44.156017+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.006799+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.977080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":710,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676559+0000","last_change":"2024-04-06T03:43:19.191651+0000","last_active":"2024-04-06T03:46:33.676559+0000","last_peered":"2024-04-06T03:46:33.676559+0000","last_clean":"2024-04-06T03:46:33.676559+0000","last_became_active":"2024-04-06T03:42:23.576890+0000","last_became_peered":"2024-04-06T03:42:23.576890+0000","last_unstale":"2024-04-06T03:46:33.676559+0000","last_undegraded":"2024-04-06T03:46:33.676559+0000","last_fullsized":"2024-04-06T03:46:33.676559+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:19.191417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:19.191417+0000","last_clean_scrub_stamp":"2024-04-06T03:43:19.191417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.229995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"338'13","reported_seq":1020,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669033+0000","last_change":"2024-04-06T03:45:20.921396+0000","last_active":"2024-04-06T03:46:33.669033+0000","last_peered":"2024-04-06T03:46:33.669033+0000","last_clean":"2024-04-06T03:46:33.669033+0000","last_became_active":"2024-04-06T03:42:57.094766+0000","last_became_peered":"2024-04-06T03:42:57.094766+0000","last_unstale":"2024-04-06T03:46:33.669033+0000","last_undegraded":"2024-04-06T03:46:33.669033+0000","last_fullsized":"2024-04-06T03:46:33.669033+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-04-06T03:45:20.921243+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:41:18.436223+0000","last_clean_scrub_stamp":"2024-04-06T03:45:20.921243+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:40.946488+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"375'11187","reported_seq":11320,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:38.057170+0000","last_change":"2024-04-06T03:44:58.331410+0000","last_active":"2024-04-06T03:46:38.057170+0000","last_peered":"2024-04-06T03:46:38.057170+0000","last_clean":"2024-04-06T03:46:38.057170+0000","last_became_active":"2024-04-06T03:43:25.130935+0000","last_became_peered":"2024-04-06T03:43:25.130935+0000","last_unstale":"2024-04-06T03:46:38.057170+0000","last_undegraded":"2024-04-06T03:46:38.057170+0000","last_fullsized":"2024-04-06T03:46:38.057170+0000","mapping_epoch":345,"log_start":"362'6300","ondisk_log_start":"362'6300","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'5388","last_scrub_stamp":"2024-04-06T03:44:58.331222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.331222+0000","objects_scrubbed":5324,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.996955+0000","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138215424,"num_objects":2109,"num_object_clones":0,"num_object_copies":6327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":425472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":696,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670722+0000","last_change":"2024-04-06T03:45:42.012146+0000","last_active":"2024-04-06T03:46:33.670722+0000","last_peered":"2024-04-06T03:46:33.670722+0000","last_clean":"2024-04-06T03:46:33.670722+0000","last_became_active":"2024-04-06T03:42:23.590773+0000","last_became_peered":"2024-04-06T03:42:23.590773+0000","last_unstale":"2024-04-06T03:46:33.670722+0000","last_undegraded":"2024-04-06T03:46:33.670722+0000","last_fullsized":"2024-04-06T03:46:33.670722+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:42.012040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:45:42.012040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:46.339282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":575,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678001+0000","last_change":"2024-04-06T03:45:35.465666+0000","last_active":"2024-04-06T03:46:33.678001+0000","last_peered":"2024-04-06T03:46:33.678001+0000","last_clean":"2024-04-06T03:46:33.678001+0000","last_became_active":"2024-04-06T03:42:23.595618+0000","last_became_peered":"2024-04-06T03:42:23.595618+0000","last_unstale":"2024-04-06T03:46:33.678001+0000","last_undegraded":"2024-04-06T03:46:33.678001+0000","last_fullsized":"2024-04-06T03:46:33.678001+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:35.465468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.465468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:47.618305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":700,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.682668+0000","last_change":"2024-04-06T03:45:30.229007+0000","last_active":"2024-04-06T03:46:33.682668+0000","last_peered":"2024-04-06T03:46:33.682668+0000","last_clean":"2024-04-06T03:46:33.682668+0000","last_became_active":"2024-04-06T03:42:57.115680+0000","last_became_peered":"2024-04-06T03:42:57.115680+0000","last_unstale":"2024-04-06T03:46:33.682668+0000","last_undegraded":"2024-04-06T03:46:33.682668+0000","last_fullsized":"2024-04-06T03:46:33.682668+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.228890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.228890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:46.685319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"307'43","reported_seq":764,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676465+0000","last_change":"2024-04-06T03:43:24.255032+0000","last_active":"2024-04-06T03:46:33.676465+0000","last_peered":"2024-04-06T03:46:33.676465+0000","last_clean":"2024-04-06T03:46:33.676465+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:46:33.676465+0000","last_undegraded":"2024-04-06T03:46:33.676465+0000","last_fullsized":"2024-04-06T03:46:33.676465+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'43","last_scrub_stamp":"2024-04-06T03:43:24.254836+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:43:24.254836+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"375'11078","reported_seq":11149,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:38.142705+0000","last_change":"2024-04-06T03:43:25.131506+0000","last_active":"2024-04-06T03:46:38.142705+0000","last_peered":"2024-04-06T03:46:38.142705+0000","last_clean":"2024-04-06T03:46:38.142705+0000","last_became_active":"2024-04-06T03:43:25.130566+0000","last_became_peered":"2024-04-06T03:43:25.130566+0000","last_unstale":"2024-04-06T03:46:38.142705+0000","last_undegraded":"2024-04-06T03:46:38.142705+0000","last_fullsized":"2024-04-06T03:46:38.142705+0000","mapping_epoch":345,"log_start":"362'6200","ondisk_log_start":"362'6200","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:42.989620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136052736,"num_objects":2076,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":420928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":528,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677146+0000","last_change":"2024-04-06T03:43:42.121160+0000","last_active":"2024-04-06T03:46:33.677146+0000","last_peered":"2024-04-06T03:46:33.677146+0000","last_clean":"2024-04-06T03:46:33.677146+0000","last_became_active":"2024-04-06T03:42:57.078094+0000","last_became_peered":"2024-04-06T03:42:57.078094+0000","last_unstale":"2024-04-06T03:46:33.677146+0000","last_undegraded":"2024-04-06T03:46:33.677146+0000","last_fullsized":"2024-04-06T03:46:33.677146+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.121037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.121037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.977050+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":1016,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678364+0000","last_change":"2024-04-06T03:43:30.210944+0000","last_active":"2024-04-06T03:46:33.678364+0000","last_peered":"2024-04-06T03:46:33.678364+0000","last_clean":"2024-04-06T03:46:33.678364+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:46:33.678364+0000","last_undegraded":"2024-04-06T03:46:33.678364+0000","last_fullsized":"2024-04-06T03:46:33.678364+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:43:30.210531+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:35:33.269156+0000","last_clean_scrub_stamp":"2024-04-06T03:43:30.210531+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230137+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":95,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678577+0000","last_change":"2024-04-06T03:43:24.206838+0000","last_active":"2024-04-06T03:46:33.678577+0000","last_peered":"2024-04-06T03:46:33.678577+0000","last_clean":"2024-04-06T03:46:33.678577+0000","last_became_active":"2024-04-06T03:42:57.092193+0000","last_became_peered":"2024-04-06T03:42:57.092193+0000","last_unstale":"2024-04-06T03:46:33.678577+0000","last_undegraded":"2024-04-06T03:46:33.678577+0000","last_fullsized":"2024-04-06T03:46:33.678577+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:24.206535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:24.206535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"307'51","reported_seq":795,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.668799+0000","last_change":"2024-04-06T03:42:59.058217+0000","last_active":"2024-04-06T03:46:33.668799+0000","last_peered":"2024-04-06T03:46:33.668799+0000","last_clean":"2024-04-06T03:46:33.668799+0000","last_became_active":"2024-04-06T03:42:57.113030+0000","last_became_peered":"2024-04-06T03:42:57.113030+0000","last_unstale":"2024-04-06T03:46:33.668799+0000","last_undegraded":"2024-04-06T03:46:33.668799+0000","last_fullsized":"2024-04-06T03:46:33.668799+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'51","last_scrub_stamp":"2024-04-06T03:42:59.057819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:42:59.057819+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"375'10970","reported_seq":11041,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:34.599095+0000","last_change":"2024-04-06T03:43:25.129974+0000","last_active":"2024-04-06T03:46:34.599095+0000","last_peered":"2024-04-06T03:46:34.599095+0000","last_clean":"2024-04-06T03:46:34.599095+0000","last_became_active":"2024-04-06T03:43:25.129076+0000","last_became_peered":"2024-04-06T03:43:25.129076+0000","last_unstale":"2024-04-06T03:46:34.599095+0000","last_undegraded":"2024-04-06T03:46:34.599095+0000","last_fullsized":"2024-04-06T03:46:34.599095+0000","mapping_epoch":345,"log_start":"364'6700","ondisk_log_start":"364'6700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156499968,"num_objects":2388,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10970,"num_write_kb":427456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":698,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.681532+0000","last_change":"2024-04-06T03:45:37.526151+0000","last_active":"2024-04-06T03:46:33.681532+0000","last_peered":"2024-04-06T03:46:33.681532+0000","last_clean":"2024-04-06T03:46:33.681532+0000","last_became_active":"2024-04-06T03:42:57.102442+0000","last_became_peered":"2024-04-06T03:42:57.102442+0000","last_unstale":"2024-04-06T03:46:33.681532+0000","last_undegraded":"2024-04-06T03:46:33.681532+0000","last_fullsized":"2024-04-06T03:46:33.681532+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:37.525986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.525986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:49.532812+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":1040,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671805+0000","last_change":"2024-04-06T03:44:57.608200+0000","last_active":"2024-04-06T03:46:33.671805+0000","last_peered":"2024-04-06T03:46:33.671805+0000","last_clean":"2024-04-06T03:46:33.671805+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:46:33.671805+0000","last_undegraded":"2024-04-06T03:46:33.671805+0000","last_fullsized":"2024-04-06T03:46:33.671805+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:44:57.607946+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.607946+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294626+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":579,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671723+0000","last_change":"2024-04-06T03:43:32.152585+0000","last_active":"2024-04-06T03:46:33.671723+0000","last_peered":"2024-04-06T03:46:33.671723+0000","last_clean":"2024-04-06T03:46:33.671723+0000","last_became_active":"2024-04-06T03:42:57.078347+0000","last_became_peered":"2024-04-06T03:42:57.078347+0000","last_unstale":"2024-04-06T03:46:33.671723+0000","last_undegraded":"2024-04-06T03:46:33.671723+0000","last_fullsized":"2024-04-06T03:46:33.671723+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:32.152478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:23.908290+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.152478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.775060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"307'27","reported_seq":738,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671176+0000","last_change":"2024-04-06T03:45:10.955262+0000","last_active":"2024-04-06T03:46:33.671176+0000","last_peered":"2024-04-06T03:46:33.671176+0000","last_clean":"2024-04-06T03:46:33.671176+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:46:33.671176+0000","last_undegraded":"2024-04-06T03:46:33.671176+0000","last_fullsized":"2024-04-06T03:46:33.671176+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:45:10.955163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:10.955163+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:38.700162+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"375'10769","reported_seq":10838,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:35.359572+0000","last_change":"2024-04-06T03:43:25.029297+0000","last_active":"2024-04-06T03:46:35.359572+0000","last_peered":"2024-04-06T03:46:35.359572+0000","last_clean":"2024-04-06T03:46:35.359572+0000","last_became_active":"2024-04-06T03:43:25.028867+0000","last_became_peered":"2024-04-06T03:43:25.028867+0000","last_unstale":"2024-04-06T03:46:35.359572+0000","last_undegraded":"2024-04-06T03:46:35.359572+0000","last_fullsized":"2024-04-06T03:46:35.359572+0000","mapping_epoch":345,"log_start":"359'5300","ondisk_log_start":"359'5300","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":5469,"log_dups_size":0,"ondisk_log_size":5469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144375808,"num_objects":2203,"num_object_clones":0,"num_object_copies":6609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":415104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":716,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.672069+0000","last_change":"2024-04-06T03:45:17.270096+0000","last_active":"2024-04-06T03:46:33.672069+0000","last_peered":"2024-04-06T03:46:33.672069+0000","last_clean":"2024-04-06T03:46:33.672069+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:46:33.672069+0000","last_undegraded":"2024-04-06T03:46:33.672069+0000","last_fullsized":"2024-04-06T03:46:33.672069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:17.269763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.269763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294681+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":525,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678908+0000","last_change":"2024-04-06T03:45:02.458926+0000","last_active":"2024-04-06T03:46:33.678908+0000","last_peered":"2024-04-06T03:46:33.678908+0000","last_clean":"2024-04-06T03:46:33.678908+0000","last_became_active":"2024-04-06T03:42:57.110057+0000","last_became_peered":"2024-04-06T03:42:57.110057+0000","last_unstale":"2024-04-06T03:46:33.678908+0000","last_undegraded":"2024-04-06T03:46:33.678908+0000","last_fullsized":"2024-04-06T03:46:33.678908+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:02.458841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.458841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":702,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678828+0000","last_change":"2024-04-06T03:45:24.308229+0000","last_active":"2024-04-06T03:46:33.678828+0000","last_peered":"2024-04-06T03:46:33.678828+0000","last_clean":"2024-04-06T03:46:33.678828+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:46:33.678828+0000","last_undegraded":"2024-04-06T03:46:33.678828+0000","last_fullsized":"2024-04-06T03:46:33.678828+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:24.308115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:45:24.308115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:48.379923+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"307'57","reported_seq":759,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.677182+0000","last_change":"2024-04-06T03:43:43.053137+0000","last_active":"2024-04-06T03:46:33.677182+0000","last_peered":"2024-04-06T03:46:33.677182+0000","last_clean":"2024-04-06T03:46:33.677182+0000","last_became_active":"2024-04-06T03:42:57.110277+0000","last_became_peered":"2024-04-06T03:42:57.110277+0000","last_unstale":"2024-04-06T03:46:33.677182+0000","last_undegraded":"2024-04-06T03:46:33.677182+0000","last_fullsized":"2024-04-06T03:46:33.677182+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'57","last_scrub_stamp":"2024-04-06T03:43:43.052879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:43.052879+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230389+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"375'10931","reported_seq":11002,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:34.587644+0000","last_change":"2024-04-06T03:43:25.132069+0000","last_active":"2024-04-06T03:46:34.587644+0000","last_peered":"2024-04-06T03:46:34.587644+0000","last_clean":"2024-04-06T03:46:34.587644+0000","last_became_active":"2024-04-06T03:43:25.131461+0000","last_became_peered":"2024-04-06T03:43:25.131461+0000","last_unstale":"2024-04-06T03:46:34.587644+0000","last_undegraded":"2024-04-06T03:46:34.587644+0000","last_fullsized":"2024-04-06T03:46:34.587644+0000","mapping_epoch":345,"log_start":"364'6700","ondisk_log_start":"364'6700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4231,"log_dups_size":0,"ondisk_log_size":4231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147652608,"num_objects":2253,"num_object_clones":0,"num_object_copies":6759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10931,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":682,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678884+0000","last_change":"2024-04-06T03:45:14.363616+0000","last_active":"2024-04-06T03:46:33.678884+0000","last_peered":"2024-04-06T03:46:33.678884+0000","last_clean":"2024-04-06T03:46:33.678884+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:46:33.678884+0000","last_undegraded":"2024-04-06T03:46:33.678884+0000","last_fullsized":"2024-04-06T03:46:33.678884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:14.363468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:14.363468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:33.831237+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":720,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670477+0000","last_change":"2024-04-06T03:45:34.011396+0000","last_active":"2024-04-06T03:46:33.670477+0000","last_peered":"2024-04-06T03:46:33.670477+0000","last_clean":"2024-04-06T03:46:33.670477+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:46:33.670477+0000","last_undegraded":"2024-04-06T03:46:33.670477+0000","last_fullsized":"2024-04-06T03:46:33.670477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.011285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:33.770813+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.011285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:45.106276+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":720,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.670674+0000","last_change":"2024-04-06T03:45:10.895775+0000","last_active":"2024-04-06T03:46:33.670674+0000","last_peered":"2024-04-06T03:46:33.670674+0000","last_clean":"2024-04-06T03:46:33.670674+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:46:33.670674+0000","last_undegraded":"2024-04-06T03:46:33.670674+0000","last_fullsized":"2024-04-06T03:46:33.670674+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:10.895208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:45:10.895208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.672363+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"307'53","reported_seq":816,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.680056+0000","last_change":"2024-04-06T03:45:02.412015+0000","last_active":"2024-04-06T03:46:33.680056+0000","last_peered":"2024-04-06T03:46:33.680056+0000","last_clean":"2024-04-06T03:46:33.680056+0000","last_became_active":"2024-04-06T03:42:23.570628+0000","last_became_peered":"2024-04-06T03:42:23.570628+0000","last_unstale":"2024-04-06T03:46:33.680056+0000","last_undegraded":"2024-04-06T03:46:33.680056+0000","last_fullsized":"2024-04-06T03:46:33.680056+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'53","last_scrub_stamp":"2024-04-06T03:45:02.411569+0000","last_deep_scrub":"307'53","last_deep_scrub_stamp":"2024-04-06T03:41:26.233369+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.411569+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110590+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"375'10799","reported_seq":10870,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:34.587129+0000","last_change":"2024-04-06T03:43:25.130138+0000","last_active":"2024-04-06T03:46:34.587129+0000","last_peered":"2024-04-06T03:46:34.587129+0000","last_clean":"2024-04-06T03:46:34.587129+0000","last_became_active":"2024-04-06T03:43:25.129488+0000","last_became_peered":"2024-04-06T03:43:25.129488+0000","last_unstale":"2024-04-06T03:46:34.587129+0000","last_undegraded":"2024-04-06T03:46:34.587129+0000","last_fullsized":"2024-04-06T03:46:34.587129+0000","mapping_epoch":345,"log_start":"364'6500","ondisk_log_start":"364'6500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148307968,"num_objects":2263,"num_object_clones":0,"num_object_copies":6789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10799,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":690,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.683758+0000","last_change":"2024-04-06T03:43:35.178431+0000","last_active":"2024-04-06T03:46:33.683758+0000","last_peered":"2024-04-06T03:46:33.683758+0000","last_clean":"2024-04-06T03:46:33.683758+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:46:33.683758+0000","last_undegraded":"2024-04-06T03:46:33.683758+0000","last_fullsized":"2024-04-06T03:46:33.683758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:35.178312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.178312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.976882+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":712,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676693+0000","last_change":"2024-04-06T03:43:20.948322+0000","last_active":"2024-04-06T03:46:33.676693+0000","last_peered":"2024-04-06T03:46:33.676693+0000","last_clean":"2024-04-06T03:46:33.676693+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:46:33.676693+0000","last_undegraded":"2024-04-06T03:46:33.676693+0000","last_fullsized":"2024-04-06T03:46:33.676693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:20.948160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_clean_scrub_stamp":"2024-04-06T03:43:20.948160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":95,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.676908+0000","last_change":"2024-04-06T03:45:01.559313+0000","last_active":"2024-04-06T03:46:33.676908+0000","last_peered":"2024-04-06T03:46:33.676908+0000","last_clean":"2024-04-06T03:46:33.676908+0000","last_became_active":"2024-04-06T03:42:57.102050+0000","last_became_peered":"2024-04-06T03:42:57.102050+0000","last_unstale":"2024-04-06T03:46:33.676908+0000","last_undegraded":"2024-04-06T03:46:33.676908+0000","last_fullsized":"2024-04-06T03:46:33.676908+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.559226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.559226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230551+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"307'35","reported_seq":748,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.669409+0000","last_change":"2024-04-06T03:43:19.078582+0000","last_active":"2024-04-06T03:46:33.669409+0000","last_peered":"2024-04-06T03:46:33.669409+0000","last_clean":"2024-04-06T03:46:33.669409+0000","last_became_active":"2024-04-06T03:42:23.590160+0000","last_became_peered":"2024-04-06T03:42:23.590160+0000","last_unstale":"2024-04-06T03:46:33.669409+0000","last_undegraded":"2024-04-06T03:46:33.669409+0000","last_fullsized":"2024-04-06T03:46:33.669409+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'35","last_scrub_stamp":"2024-04-06T03:43:19.078336+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:43:19.078336+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:36.774799+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"375'11035","reported_seq":11106,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:35.039922+0000","last_change":"2024-04-06T03:43:25.130562+0000","last_active":"2024-04-06T03:46:35.039922+0000","last_peered":"2024-04-06T03:46:35.039922+0000","last_clean":"2024-04-06T03:46:35.039922+0000","last_became_active":"2024-04-06T03:43:25.129636+0000","last_became_peered":"2024-04-06T03:43:25.129636+0000","last_unstale":"2024-04-06T03:46:35.039922+0000","last_undegraded":"2024-04-06T03:46:35.039922+0000","last_fullsized":"2024-04-06T03:46:35.039922+0000","mapping_epoch":345,"log_start":"363'6400","ondisk_log_start":"363'6400","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.230226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149225472,"num_objects":2277,"num_object_clones":0,"num_object_copies":6831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":573,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.682296+0000","last_change":"2024-04-06T03:45:11.156530+0000","last_active":"2024-04-06T03:46:33.682296+0000","last_peered":"2024-04-06T03:46:33.682296+0000","last_clean":"2024-04-06T03:46:33.682296+0000","last_became_active":"2024-04-06T03:42:23.569567+0000","last_became_peered":"2024-04-06T03:42:23.569567+0000","last_unstale":"2024-04-06T03:46:33.682296+0000","last_undegraded":"2024-04-06T03:46:33.682296+0000","last_fullsized":"2024-04-06T03:46:33.682296+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:11.156227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.156227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:35.977159+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":728,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.671919+0000","last_change":"2024-04-06T03:45:06.959750+0000","last_active":"2024-04-06T03:46:33.671919+0000","last_peered":"2024-04-06T03:46:33.671919+0000","last_clean":"2024-04-06T03:46:33.671919+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:46:33.671919+0000","last_undegraded":"2024-04-06T03:46:33.671919+0000","last_fullsized":"2024-04-06T03:46:33.671919+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:06.959590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:45:06.959590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:37.294558+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"307'46","reported_seq":758,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.679472+0000","last_change":"2024-04-06T03:43:07.288106+0000","last_active":"2024-04-06T03:46:33.679472+0000","last_peered":"2024-04-06T03:46:33.679472+0000","last_clean":"2024-04-06T03:46:33.679472+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:46:33.679472+0000","last_undegraded":"2024-04-06T03:46:33.679472+0000","last_fullsized":"2024-04-06T03:46:33.679472+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'46","last_scrub_stamp":"2024-04-06T03:43:07.287843+0000","last_deep_scrub":"291'34","last_deep_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_clean_scrub_stamp":"2024-04-06T03:43:07.287843+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.099454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"327'14","reported_seq":1036,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-04-06T03:46:33.678756+0000","last_change":"2024-04-06T03:43:35.224637+0000","last_active":"2024-04-06T03:46:33.678756+0000","last_peered":"2024-04-06T03:46:33.678756+0000","last_clean":"2024-04-06T03:46:33.678756+0000","last_became_active":"2024-04-06T03:42:23.576196+0000","last_became_peered":"2024-04-06T03:42:23.576196+0000","last_unstale":"2024-04-06T03:46:33.678756+0000","last_undegraded":"2024-04-06T03:46:33.678756+0000","last_fullsized":"2024-04-06T03:46:33.678756+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'14","last_scrub_stamp":"2024-04-06T03:43:35.224536+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.224536+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:34.110521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2305490968,"num_objects":35180,"num_object_clones":0,"num_object_copies":105540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35180,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176764,"num_write_kb":6782145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7027372032,"data_stored":7027359816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76564,"ondisk_log_size":76564,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1842,"num_read_kb":1567,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1301,"ondisk_log_size":1301,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1513,"num_read_kb":1508,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743743,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573180,"kb_used_data":1040816,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92171972,"statfs":{"total":95995035648,"available":94384099328,"internally_reserved":0,"allocated":1065795584,"data_stored":1065434426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":545126858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136201,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1894160,"kb_used_data":1339464,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91850992,"statfs":{"total":95995035648,"available":94055415808,"internally_reserved":0,"allocated":1371611136,"data_stored":1371289104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575230,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738760,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736060,"kb_used_data":1109300,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92009092,"statfs":{"total":95995035648,"available":94217310208,"internally_reserved":0,"allocated":1135923200,"data_stored":1135557738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378996,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1413840,"kb_used_data":864840,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":92331312,"statfs":{"total":95995035648,"available":94547263488,"internally_reserved":0,"allocated":885596160,"data_stored":885273470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084648,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557384,"kb_used_data":944512,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92187768,"statfs":{"total":95995035648,"available":94400274432,"internally_reserved":0,"allocated":967180288,"data_stored":966865438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":627571143},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855711,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1432692,"kb_used_data":887848,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92312460,"statfs":{"total":95995035648,"available":94527959040,"internally_reserved":0,"allocated":909156352,"data_stored":908800981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495957,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326196,"kb_used_data":757736,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92418956,"statfs":{"total":95995035648,"available":94637010944,"internally_reserved":0,"allocated":775921664,"data_stored":775630748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1118306304,"data_stored":1118306304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":881004544,"data_stored":881000472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":959905792,"data_stored":959905792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":892604416,"data_stored":892600344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":762314752,"data_stored":762314752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1363283968,"data_stored":1363279896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1049952256,"data_stored":1049952256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:46:41.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:46:41.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:41.177 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:41.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:40 smithi153 bash[23852]: audit 2024-04-06T03:46:39.069000+0000 mgr.y (mgr.24368) 942 : audit 0 from='client.25481 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:46:41.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:40 smithi153 bash[23852]: audit 2024-04-06T03:46:39.968052+0000 mon.a (mon.0) 1750 : audit 0 from='client.? 172.21.15.50:0/1895352319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:46:41.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:40 smithi153 bash[23852]: audit 2024-04-06T03:46:40.371881+0000 mon.c (mon.2) 113 : audit 1 from='client.? 172.21.15.50:0/402463763' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:46:41.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:40 smithi153 bash[23852]: audit 2024-04-06T03:46:40.372485+0000 mon.a (mon.0) 1751 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:46:41.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:41.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:41.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:41.457 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:41.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:41.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:41.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:41.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:41.724 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:41.827 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:41.851 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:41.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[21467]: audit 2024-04-06T03:46:40.317452+0000 mgr.y (mgr.24368) 943 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[21467]: cluster 2024-04-06T03:46:40.554350+0000 mgr.y (mgr.24368) 944 : cluster 0 pgmap v1026: 148 pgs: 148 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 838 op/s 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[21467]: cluster 2024-04-06T03:46:40.681231+0000 mon.a (mon.0) 1752 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[21467]: audit 2024-04-06T03:46:40.686819+0000 mon.a (mon.0) 1753 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[21467]: cluster 2024-04-06T03:46:40.692005+0000 mon.a (mon.0) 1754 : cluster 0 osdmap e376: 8 total, 8 up, 7 in 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[21467]: cluster 2024-04-06T03:46:41.288025+0000 osd.2 (osd.2) 477 : cluster 0 6.14 scrub starts 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[29221]: audit 2024-04-06T03:46:40.317452+0000 mgr.y (mgr.24368) 943 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[29221]: cluster 2024-04-06T03:46:40.554350+0000 mgr.y (mgr.24368) 944 : cluster 0 pgmap v1026: 148 pgs: 148 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 838 op/s 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[29221]: cluster 2024-04-06T03:46:40.681231+0000 mon.a (mon.0) 1752 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[29221]: audit 2024-04-06T03:46:40.686819+0000 mon.a (mon.0) 1753 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:46:41.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[29221]: cluster 2024-04-06T03:46:40.692005+0000 mon.a (mon.0) 1754 : cluster 0 osdmap e376: 8 total, 8 up, 7 in 2024-04-06T03:46:41.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:41 smithi050 bash[29221]: cluster 2024-04-06T03:46:41.288025+0000 osd.2 (osd.2) 477 : cluster 0 6.14 scrub starts 2024-04-06T03:46:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:41.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:41.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:42.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:41 smithi153 bash[23852]: audit 2024-04-06T03:46:40.317452+0000 mgr.y (mgr.24368) 943 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:42.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:41 smithi153 bash[23852]: cluster 2024-04-06T03:46:40.554350+0000 mgr.y (mgr.24368) 944 : cluster 0 pgmap v1026: 148 pgs: 148 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 838 op/s 2024-04-06T03:46:42.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:41 smithi153 bash[23852]: cluster 2024-04-06T03:46:40.681231+0000 mon.a (mon.0) 1752 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:46:42.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:41 smithi153 bash[23852]: audit 2024-04-06T03:46:40.686819+0000 mon.a (mon.0) 1753 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:46:42.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:41 smithi153 bash[23852]: cluster 2024-04-06T03:46:40.692005+0000 mon.a (mon.0) 1754 : cluster 0 osdmap e376: 8 total, 8 up, 7 in 2024-04-06T03:46:42.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:41 smithi153 bash[23852]: cluster 2024-04-06T03:46:41.288025+0000 osd.2 (osd.2) 477 : cluster 0 6.14 scrub starts 2024-04-06T03:46:42.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:42.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:42.223 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:42.254 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:42.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:42.388 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:42.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:42.521 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:42.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:42.655 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:42.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:42.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:42.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:42.910 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:42 smithi153 bash[23852]: cluster 2024-04-06T03:46:41.301092+0000 osd.2 (osd.2) 478 : cluster 0 6.14 scrub ok 2024-04-06T03:46:42.910 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:42 smithi153 bash[23852]: cluster 2024-04-06T03:46:42.273552+0000 osd.2 (osd.2) 479 : cluster 0 5.a scrub starts 2024-04-06T03:46:42.910 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:42 smithi153 bash[23852]: cluster 2024-04-06T03:46:42.275010+0000 osd.2 (osd.2) 480 : cluster 0 5.a scrub ok 2024-04-06T03:46:42.928 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:42.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:43.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:43.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:42 smithi050 bash[29221]: cluster 2024-04-06T03:46:41.301092+0000 osd.2 (osd.2) 478 : cluster 0 6.14 scrub ok 2024-04-06T03:46:43.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:42 smithi050 bash[29221]: cluster 2024-04-06T03:46:42.273552+0000 osd.2 (osd.2) 479 : cluster 0 5.a scrub starts 2024-04-06T03:46:43.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:42 smithi050 bash[29221]: cluster 2024-04-06T03:46:42.275010+0000 osd.2 (osd.2) 480 : cluster 0 5.a scrub ok 2024-04-06T03:46:43.044 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:42 smithi050 bash[21467]: cluster 2024-04-06T03:46:41.301092+0000 osd.2 (osd.2) 478 : cluster 0 6.14 scrub ok 2024-04-06T03:46:43.044 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:42 smithi050 bash[21467]: cluster 2024-04-06T03:46:42.273552+0000 osd.2 (osd.2) 479 : cluster 0 5.a scrub starts 2024-04-06T03:46:43.044 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:42 smithi050 bash[21467]: cluster 2024-04-06T03:46:42.275010+0000 osd.2 (osd.2) 480 : cluster 0 5.a scrub ok 2024-04-06T03:46:43.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:43.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:43.188 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:43.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:43.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:43.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:43.452 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:43.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:43.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:43.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:43.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:43.822 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:43.858 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:43.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:43.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:43 smithi050 bash[21467]: cluster 2024-04-06T03:46:42.554925+0000 mgr.y (mgr.24368) 945 : cluster 0 pgmap v1028: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 576 B/s rd, 0 B/s wr, 787 op/s 2024-04-06T03:46:43.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:43 smithi050 bash[29221]: cluster 2024-04-06T03:46:42.554925+0000 mgr.y (mgr.24368) 945 : cluster 0 pgmap v1028: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 576 B/s rd, 0 B/s wr, 787 op/s 2024-04-06T03:46:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:43.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:43.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:44.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:43 smithi153 bash[23852]: cluster 2024-04-06T03:46:42.554925+0000 mgr.y (mgr.24368) 945 : cluster 0 pgmap v1028: 148 pgs: 148 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 576 B/s rd, 0 B/s wr, 787 op/s 2024-04-06T03:46:44.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:44.119 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:44.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:44.251 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:44.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:44.380 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:44.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:44.524 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:44.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:44.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:44.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:44.790 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:44.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:44.927 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:44.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:44.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:44 smithi050 bash[21467]: cluster 2024-04-06T03:46:43.995022+0000 osd.0 (osd.0) 591 : cluster 0 3.17 scrub starts 2024-04-06T03:46:44.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:44 smithi050 bash[21467]: cluster 2024-04-06T03:46:44.002413+0000 osd.0 (osd.0) 592 : cluster 0 3.17 scrub ok 2024-04-06T03:46:44.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:44 smithi050 bash[29221]: cluster 2024-04-06T03:46:43.995022+0000 osd.0 (osd.0) 591 : cluster 0 3.17 scrub starts 2024-04-06T03:46:44.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:44 smithi050 bash[29221]: cluster 2024-04-06T03:46:44.002413+0000 osd.0 (osd.0) 592 : cluster 0 3.17 scrub ok 2024-04-06T03:46:45.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:45.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:44 smithi153 bash[23852]: cluster 2024-04-06T03:46:43.995022+0000 osd.0 (osd.0) 591 : cluster 0 3.17 scrub starts 2024-04-06T03:46:45.042 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:44 smithi153 bash[23852]: cluster 2024-04-06T03:46:44.002413+0000 osd.0 (osd.0) 592 : cluster 0 3.17 scrub ok 2024-04-06T03:46:45.062 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:45.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:45.186 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:45.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:46:45.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:45.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:45.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:46:45.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:45.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:45.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:45.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:45.720 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:45.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:45.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:44.061807+0000 osd.3 (osd.3) 595 : cluster 0 3.8 scrub starts 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:44.064516+0000 osd.3 (osd.3) 596 : cluster 0 3.8 scrub ok 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:44.555426+0000 mgr.y (mgr.24368) 946 : cluster 0 pgmap v1029: 148 pgs: 148 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 859 op/s 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:44.984946+0000 osd.0 (osd.0) 593 : cluster 0 5.15 scrub starts 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:44.991124+0000 osd.0 (osd.0) 594 : cluster 0 5.15 scrub ok 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:45.157388+0000 osd.5 (osd.5) 153 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:45.157450+0000 osd.5 (osd.5) 154 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[21467]: cluster 2024-04-06T03:46:45.220438+0000 osd.2 (osd.2) 481 : cluster 0 11.2 scrub starts 2024-04-06T03:46:45.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:44.061807+0000 osd.3 (osd.3) 595 : cluster 0 3.8 scrub starts 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:44.064516+0000 osd.3 (osd.3) 596 : cluster 0 3.8 scrub ok 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:44.555426+0000 mgr.y (mgr.24368) 946 : cluster 0 pgmap v1029: 148 pgs: 148 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 859 op/s 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:44.984946+0000 osd.0 (osd.0) 593 : cluster 0 5.15 scrub starts 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:44.991124+0000 osd.0 (osd.0) 594 : cluster 0 5.15 scrub ok 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:45.157388+0000 osd.5 (osd.5) 153 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:45.157450+0000 osd.5 (osd.5) 154 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:45.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:45 smithi050 bash[29221]: cluster 2024-04-06T03:46:45.220438+0000 osd.2 (osd.2) 481 : cluster 0 11.2 scrub starts 2024-04-06T03:46:45.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:45.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:45.946 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:46:45.985 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:44.061807+0000 osd.3 (osd.3) 595 : cluster 0 3.8 scrub starts 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:44.064516+0000 osd.3 (osd.3) 596 : cluster 0 3.8 scrub ok 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:44.555426+0000 mgr.y (mgr.24368) 946 : cluster 0 pgmap v1029: 148 pgs: 148 active+clean; 1.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 859 op/s 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:44.984946+0000 osd.0 (osd.0) 593 : cluster 0 5.15 scrub starts 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:44.991124+0000 osd.0 (osd.0) 594 : cluster 0 5.15 scrub ok 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:45.157388+0000 osd.5 (osd.5) 153 : cluster 0 purged_snaps scrub starts 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:45.157450+0000 osd.5 (osd.5) 154 : cluster 0 purged_snaps scrub ok 2024-04-06T03:46:45.995 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:45 smithi153 bash[23852]: cluster 2024-04-06T03:46:45.220438+0000 osd.2 (osd.2) 481 : cluster 0 11.2 scrub starts 2024-04-06T03:46:46.011 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:46.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:46.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:46.243 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:46.264 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:46.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:46.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:46.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:46.528 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:46.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:46.768 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:46 smithi153 bash[23852]: cluster 2024-04-06T03:46:45.710460+0000 mon.a (mon.0) 1755 : cluster 0 osdmap e377: 8 total, 8 up, 7 in 2024-04-06T03:46:46.768 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:46 smithi153 bash[23852]: cluster 2024-04-06T03:46:45.928164+0000 osd.2 (osd.2) 482 : cluster 0 11.2 scrub ok 2024-04-06T03:46:46.774 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.802 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:46.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:46.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:46.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:46 smithi050 bash[21467]: cluster 2024-04-06T03:46:45.710460+0000 mon.a (mon.0) 1755 : cluster 0 osdmap e377: 8 total, 8 up, 7 in 2024-04-06T03:46:46.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:46 smithi050 bash[21467]: cluster 2024-04-06T03:46:45.928164+0000 osd.2 (osd.2) 482 : cluster 0 11.2 scrub ok 2024-04-06T03:46:46.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:46 smithi050 bash[29221]: cluster 2024-04-06T03:46:45.710460+0000 mon.a (mon.0) 1755 : cluster 0 osdmap e377: 8 total, 8 up, 7 in 2024-04-06T03:46:46.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:46 smithi050 bash[29221]: cluster 2024-04-06T03:46:45.928164+0000 osd.2 (osd.2) 482 : cluster 0 11.2 scrub ok 2024-04-06T03:46:46.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:46.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:46.985 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:46:46.986 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:46:47.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:47.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:47.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:47.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:47.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:47.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:47.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:47.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:47.543 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:47.580 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:47.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:47.710 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:47.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:47.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:47.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:47.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:47.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:47.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:46.088579+0000 osd.3 (osd.3) 597 : cluster 0 4.0 scrub starts 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:46.090637+0000 osd.3 (osd.3) 598 : cluster 0 4.0 scrub ok 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:46.339283+0000 osd.4 (osd.4) 577 : cluster 0 4.1 scrub starts 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:46.343002+0000 osd.4 (osd.4) 578 : cluster 0 4.1 scrub ok 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:46.555900+0000 mgr.y (mgr.24368) 947 : cluster 0 pgmap v1031: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:46.991325+0000 osd.7 (osd.7) 595 : cluster 0 6.12 scrub starts 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:47.026659+0000 osd.7 (osd.7) 596 : cluster 0 6.12 scrub ok 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:47.215252+0000 osd.2 (osd.2) 483 : cluster 0 2.16 scrub starts 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[29221]: cluster 2024-04-06T03:46:47.216925+0000 osd.2 (osd.2) 484 : cluster 0 2.16 scrub ok 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:46.088579+0000 osd.3 (osd.3) 597 : cluster 0 4.0 scrub starts 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:46.090637+0000 osd.3 (osd.3) 598 : cluster 0 4.0 scrub ok 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:46.339283+0000 osd.4 (osd.4) 577 : cluster 0 4.1 scrub starts 2024-04-06T03:46:47.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:46.343002+0000 osd.4 (osd.4) 578 : cluster 0 4.1 scrub ok 2024-04-06T03:46:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:46.555900+0000 mgr.y (mgr.24368) 947 : cluster 0 pgmap v1031: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-04-06T03:46:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:46.991325+0000 osd.7 (osd.7) 595 : cluster 0 6.12 scrub starts 2024-04-06T03:46:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:47.026659+0000 osd.7 (osd.7) 596 : cluster 0 6.12 scrub ok 2024-04-06T03:46:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:47.215252+0000 osd.2 (osd.2) 483 : cluster 0 2.16 scrub starts 2024-04-06T03:46:47.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:47 smithi050 bash[21467]: cluster 2024-04-06T03:46:47.216925+0000 osd.2 (osd.2) 484 : cluster 0 2.16 scrub ok 2024-04-06T03:46:48.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:46.088579+0000 osd.3 (osd.3) 597 : cluster 0 4.0 scrub starts 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:46.090637+0000 osd.3 (osd.3) 598 : cluster 0 4.0 scrub ok 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:46.339283+0000 osd.4 (osd.4) 577 : cluster 0 4.1 scrub starts 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:46.343002+0000 osd.4 (osd.4) 578 : cluster 0 4.1 scrub ok 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:46.555900+0000 mgr.y (mgr.24368) 947 : cluster 0 pgmap v1031: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:46.991325+0000 osd.7 (osd.7) 595 : cluster 0 6.12 scrub starts 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:47.026659+0000 osd.7 (osd.7) 596 : cluster 0 6.12 scrub ok 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:47.215252+0000 osd.2 (osd.2) 483 : cluster 0 2.16 scrub starts 2024-04-06T03:46:48.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:47 smithi153 bash[23852]: cluster 2024-04-06T03:46:47.216925+0000 osd.2 (osd.2) 484 : cluster 0 2.16 scrub ok 2024-04-06T03:46:48.065 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:48.099 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:48.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-06T03:46:48.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:48.263 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:48.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:48.397 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:48.501 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:48.522 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:48.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:48.653 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:48.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:48.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:48.792 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:48.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:48.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:48.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:49.029 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:49.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:49.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:49.325 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:49.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:49.439 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:46:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:46:49.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:49.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:49.586 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:49.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:49.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:49.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:49.852 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:49.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:49.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:49 smithi153 bash[23852]: cluster 2024-04-06T03:46:48.556618+0000 mgr.y (mgr.24368) 948 : cluster 0 pgmap v1032: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-04-06T03:46:49.964 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:49 smithi153 bash[23852]: cluster 2024-04-06T03:46:49.061009+0000 osd.7 (osd.7) 597 : cluster 0 2.1c scrub starts 2024-04-06T03:46:49.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:49 smithi153 bash[23852]: cluster 2024-04-06T03:46:49.067332+0000 osd.7 (osd.7) 598 : cluster 0 2.1c scrub ok 2024-04-06T03:46:49.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:49 smithi153 bash[23852]: cluster 2024-04-06T03:46:49.143734+0000 osd.2 (osd.2) 485 : cluster 0 2.1 scrub starts 2024-04-06T03:46:49.965 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:49 smithi153 bash[23852]: cluster 2024-04-06T03:46:49.152612+0000 osd.2 (osd.2) 486 : cluster 0 2.1 scrub ok 2024-04-06T03:46:49.976 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:49.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:49.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:50.066 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[21467]: cluster 2024-04-06T03:46:48.556618+0000 mgr.y (mgr.24368) 948 : cluster 0 pgmap v1032: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[21467]: cluster 2024-04-06T03:46:49.061009+0000 osd.7 (osd.7) 597 : cluster 0 2.1c scrub starts 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[21467]: cluster 2024-04-06T03:46:49.067332+0000 osd.7 (osd.7) 598 : cluster 0 2.1c scrub ok 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[21467]: cluster 2024-04-06T03:46:49.143734+0000 osd.2 (osd.2) 485 : cluster 0 2.1 scrub starts 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[21467]: cluster 2024-04-06T03:46:49.152612+0000 osd.2 (osd.2) 486 : cluster 0 2.1 scrub ok 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[29221]: cluster 2024-04-06T03:46:48.556618+0000 mgr.y (mgr.24368) 948 : cluster 0 pgmap v1032: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[29221]: cluster 2024-04-06T03:46:49.061009+0000 osd.7 (osd.7) 597 : cluster 0 2.1c scrub starts 2024-04-06T03:46:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[29221]: cluster 2024-04-06T03:46:49.067332+0000 osd.7 (osd.7) 598 : cluster 0 2.1c scrub ok 2024-04-06T03:46:50.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[29221]: cluster 2024-04-06T03:46:49.143734+0000 osd.2 (osd.2) 485 : cluster 0 2.1 scrub starts 2024-04-06T03:46:50.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:49 smithi050 bash[29221]: cluster 2024-04-06T03:46:49.152612+0000 osd.2 (osd.2) 486 : cluster 0 2.1 scrub ok 2024-04-06T03:46:50.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:50.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:50.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:50.336 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.379 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:50.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:50.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:50.514 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:50.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:50.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:50.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:46:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:46:50.746 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:50.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:50.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:50.949 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:50.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[21467]: cluster 2024-04-06T03:46:49.072166+0000 osd.3 (osd.3) 599 : cluster 0 5.17 scrub starts 2024-04-06T03:46:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[21467]: cluster 2024-04-06T03:46:49.107832+0000 osd.3 (osd.3) 600 : cluster 0 5.17 scrub ok 2024-04-06T03:46:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[21467]: cluster 2024-04-06T03:46:50.078021+0000 osd.0 (osd.0) 595 : cluster 0 3.1f scrub starts 2024-04-06T03:46:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[21467]: cluster 2024-04-06T03:46:50.084581+0000 osd.0 (osd.0) 596 : cluster 0 3.1f scrub ok 2024-04-06T03:46:50.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[21467]: audit 2024-04-06T03:46:50.620925+0000 mon.a (mon.0) 1756 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[29221]: cluster 2024-04-06T03:46:49.072166+0000 osd.3 (osd.3) 599 : cluster 0 5.17 scrub starts 2024-04-06T03:46:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[29221]: cluster 2024-04-06T03:46:49.107832+0000 osd.3 (osd.3) 600 : cluster 0 5.17 scrub ok 2024-04-06T03:46:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[29221]: cluster 2024-04-06T03:46:50.078021+0000 osd.0 (osd.0) 595 : cluster 0 3.1f scrub starts 2024-04-06T03:46:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[29221]: cluster 2024-04-06T03:46:50.084581+0000 osd.0 (osd.0) 596 : cluster 0 3.1f scrub ok 2024-04-06T03:46:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:50 smithi050 bash[29221]: audit 2024-04-06T03:46:50.620925+0000 mon.a (mon.0) 1756 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:50 smithi153 bash[23852]: cluster 2024-04-06T03:46:49.072166+0000 osd.3 (osd.3) 599 : cluster 0 5.17 scrub starts 2024-04-06T03:46:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:50 smithi153 bash[23852]: cluster 2024-04-06T03:46:49.107832+0000 osd.3 (osd.3) 600 : cluster 0 5.17 scrub ok 2024-04-06T03:46:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:50 smithi153 bash[23852]: cluster 2024-04-06T03:46:50.078021+0000 osd.0 (osd.0) 595 : cluster 0 3.1f scrub starts 2024-04-06T03:46:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:50 smithi153 bash[23852]: cluster 2024-04-06T03:46:50.084581+0000 osd.0 (osd.0) 596 : cluster 0 3.1f scrub ok 2024-04-06T03:46:51.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:50 smithi153 bash[23852]: audit 2024-04-06T03:46:50.620925+0000 mon.a (mon.0) 1756 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:46:51.053 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:51.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:51.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:51.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:51.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:51.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:51.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:51.480 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:51.536 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:46:51.537 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:46:51.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:51.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:51.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:51.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:51.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:51.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:51.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:51.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:51.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:52.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:51 smithi153 bash[23852]: audit 2024-04-06T03:46:50.329925+0000 mgr.y (mgr.24368) 949 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:51 smithi153 bash[23852]: cluster 2024-04-06T03:46:50.557317+0000 mgr.y (mgr.24368) 950 : cluster 0 pgmap v1033: 148 pgs: 148 active+clean; 1.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 968 op/s 2024-04-06T03:46:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:51 smithi153 bash[23852]: cluster 2024-04-06T03:46:51.171929+0000 osd.2 (osd.2) 487 : cluster 0 2.17 scrub starts 2024-04-06T03:46:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:51 smithi153 bash[23852]: cluster 2024-04-06T03:46:51.174154+0000 osd.2 (osd.2) 488 : cluster 0 2.17 scrub ok 2024-04-06T03:46:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:51 smithi153 bash[23852]: cluster 2024-04-06T03:46:51.286098+0000 osd.4 (osd.4) 579 : cluster 0 2.6 scrub starts 2024-04-06T03:46:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:51 smithi153 bash[23852]: cluster 2024-04-06T03:46:51.289376+0000 osd.4 (osd.4) 580 : cluster 0 2.6 scrub ok 2024-04-06T03:46:52.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:52.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[21467]: audit 2024-04-06T03:46:50.329925+0000 mgr.y (mgr.24368) 949 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:52.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[21467]: cluster 2024-04-06T03:46:50.557317+0000 mgr.y (mgr.24368) 950 : cluster 0 pgmap v1033: 148 pgs: 148 active+clean; 1.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 968 op/s 2024-04-06T03:46:52.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[21467]: cluster 2024-04-06T03:46:51.171929+0000 osd.2 (osd.2) 487 : cluster 0 2.17 scrub starts 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[21467]: cluster 2024-04-06T03:46:51.174154+0000 osd.2 (osd.2) 488 : cluster 0 2.17 scrub ok 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[21467]: cluster 2024-04-06T03:46:51.286098+0000 osd.4 (osd.4) 579 : cluster 0 2.6 scrub starts 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[21467]: cluster 2024-04-06T03:46:51.289376+0000 osd.4 (osd.4) 580 : cluster 0 2.6 scrub ok 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[29221]: audit 2024-04-06T03:46:50.329925+0000 mgr.y (mgr.24368) 949 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[29221]: cluster 2024-04-06T03:46:50.557317+0000 mgr.y (mgr.24368) 950 : cluster 0 pgmap v1033: 148 pgs: 148 active+clean; 1.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 968 op/s 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[29221]: cluster 2024-04-06T03:46:51.171929+0000 osd.2 (osd.2) 487 : cluster 0 2.17 scrub starts 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[29221]: cluster 2024-04-06T03:46:51.174154+0000 osd.2 (osd.2) 488 : cluster 0 2.17 scrub ok 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[29221]: cluster 2024-04-06T03:46:51.286098+0000 osd.4 (osd.4) 579 : cluster 0 2.6 scrub starts 2024-04-06T03:46:52.107 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:51 smithi050 bash[29221]: cluster 2024-04-06T03:46:51.289376+0000 osd.4 (osd.4) 580 : cluster 0 2.6 scrub ok 2024-04-06T03:46:52.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:52.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:52.257 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:52.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:52.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:52.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:52.520 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:52.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:52.656 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:52.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:52.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:52.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:52 smithi050 bash[21467]: cluster 2024-04-06T03:46:52.012703+0000 osd.0 (osd.0) 597 : cluster 0 2.2 scrub starts 2024-04-06T03:46:52.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:52 smithi050 bash[21467]: cluster 2024-04-06T03:46:52.016712+0000 osd.0 (osd.0) 598 : cluster 0 2.2 scrub ok 2024-04-06T03:46:52.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:52 smithi050 bash[29221]: cluster 2024-04-06T03:46:52.012703+0000 osd.0 (osd.0) 597 : cluster 0 2.2 scrub starts 2024-04-06T03:46:52.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:52 smithi050 bash[29221]: cluster 2024-04-06T03:46:52.016712+0000 osd.0 (osd.0) 598 : cluster 0 2.2 scrub ok 2024-04-06T03:46:52.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:52.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:52.980 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:46:53.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:52 smithi153 bash[23852]: cluster 2024-04-06T03:46:52.012703+0000 osd.0 (osd.0) 597 : cluster 0 2.2 scrub starts 2024-04-06T03:46:53.010 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:52 smithi153 bash[23852]: cluster 2024-04-06T03:46:52.016712+0000 osd.0 (osd.0) 598 : cluster 0 2.2 scrub ok 2024-04-06T03:46:53.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:53.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:53.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:53.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:53.441 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:53.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:53.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:53.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:53.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:53.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:53.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:53.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:53.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:53.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:53.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:53 smithi050 bash[21467]: cluster 2024-04-06T03:46:52.557839+0000 mgr.y (mgr.24368) 951 : cluster 0 pgmap v1034: 148 pgs: 148 active+clean; 1.4 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 963 op/s 2024-04-06T03:46:53.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:53 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.035944+0000 osd.0 (osd.0) 599 : cluster 0 6.16 scrub starts 2024-04-06T03:46:53.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:53 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.122054+0000 osd.0 (osd.0) 600 : cluster 0 6.16 scrub ok 2024-04-06T03:46:53.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:53 smithi050 bash[29221]: cluster 2024-04-06T03:46:52.557839+0000 mgr.y (mgr.24368) 951 : cluster 0 pgmap v1034: 148 pgs: 148 active+clean; 1.4 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 963 op/s 2024-04-06T03:46:53.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:53 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.035944+0000 osd.0 (osd.0) 599 : cluster 0 6.16 scrub starts 2024-04-06T03:46:53.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:53 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.122054+0000 osd.0 (osd.0) 600 : cluster 0 6.16 scrub ok 2024-04-06T03:46:53.824 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:53.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:53.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:53.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:53.968 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:53.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:53 smithi153 bash[23852]: cluster 2024-04-06T03:46:52.557839+0000 mgr.y (mgr.24368) 951 : cluster 0 pgmap v1034: 148 pgs: 148 active+clean; 1.4 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 963 op/s 2024-04-06T03:46:54.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:53 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.035944+0000 osd.0 (osd.0) 599 : cluster 0 6.16 scrub starts 2024-04-06T03:46:54.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:53 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.122054+0000 osd.0 (osd.0) 600 : cluster 0 6.16 scrub ok 2024-04-06T03:46:54.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:54.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:54.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:54.250 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:54.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:54.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:46:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:46:54.482 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:54.510 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:54.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:54.640 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:54.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:54.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:54.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:54.910 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:54.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:55.013 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.099045+0000 osd.3 (osd.3) 601 : cluster 0 6.1f scrub starts 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.374073+0000 osd.2 (osd.2) 489 : cluster 0 3.4 scrub starts 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.395287+0000 osd.2 (osd.2) 490 : cluster 0 3.4 scrub ok 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.453150+0000 osd.3 (osd.3) 602 : cluster 0 6.1f scrub ok 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:53.976120+0000 osd.7 (osd.7) 599 : cluster 0 5.10 scrub starts 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:54.004794+0000 osd.7 (osd.7) 600 : cluster 0 5.10 scrub ok 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:54.026005+0000 osd.0 (osd.0) 601 : cluster 0 2.4 scrub starts 2024-04-06T03:46:55.024 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:54 smithi153 bash[23852]: cluster 2024-04-06T03:46:54.030383+0000 osd.0 (osd.0) 602 : cluster 0 2.4 scrub ok 2024-04-06T03:46:55.039 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:55.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:55.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.099045+0000 osd.3 (osd.3) 601 : cluster 0 6.1f scrub starts 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.374073+0000 osd.2 (osd.2) 489 : cluster 0 3.4 scrub starts 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.395287+0000 osd.2 (osd.2) 490 : cluster 0 3.4 scrub ok 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.453150+0000 osd.3 (osd.3) 602 : cluster 0 6.1f scrub ok 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:53.976120+0000 osd.7 (osd.7) 599 : cluster 0 5.10 scrub starts 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:54.004794+0000 osd.7 (osd.7) 600 : cluster 0 5.10 scrub ok 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:54.026005+0000 osd.0 (osd.0) 601 : cluster 0 2.4 scrub starts 2024-04-06T03:46:55.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[21467]: cluster 2024-04-06T03:46:54.030383+0000 osd.0 (osd.0) 602 : cluster 0 2.4 scrub ok 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.099045+0000 osd.3 (osd.3) 601 : cluster 0 6.1f scrub starts 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.374073+0000 osd.2 (osd.2) 489 : cluster 0 3.4 scrub starts 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.395287+0000 osd.2 (osd.2) 490 : cluster 0 3.4 scrub ok 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.453150+0000 osd.3 (osd.3) 602 : cluster 0 6.1f scrub ok 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:53.976120+0000 osd.7 (osd.7) 599 : cluster 0 5.10 scrub starts 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:54.004794+0000 osd.7 (osd.7) 600 : cluster 0 5.10 scrub ok 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:54.026005+0000 osd.0 (osd.0) 601 : cluster 0 2.4 scrub starts 2024-04-06T03:46:55.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:54 smithi050 bash[29221]: cluster 2024-04-06T03:46:54.030383+0000 osd.0 (osd.0) 602 : cluster 0 2.4 scrub ok 2024-04-06T03:46:55.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:55.314 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:55.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:55.448 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:55.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:55.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:55.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:55.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:55.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:55 smithi153 bash[23852]: cluster 2024-04-06T03:46:54.558314+0000 mgr.y (mgr.24368) 952 : cluster 0 pgmap v1035: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:46:55.763 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:55 smithi153 bash[23852]: cluster 2024-04-06T03:46:55.012973+0000 osd.7 (osd.7) 601 : cluster 0 3.f scrub starts 2024-04-06T03:46:55.764 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:55 smithi153 bash[23852]: cluster 2024-04-06T03:46:55.024878+0000 osd.7 (osd.7) 602 : cluster 0 3.f scrub ok 2024-04-06T03:46:55.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:55.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:55 smithi050 bash[21467]: cluster 2024-04-06T03:46:54.558314+0000 mgr.y (mgr.24368) 952 : cluster 0 pgmap v1035: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:46:55.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:55 smithi050 bash[21467]: cluster 2024-04-06T03:46:55.012973+0000 osd.7 (osd.7) 601 : cluster 0 3.f scrub starts 2024-04-06T03:46:55.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:55 smithi050 bash[21467]: cluster 2024-04-06T03:46:55.024878+0000 osd.7 (osd.7) 602 : cluster 0 3.f scrub ok 2024-04-06T03:46:55.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:55 smithi050 bash[29221]: cluster 2024-04-06T03:46:54.558314+0000 mgr.y (mgr.24368) 952 : cluster 0 pgmap v1035: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-04-06T03:46:55.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:55 smithi050 bash[29221]: cluster 2024-04-06T03:46:55.012973+0000 osd.7 (osd.7) 601 : cluster 0 3.f scrub starts 2024-04-06T03:46:55.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:55 smithi050 bash[29221]: cluster 2024-04-06T03:46:55.024878+0000 osd.7 (osd.7) 602 : cluster 0 3.f scrub ok 2024-04-06T03:46:55.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:55.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:55.958 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:56.075 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:56.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:56.179 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:56.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:56.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:56.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:56.336 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:56.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:56.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:56.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:56.565 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:56.589 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:56.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:56.692 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:46:56.974 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:46:56.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:56 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.033190+0000 osd.0 (osd.0) 603 : cluster 0 2.15 scrub starts 2024-04-06T03:46:56.993 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:56 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.048187+0000 osd.0 (osd.0) 604 : cluster 0 2.15 scrub ok 2024-04-06T03:46:57.019 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:56 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.033190+0000 osd.0 (osd.0) 603 : cluster 0 2.15 scrub starts 2024-04-06T03:46:57.019 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:56 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.048187+0000 osd.0 (osd.0) 604 : cluster 0 2.15 scrub ok 2024-04-06T03:46:57.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:56 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.033190+0000 osd.0 (osd.0) 603 : cluster 0 2.15 scrub starts 2024-04-06T03:46:57.020 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:56 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.048187+0000 osd.0 (osd.0) 604 : cluster 0 2.15 scrub ok 2024-04-06T03:46:57.375 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:57.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:57.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:57.503 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:57.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:57.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:57.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:46:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:57.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:46:57.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:57.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:57.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:57 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.530044+0000 osd.4 (osd.4) 581 : cluster 0 5.1 scrub starts 2024-04-06T03:46:57.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:57 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.535895+0000 osd.4 (osd.4) 582 : cluster 0 5.1 scrub ok 2024-04-06T03:46:57.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:57 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.558903+0000 mgr.y (mgr.24368) 953 : cluster 0 pgmap v1036: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 907 op/s 2024-04-06T03:46:57.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:57 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.988921+0000 osd.7 (osd.7) 603 : cluster 0 2.7 scrub starts 2024-04-06T03:46:57.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:57 smithi153 bash[23852]: cluster 2024-04-06T03:46:56.994443+0000 osd.7 (osd.7) 604 : cluster 0 2.7 scrub ok 2024-04-06T03:46:57.869 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:57.897 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:57.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:57.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.530044+0000 osd.4 (osd.4) 581 : cluster 0 5.1 scrub starts 2024-04-06T03:46:57.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.535895+0000 osd.4 (osd.4) 582 : cluster 0 5.1 scrub ok 2024-04-06T03:46:57.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.558903+0000 mgr.y (mgr.24368) 953 : cluster 0 pgmap v1036: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 907 op/s 2024-04-06T03:46:57.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.988921+0000 osd.7 (osd.7) 603 : cluster 0 2.7 scrub starts 2024-04-06T03:46:57.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[21467]: cluster 2024-04-06T03:46:56.994443+0000 osd.7 (osd.7) 604 : cluster 0 2.7 scrub ok 2024-04-06T03:46:57.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.530044+0000 osd.4 (osd.4) 581 : cluster 0 5.1 scrub starts 2024-04-06T03:46:57.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.535895+0000 osd.4 (osd.4) 582 : cluster 0 5.1 scrub ok 2024-04-06T03:46:57.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.558903+0000 mgr.y (mgr.24368) 953 : cluster 0 pgmap v1036: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 907 op/s 2024-04-06T03:46:57.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.988921+0000 osd.7 (osd.7) 603 : cluster 0 2.7 scrub starts 2024-04-06T03:46:57.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:57 smithi050 bash[29221]: cluster 2024-04-06T03:46:56.994443+0000 osd.7 (osd.7) 604 : cluster 0 2.7 scrub ok 2024-04-06T03:46:58.001 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:58.033 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:58.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:58.147 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:58.617 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:58.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:58.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:58.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:58.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:58.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:58.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:58.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:58.978 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:58.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:58 smithi153 bash[23852]: audit 2024-04-06T03:46:56.941173+0000 mgr.y (mgr.24368) 954 : audit 0 from='client.25499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:46:58.985 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:58 smithi153 bash[23852]: cluster 2024-04-06T03:46:58.011415+0000 osd.2 (osd.2) 491 : cluster 0 5.8 scrub starts 2024-04-06T03:46:58.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:58 smithi153 bash[23852]: cluster 2024-04-06T03:46:58.015508+0000 osd.2 (osd.2) 492 : cluster 0 5.8 scrub ok 2024-04-06T03:46:58.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:58 smithi153 bash[23852]: cluster 2024-04-06T03:46:58.023732+0000 osd.7 (osd.7) 605 : cluster 0 11.4 scrub starts 2024-04-06T03:46:58.986 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:58 smithi153 bash[23852]: cluster 2024-04-06T03:46:58.460280+0000 osd.7 (osd.7) 606 : cluster 0 11.4 scrub ok 2024-04-06T03:46:58.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:58.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:59.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:59.120 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:46:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:59.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:46:59.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:46:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:59.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:46:59.264 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:46:59.264 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:46:59.264 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:46:59.264 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:46:59.264 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.264 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:83119 10.c 10:383ea5c5:::benchmark_data_smithi050_121293_object83118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:40:15.521463+0000", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 403.73821798900002, 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43933383100000001, 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 83119 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521463+0000", 2024-04-06T03:46:59.265 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521463+0000", 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521465+0000", 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6950000000000001e-06 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521503+0000", 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8680000000000002e-05 2024-04-06T03:46:59.266 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521518+0000", 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4541e-05 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521525+0000", 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1539999999999996e-06 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521579+0000", 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4063000000000001e-05 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.267 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521679+0000", 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9727000000000002e-05 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.521778+0000", 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9328999999999997e-05 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.523825+0000", 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020471349999999998 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.268 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.524073+0000", 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024731500000000001 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.955752+0000", 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43167981799999999 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.960725+0000", 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0049720270000000004 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.269 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:15.960797+0000", 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2347000000000004e-05 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:84316 10.c 10:3769ea5d:::benchmark_data_smithi050_121293_object84315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:40:17.278580+0000", 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 401.981100744, 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.52345149700000004, 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.270 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84316 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278580+0000", 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278580+0000", 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.271 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278582+0000", 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7260000000000001e-06 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278634+0000", 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1569e-05 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.272 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278646+0000", 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2174999999999999e-05 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278652+0000", 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1580000000000002e-06 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278704+0000", 2024-04-06T03:46:59.273 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2153e-05 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.278852+0000", 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014809299999999999 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.279040+0000", 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00018819600000000001 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.274 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.281407+0000", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0023669609999999999 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.281554+0000", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000146279 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.801867+0000", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.52031344700000004 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.275 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.801961+0000", 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4335999999999999e-05 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:40:17.802032+0000", 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0403999999999999e-05 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:126117 10.c 10:352f536f:::benchmark_data_smithi050_121293_object27300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:06.954295+0000", 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 352.30538559600001, 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41383709899999999, 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.276 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 126117 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.954295+0000", 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.954295+0000", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.954297+0000", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.373e-06 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.954303+0000", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2879999999999998e-06 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.278 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.954316+0000", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2627e-05 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.954324+0000", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9729999999999997e-06 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.956495+0000", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002171348 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.956624+0000", 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012892900000000001 2024-04-06T03:46:59.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.956745+0000", 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121383 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.960319+0000", 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0035732110000000002 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:06.968990+0000", 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0086709549999999993 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.280 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:07.367833+0000", 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.398843163 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:07.368013+0000", 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000179851 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:07.368133+0000", 2024-04-06T03:46:59.281 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000119998 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:132572 10.c 10:33469d24:::benchmark_data_smithi050_121293_object33755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:15.050520+0000", 2024-04-06T03:46:59.282 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 344.20916053600001, 2024-04-06T03:46:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48138919800000002, 2024-04-06T03:46:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.283 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 132572 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050520+0000", 2024-04-06T03:46:59.285 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050520+0000", 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050522+0000", 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4619999999999999e-06 2024-04-06T03:46:59.286 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.287 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.287 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.287 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050531+0000", 2024-04-06T03:46:59.287 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9719999999999998e-06 2024-04-06T03:46:59.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050546+0000", 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5579000000000001e-05 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050558+0000", 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1556000000000001e-05 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.288 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.050641+0000", 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2690999999999998e-05 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.051049+0000", 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00040863199999999997 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.289 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.051170+0000", 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121119 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.054060+0000", 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002889736 2024-04-06T03:46:59.290 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.058849+0000", 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0047883429999999996 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.531653+0000", 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47280399899999997 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.531800+0000", 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014720999999999999 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.291 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.531910+0000", 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000109899 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:132577 10.c 10:39a63a24:::benchmark_data_smithi050_121293_object33760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:15.054997+0000", 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 344.20468416400001, 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.477235721, 2024-04-06T03:46:59.292 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 132577 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.054997+0000", 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.293 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.054997+0000", 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.054998+0000", 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.344e-06 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.055006+0000", 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4150000000000002e-06 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.294 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.055024+0000", 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8610999999999999e-05 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.055034+0000", 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6169999999999996e-06 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.055111+0000", 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7310000000000004e-05 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.295 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.055271+0000", 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015945400000000001 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.055431+0000", 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016077600000000001 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.057655+0000", 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0022232100000000002 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.296 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059192+0000", 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0015370819999999999 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.532003+0000", 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47281128500000003 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.532111+0000", 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000108102 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.297 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.532233+0000", 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121515 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:132579 10.c 10:3d5754e6:::benchmark_data_smithi050_121293_object33762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:15.059394+0000", 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 344.200286742, 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48424179499999997, 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.298 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 132579 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059394+0000", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059394+0000", 2024-04-06T03:46:59.299 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059395+0000", 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2100000000000001e-06 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059402+0000", 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6839999999999999e-06 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.300 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059413+0000", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0485e-05 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059418+0000", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6450000000000002e-06 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059465+0000", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7145999999999999e-05 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.301 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059600+0000", 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000134752 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.059741+0000", 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000140498 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.061842+0000", 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002101159 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.062515+0000", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00067284899999999999 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.543354+0000", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48083932299999999 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.543509+0000", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015521 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.303 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:15.543636+0000", 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012683400000000001 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:165871 10.c 10:3b4360f5:::benchmark_data_smithi050_121293_object67054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:58.596624+0000", 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 300.663057369, 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48529114499999998, 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.304 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 165871 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596624+0000", 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.305 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596624+0000", 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596625+0000", 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4279999999999999e-06 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596630+0000", 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7419999999999997e-06 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.306 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596641+0000", 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1039999999999999e-05 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596649+0000", 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3289999999999994e-06 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596724+0000", 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5227000000000003e-05 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.307 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596841+0000", 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000116302 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596950+0000", 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010900600000000001 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.599506+0000", 2024-04-06T03:46:59.308 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025567010000000002 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.607899+0000", 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0083922719999999992 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081812+0000", 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47391306599999999 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081877+0000", 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5177000000000003e-05 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.309 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081915+0000", 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7855000000000003e-05 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:165873 10.c 10:353b0683:::benchmark_data_smithi050_121293_object67056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:58.601575+0000", 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 300.65810633400002, 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48048355300000001, 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.310 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 165873 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.311 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601575+0000", 2024-04-06T03:46:59.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.312 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.312 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.312 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.312 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601575+0000", 2024-04-06T03:46:59.312 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.313 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.313 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.313 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.313 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601576+0000", 2024-04-06T03:46:59.313 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3069999999999999e-06 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601582+0000", 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5199999999999997e-06 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601595+0000", 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3142999999999999e-05 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.314 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601601+0000", 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6769999999999998e-06 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601658+0000", 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6910999999999998e-05 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601797+0000", 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000138935 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.315 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601953+0000", 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000156224 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.604454+0000", 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025002940000000001 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.607939+0000", 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003485546 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.316 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081957+0000", 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47401758300000002 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081997+0000", 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0373000000000001e-05 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082058+0000", 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1039999999999998e-05 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.317 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:165880 10.c 10:3a9cc1ff:::benchmark_data_smithi050_121293_object67063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:58.611324+0000", 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 300.64835652199997, 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47127398500000001, 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 165880 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.318 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611324+0000", 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611324+0000", 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611326+0000", 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.449e-06 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.319 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611331+0000", 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.169e-06 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611345+0000", 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3529e-05 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611354+0000", 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9439999999999994e-06 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.320 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611413+0000", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9459000000000001e-05 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611528+0000", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000115103 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611643+0000", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011493999999999999 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.321 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.613786+0000", 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021425789999999999 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.615791+0000", 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020058070000000001 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082545+0000", 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.46675355499999999 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.322 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082575+0000", 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9646999999999999e-05 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082598+0000", 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3804000000000001e-05 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:180541 10.c 10:32bb1016:::benchmark_data_smithi050_121293_object81724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.651564+0000", 2024-04-06T03:46:59.323 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 278.60811738299998, 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55442063399999997, 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 180541 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651564+0000", 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.324 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651564+0000", 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651565+0000", 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.61e-06 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651571+0000", 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.604e-06 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.325 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651583+0000", 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2351000000000001e-05 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651590+0000", 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1060000000000002e-06 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651659+0000", 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8736000000000001e-05 2024-04-06T03:46:59.326 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651751+0000", 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2001000000000002e-05 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651827+0000", 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6103e-05 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654354+0000", 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025271080000000001 2024-04-06T03:46:59.327 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654607+0000", 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025244599999999997 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.205724+0000", 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55111694300000003 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.205836+0000", 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112645 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.328 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.205984+0000", 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014798099999999999 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:180542 10.c 10:3a06efca:::benchmark_data_smithi050_121293_object81725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.654984+0000", 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 278.604697273, 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55130753499999996, 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.329 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 180542 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654984+0000", 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654984+0000", 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.330 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654985+0000", 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.47e-07 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654988+0000", 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8909999999999997e-06 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654997+0000", 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9989999999999997e-06 2024-04-06T03:46:59.331 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655003+0000", 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4140000000000002e-06 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655044+0000", 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1588999999999998e-05 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.332 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655138+0000", 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3553000000000004e-05 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655213+0000", 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4663e-05 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.656889+0000", 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016761370000000001 2024-04-06T03:46:59.333 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.657245+0000", 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00035658899999999998 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.206122+0000", 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54887692600000004 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.334 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.206215+0000", 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2174000000000002e-05 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.206291+0000", 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6753000000000002e-05 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84079 11.f 11:f336f4e4:::benchmark_data_smithi050_138698_object84078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.082199+0000", 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 123.177482372, 2024-04-06T03:46:59.335 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48952396300000001, 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84079 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082199+0000", 2024-04-06T03:46:59.336 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082199+0000", 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082200+0000", 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0279999999999999e-06 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082285+0000", 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5451000000000006e-05 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082297+0000", 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.184e-05 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082326+0000", 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8679000000000001e-05 2024-04-06T03:46:59.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.084009+0000", 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016828800000000001 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.084144+0000", 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013520200000000001 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.339 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.084268+0000", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000124234 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.087490+0000", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003222291 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.087956+0000", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00046621299999999999 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.571387+0000", 2024-04-06T03:46:59.340 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.483430154 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.571586+0000", 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000199583 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.571723+0000", 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000136408 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84093 11.f 11:f22a7ccd:::benchmark_data_smithi050_138698_object84092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.094231+0000", 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 123.165449542, 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48593634800000002, 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84093 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094231+0000", 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094231+0000", 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094232+0000", 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5600000000000004e-07 2024-04-06T03:46:59.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094367+0000", 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013476100000000001 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094379+0000", 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1525999999999999e-05 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.344 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094386+0000", 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5730000000000002e-06 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094440+0000", 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4128000000000001e-05 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094559+0000", 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011815399999999999 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.345 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094650+0000", 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1713999999999996e-05 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.096736+0000", 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002085408 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.098839+0000", 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002103705 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580100+0000", 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48126070799999998 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580135+0000", 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5182999999999998e-05 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580168+0000", 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2532e-05 2024-04-06T03:46:59.347 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84100 11.f 11:f2bff20b:::benchmark_data_smithi050_138698_object84099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.102298+0000", 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 123.15738261200001, 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48398215500000002, 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.348 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84100 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102298+0000", 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102298+0000", 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.349 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102300+0000", 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.108e-06 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102343+0000", 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3961000000000001e-05 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102356+0000", 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2659e-05 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.350 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102363+0000", 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.097e-06 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102421+0000", 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7815000000000001e-05 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102542+0000", 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121045 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.351 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102648+0000", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000105849 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.104642+0000", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019937079999999999 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580240+0000", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47559864499999999 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586194+0000", 2024-04-06T03:46:59.352 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.005953814 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586237+0000", 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.244e-05 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586281+0000", 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4014000000000002e-05 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.353 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84103 11.f 11:f0c0dc0b:::benchmark_data_smithi050_138698_object84102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.106886+0000", 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 123.152794777, 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47961802100000001, 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84103 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.354 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106886+0000", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106886+0000", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106894+0000", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5650000000000004e-06 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106944+0000", 2024-04-06T03:46:59.355 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0262000000000001e-05 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106955+0000", 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1015999999999999e-05 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106982+0000", 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7263e-05 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.107027+0000", 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5124999999999999e-05 2024-04-06T03:46:59.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.107144+0000", 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011633600000000001 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.107253+0000", 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000108849 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.109003+0000", 2024-04-06T03:46:59.357 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0017502349999999999 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.581085+0000", 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47208220400000001 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586450+0000", 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0053647670000000003 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586484+0000", 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4044000000000001e-05 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586504+0000", 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0355000000000001e-05 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:85899 11.f 11:f7a53cd3:::benchmark_data_smithi050_138698_object85898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:58.381838+0000", 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 120.87784340899999, 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55626176100000002, 2024-04-06T03:46:59.359 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 85899 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381838+0000", 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.360 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381838+0000", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381839+0000", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.463e-06 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381890+0000", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1119999999999998e-05 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381901+0000", 2024-04-06T03:46:59.361 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1185999999999999e-05 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381909+0000", 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.123e-06 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381968+0000", 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8616000000000003e-05 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.362 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.382079+0000", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000110776 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.382180+0000", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000101068 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.385840+0000", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036598479999999998 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.393293+0000", 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00745366 2024-04-06T03:46:59.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.937979+0000", 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54468579900000003 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.938045+0000", 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5513999999999996e-05 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.938099+0000", 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4588e-05 2024-04-06T03:46:59.364 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:85907 11.f 11:f6c6d927:::benchmark_data_smithi050_138698_object85906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:58.387295+0000", 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 120.872385596, 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.56115197800000005, 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 85907 2024-04-06T03:46:59.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387295+0000", 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387295+0000", 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387296+0000", 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7100000000000002e-07 2024-04-06T03:46:59.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387354+0000", 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8009e-05 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387360+0000", 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7570000000000001e-06 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387364+0000", 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4789999999999998e-06 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.367 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387422+0000", 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7383999999999998e-05 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387528+0000", 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010686100000000001 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387629+0000", 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000100826 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.368 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.396378+0000", 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0087485410000000003 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397753+0000", 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013752440000000001 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948329+0000", 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55057592899999996 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948395+0000", 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5655999999999996e-05 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948447+0000", 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2720999999999999e-05 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.370 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:85914 11.f 11:f40a4adc:::benchmark_data_smithi050_138698_object85913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:58.397788+0000", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 120.861893263, 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55124815999999999, 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 85914 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397788+0000", 2024-04-06T03:46:59.371 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397788+0000", 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397789+0000", 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.09e-07 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397810+0000", 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0930000000000001e-05 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.372 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397816+0000", 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2600000000000002e-06 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397820+0000", 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3309999999999997e-06 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.399239+0000", 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001419078 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.373 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.399391+0000", 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015199899999999999 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.399795+0000", 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00040332299999999997 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.403164+0000", 2024-04-06T03:46:59.374 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033698999999999999 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.403171+0000", 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6660000000000002e-06 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948970+0000", 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54579887100000002 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.949010+0000", 2024-04-06T03:46:59.375 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9861999999999999e-05 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.949036+0000", 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6030999999999999e-05 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:149170 11.4 11:20e216ff:::benchmark_data_smithi050_138698_object43196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:05.458743+0000", 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 53.800938174000002, 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43579931799999999, 2024-04-06T03:46:59.376 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 149170 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458743+0000", 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.377 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458743+0000", 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458744+0000", 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4900000000000005e-07 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458747+0000", 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.613e-06 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.378 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458756+0000", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6170000000000007e-06 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458762+0000", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4790000000000001e-06 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458815+0000", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3087999999999997e-05 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.379 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458899+0000", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3294999999999994e-05 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,6", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458989+0000", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0365999999999995e-05 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.462025+0000", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003035921 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.462702+0000", 2024-04-06T03:46:59.380 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00067719500000000001 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.894448+0000", 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.431745555 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.894499+0000", 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0955000000000003e-05 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.894542+0000", 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3485000000000001e-05 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.381 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:186832 11.f 11:f046fa0f:::benchmark_data_smithi050_138698_object80858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:47.059724+0000", 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 12.199956797, 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41681425999999999, 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:46:59.382 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 186832 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.059724+0000", 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.059724+0000", 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:46:59.383 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.059725+0000", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5199999999999998e-07 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.059727+0000", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2359999999999999e-06 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.059733+0000", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9710000000000001e-06 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.059738+0000", 2024-04-06T03:46:59.384 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9930000000000003e-06 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.061066+0000", 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001328224 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.061164+0000", 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8189000000000005e-05 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.385 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.061268+0000", 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000103514 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.065427+0000", 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0041595440000000003 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.065441+0000", 2024-04-06T03:46:59.386 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3801000000000001e-05 2024-04-06T03:46:59.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.387 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.387 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:46:59.387 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.476286+0000", 2024-04-06T03:46:59.387 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.41084452500000002 2024-04-06T03:46:59.387 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.476457+0000", 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0001719 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:47.476538+0000", 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1011000000000001e-05 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.388 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.389 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:46:59.389 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:46:59.389 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:46:59.389 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:59.390 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[21467]: audit 2024-04-06T03:46:56.941173+0000 mgr.y (mgr.24368) 954 : audit 0 from='client.25499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:46:59.390 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[21467]: cluster 2024-04-06T03:46:58.011415+0000 osd.2 (osd.2) 491 : cluster 0 5.8 scrub starts 2024-04-06T03:46:59.390 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[21467]: cluster 2024-04-06T03:46:58.015508+0000 osd.2 (osd.2) 492 : cluster 0 5.8 scrub ok 2024-04-06T03:46:59.390 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[21467]: cluster 2024-04-06T03:46:58.023732+0000 osd.7 (osd.7) 605 : cluster 0 11.4 scrub starts 2024-04-06T03:46:59.390 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[21467]: cluster 2024-04-06T03:46:58.460280+0000 osd.7 (osd.7) 606 : cluster 0 11.4 scrub ok 2024-04-06T03:46:59.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[29221]: audit 2024-04-06T03:46:56.941173+0000 mgr.y (mgr.24368) 954 : audit 0 from='client.25499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:46:59.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[29221]: cluster 2024-04-06T03:46:58.011415+0000 osd.2 (osd.2) 491 : cluster 0 5.8 scrub starts 2024-04-06T03:46:59.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[29221]: cluster 2024-04-06T03:46:58.015508+0000 osd.2 (osd.2) 492 : cluster 0 5.8 scrub ok 2024-04-06T03:46:59.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[29221]: cluster 2024-04-06T03:46:58.023732+0000 osd.7 (osd.7) 605 : cluster 0 11.4 scrub starts 2024-04-06T03:46:59.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:58 smithi050 bash[29221]: cluster 2024-04-06T03:46:58.460280+0000 osd.7 (osd.7) 606 : cluster 0 11.4 scrub ok 2024-04-06T03:46:59.424 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:46:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:59.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:46:59.528 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:59.614 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:46:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:59.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:46:59.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:59.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:46:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:46:59.747 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:46:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:59.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:46:59.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:59.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:46:59.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:46:59.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:46:59.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:00.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[29221]: cluster 2024-04-06T03:46:58.559456+0000 mgr.y (mgr.24368) 955 : cluster 0 pgmap v1037: 148 pgs: 148 active+clean; 1.0 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 980 op/s 2024-04-06T03:47:00.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[29221]: cluster 2024-04-06T03:46:58.959314+0000 osd.0 (osd.0) 605 : cluster 0 6.9 scrub starts 2024-04-06T03:47:00.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[29221]: cluster 2024-04-06T03:46:59.382555+0000 osd.0 (osd.0) 606 : cluster 0 6.9 scrub ok 2024-04-06T03:47:00.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[29221]: cluster 2024-04-06T03:46:59.383468+0000 osd.7 (osd.7) 607 : cluster 0 3.13 scrub starts 2024-04-06T03:47:00.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[29221]: cluster 2024-04-06T03:46:59.393862+0000 osd.7 (osd.7) 608 : cluster 0 3.13 scrub ok 2024-04-06T03:47:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:00.026 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[21467]: cluster 2024-04-06T03:46:58.559456+0000 mgr.y (mgr.24368) 955 : cluster 0 pgmap v1037: 148 pgs: 148 active+clean; 1.0 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 980 op/s 2024-04-06T03:47:00.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[21467]: cluster 2024-04-06T03:46:58.959314+0000 osd.0 (osd.0) 605 : cluster 0 6.9 scrub starts 2024-04-06T03:47:00.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[21467]: cluster 2024-04-06T03:46:59.382555+0000 osd.0 (osd.0) 606 : cluster 0 6.9 scrub ok 2024-04-06T03:47:00.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[21467]: cluster 2024-04-06T03:46:59.383468+0000 osd.7 (osd.7) 607 : cluster 0 3.13 scrub starts 2024-04-06T03:47:00.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:46:59 smithi050 bash[21467]: cluster 2024-04-06T03:46:59.393862+0000 osd.7 (osd.7) 608 : cluster 0 3.13 scrub ok 2024-04-06T03:47:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:59 smithi153 bash[23852]: cluster 2024-04-06T03:46:58.559456+0000 mgr.y (mgr.24368) 955 : cluster 0 pgmap v1037: 148 pgs: 148 active+clean; 1.0 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 980 op/s 2024-04-06T03:47:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:59 smithi153 bash[23852]: cluster 2024-04-06T03:46:58.959314+0000 osd.0 (osd.0) 605 : cluster 0 6.9 scrub starts 2024-04-06T03:47:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:59 smithi153 bash[23852]: cluster 2024-04-06T03:46:59.382555+0000 osd.0 (osd.0) 606 : cluster 0 6.9 scrub ok 2024-04-06T03:47:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:59 smithi153 bash[23852]: cluster 2024-04-06T03:46:59.383468+0000 osd.7 (osd.7) 607 : cluster 0 3.13 scrub starts 2024-04-06T03:47:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:46:59 smithi153 bash[23852]: cluster 2024-04-06T03:46:59.393862+0000 osd.7 (osd.7) 608 : cluster 0 3.13 scrub ok 2024-04-06T03:47:00.127 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:00.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:00.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:00.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:00.405 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:47:00.425 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:00.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-06T03:47:00.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:00.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:00.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:00.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:00.698 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:00.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:00.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.008150+0000 osd.0 (osd.0) 607 : cluster 0 5.1e scrub starts 2024-04-06T03:47:00.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.011075+0000 osd.0 (osd.0) 608 : cluster 0 5.1e scrub ok 2024-04-06T03:47:00.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.039250+0000 osd.7 (osd.7) 609 : cluster 0 1.0 scrub starts 2024-04-06T03:47:00.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.044945+0000 osd.7 (osd.7) 610 : cluster 0 1.0 scrub ok 2024-04-06T03:47:00.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.336130+0000 osd.2 (osd.2) 493 : cluster 0 2.e scrub starts 2024-04-06T03:47:00.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.344368+0000 osd.2 (osd.2) 494 : cluster 0 2.e scrub ok 2024-04-06T03:47:00.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.008150+0000 osd.0 (osd.0) 607 : cluster 0 5.1e scrub starts 2024-04-06T03:47:00.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.011075+0000 osd.0 (osd.0) 608 : cluster 0 5.1e scrub ok 2024-04-06T03:47:00.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.039250+0000 osd.7 (osd.7) 609 : cluster 0 1.0 scrub starts 2024-04-06T03:47:00.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.044945+0000 osd.7 (osd.7) 610 : cluster 0 1.0 scrub ok 2024-04-06T03:47:00.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.336130+0000 osd.2 (osd.2) 493 : cluster 0 2.e scrub starts 2024-04-06T03:47:00.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.344368+0000 osd.2 (osd.2) 494 : cluster 0 2.e scrub ok 2024-04-06T03:47:00.838 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:00.882 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1036,"stamp":"2024-04-06T03:46:56.558469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1208162678,"num_objects":18657,"num_object_clones":0,"num_object_copies":55971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18657,"num_whiteouts":0,"num_read":3620,"num_read_kb":3434,"num_write":194896,"num_write_kb":6784606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":2958161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78175,"ondisk_log_size":78175,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7598372,"kb_used_data":3609260,"kb_used_omap":10,"kb_used_meta":3988789,"kb_avail":648617692,"statfs":{"total":671965249536,"available":664184516608,"internally_reserved":0,"allocated":3695882240,"data_stored":3693491143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4084520043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,7,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-644743168,"num_objects":-9838,"num_object_clones":0,"num_object_copies":-29514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9838,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9838,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.850986"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":699,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725250+0000","last_change":"2024-04-06T03:43:33.261450+0000","last_active":"2024-04-06T03:46:45.725250+0000","last_peered":"2024-04-06T03:46:45.725250+0000","last_clean":"2024-04-06T03:46:45.725250+0000","last_became_active":"2024-04-06T03:42:23.588459+0000","last_became_peered":"2024-04-06T03:42:23.588459+0000","last_unstale":"2024-04-06T03:46:45.725250+0000","last_undegraded":"2024-04-06T03:46:45.725250+0000","last_fullsized":"2024-04-06T03:46:45.725250+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_clean_scrub_stamp":"2024-04-06T03:43:33.261299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":709,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710157+0000","last_change":"2024-04-06T03:45:17.323934+0000","last_active":"2024-04-06T03:46:45.710157+0000","last_peered":"2024-04-06T03:46:45.710157+0000","last_clean":"2024-04-06T03:46:45.710157+0000","last_became_active":"2024-04-06T03:42:57.110401+0000","last_became_peered":"2024-04-06T03:42:57.110401+0000","last_unstale":"2024-04-06T03:46:45.710157+0000","last_undegraded":"2024-04-06T03:46:45.710157+0000","last_fullsized":"2024-04-06T03:46:45.710157+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:17.323812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.323812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":727,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.716074+0000","last_change":"2024-04-06T03:45:21.991669+0000","last_active":"2024-04-06T03:46:45.716074+0000","last_peered":"2024-04-06T03:46:45.716074+0000","last_clean":"2024-04-06T03:46:45.716074+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:46:45.716074+0000","last_undegraded":"2024-04-06T03:46:45.716074+0000","last_fullsized":"2024-04-06T03:46:45.716074+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_clean_scrub_stamp":"2024-04-06T03:45:21.990975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"307'42","reported_seq":800,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724196+0000","last_change":"2024-04-06T03:44:58.409103+0000","last_active":"2024-04-06T03:46:45.724196+0000","last_peered":"2024-04-06T03:46:45.724196+0000","last_clean":"2024-04-06T03:46:45.724196+0000","last_became_active":"2024-04-06T03:42:23.588857+0000","last_became_peered":"2024-04-06T03:42:23.588857+0000","last_unstale":"2024-04-06T03:46:45.724196+0000","last_undegraded":"2024-04-06T03:46:45.724196+0000","last_fullsized":"2024-04-06T03:46:45.724196+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'42","last_scrub_stamp":"2024-04-06T03:44:58.408958+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.408958+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":701,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710488+0000","last_change":"2024-04-06T03:45:38.544714+0000","last_active":"2024-04-06T03:46:45.710488+0000","last_peered":"2024-04-06T03:46:45.710488+0000","last_clean":"2024-04-06T03:46:45.710488+0000","last_became_active":"2024-04-06T03:42:57.091902+0000","last_became_peered":"2024-04-06T03:42:57.091902+0000","last_unstale":"2024-04-06T03:46:45.710488+0000","last_undegraded":"2024-04-06T03:46:45.710488+0000","last_fullsized":"2024-04-06T03:46:45.710488+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:38.544490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:38.544490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:57.209066+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":584,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724846+0000","last_change":"2024-04-06T03:45:37.144389+0000","last_active":"2024-04-06T03:46:45.724846+0000","last_peered":"2024-04-06T03:46:45.724846+0000","last_clean":"2024-04-06T03:46:45.724846+0000","last_became_active":"2024-04-06T03:42:57.100816+0000","last_became_peered":"2024-04-06T03:42:57.100816+0000","last_unstale":"2024-04-06T03:46:45.724846+0000","last_undegraded":"2024-04-06T03:46:45.724846+0000","last_fullsized":"2024-04-06T03:46:45.724846+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:37.144305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:37.144305+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.144305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":704,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724036+0000","last_change":"2024-04-06T03:44:57.401411+0000","last_active":"2024-04-06T03:46:45.724036+0000","last_peered":"2024-04-06T03:46:45.724036+0000","last_clean":"2024-04-06T03:46:45.724036+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:46:45.724036+0000","last_undegraded":"2024-04-06T03:46:45.724036+0000","last_fullsized":"2024-04-06T03:46:45.724036+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:57.401286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.401286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"307'39","reported_seq":772,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:50.084961+0000","last_change":"2024-04-06T03:46:50.085063+0000","last_active":"2024-04-06T03:46:50.084961+0000","last_peered":"2024-04-06T03:46:50.084961+0000","last_clean":"2024-04-06T03:46:50.084961+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:46:50.084961+0000","last_undegraded":"2024-04-06T03:46:50.084961+0000","last_fullsized":"2024-04-06T03:46:50.084961+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:46:50.084593+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:46:50.084593+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:15.546746+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":103,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.727543+0000","last_change":"2024-04-06T03:45:08.003956+0000","last_active":"2024-04-06T03:46:45.727543+0000","last_peered":"2024-04-06T03:46:45.727543+0000","last_clean":"2024-04-06T03:46:45.727543+0000","last_became_active":"2024-04-06T03:42:57.078887+0000","last_became_peered":"2024-04-06T03:42:57.078887+0000","last_unstale":"2024-04-06T03:46:45.727543+0000","last_undegraded":"2024-04-06T03:46:45.727543+0000","last_fullsized":"2024-04-06T03:46:45.727543+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:08.003828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:45:08.003828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":695,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724812+0000","last_change":"2024-04-06T03:45:25.274860+0000","last_active":"2024-04-06T03:46:45.724812+0000","last_peered":"2024-04-06T03:46:45.724812+0000","last_clean":"2024-04-06T03:46:45.724812+0000","last_became_active":"2024-04-06T03:42:57.112847+0000","last_became_peered":"2024-04-06T03:42:57.112847+0000","last_unstale":"2024-04-06T03:46:45.724812+0000","last_undegraded":"2024-04-06T03:46:45.724812+0000","last_fullsized":"2024-04-06T03:46:45.724812+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_clean_scrub_stamp":"2024-04-06T03:45:25.274778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":733,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726488+0000","last_change":"2024-04-06T03:45:04.452779+0000","last_active":"2024-04-06T03:46:45.726488+0000","last_peered":"2024-04-06T03:46:45.726488+0000","last_clean":"2024-04-06T03:46:45.726488+0000","last_became_active":"2024-04-06T03:42:23.570089+0000","last_became_peered":"2024-04-06T03:42:23.570089+0000","last_unstale":"2024-04-06T03:46:45.726488+0000","last_undegraded":"2024-04-06T03:46:45.726488+0000","last_fullsized":"2024-04-06T03:46:45.726488+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:45:04.452660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.452660+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"307'46","reported_seq":735,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710493+0000","last_change":"2024-04-06T03:44:57.339512+0000","last_active":"2024-04-06T03:46:45.710493+0000","last_peered":"2024-04-06T03:46:45.710493+0000","last_clean":"2024-04-06T03:46:45.710493+0000","last_became_active":"2024-04-06T03:42:57.110510+0000","last_became_peered":"2024-04-06T03:42:57.110510+0000","last_unstale":"2024-04-06T03:46:45.710493+0000","last_undegraded":"2024-04-06T03:46:45.710493+0000","last_fullsized":"2024-04-06T03:46:45.710493+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'46","last_scrub_stamp":"2024-04-06T03:44:57.339353+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.339353+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":699,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712040+0000","last_change":"2024-04-06T03:45:05.101359+0000","last_active":"2024-04-06T03:46:45.712040+0000","last_peered":"2024-04-06T03:46:45.712040+0000","last_clean":"2024-04-06T03:46:45.712040+0000","last_became_active":"2024-04-06T03:42:57.092354+0000","last_became_peered":"2024-04-06T03:42:57.092354+0000","last_unstale":"2024-04-06T03:46:45.712040+0000","last_undegraded":"2024-04-06T03:46:45.712040+0000","last_fullsized":"2024-04-06T03:46:45.712040+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:05.101251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:05.101251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":574,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712272+0000","last_change":"2024-04-06T03:45:02.133616+0000","last_active":"2024-04-06T03:46:45.712272+0000","last_peered":"2024-04-06T03:46:45.712272+0000","last_clean":"2024-04-06T03:46:45.712272+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:46:45.712272+0000","last_undegraded":"2024-04-06T03:46:45.712272+0000","last_fullsized":"2024-04-06T03:46:45.712272+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:02.132904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.132904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:49.067415+0000","last_change":"2024-04-06T03:46:49.067449+0000","last_active":"2024-04-06T03:46:49.067415+0000","last_peered":"2024-04-06T03:46:49.067415+0000","last_clean":"2024-04-06T03:46:49.067415+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:46:49.067415+0000","last_undegraded":"2024-04-06T03:46:49.067415+0000","last_fullsized":"2024-04-06T03:46:49.067415+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:49.067339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:46:49.067339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:11.262399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"307'36","reported_seq":722,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711221+0000","last_change":"2024-04-06T03:45:18.356128+0000","last_active":"2024-04-06T03:46:45.711221+0000","last_peered":"2024-04-06T03:46:45.711221+0000","last_clean":"2024-04-06T03:46:45.711221+0000","last_became_active":"2024-04-06T03:42:23.592623+0000","last_became_peered":"2024-04-06T03:42:23.592623+0000","last_unstale":"2024-04-06T03:46:45.711221+0000","last_undegraded":"2024-04-06T03:46:45.711221+0000","last_fullsized":"2024-04-06T03:46:45.711221+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'36","last_scrub_stamp":"2024-04-06T03:45:18.356021+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:45:18.356021+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean+scrubbing","last_fresh":"2024-04-06T03:46:53.099092+0000","last_change":"2024-04-06T03:46:53.097783+0000","last_active":"2024-04-06T03:46:53.099092+0000","last_peered":"2024-04-06T03:46:53.099092+0000","last_clean":"2024-04-06T03:46:53.099092+0000","last_became_active":"2024-04-06T03:42:23.577258+0000","last_became_peered":"2024-04-06T03:42:23.577258+0000","last_unstale":"2024-04-06T03:46:53.099092+0000","last_undegraded":"2024-04-06T03:46:53.099092+0000","last_fullsized":"2024-04-06T03:46:53.099092+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:04.325567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:04.325567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":699,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710825+0000","last_change":"2024-04-06T03:43:23.225743+0000","last_active":"2024-04-06T03:46:45.710825+0000","last_peered":"2024-04-06T03:46:45.710825+0000","last_clean":"2024-04-06T03:46:45.710825+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:46:45.710825+0000","last_undegraded":"2024-04-06T03:46:45.710825+0000","last_fullsized":"2024-04-06T03:46:45.710825+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.225643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.225643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":703,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725484+0000","last_change":"2024-04-06T03:45:01.585396+0000","last_active":"2024-04-06T03:46:45.725484+0000","last_peered":"2024-04-06T03:46:45.725484+0000","last_clean":"2024-04-06T03:46:45.725484+0000","last_became_active":"2024-04-06T03:42:23.588672+0000","last_became_peered":"2024-04-06T03:42:23.588672+0000","last_unstale":"2024-04-06T03:46:45.725484+0000","last_undegraded":"2024-04-06T03:46:45.725484+0000","last_fullsized":"2024-04-06T03:46:45.725484+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.585125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.585125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"307'39","reported_seq":787,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710649+0000","last_change":"2024-04-06T03:45:04.428891+0000","last_active":"2024-04-06T03:46:45.710649+0000","last_peered":"2024-04-06T03:46:45.710649+0000","last_clean":"2024-04-06T03:46:45.710649+0000","last_became_active":"2024-04-06T03:42:57.096631+0000","last_became_peered":"2024-04-06T03:42:57.096631+0000","last_unstale":"2024-04-06T03:46:45.710649+0000","last_undegraded":"2024-04-06T03:46:45.710649+0000","last_fullsized":"2024-04-06T03:46:45.710649+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:45:04.428726+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.428726+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":701,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711973+0000","last_change":"2024-04-06T03:43:35.230118+0000","last_active":"2024-04-06T03:46:45.711973+0000","last_peered":"2024-04-06T03:46:45.711973+0000","last_clean":"2024-04-06T03:46:45.711973+0000","last_became_active":"2024-04-06T03:42:23.594112+0000","last_became_peered":"2024-04-06T03:42:23.594112+0000","last_unstale":"2024-04-06T03:46:45.711973+0000","last_undegraded":"2024-04-06T03:46:45.711973+0000","last_fullsized":"2024-04-06T03:46:45.711973+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:35.230016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:07.110775+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.230016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":100,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711376+0000","last_change":"2024-04-06T03:43:36.185334+0000","last_active":"2024-04-06T03:46:45.711376+0000","last_peered":"2024-04-06T03:46:45.711376+0000","last_clean":"2024-04-06T03:46:45.711376+0000","last_became_active":"2024-04-06T03:42:57.103063+0000","last_became_peered":"2024-04-06T03:42:57.103063+0000","last_unstale":"2024-04-06T03:46:45.711376+0000","last_undegraded":"2024-04-06T03:46:45.711376+0000","last_fullsized":"2024-04-06T03:46:45.711376+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:36.185042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:43:36.185042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":572,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710818+0000","last_change":"2024-04-06T03:45:26.399149+0000","last_active":"2024-04-06T03:46:45.710818+0000","last_peered":"2024-04-06T03:46:45.710818+0000","last_clean":"2024-04-06T03:46:45.710818+0000","last_became_active":"2024-04-06T03:42:23.593721+0000","last_became_peered":"2024-04-06T03:42:23.593721+0000","last_unstale":"2024-04-06T03:46:45.710818+0000","last_undegraded":"2024-04-06T03:46:45.710818+0000","last_fullsized":"2024-04-06T03:46:45.710818+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:26.399024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:26.399024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:52.434811+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"307'18","reported_seq":740,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.713460+0000","last_change":"2024-04-06T03:45:37.208824+0000","last_active":"2024-04-06T03:46:45.713460+0000","last_peered":"2024-04-06T03:46:45.713460+0000","last_clean":"2024-04-06T03:46:45.713460+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:46:45.713460+0000","last_undegraded":"2024-04-06T03:46:45.713460+0000","last_fullsized":"2024-04-06T03:46:45.713460+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'18","last_scrub_stamp":"2024-04-06T03:45:37.208678+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.208678+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":575,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710596+0000","last_change":"2024-04-06T03:43:44.166907+0000","last_active":"2024-04-06T03:46:45.710596+0000","last_peered":"2024-04-06T03:46:45.710596+0000","last_clean":"2024-04-06T03:46:45.710596+0000","last_became_active":"2024-04-06T03:42:57.096442+0000","last_became_peered":"2024-04-06T03:42:57.096442+0000","last_unstale":"2024-04-06T03:46:45.710596+0000","last_undegraded":"2024-04-06T03:46:45.710596+0000","last_fullsized":"2024-04-06T03:46:45.710596+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:44.166672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:44.166672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":699,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.714743+0000","last_change":"2024-04-06T03:43:26.134513+0000","last_active":"2024-04-06T03:46:45.714743+0000","last_peered":"2024-04-06T03:46:45.714743+0000","last_clean":"2024-04-06T03:46:45.714743+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:46:45.714743+0000","last_undegraded":"2024-04-06T03:46:45.714743+0000","last_fullsized":"2024-04-06T03:46:45.714743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.134284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":712,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.734538+0000","last_change":"2024-04-06T03:43:28.171388+0000","last_active":"2024-04-06T03:46:45.734538+0000","last_peered":"2024-04-06T03:46:45.734538+0000","last_clean":"2024-04-06T03:46:45.734538+0000","last_became_active":"2024-04-06T03:42:23.577120+0000","last_became_peered":"2024-04-06T03:42:23.577120+0000","last_unstale":"2024-04-06T03:46:45.734538+0000","last_undegraded":"2024-04-06T03:46:45.734538+0000","last_fullsized":"2024-04-06T03:46:45.734538+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:43:28.171183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:28.171183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"307'32","reported_seq":756,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725638+0000","last_change":"2024-04-06T03:45:17.921718+0000","last_active":"2024-04-06T03:46:45.725638+0000","last_peered":"2024-04-06T03:46:45.725638+0000","last_clean":"2024-04-06T03:46:45.725638+0000","last_became_active":"2024-04-06T03:42:57.070724+0000","last_became_peered":"2024-04-06T03:42:57.070724+0000","last_unstale":"2024-04-06T03:46:45.725638+0000","last_undegraded":"2024-04-06T03:46:45.725638+0000","last_fullsized":"2024-04-06T03:46:45.725638+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'32","last_scrub_stamp":"2024-04-06T03:45:17.921596+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.921596+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":688,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725684+0000","last_change":"2024-04-06T03:45:11.392248+0000","last_active":"2024-04-06T03:46:45.725684+0000","last_peered":"2024-04-06T03:46:45.725684+0000","last_clean":"2024-04-06T03:46:45.725684+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:46:45.725684+0000","last_undegraded":"2024-04-06T03:46:45.725684+0000","last_fullsized":"2024-04-06T03:46:45.725684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:45:11.392164+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:41:29.127643+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.392164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"327'15","reported_seq":913,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.416705+0000","last_change":"2024-04-06T03:45:36.499577+0000","last_active":"2024-04-06T03:46:47.416705+0000","last_peered":"2024-04-06T03:46:47.416705+0000","last_clean":"2024-04-06T03:46:47.416705+0000","last_became_active":"2024-04-06T03:42:23.594397+0000","last_became_peered":"2024-04-06T03:42:23.594397+0000","last_unstale":"2024-04-06T03:46:47.416705+0000","last_undegraded":"2024-04-06T03:46:47.416705+0000","last_fullsized":"2024-04-06T03:46:47.416705+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-04-06T03:45:36.499448+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:45:36.499448+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":574,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710222+0000","last_change":"2024-04-06T03:45:31.513869+0000","last_active":"2024-04-06T03:46:45.710222+0000","last_peered":"2024-04-06T03:46:45.710222+0000","last_clean":"2024-04-06T03:46:45.710222+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:46:45.710222+0000","last_undegraded":"2024-04-06T03:46:45.710222+0000","last_fullsized":"2024-04-06T03:46:45.710222+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:31.513718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:45:31.513718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:57.374026+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"307'74","reported_seq":800,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.734805+0000","last_change":"2024-04-06T03:45:13.928026+0000","last_active":"2024-04-06T03:46:45.734805+0000","last_peered":"2024-04-06T03:46:45.734805+0000","last_clean":"2024-04-06T03:46:45.734805+0000","last_became_active":"2024-04-06T03:42:57.093866+0000","last_became_peered":"2024-04-06T03:42:57.093866+0000","last_unstale":"2024-04-06T03:46:45.734805+0000","last_undegraded":"2024-04-06T03:46:45.734805+0000","last_fullsized":"2024-04-06T03:46:45.734805+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'74","last_scrub_stamp":"2024-04-06T03:45:13.927878+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:45:13.927878+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":692,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725857+0000","last_change":"2024-04-06T03:43:34.209777+0000","last_active":"2024-04-06T03:46:45.725857+0000","last_peered":"2024-04-06T03:46:45.725857+0000","last_clean":"2024-04-06T03:46:45.725857+0000","last_became_active":"2024-04-06T03:42:23.588998+0000","last_became_peered":"2024-04-06T03:42:23.588998+0000","last_unstale":"2024-04-06T03:46:45.725857+0000","last_undegraded":"2024-04-06T03:46:45.725857+0000","last_fullsized":"2024-04-06T03:46:45.725857+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:34.209617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:43:34.209617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":710,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:54.004874+0000","last_change":"2024-04-06T03:46:54.004908+0000","last_active":"2024-04-06T03:46:54.004874+0000","last_peered":"2024-04-06T03:46:54.004874+0000","last_clean":"2024-04-06T03:46:54.004874+0000","last_became_active":"2024-04-06T03:42:23.577555+0000","last_became_peered":"2024-04-06T03:42:23.577555+0000","last_unstale":"2024-04-06T03:46:54.004874+0000","last_undegraded":"2024-04-06T03:46:54.004874+0000","last_fullsized":"2024-04-06T03:46:54.004874+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:54.004800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:46:54.004800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:13.380811+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":596,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:51.174821+0000","last_change":"2024-04-06T03:46:51.174954+0000","last_active":"2024-04-06T03:46:51.174821+0000","last_peered":"2024-04-06T03:46:51.174821+0000","last_clean":"2024-04-06T03:46:51.174821+0000","last_became_active":"2024-04-06T03:42:23.582180+0000","last_became_peered":"2024-04-06T03:42:23.582180+0000","last_unstale":"2024-04-06T03:46:51.174821+0000","last_undegraded":"2024-04-06T03:46:51.174821+0000","last_fullsized":"2024-04-06T03:46:51.174821+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:51.174158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:46:51.174158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:13.918372+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"344'39","reported_seq":669,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.727195+0000","last_change":"2024-04-06T03:43:22.294700+0000","last_active":"2024-04-06T03:46:45.727195+0000","last_peered":"2024-04-06T03:46:45.727195+0000","last_clean":"2024-04-06T03:46:45.727195+0000","last_became_active":"2024-04-06T03:42:57.093955+0000","last_became_peered":"2024-04-06T03:42:57.093955+0000","last_unstale":"2024-04-06T03:46:45.727195+0000","last_undegraded":"2024-04-06T03:46:45.727195+0000","last_fullsized":"2024-04-06T03:46:45.727195+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'39","last_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_deep_scrub":"344'39","last_deep_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_clean_scrub_stamp":"2024-04-06T03:43:22.294366+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":694,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.027044+0000","last_change":"2024-04-06T03:46:47.027083+0000","last_active":"2024-04-06T03:46:47.027044+0000","last_peered":"2024-04-06T03:46:47.027044+0000","last_clean":"2024-04-06T03:46:47.027044+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:46:47.027044+0000","last_undegraded":"2024-04-06T03:46:47.027044+0000","last_fullsized":"2024-04-06T03:46:47.027044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:47.026665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:46:47.026665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:00.658796+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":573,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710143+0000","last_change":"2024-04-06T03:45:28.454818+0000","last_active":"2024-04-06T03:46:45.710143+0000","last_peered":"2024-04-06T03:46:45.710143+0000","last_clean":"2024-04-06T03:46:45.710143+0000","last_became_active":"2024-04-06T03:42:23.580807+0000","last_became_peered":"2024-04-06T03:42:23.580807+0000","last_unstale":"2024-04-06T03:46:45.710143+0000","last_undegraded":"2024-04-06T03:46:45.710143+0000","last_fullsized":"2024-04-06T03:46:45.710143+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:28.454731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:28.454731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":599,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.217011+0000","last_change":"2024-04-06T03:46:47.217038+0000","last_active":"2024-04-06T03:46:47.217011+0000","last_peered":"2024-04-06T03:46:47.217011+0000","last_clean":"2024-04-06T03:46:47.217011+0000","last_became_active":"2024-04-06T03:42:23.578658+0000","last_became_peered":"2024-04-06T03:42:23.578658+0000","last_unstale":"2024-04-06T03:46:47.217011+0000","last_undegraded":"2024-04-06T03:46:47.217011+0000","last_fullsized":"2024-04-06T03:46:47.217011+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:47.216929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:46:47.216929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:49.286842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"307'28","reported_seq":768,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.715748+0000","last_change":"2024-04-06T03:46:44.002540+0000","last_active":"2024-04-06T03:46:45.715748+0000","last_peered":"2024-04-06T03:46:45.715748+0000","last_clean":"2024-04-06T03:46:45.715748+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:46:45.715748+0000","last_undegraded":"2024-04-06T03:46:45.715748+0000","last_fullsized":"2024-04-06T03:46:45.715748+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'28","last_scrub_stamp":"2024-04-06T03:46:44.002421+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:46:44.002421+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:04.666967+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":680,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.732314+0000","last_change":"2024-04-06T03:43:29.215763+0000","last_active":"2024-04-06T03:46:45.732314+0000","last_peered":"2024-04-06T03:46:45.732314+0000","last_clean":"2024-04-06T03:46:45.732314+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:46:45.732314+0000","last_undegraded":"2024-04-06T03:46:45.732314+0000","last_fullsized":"2024-04-06T03:46:45.732314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:29.215509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:29.215509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":100,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723886+0000","last_change":"2024-04-06T03:45:15.944386+0000","last_active":"2024-04-06T03:46:45.723886+0000","last_peered":"2024-04-06T03:46:45.723886+0000","last_clean":"2024-04-06T03:46:45.723886+0000","last_became_active":"2024-04-06T03:42:57.085102+0000","last_became_peered":"2024-04-06T03:42:57.085102+0000","last_unstale":"2024-04-06T03:46:45.723886+0000","last_undegraded":"2024-04-06T03:46:45.723886+0000","last_fullsized":"2024-04-06T03:46:45.723886+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:15.944262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:15.944262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":99,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.713804+0000","last_change":"2024-04-06T03:43:29.201914+0000","last_active":"2024-04-06T03:46:45.713804+0000","last_peered":"2024-04-06T03:46:45.713804+0000","last_clean":"2024-04-06T03:46:45.713804+0000","last_became_active":"2024-04-06T03:42:57.091715+0000","last_became_peered":"2024-04-06T03:42:57.091715+0000","last_unstale":"2024-04-06T03:46:45.713804+0000","last_undegraded":"2024-04-06T03:46:45.713804+0000","last_fullsized":"2024-04-06T03:46:45.713804+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:29.201580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:43:29.201580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"307'43","reported_seq":818,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710694+0000","last_change":"2024-04-06T03:43:26.202165+0000","last_active":"2024-04-06T03:46:45.710694+0000","last_peered":"2024-04-06T03:46:45.710694+0000","last_clean":"2024-04-06T03:46:45.710694+0000","last_became_active":"2024-04-06T03:42:23.592079+0000","last_became_peered":"2024-04-06T03:42:23.592079+0000","last_unstale":"2024-04-06T03:46:45.710694+0000","last_undegraded":"2024-04-06T03:46:45.710694+0000","last_fullsized":"2024-04-06T03:46:45.710694+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'43","last_scrub_stamp":"2024-04-06T03:43:26.202029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.202029+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":700,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711910+0000","last_change":"2024-04-06T03:43:45.957138+0000","last_active":"2024-04-06T03:46:45.711910+0000","last_peered":"2024-04-06T03:46:45.711910+0000","last_clean":"2024-04-06T03:46:45.711910+0000","last_became_active":"2024-04-06T03:42:57.089541+0000","last_became_peered":"2024-04-06T03:42:57.089541+0000","last_unstale":"2024-04-06T03:46:45.711910+0000","last_undegraded":"2024-04-06T03:46:45.711910+0000","last_fullsized":"2024-04-06T03:46:45.711910+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_clean_scrub_stamp":"2024-04-06T03:43:45.956977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":690,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725650+0000","last_change":"2024-04-06T03:43:26.163518+0000","last_active":"2024-04-06T03:46:45.725650+0000","last_peered":"2024-04-06T03:46:45.725650+0000","last_clean":"2024-04-06T03:46:45.725650+0000","last_became_active":"2024-04-06T03:42:57.079386+0000","last_became_peered":"2024-04-06T03:42:57.079386+0000","last_unstale":"2024-04-06T03:46:45.725650+0000","last_undegraded":"2024-04-06T03:46:45.725650+0000","last_fullsized":"2024-04-06T03:46:45.725650+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:26.163320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.163320+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.163320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":564,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724290+0000","last_change":"2024-04-06T03:43:41.985445+0000","last_active":"2024-04-06T03:46:45.724290+0000","last_peered":"2024-04-06T03:46:45.724290+0000","last_clean":"2024-04-06T03:46:45.724290+0000","last_became_active":"2024-04-06T03:42:23.589224+0000","last_became_peered":"2024-04-06T03:42:23.589224+0000","last_unstale":"2024-04-06T03:46:45.724290+0000","last_undegraded":"2024-04-06T03:46:45.724290+0000","last_fullsized":"2024-04-06T03:46:45.724290+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:41.985333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:43:41.985333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"307'27","reported_seq":739,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726589+0000","last_change":"2024-04-06T03:43:38.248493+0000","last_active":"2024-04-06T03:46:45.726589+0000","last_peered":"2024-04-06T03:46:45.726589+0000","last_clean":"2024-04-06T03:46:45.726589+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:46:45.726589+0000","last_undegraded":"2024-04-06T03:46:45.726589+0000","last_fullsized":"2024-04-06T03:46:45.726589+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:43:38.248258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:38.248258+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":695,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711108+0000","last_change":"2024-04-06T03:44:58.475184+0000","last_active":"2024-04-06T03:46:45.711108+0000","last_peered":"2024-04-06T03:46:45.711108+0000","last_clean":"2024-04-06T03:46:45.711108+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:46:45.711108+0000","last_undegraded":"2024-04-06T03:46:45.711108+0000","last_fullsized":"2024-04-06T03:46:45.711108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:58.475083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.475083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":1016,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.417693+0000","last_change":"2024-04-06T03:45:41.030902+0000","last_active":"2024-04-06T03:46:47.417693+0000","last_peered":"2024-04-06T03:46:47.417693+0000","last_clean":"2024-04-06T03:46:47.417693+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:46:47.417693+0000","last_undegraded":"2024-04-06T03:46:47.417693+0000","last_fullsized":"2024-04-06T03:46:47.417693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:45:41.030592+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.030592+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:04.211736+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":714,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712547+0000","last_change":"2024-04-06T03:45:34.135652+0000","last_active":"2024-04-06T03:46:45.712547+0000","last_peered":"2024-04-06T03:46:45.712547+0000","last_clean":"2024-04-06T03:46:45.712547+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:46:45.712547+0000","last_undegraded":"2024-04-06T03:46:45.712547+0000","last_fullsized":"2024-04-06T03:46:45.712547+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.135479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.135479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:51.604553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"307'21","reported_seq":736,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.714402+0000","last_change":"2024-04-06T03:43:37.054237+0000","last_active":"2024-04-06T03:46:45.714402+0000","last_peered":"2024-04-06T03:46:45.714402+0000","last_clean":"2024-04-06T03:46:45.714402+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:46:45.714402+0000","last_undegraded":"2024-04-06T03:46:45.714402+0000","last_fullsized":"2024-04-06T03:46:45.714402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'21","last_scrub_stamp":"2024-04-06T03:43:37.053855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:37.053855+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":679,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.714933+0000","last_change":"2024-04-06T03:43:15.163263+0000","last_active":"2024-04-06T03:46:45.714933+0000","last_peered":"2024-04-06T03:46:45.714933+0000","last_clean":"2024-04-06T03:46:45.714933+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:46:45.714933+0000","last_undegraded":"2024-04-06T03:46:45.714933+0000","last_fullsized":"2024-04-06T03:46:45.714933+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:15.163100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:43:15.163100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"338'15","reported_seq":813,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.459508+0000","last_change":"2024-04-06T03:46:44.991251+0000","last_active":"2024-04-06T03:46:47.459508+0000","last_peered":"2024-04-06T03:46:47.459508+0000","last_clean":"2024-04-06T03:46:47.459508+0000","last_became_active":"2024-04-06T03:42:57.087361+0000","last_became_peered":"2024-04-06T03:42:57.087361+0000","last_unstale":"2024-04-06T03:46:47.459508+0000","last_undegraded":"2024-04-06T03:46:47.459508+0000","last_fullsized":"2024-04-06T03:46:47.459508+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'15","last_scrub_stamp":"2024-04-06T03:46:44.991131+0000","last_deep_scrub":"309'14","last_deep_scrub_stamp":"2024-04-06T03:41:21.025392+0000","last_clean_scrub_stamp":"2024-04-06T03:46:44.991131+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:57.162775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":564,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723754+0000","last_change":"2024-04-06T03:45:35.931813+0000","last_active":"2024-04-06T03:46:45.723754+0000","last_peered":"2024-04-06T03:46:45.723754+0000","last_clean":"2024-04-06T03:46:45.723754+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:46:45.723754+0000","last_undegraded":"2024-04-06T03:46:45.723754+0000","last_fullsized":"2024-04-06T03:46:45.723754+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:35.931720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:21.406614+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.931720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:55.171073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"307'32","reported_seq":742,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725499+0000","last_change":"2024-04-06T03:43:10.275147+0000","last_active":"2024-04-06T03:46:45.725499+0000","last_peered":"2024-04-06T03:46:45.725499+0000","last_clean":"2024-04-06T03:46:45.725499+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:46:45.725499+0000","last_undegraded":"2024-04-06T03:46:45.725499+0000","last_fullsized":"2024-04-06T03:46:45.725499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'32","last_scrub_stamp":"2024-04-06T03:43:10.274736+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:43:10.274736+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726216+0000","last_change":"2024-04-06T03:44:59.520843+0000","last_active":"2024-04-06T03:46:45.726216+0000","last_peered":"2024-04-06T03:46:45.726216+0000","last_clean":"2024-04-06T03:46:45.726216+0000","last_became_active":"2024-04-06T03:42:23.574848+0000","last_became_peered":"2024-04-06T03:42:23.574848+0000","last_unstale":"2024-04-06T03:46:45.726216+0000","last_undegraded":"2024-04-06T03:46:45.726216+0000","last_fullsized":"2024-04-06T03:46:45.726216+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:59.520731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:44:59.520731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":564,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723990+0000","last_change":"2024-04-06T03:45:11.925553+0000","last_active":"2024-04-06T03:46:45.723990+0000","last_peered":"2024-04-06T03:46:45.723990+0000","last_clean":"2024-04-06T03:46:45.723990+0000","last_became_active":"2024-04-06T03:42:57.092555+0000","last_became_peered":"2024-04-06T03:42:57.092555+0000","last_unstale":"2024-04-06T03:46:45.723990+0000","last_undegraded":"2024-04-06T03:46:45.723990+0000","last_fullsized":"2024-04-06T03:46:45.723990+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:11.925422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.925422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":578,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710733+0000","last_change":"2024-04-06T03:45:29.469379+0000","last_active":"2024-04-06T03:46:45.710733+0000","last_peered":"2024-04-06T03:46:45.710733+0000","last_clean":"2024-04-06T03:46:45.710733+0000","last_became_active":"2024-04-06T03:42:57.109758+0000","last_became_peered":"2024-04-06T03:42:57.109758+0000","last_unstale":"2024-04-06T03:46:45.710733+0000","last_undegraded":"2024-04-06T03:46:45.710733+0000","last_fullsized":"2024-04-06T03:46:45.710733+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.469254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.469254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:57.458772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"307'14","reported_seq":693,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712357+0000","last_change":"2024-04-06T03:45:35.162259+0000","last_active":"2024-04-06T03:46:45.712357+0000","last_peered":"2024-04-06T03:46:45.712357+0000","last_clean":"2024-04-06T03:46:45.712357+0000","last_became_active":"2024-04-06T03:42:23.581903+0000","last_became_peered":"2024-04-06T03:42:23.581903+0000","last_unstale":"2024-04-06T03:46:45.712357+0000","last_undegraded":"2024-04-06T03:46:45.712357+0000","last_fullsized":"2024-04-06T03:46:45.712357+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'14","last_scrub_stamp":"2024-04-06T03:45:35.162112+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.162112+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":722,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.728514+0000","last_change":"2024-04-06T03:46:41.301282+0000","last_active":"2024-04-06T03:46:45.728514+0000","last_peered":"2024-04-06T03:46:45.728514+0000","last_clean":"2024-04-06T03:46:45.728514+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:46:45.728514+0000","last_undegraded":"2024-04-06T03:46:45.728514+0000","last_fullsized":"2024-04-06T03:46:45.728514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:46:41.301153+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:46:41.301153+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:55.802820+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:49.107895+0000","last_change":"2024-04-06T03:46:49.107917+0000","last_active":"2024-04-06T03:46:49.107895+0000","last_peered":"2024-04-06T03:46:49.107895+0000","last_clean":"2024-04-06T03:46:49.107895+0000","last_became_active":"2024-04-06T03:42:57.084986+0000","last_became_peered":"2024-04-06T03:42:57.084986+0000","last_unstale":"2024-04-06T03:46:49.107895+0000","last_undegraded":"2024-04-06T03:46:49.107895+0000","last_fullsized":"2024-04-06T03:46:49.107895+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:49.107837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:46:49.107837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:01.195456+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":738,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.719368+0000","last_change":"2024-04-06T03:45:29.043189+0000","last_active":"2024-04-06T03:46:45.719368+0000","last_peered":"2024-04-06T03:46:45.719368+0000","last_clean":"2024-04-06T03:46:45.719368+0000","last_became_active":"2024-04-06T03:42:57.071688+0000","last_became_peered":"2024-04-06T03:42:57.071688+0000","last_unstale":"2024-04-06T03:46:45.719368+0000","last_undegraded":"2024-04-06T03:46:45.719368+0000","last_fullsized":"2024-04-06T03:46:45.719368+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.043086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.043086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"307'39","reported_seq":803,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724737+0000","last_change":"2024-04-06T03:43:26.230289+0000","last_active":"2024-04-06T03:46:45.724737+0000","last_peered":"2024-04-06T03:46:45.724737+0000","last_clean":"2024-04-06T03:46:45.724737+0000","last_became_active":"2024-04-06T03:42:23.574886+0000","last_became_peered":"2024-04-06T03:42:23.574886+0000","last_unstale":"2024-04-06T03:46:45.724737+0000","last_undegraded":"2024-04-06T03:46:45.724737+0000","last_fullsized":"2024-04-06T03:46:45.724737+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:43:26.230137+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.230137+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"377'12249","reported_seq":12388,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:52.475547+0000","last_change":"2024-04-06T03:45:41.170863+0000","last_active":"2024-04-06T03:46:52.475547+0000","last_peered":"2024-04-06T03:46:52.475547+0000","last_clean":"2024-04-06T03:46:52.475547+0000","last_became_active":"2024-04-06T03:43:25.018581+0000","last_became_peered":"2024-04-06T03:43:25.018581+0000","last_unstale":"2024-04-06T03:46:52.475547+0000","last_undegraded":"2024-04-06T03:46:52.475547+0000","last_fullsized":"2024-04-06T03:46:52.475547+0000","mapping_epoch":345,"log_start":"364'7700","ondisk_log_start":"364'7700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'8005","last_scrub_stamp":"2024-04-06T03:45:41.170736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.170736+0000","objects_scrubbed":5473,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:05.311420+0000","scrub_duration":2019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12249,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":695,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725730+0000","last_change":"2024-04-06T03:45:30.126413+0000","last_active":"2024-04-06T03:46:45.725730+0000","last_peered":"2024-04-06T03:46:45.725730+0000","last_clean":"2024-04-06T03:46:45.725730+0000","last_became_active":"2024-04-06T03:42:57.093752+0000","last_became_peered":"2024-04-06T03:42:57.093752+0000","last_unstale":"2024-04-06T03:46:45.725730+0000","last_undegraded":"2024-04-06T03:46:45.725730+0000","last_fullsized":"2024-04-06T03:46:45.725730+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.126246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.126246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":726,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710575+0000","last_change":"2024-04-06T03:44:59.372004+0000","last_active":"2024-04-06T03:46:45.710575+0000","last_peered":"2024-04-06T03:46:45.710575+0000","last_clean":"2024-04-06T03:46:45.710575+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:46:45.710575+0000","last_undegraded":"2024-04-06T03:46:45.710575+0000","last_fullsized":"2024-04-06T03:46:45.710575+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:44:59.371923+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:44:59.371923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"307'39","reported_seq":791,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726989+0000","last_change":"2024-04-06T03:45:38.163802+0000","last_active":"2024-04-06T03:46:45.726989+0000","last_peered":"2024-04-06T03:46:45.726989+0000","last_clean":"2024-04-06T03:46:45.726989+0000","last_became_active":"2024-04-06T03:42:57.113147+0000","last_became_peered":"2024-04-06T03:42:57.113147+0000","last_unstale":"2024-04-06T03:46:45.726989+0000","last_undegraded":"2024-04-06T03:46:45.726989+0000","last_fullsized":"2024-04-06T03:46:45.726989+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:45:38.163676+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:45:38.163676+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:55.642070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":722,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.719458+0000","last_change":"2024-04-06T03:43:39.845859+0000","last_active":"2024-04-06T03:46:45.719458+0000","last_peered":"2024-04-06T03:46:45.719458+0000","last_clean":"2024-04-06T03:46:45.719458+0000","last_became_active":"2024-04-06T03:42:57.074804+0000","last_became_peered":"2024-04-06T03:42:57.074804+0000","last_unstale":"2024-04-06T03:46:45.719458+0000","last_undegraded":"2024-04-06T03:46:45.719458+0000","last_fullsized":"2024-04-06T03:46:45.719458+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:39.845625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:43:39.845625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"377'12388","reported_seq":12536,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:55.355348+0000","last_change":"2024-04-06T03:45:27.058446+0000","last_active":"2024-04-06T03:46:55.355348+0000","last_peered":"2024-04-06T03:46:55.355348+0000","last_clean":"2024-04-06T03:46:55.355348+0000","last_became_active":"2024-04-06T03:43:25.130333+0000","last_became_peered":"2024-04-06T03:43:25.130333+0000","last_unstale":"2024-04-06T03:46:55.355348+0000","last_undegraded":"2024-04-06T03:46:55.355348+0000","last_fullsized":"2024-04-06T03:46:55.355348+0000","mapping_epoch":345,"log_start":"364'6900","ondisk_log_start":"364'6900","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"364'7153","last_scrub_stamp":"2024-04-06T03:45:27.058114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:27.058114+0000","objects_scrubbed":6395,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70975512,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12388,"num_write_kb":431041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":703,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711918+0000","last_change":"2024-04-06T03:45:28.217075+0000","last_active":"2024-04-06T03:46:45.711918+0000","last_peered":"2024-04-06T03:46:45.711918+0000","last_clean":"2024-04-06T03:46:45.711918+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:46:45.711918+0000","last_undegraded":"2024-04-06T03:46:45.711918+0000","last_fullsized":"2024-04-06T03:46:45.711918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:28.216763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:45:28.216763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":665,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712658+0000","last_change":"2024-04-06T03:43:34.098016+0000","last_active":"2024-04-06T03:46:45.712658+0000","last_peered":"2024-04-06T03:46:45.712658+0000","last_clean":"2024-04-06T03:46:45.712658+0000","last_became_active":"2024-04-06T03:42:57.101420+0000","last_became_peered":"2024-04-06T03:42:57.101420+0000","last_unstale":"2024-04-06T03:46:45.712658+0000","last_undegraded":"2024-04-06T03:46:45.712658+0000","last_fullsized":"2024-04-06T03:46:45.712658+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:43:34.097807+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:43:34.097807+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"307'49","reported_seq":899,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:48.274120+0000","last_change":"2024-04-06T03:43:10.099251+0000","last_active":"2024-04-06T03:46:48.274120+0000","last_peered":"2024-04-06T03:46:48.274120+0000","last_clean":"2024-04-06T03:46:48.274120+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:46:48.274120+0000","last_undegraded":"2024-04-06T03:46:48.274120+0000","last_fullsized":"2024-04-06T03:46:48.274120+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'49","last_scrub_stamp":"2024-04-06T03:43:10.099011+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:43:10.099011+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":173,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711169+0000","last_change":"2024-04-06T03:43:32.211974+0000","last_active":"2024-04-06T03:46:45.711169+0000","last_peered":"2024-04-06T03:46:45.711169+0000","last_clean":"2024-04-06T03:46:45.711169+0000","last_became_active":"2024-04-06T03:42:57.119712+0000","last_became_peered":"2024-04-06T03:42:57.119712+0000","last_unstale":"2024-04-06T03:46:45.711169+0000","last_undegraded":"2024-04-06T03:46:45.711169+0000","last_fullsized":"2024-04-06T03:46:45.711169+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:43:32.211738+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.211738+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"327'13","reported_seq":1044,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.459547+0000","last_change":"2024-04-06T03:43:33.097564+0000","last_active":"2024-04-06T03:46:47.459547+0000","last_peered":"2024-04-06T03:46:47.459547+0000","last_clean":"2024-04-06T03:46:47.459547+0000","last_became_active":"2024-04-06T03:42:23.582156+0000","last_became_peered":"2024-04-06T03:42:23.582156+0000","last_unstale":"2024-04-06T03:46:47.459547+0000","last_undegraded":"2024-04-06T03:46:47.459547+0000","last_fullsized":"2024-04-06T03:46:47.459547+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'13","last_scrub_stamp":"2024-04-06T03:43:33.096089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:43:33.096089+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"377'11893","reported_seq":11968,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.144682+0000","last_change":"2024-04-06T03:43:25.029317+0000","last_active":"2024-04-06T03:46:53.144682+0000","last_peered":"2024-04-06T03:46:53.144682+0000","last_clean":"2024-04-06T03:46:53.144682+0000","last_became_active":"2024-04-06T03:43:25.028550+0000","last_became_peered":"2024-04-06T03:43:25.028550+0000","last_unstale":"2024-04-06T03:46:53.144682+0000","last_undegraded":"2024-04-06T03:46:53.144682+0000","last_fullsized":"2024-04-06T03:46:53.144682+0000","mapping_epoch":345,"log_start":"364'7000","ondisk_log_start":"364'7000","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77398016,"num_objects":1181,"num_object_clones":0,"num_object_copies":3543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":418368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":688,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724137+0000","last_change":"2024-04-06T03:45:29.035614+0000","last_active":"2024-04-06T03:46:45.724137+0000","last_peered":"2024-04-06T03:46:45.724137+0000","last_clean":"2024-04-06T03:46:45.724137+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:46:45.724137+0000","last_undegraded":"2024-04-06T03:46:45.724137+0000","last_fullsized":"2024-04-06T03:46:45.724137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.035484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.035484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:53.348498+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":106,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:52.016793+0000","last_change":"2024-04-06T03:46:52.016829+0000","last_active":"2024-04-06T03:46:52.016793+0000","last_peered":"2024-04-06T03:46:52.016793+0000","last_clean":"2024-04-06T03:46:52.016793+0000","last_became_active":"2024-04-06T03:42:57.089778+0000","last_became_peered":"2024-04-06T03:42:57.089778+0000","last_unstale":"2024-04-06T03:46:52.016793+0000","last_undegraded":"2024-04-06T03:46:52.016793+0000","last_fullsized":"2024-04-06T03:46:52.016793+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:52.016719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:46:52.016719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:53.366936+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"307'54","reported_seq":836,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711055+0000","last_change":"2024-04-06T03:43:39.143819+0000","last_active":"2024-04-06T03:46:45.711055+0000","last_peered":"2024-04-06T03:46:45.711055+0000","last_clean":"2024-04-06T03:46:45.711055+0000","last_became_active":"2024-04-06T03:42:23.591518+0000","last_became_peered":"2024-04-06T03:42:23.591518+0000","last_unstale":"2024-04-06T03:46:45.711055+0000","last_undegraded":"2024-04-06T03:46:45.711055+0000","last_fullsized":"2024-04-06T03:46:45.711055+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'54","last_scrub_stamp":"2024-04-06T03:43:39.143539+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:43:39.143539+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.713568+0000","last_change":"2024-04-06T03:43:38.985859+0000","last_active":"2024-04-06T03:46:45.713568+0000","last_peered":"2024-04-06T03:46:45.713568+0000","last_clean":"2024-04-06T03:46:45.713568+0000","last_became_active":"2024-04-06T03:42:57.097491+0000","last_became_peered":"2024-04-06T03:42:57.097491+0000","last_unstale":"2024-04-06T03:46:45.713568+0000","last_undegraded":"2024-04-06T03:46:45.713568+0000","last_fullsized":"2024-04-06T03:46:45.713568+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:38.985733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:31.042201+0000","last_clean_scrub_stamp":"2024-04-06T03:43:38.985733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"377'12367","reported_seq":12442,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.142278+0000","last_change":"2024-04-06T03:43:25.131438+0000","last_active":"2024-04-06T03:46:53.142278+0000","last_peered":"2024-04-06T03:46:53.142278+0000","last_clean":"2024-04-06T03:46:53.142278+0000","last_became_active":"2024-04-06T03:43:25.130484+0000","last_became_peered":"2024-04-06T03:43:25.130484+0000","last_unstale":"2024-04-06T03:46:53.142278+0000","last_undegraded":"2024-04-06T03:46:53.142278+0000","last_fullsized":"2024-04-06T03:46:53.142278+0000","mapping_epoch":345,"log_start":"364'7500","ondisk_log_start":"364'7500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75431936,"num_objects":1151,"num_object_clones":0,"num_object_copies":3453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12367,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":707,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726317+0000","last_change":"2024-04-06T03:45:23.445396+0000","last_active":"2024-04-06T03:46:45.726317+0000","last_peered":"2024-04-06T03:46:45.726317+0000","last_clean":"2024-04-06T03:46:45.726317+0000","last_became_active":"2024-04-06T03:42:23.569974+0000","last_became_peered":"2024-04-06T03:42:23.569974+0000","last_unstale":"2024-04-06T03:46:45.726317+0000","last_undegraded":"2024-04-06T03:46:45.726317+0000","last_fullsized":"2024-04-06T03:46:45.726317+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:23.445295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:45:23.445295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"307'54","reported_seq":742,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.719310+0000","last_change":"2024-04-06T03:45:14.950304+0000","last_active":"2024-04-06T03:46:45.719310+0000","last_peered":"2024-04-06T03:46:45.719310+0000","last_clean":"2024-04-06T03:46:45.719310+0000","last_became_active":"2024-04-06T03:42:57.096119+0000","last_became_peered":"2024-04-06T03:42:57.096119+0000","last_unstale":"2024-04-06T03:46:45.719310+0000","last_undegraded":"2024-04-06T03:46:45.719310+0000","last_fullsized":"2024-04-06T03:46:45.719310+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'54","last_scrub_stamp":"2024-04-06T03:45:14.950155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:14.950155+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":733,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:49.154457+0000","last_change":"2024-04-06T03:46:49.154499+0000","last_active":"2024-04-06T03:46:49.154457+0000","last_peered":"2024-04-06T03:46:49.154457+0000","last_clean":"2024-04-06T03:46:49.154457+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:46:49.154457+0000","last_undegraded":"2024-04-06T03:46:49.154457+0000","last_fullsized":"2024-04-06T03:46:49.154457+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:49.152620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:46:49.152620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.417526+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":724,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725245+0000","last_change":"2024-04-06T03:44:58.598242+0000","last_active":"2024-04-06T03:46:45.725245+0000","last_peered":"2024-04-06T03:46:45.725245+0000","last_clean":"2024-04-06T03:46:45.725245+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:46:45.725245+0000","last_undegraded":"2024-04-06T03:46:45.725245+0000","last_fullsized":"2024-04-06T03:46:45.725245+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:58.598017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.598017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"377'11926","reported_seq":12001,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.136159+0000","last_change":"2024-04-06T03:43:25.131247+0000","last_active":"2024-04-06T03:46:53.136159+0000","last_peered":"2024-04-06T03:46:53.136159+0000","last_clean":"2024-04-06T03:46:53.136159+0000","last_became_active":"2024-04-06T03:43:25.130333+0000","last_became_peered":"2024-04-06T03:43:25.130333+0000","last_unstale":"2024-04-06T03:46:53.136159+0000","last_undegraded":"2024-04-06T03:46:53.136159+0000","last_fullsized":"2024-04-06T03:46:53.136159+0000","mapping_epoch":345,"log_start":"364'7000","ondisk_log_start":"364'7000","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":419072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":552,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724061+0000","last_change":"2024-04-06T03:43:21.171626+0000","last_active":"2024-04-06T03:46:45.724061+0000","last_peered":"2024-04-06T03:46:45.724061+0000","last_clean":"2024-04-06T03:46:45.724061+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:46:45.724061+0000","last_undegraded":"2024-04-06T03:46:45.724061+0000","last_fullsized":"2024-04-06T03:46:45.724061+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:21.171511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:21.171511+0000","last_clean_scrub_stamp":"2024-04-06T03:43:21.171511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"307'42","reported_seq":800,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723590+0000","last_change":"2024-04-06T03:45:33.957089+0000","last_active":"2024-04-06T03:46:45.723590+0000","last_peered":"2024-04-06T03:46:45.723590+0000","last_clean":"2024-04-06T03:46:45.723590+0000","last_became_active":"2024-04-06T03:42:23.588643+0000","last_became_peered":"2024-04-06T03:42:23.588643+0000","last_unstale":"2024-04-06T03:46:45.723590+0000","last_undegraded":"2024-04-06T03:46:45.723590+0000","last_fullsized":"2024-04-06T03:46:45.723590+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'42","last_scrub_stamp":"2024-04-06T03:45:33.956933+0000","last_deep_scrub":"307'42","last_deep_scrub_stamp":"2024-04-06T03:41:26.470096+0000","last_clean_scrub_stamp":"2024-04-06T03:45:33.956933+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":592,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.719893+0000","last_change":"2024-04-06T03:45:30.075970+0000","last_active":"2024-04-06T03:46:45.719893+0000","last_peered":"2024-04-06T03:46:45.719893+0000","last_clean":"2024-04-06T03:46:45.719893+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:46:45.719893+0000","last_undegraded":"2024-04-06T03:46:45.719893+0000","last_fullsized":"2024-04-06T03:46:45.719893+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.075882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.075882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":989,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725564+0000","last_change":"2024-04-06T03:43:11.279746+0000","last_active":"2024-04-06T03:46:45.725564+0000","last_peered":"2024-04-06T03:46:45.725564+0000","last_clean":"2024-04-06T03:46:45.725564+0000","last_became_active":"2024-04-06T03:42:26.582173+0000","last_became_peered":"2024-04-06T03:42:26.582173+0000","last_unstale":"2024-04-06T03:46:45.725564+0000","last_undegraded":"2024-04-06T03:46:45.725564+0000","last_fullsized":"2024-04-06T03:46:45.725564+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:43:11.279588+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:43:11.279588+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":699,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725548+0000","last_change":"2024-04-06T03:45:12.385460+0000","last_active":"2024-04-06T03:46:45.725548+0000","last_peered":"2024-04-06T03:46:45.725548+0000","last_clean":"2024-04-06T03:46:45.725548+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:46:45.725548+0000","last_undegraded":"2024-04-06T03:46:45.725548+0000","last_fullsized":"2024-04-06T03:46:45.725548+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:12.385092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:45:12.385092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"377'12305","reported_seq":12380,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.148875+0000","last_change":"2024-04-06T03:43:25.133000+0000","last_active":"2024-04-06T03:46:53.148875+0000","last_peered":"2024-04-06T03:46:53.148875+0000","last_clean":"2024-04-06T03:46:53.148875+0000","last_became_active":"2024-04-06T03:43:25.130714+0000","last_became_peered":"2024-04-06T03:43:25.130714+0000","last_unstale":"2024-04-06T03:46:53.148875+0000","last_undegraded":"2024-04-06T03:46:53.148875+0000","last_fullsized":"2024-04-06T03:46:53.148875+0000","mapping_epoch":345,"log_start":"364'7400","ondisk_log_start":"364'7400","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4905,"log_dups_size":0,"ondisk_log_size":4905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78315520,"num_objects":1195,"num_object_clones":0,"num_object_copies":3585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12305,"num_write_kb":432000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":99,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724487+0000","last_change":"2024-04-06T03:43:42.965111+0000","last_active":"2024-04-06T03:46:45.724487+0000","last_peered":"2024-04-06T03:46:45.724487+0000","last_clean":"2024-04-06T03:46:45.724487+0000","last_became_active":"2024-04-06T03:42:57.093548+0000","last_became_peered":"2024-04-06T03:42:57.093548+0000","last_unstale":"2024-04-06T03:46:45.724487+0000","last_undegraded":"2024-04-06T03:46:45.724487+0000","last_fullsized":"2024-04-06T03:46:45.724487+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.964733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:37.513299+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.964733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"307'49","reported_seq":787,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712548+0000","last_change":"2024-04-06T03:45:17.013250+0000","last_active":"2024-04-06T03:46:45.712548+0000","last_peered":"2024-04-06T03:46:45.712548+0000","last_clean":"2024-04-06T03:46:45.712548+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:46:45.712548+0000","last_undegraded":"2024-04-06T03:46:45.712548+0000","last_fullsized":"2024-04-06T03:46:45.712548+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'49","last_scrub_stamp":"2024-04-06T03:45:17.013155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.013155+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725465+0000","last_change":"2024-04-06T03:43:42.231771+0000","last_active":"2024-04-06T03:46:45.725465+0000","last_peered":"2024-04-06T03:46:45.725465+0000","last_clean":"2024-04-06T03:46:45.725465+0000","last_became_active":"2024-04-06T03:42:57.093654+0000","last_became_peered":"2024-04-06T03:42:57.093654+0000","last_unstale":"2024-04-06T03:46:45.725465+0000","last_undegraded":"2024-04-06T03:46:45.725465+0000","last_fullsized":"2024-04-06T03:46:45.725465+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.231565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.231565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":110,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.715621+0000","last_change":"2024-04-06T03:45:32.117873+0000","last_active":"2024-04-06T03:46:45.715621+0000","last_peered":"2024-04-06T03:46:45.715621+0000","last_clean":"2024-04-06T03:46:45.715621+0000","last_became_active":"2024-04-06T03:42:57.098171+0000","last_became_peered":"2024-04-06T03:42:57.098171+0000","last_unstale":"2024-04-06T03:46:45.715621+0000","last_undegraded":"2024-04-06T03:46:45.715621+0000","last_fullsized":"2024-04-06T03:46:45.715621+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_clean_scrub_stamp":"2024-04-06T03:45:32.117767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:54.670260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"377'12004","reported_seq":12141,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:55.406484+0000","last_change":"2024-04-06T03:45:03.993018+0000","last_active":"2024-04-06T03:46:55.406484+0000","last_peered":"2024-04-06T03:46:55.406484+0000","last_clean":"2024-04-06T03:46:55.406484+0000","last_became_active":"2024-04-06T03:43:25.027401+0000","last_became_peered":"2024-04-06T03:43:25.027401+0000","last_unstale":"2024-04-06T03:46:55.406484+0000","last_undegraded":"2024-04-06T03:46:55.406484+0000","last_fullsized":"2024-04-06T03:46:55.406484+0000","mapping_epoch":345,"log_start":"364'6500","ondisk_log_start":"364'6500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'5530","last_scrub_stamp":"2024-04-06T03:45:03.992912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:03.992912+0000","objects_scrubbed":5518,"log_size":5504,"log_dups_size":0,"ondisk_log_size":5504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1415,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76152832,"num_objects":1162,"num_object_clones":0,"num_object_copies":3486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":703,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.719631+0000","last_change":"2024-04-06T03:45:01.597564+0000","last_active":"2024-04-06T03:46:45.719631+0000","last_peered":"2024-04-06T03:46:45.719631+0000","last_clean":"2024-04-06T03:46:45.719631+0000","last_became_active":"2024-04-06T03:42:57.095020+0000","last_became_peered":"2024-04-06T03:42:57.095020+0000","last_unstale":"2024-04-06T03:46:45.719631+0000","last_undegraded":"2024-04-06T03:46:45.719631+0000","last_fullsized":"2024-04-06T03:46:45.719631+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.597125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.597125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":707,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726391+0000","last_change":"2024-04-06T03:45:34.168883+0000","last_active":"2024-04-06T03:46:45.726391+0000","last_peered":"2024-04-06T03:46:45.726391+0000","last_clean":"2024-04-06T03:46:45.726391+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:46:45.726391+0000","last_undegraded":"2024-04-06T03:46:45.726391+0000","last_fullsized":"2024-04-06T03:46:45.726391+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.165081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.165081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"338'64","reported_seq":1050,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.395568+0000","last_change":"2024-04-06T03:46:53.395723+0000","last_active":"2024-04-06T03:46:53.395568+0000","last_peered":"2024-04-06T03:46:53.395568+0000","last_clean":"2024-04-06T03:46:53.395568+0000","last_became_active":"2024-04-06T03:42:57.101642+0000","last_became_peered":"2024-04-06T03:42:57.101642+0000","last_unstale":"2024-04-06T03:46:53.395568+0000","last_undegraded":"2024-04-06T03:46:53.395568+0000","last_fullsized":"2024-04-06T03:46:53.395568+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'64","last_scrub_stamp":"2024-04-06T03:46:53.395305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:46:53.395305+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:01.452463+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":582,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.714293+0000","last_change":"2024-04-06T03:45:07.981010+0000","last_active":"2024-04-06T03:46:45.714293+0000","last_peered":"2024-04-06T03:46:45.714293+0000","last_clean":"2024-04-06T03:46:45.714293+0000","last_became_active":"2024-04-06T03:42:23.569142+0000","last_became_peered":"2024-04-06T03:42:23.569142+0000","last_unstale":"2024-04-06T03:46:45.714293+0000","last_undegraded":"2024-04-06T03:46:45.714293+0000","last_fullsized":"2024-04-06T03:46:45.714293+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:07.980779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:07.980779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"377'11990","reported_seq":12065,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:54.488304+0000","last_change":"2024-04-06T03:43:25.029080+0000","last_active":"2024-04-06T03:46:54.488304+0000","last_peered":"2024-04-06T03:46:54.488304+0000","last_clean":"2024-04-06T03:46:54.488304+0000","last_became_active":"2024-04-06T03:43:25.028236+0000","last_became_peered":"2024-04-06T03:43:25.028236+0000","last_unstale":"2024-04-06T03:46:54.488304+0000","last_undegraded":"2024-04-06T03:46:54.488304+0000","last_fullsized":"2024-04-06T03:46:54.488304+0000","mapping_epoch":345,"log_start":"365'7700","ondisk_log_start":"365'7700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75366400,"num_objects":1150,"num_object_clones":0,"num_object_copies":3450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11990,"num_write_kb":420480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":697,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710595+0000","last_change":"2024-04-06T03:45:41.425674+0000","last_active":"2024-04-06T03:46:45.710595+0000","last_peered":"2024-04-06T03:46:45.710595+0000","last_clean":"2024-04-06T03:46:45.710595+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:46:45.710595+0000","last_undegraded":"2024-04-06T03:46:45.710595+0000","last_fullsized":"2024-04-06T03:46:45.710595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:41.425546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.425546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:00.450786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":119,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:51.289584+0000","last_change":"2024-04-06T03:46:51.289627+0000","last_active":"2024-04-06T03:46:51.289584+0000","last_peered":"2024-04-06T03:46:51.289584+0000","last_clean":"2024-04-06T03:46:51.289584+0000","last_became_active":"2024-04-06T03:42:57.116773+0000","last_became_peered":"2024-04-06T03:42:57.116773+0000","last_unstale":"2024-04-06T03:46:51.289584+0000","last_undegraded":"2024-04-06T03:46:51.289584+0000","last_fullsized":"2024-04-06T03:46:51.289584+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:46:51.289384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:46:51.289384+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:13.202157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"307'27","reported_seq":747,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723711+0000","last_change":"2024-04-06T03:45:36.917456+0000","last_active":"2024-04-06T03:46:45.723711+0000","last_peered":"2024-04-06T03:46:45.723711+0000","last_clean":"2024-04-06T03:46:45.723711+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:46:45.723711+0000","last_undegraded":"2024-04-06T03:46:45.723711+0000","last_fullsized":"2024-04-06T03:46:45.723711+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:45:36.917321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:36.917321+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:02.293191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2233,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:52.074552+0000","last_change":"2024-04-06T03:46:46.090765+0000","last_active":"2024-04-06T03:46:52.074552+0000","last_peered":"2024-04-06T03:46:52.074552+0000","last_clean":"2024-04-06T03:46:52.074552+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:46:52.074552+0000","last_undegraded":"2024-04-06T03:46:52.074552+0000","last_fullsized":"2024-04-06T03:46:52.074552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:46:46.090642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:46:46.090642+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:11.641079+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1528,"num_read_kb":1523,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":693,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710334+0000","last_change":"2024-04-06T03:43:18.143508+0000","last_active":"2024-04-06T03:46:45.710334+0000","last_peered":"2024-04-06T03:46:45.710334+0000","last_clean":"2024-04-06T03:46:45.710334+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:46:45.710334+0000","last_undegraded":"2024-04-06T03:46:45.710334+0000","last_fullsized":"2024-04-06T03:46:45.710334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:18.143371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_clean_scrub_stamp":"2024-04-06T03:43:18.143371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"377'12126","reported_seq":12201,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:55.031194+0000","last_change":"2024-04-06T03:43:25.025446+0000","last_active":"2024-04-06T03:46:55.031194+0000","last_peered":"2024-04-06T03:46:55.031194+0000","last_clean":"2024-04-06T03:46:55.031194+0000","last_became_active":"2024-04-06T03:43:25.024471+0000","last_became_peered":"2024-04-06T03:43:25.024471+0000","last_unstale":"2024-04-06T03:46:55.031194+0000","last_undegraded":"2024-04-06T03:46:55.031194+0000","last_fullsized":"2024-04-06T03:46:55.031194+0000","mapping_epoch":345,"log_start":"364'7500","ondisk_log_start":"364'7500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4626,"log_dups_size":0,"ondisk_log_size":4626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":699,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725073+0000","last_change":"2024-04-06T03:43:23.337277+0000","last_active":"2024-04-06T03:46:45.725073+0000","last_peered":"2024-04-06T03:46:45.725073+0000","last_clean":"2024-04-06T03:46:45.725073+0000","last_became_active":"2024-04-06T03:42:23.562715+0000","last_became_peered":"2024-04-06T03:42:23.562715+0000","last_unstale":"2024-04-06T03:46:45.725073+0000","last_undegraded":"2024-04-06T03:46:45.725073+0000","last_fullsized":"2024-04-06T03:46:45.725073+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.337146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.337146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":573,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726315+0000","last_change":"2024-04-06T03:43:14.348107+0000","last_active":"2024-04-06T03:46:45.726315+0000","last_peered":"2024-04-06T03:46:45.726315+0000","last_clean":"2024-04-06T03:46:45.726315+0000","last_became_active":"2024-04-06T03:42:23.567776+0000","last_became_peered":"2024-04-06T03:42:23.567776+0000","last_unstale":"2024-04-06T03:46:45.726315+0000","last_undegraded":"2024-04-06T03:46:45.726315+0000","last_fullsized":"2024-04-06T03:46:45.726315+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:14.347877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:43:14.347877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"312'42","reported_seq":790,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.714651+0000","last_change":"2024-04-06T03:45:04.006917+0000","last_active":"2024-04-06T03:46:45.714651+0000","last_peered":"2024-04-06T03:46:45.714651+0000","last_clean":"2024-04-06T03:46:45.714651+0000","last_became_active":"2024-04-06T03:42:57.102077+0000","last_became_peered":"2024-04-06T03:42:57.102077+0000","last_unstale":"2024-04-06T03:46:45.714651+0000","last_undegraded":"2024-04-06T03:46:45.714651+0000","last_fullsized":"2024-04-06T03:46:45.714651+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'42","last_scrub_stamp":"2024-04-06T03:45:04.006799+0000","last_deep_scrub":"312'42","last_deep_scrub_stamp":"2024-04-06T03:41:44.156017+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.006799+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":722,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:46.343086+0000","last_change":"2024-04-06T03:46:46.343119+0000","last_active":"2024-04-06T03:46:46.343086+0000","last_peered":"2024-04-06T03:46:46.343086+0000","last_clean":"2024-04-06T03:46:46.343086+0000","last_became_active":"2024-04-06T03:42:23.576890+0000","last_became_peered":"2024-04-06T03:42:23.576890+0000","last_unstale":"2024-04-06T03:46:46.343086+0000","last_undegraded":"2024-04-06T03:46:46.343086+0000","last_fullsized":"2024-04-06T03:46:46.343086+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:46.343009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:19.191417+0000","last_clean_scrub_stamp":"2024-04-06T03:46:46.343009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:58.257816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"338'13","reported_seq":1027,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.417528+0000","last_change":"2024-04-06T03:45:20.921396+0000","last_active":"2024-04-06T03:46:47.417528+0000","last_peered":"2024-04-06T03:46:47.417528+0000","last_clean":"2024-04-06T03:46:47.417528+0000","last_became_active":"2024-04-06T03:42:57.094766+0000","last_became_peered":"2024-04-06T03:42:57.094766+0000","last_unstale":"2024-04-06T03:46:47.417528+0000","last_undegraded":"2024-04-06T03:46:47.417528+0000","last_fullsized":"2024-04-06T03:46:47.417528+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-04-06T03:45:20.921243+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:41:18.436223+0000","last_clean_scrub_stamp":"2024-04-06T03:45:20.921243+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"377'12127","reported_seq":12264,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.148159+0000","last_change":"2024-04-06T03:44:58.331410+0000","last_active":"2024-04-06T03:46:53.148159+0000","last_peered":"2024-04-06T03:46:53.148159+0000","last_clean":"2024-04-06T03:46:53.148159+0000","last_became_active":"2024-04-06T03:43:25.130935+0000","last_became_peered":"2024-04-06T03:43:25.130935+0000","last_unstale":"2024-04-06T03:46:53.148159+0000","last_undegraded":"2024-04-06T03:46:53.148159+0000","last_fullsized":"2024-04-06T03:46:53.148159+0000","mapping_epoch":345,"log_start":"364'7200","ondisk_log_start":"364'7200","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'5388","last_scrub_stamp":"2024-04-06T03:44:58.331222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.331222+0000","objects_scrubbed":5324,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76611584,"num_objects":1169,"num_object_clones":0,"num_object_copies":3507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":425472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":700,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724386+0000","last_change":"2024-04-06T03:45:42.012146+0000","last_active":"2024-04-06T03:46:45.724386+0000","last_peered":"2024-04-06T03:46:45.724386+0000","last_clean":"2024-04-06T03:46:45.724386+0000","last_became_active":"2024-04-06T03:42:23.590773+0000","last_became_peered":"2024-04-06T03:42:23.590773+0000","last_unstale":"2024-04-06T03:46:45.724386+0000","last_undegraded":"2024-04-06T03:46:45.724386+0000","last_fullsized":"2024-04-06T03:46:45.724386+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:42.012040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:45:42.012040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:46.339282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":579,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711454+0000","last_change":"2024-04-06T03:45:35.465666+0000","last_active":"2024-04-06T03:46:45.711454+0000","last_peered":"2024-04-06T03:46:45.711454+0000","last_clean":"2024-04-06T03:46:45.711454+0000","last_became_active":"2024-04-06T03:42:23.595618+0000","last_became_peered":"2024-04-06T03:42:23.595618+0000","last_unstale":"2024-04-06T03:46:45.711454+0000","last_undegraded":"2024-04-06T03:46:45.711454+0000","last_fullsized":"2024-04-06T03:46:45.711454+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:35.465468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.465468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:47.618305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":705,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725719+0000","last_change":"2024-04-06T03:45:30.229007+0000","last_active":"2024-04-06T03:46:45.725719+0000","last_peered":"2024-04-06T03:46:45.725719+0000","last_clean":"2024-04-06T03:46:45.725719+0000","last_became_active":"2024-04-06T03:42:57.115680+0000","last_became_peered":"2024-04-06T03:42:57.115680+0000","last_unstale":"2024-04-06T03:46:45.725719+0000","last_undegraded":"2024-04-06T03:46:45.725719+0000","last_fullsized":"2024-04-06T03:46:45.725719+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.228890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.228890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"307'43","reported_seq":768,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710499+0000","last_change":"2024-04-06T03:43:24.255032+0000","last_active":"2024-04-06T03:46:45.710499+0000","last_peered":"2024-04-06T03:46:45.710499+0000","last_clean":"2024-04-06T03:46:45.710499+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:46:45.710499+0000","last_undegraded":"2024-04-06T03:46:45.710499+0000","last_fullsized":"2024-04-06T03:46:45.710499+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'43","last_scrub_stamp":"2024-04-06T03:43:24.254836+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:43:24.254836+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"377'11958","reported_seq":12033,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:53.139065+0000","last_change":"2024-04-06T03:43:25.131506+0000","last_active":"2024-04-06T03:46:53.139065+0000","last_peered":"2024-04-06T03:46:53.139065+0000","last_clean":"2024-04-06T03:46:53.139065+0000","last_became_active":"2024-04-06T03:43:25.130566+0000","last_became_peered":"2024-04-06T03:43:25.130566+0000","last_unstale":"2024-04-06T03:46:53.139065+0000","last_undegraded":"2024-04-06T03:46:53.139065+0000","last_fullsized":"2024-04-06T03:46:53.139065+0000","mapping_epoch":345,"log_start":"364'7100","ondisk_log_start":"364'7100","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11958,"num_write_kb":420928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":532,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712784+0000","last_change":"2024-04-06T03:43:42.121160+0000","last_active":"2024-04-06T03:46:45.712784+0000","last_peered":"2024-04-06T03:46:45.712784+0000","last_clean":"2024-04-06T03:46:45.712784+0000","last_became_active":"2024-04-06T03:42:57.078094+0000","last_became_peered":"2024-04-06T03:42:57.078094+0000","last_unstale":"2024-04-06T03:46:45.712784+0000","last_undegraded":"2024-04-06T03:46:45.712784+0000","last_fullsized":"2024-04-06T03:46:45.712784+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.121037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.121037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":1024,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.416723+0000","last_change":"2024-04-06T03:43:30.210944+0000","last_active":"2024-04-06T03:46:47.416723+0000","last_peered":"2024-04-06T03:46:47.416723+0000","last_clean":"2024-04-06T03:46:47.416723+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:46:47.416723+0000","last_undegraded":"2024-04-06T03:46:47.416723+0000","last_fullsized":"2024-04-06T03:46:47.416723+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:43:30.210531+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:35:33.269156+0000","last_clean_scrub_stamp":"2024-04-06T03:43:30.210531+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":99,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724929+0000","last_change":"2024-04-06T03:43:24.206838+0000","last_active":"2024-04-06T03:46:45.724929+0000","last_peered":"2024-04-06T03:46:45.724929+0000","last_clean":"2024-04-06T03:46:45.724929+0000","last_became_active":"2024-04-06T03:42:57.092193+0000","last_became_peered":"2024-04-06T03:42:57.092193+0000","last_unstale":"2024-04-06T03:46:45.724929+0000","last_undegraded":"2024-04-06T03:46:45.724929+0000","last_fullsized":"2024-04-06T03:46:45.724929+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:24.206535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:24.206535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"307'51","reported_seq":807,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725896+0000","last_change":"2024-04-06T03:46:44.064699+0000","last_active":"2024-04-06T03:46:45.725896+0000","last_peered":"2024-04-06T03:46:45.725896+0000","last_clean":"2024-04-06T03:46:45.725896+0000","last_became_active":"2024-04-06T03:42:57.113030+0000","last_became_peered":"2024-04-06T03:42:57.113030+0000","last_unstale":"2024-04-06T03:46:45.725896+0000","last_undegraded":"2024-04-06T03:46:45.725896+0000","last_fullsized":"2024-04-06T03:46:45.725896+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'51","last_scrub_stamp":"2024-04-06T03:46:44.064524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:46:44.064524+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.715096+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"377'12140","reported_seq":12215,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:54.501565+0000","last_change":"2024-04-06T03:43:25.129974+0000","last_active":"2024-04-06T03:46:54.501565+0000","last_peered":"2024-04-06T03:46:54.501565+0000","last_clean":"2024-04-06T03:46:54.501565+0000","last_became_active":"2024-04-06T03:43:25.129076+0000","last_became_peered":"2024-04-06T03:43:25.129076+0000","last_unstale":"2024-04-06T03:46:54.501565+0000","last_undegraded":"2024-04-06T03:46:54.501565+0000","last_fullsized":"2024-04-06T03:46:54.501565+0000","mapping_epoch":345,"log_start":"367'7900","ondisk_log_start":"367'7900","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":1218,"num_object_clones":0,"num_object_copies":3654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12140,"num_write_kb":427456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":702,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711882+0000","last_change":"2024-04-06T03:45:37.526151+0000","last_active":"2024-04-06T03:46:45.711882+0000","last_peered":"2024-04-06T03:46:45.711882+0000","last_clean":"2024-04-06T03:46:45.711882+0000","last_became_active":"2024-04-06T03:42:57.102442+0000","last_became_peered":"2024-04-06T03:42:57.102442+0000","last_unstale":"2024-04-06T03:46:45.711882+0000","last_undegraded":"2024-04-06T03:46:45.711882+0000","last_fullsized":"2024-04-06T03:46:45.711882+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:37.525986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.525986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:49.532812+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":1048,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.417621+0000","last_change":"2024-04-06T03:44:57.608200+0000","last_active":"2024-04-06T03:46:47.417621+0000","last_peered":"2024-04-06T03:46:47.417621+0000","last_clean":"2024-04-06T03:46:47.417621+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:46:47.417621+0000","last_undegraded":"2024-04-06T03:46:47.417621+0000","last_fullsized":"2024-04-06T03:46:47.417621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:44:57.607946+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.607946+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":583,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723432+0000","last_change":"2024-04-06T03:43:32.152585+0000","last_active":"2024-04-06T03:46:45.723432+0000","last_peered":"2024-04-06T03:46:45.723432+0000","last_clean":"2024-04-06T03:46:45.723432+0000","last_became_active":"2024-04-06T03:42:57.078347+0000","last_became_peered":"2024-04-06T03:42:57.078347+0000","last_unstale":"2024-04-06T03:46:45.723432+0000","last_undegraded":"2024-04-06T03:46:45.723432+0000","last_fullsized":"2024-04-06T03:46:45.723432+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:32.152478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:23.908290+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.152478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"307'27","reported_seq":742,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725564+0000","last_change":"2024-04-06T03:45:10.955262+0000","last_active":"2024-04-06T03:46:45.725564+0000","last_peered":"2024-04-06T03:46:45.725564+0000","last_clean":"2024-04-06T03:46:45.725564+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:46:45.725564+0000","last_undegraded":"2024-04-06T03:46:45.725564+0000","last_fullsized":"2024-04-06T03:46:45.725564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:45:10.955163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:10.955163+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"377'11872","reported_seq":11974,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:55.401326+0000","last_change":"2024-04-06T03:46:45.928415+0000","last_active":"2024-04-06T03:46:55.401326+0000","last_peered":"2024-04-06T03:46:55.401326+0000","last_clean":"2024-04-06T03:46:55.401326+0000","last_became_active":"2024-04-06T03:43:25.028867+0000","last_became_peered":"2024-04-06T03:43:25.028867+0000","last_unstale":"2024-04-06T03:46:55.401326+0000","last_undegraded":"2024-04-06T03:46:55.401326+0000","last_fullsized":"2024-04-06T03:46:55.401326+0000","mapping_epoch":345,"log_start":"364'6400","ondisk_log_start":"364'6400","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'11351","last_scrub_stamp":"2024-04-06T03:46:45.928172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:46:45.928172+0000","objects_scrubbed":1639,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:15.367617+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11872,"num_write_kb":415104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":720,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.724894+0000","last_change":"2024-04-06T03:45:17.270096+0000","last_active":"2024-04-06T03:46:45.724894+0000","last_peered":"2024-04-06T03:46:45.724894+0000","last_clean":"2024-04-06T03:46:45.724894+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:46:45.724894+0000","last_undegraded":"2024-04-06T03:46:45.724894+0000","last_fullsized":"2024-04-06T03:46:45.724894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:17.269763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.269763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":529,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710762+0000","last_change":"2024-04-06T03:45:02.458926+0000","last_active":"2024-04-06T03:46:45.710762+0000","last_peered":"2024-04-06T03:46:45.710762+0000","last_clean":"2024-04-06T03:46:45.710762+0000","last_became_active":"2024-04-06T03:42:57.110057+0000","last_became_peered":"2024-04-06T03:42:57.110057+0000","last_unstale":"2024-04-06T03:46:45.710762+0000","last_undegraded":"2024-04-06T03:46:45.710762+0000","last_fullsized":"2024-04-06T03:46:45.710762+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:02.458841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.458841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":706,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725345+0000","last_change":"2024-04-06T03:45:24.308229+0000","last_active":"2024-04-06T03:46:45.725345+0000","last_peered":"2024-04-06T03:46:45.725345+0000","last_clean":"2024-04-06T03:46:45.725345+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:46:45.725345+0000","last_undegraded":"2024-04-06T03:46:45.725345+0000","last_fullsized":"2024-04-06T03:46:45.725345+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:24.308115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:45:24.308115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:46:48.379923+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"307'57","reported_seq":763,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.711011+0000","last_change":"2024-04-06T03:43:43.053137+0000","last_active":"2024-04-06T03:46:45.711011+0000","last_peered":"2024-04-06T03:46:45.711011+0000","last_clean":"2024-04-06T03:46:45.711011+0000","last_became_active":"2024-04-06T03:42:57.110277+0000","last_became_peered":"2024-04-06T03:42:57.110277+0000","last_unstale":"2024-04-06T03:46:45.711011+0000","last_undegraded":"2024-04-06T03:46:45.711011+0000","last_fullsized":"2024-04-06T03:46:45.711011+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'57","last_scrub_stamp":"2024-04-06T03:43:43.052879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:43.052879+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"377'12044","reported_seq":12119,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:54.498022+0000","last_change":"2024-04-06T03:43:25.132069+0000","last_active":"2024-04-06T03:46:54.498022+0000","last_peered":"2024-04-06T03:46:54.498022+0000","last_clean":"2024-04-06T03:46:54.498022+0000","last_became_active":"2024-04-06T03:43:25.131461+0000","last_became_peered":"2024-04-06T03:43:25.131461+0000","last_unstale":"2024-04-06T03:46:54.498022+0000","last_undegraded":"2024-04-06T03:46:54.498022+0000","last_fullsized":"2024-04-06T03:46:54.498022+0000","mapping_epoch":345,"log_start":"366'7800","ondisk_log_start":"366'7800","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4244,"log_dups_size":0,"ondisk_log_size":4244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74711040,"num_objects":1140,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12044,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":686,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725349+0000","last_change":"2024-04-06T03:45:14.363616+0000","last_active":"2024-04-06T03:46:45.725349+0000","last_peered":"2024-04-06T03:46:45.725349+0000","last_clean":"2024-04-06T03:46:45.725349+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:46:45.725349+0000","last_undegraded":"2024-04-06T03:46:45.725349+0000","last_fullsized":"2024-04-06T03:46:45.725349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:14.363468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:14.363468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":724,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.719804+0000","last_change":"2024-04-06T03:45:34.011396+0000","last_active":"2024-04-06T03:46:45.719804+0000","last_peered":"2024-04-06T03:46:45.719804+0000","last_clean":"2024-04-06T03:46:45.719804+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:46:45.719804+0000","last_undegraded":"2024-04-06T03:46:45.719804+0000","last_fullsized":"2024-04-06T03:46:45.719804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.011285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:33.770813+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.011285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":724,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.728293+0000","last_change":"2024-04-06T03:45:10.895775+0000","last_active":"2024-04-06T03:46:45.728293+0000","last_peered":"2024-04-06T03:46:45.728293+0000","last_clean":"2024-04-06T03:46:45.728293+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:46:45.728293+0000","last_undegraded":"2024-04-06T03:46:45.728293+0000","last_fullsized":"2024-04-06T03:46:45.728293+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:10.895208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:45:10.895208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"307'53","reported_seq":820,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.726539+0000","last_change":"2024-04-06T03:45:02.412015+0000","last_active":"2024-04-06T03:46:45.726539+0000","last_peered":"2024-04-06T03:46:45.726539+0000","last_clean":"2024-04-06T03:46:45.726539+0000","last_became_active":"2024-04-06T03:42:23.570628+0000","last_became_peered":"2024-04-06T03:42:23.570628+0000","last_unstale":"2024-04-06T03:46:45.726539+0000","last_undegraded":"2024-04-06T03:46:45.726539+0000","last_fullsized":"2024-04-06T03:46:45.726539+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'53","last_scrub_stamp":"2024-04-06T03:45:02.411569+0000","last_deep_scrub":"307'53","last_deep_scrub_stamp":"2024-04-06T03:41:26.233369+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.411569+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"377'11939","reported_seq":12014,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:54.514044+0000","last_change":"2024-04-06T03:43:25.130138+0000","last_active":"2024-04-06T03:46:54.514044+0000","last_peered":"2024-04-06T03:46:54.514044+0000","last_clean":"2024-04-06T03:46:54.514044+0000","last_became_active":"2024-04-06T03:43:25.129488+0000","last_became_peered":"2024-04-06T03:43:25.129488+0000","last_unstale":"2024-04-06T03:46:54.514044+0000","last_undegraded":"2024-04-06T03:46:54.514044+0000","last_fullsized":"2024-04-06T03:46:54.514044+0000","mapping_epoch":345,"log_start":"365'7700","ondisk_log_start":"365'7700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73596928,"num_objects":1123,"num_object_clones":0,"num_object_copies":3369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11939,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":694,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.712653+0000","last_change":"2024-04-06T03:43:35.178431+0000","last_active":"2024-04-06T03:46:45.712653+0000","last_peered":"2024-04-06T03:46:45.712653+0000","last_clean":"2024-04-06T03:46:45.712653+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:46:45.712653+0000","last_undegraded":"2024-04-06T03:46:45.712653+0000","last_fullsized":"2024-04-06T03:46:45.712653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:35.178312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.178312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":724,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.720225+0000","last_change":"2024-04-06T03:46:42.275126+0000","last_active":"2024-04-06T03:46:45.720225+0000","last_peered":"2024-04-06T03:46:45.720225+0000","last_clean":"2024-04-06T03:46:45.720225+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:46:45.720225+0000","last_undegraded":"2024-04-06T03:46:45.720225+0000","last_fullsized":"2024-04-06T03:46:45.720225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:42.275016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_clean_scrub_stamp":"2024-04-06T03:46:42.275016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:00.582617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":99,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.710806+0000","last_change":"2024-04-06T03:45:01.559313+0000","last_active":"2024-04-06T03:46:45.710806+0000","last_peered":"2024-04-06T03:46:45.710806+0000","last_clean":"2024-04-06T03:46:45.710806+0000","last_became_active":"2024-04-06T03:42:57.102050+0000","last_became_peered":"2024-04-06T03:42:57.102050+0000","last_unstale":"2024-04-06T03:46:45.710806+0000","last_undegraded":"2024-04-06T03:46:45.710806+0000","last_fullsized":"2024-04-06T03:46:45.710806+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.559226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.559226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"307'35","reported_seq":752,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.723394+0000","last_change":"2024-04-06T03:43:19.078582+0000","last_active":"2024-04-06T03:46:45.723394+0000","last_peered":"2024-04-06T03:46:45.723394+0000","last_clean":"2024-04-06T03:46:45.723394+0000","last_became_active":"2024-04-06T03:42:23.590160+0000","last_became_peered":"2024-04-06T03:42:23.590160+0000","last_unstale":"2024-04-06T03:46:45.723394+0000","last_undegraded":"2024-04-06T03:46:45.723394+0000","last_fullsized":"2024-04-06T03:46:45.723394+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'35","last_scrub_stamp":"2024-04-06T03:43:19.078336+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:43:19.078336+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"377'12189","reported_seq":12264,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:55.038386+0000","last_change":"2024-04-06T03:43:25.130562+0000","last_active":"2024-04-06T03:46:55.038386+0000","last_peered":"2024-04-06T03:46:55.038386+0000","last_clean":"2024-04-06T03:46:55.038386+0000","last_became_active":"2024-04-06T03:43:25.129636+0000","last_became_peered":"2024-04-06T03:43:25.129636+0000","last_unstale":"2024-04-06T03:46:55.038386+0000","last_undegraded":"2024-04-06T03:46:55.038386+0000","last_fullsized":"2024-04-06T03:46:55.038386+0000","mapping_epoch":345,"log_start":"364'7600","ondisk_log_start":"364'7600","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73596928,"num_objects":1123,"num_object_clones":0,"num_object_copies":3369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12189,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":577,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.714402+0000","last_change":"2024-04-06T03:45:11.156530+0000","last_active":"2024-04-06T03:46:45.714402+0000","last_peered":"2024-04-06T03:46:45.714402+0000","last_clean":"2024-04-06T03:46:45.714402+0000","last_became_active":"2024-04-06T03:42:23.569567+0000","last_became_peered":"2024-04-06T03:42:23.569567+0000","last_unstale":"2024-04-06T03:46:45.714402+0000","last_undegraded":"2024-04-06T03:46:45.714402+0000","last_fullsized":"2024-04-06T03:46:45.714402+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:11.156227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.156227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":732,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725195+0000","last_change":"2024-04-06T03:45:06.959750+0000","last_active":"2024-04-06T03:46:45.725195+0000","last_peered":"2024-04-06T03:46:45.725195+0000","last_clean":"2024-04-06T03:46:45.725195+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:46:45.725195+0000","last_undegraded":"2024-04-06T03:46:45.725195+0000","last_fullsized":"2024-04-06T03:46:45.725195+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:06.959590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:45:06.959590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"307'46","reported_seq":762,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:45.725999+0000","last_change":"2024-04-06T03:43:07.288106+0000","last_active":"2024-04-06T03:46:45.725999+0000","last_peered":"2024-04-06T03:46:45.725999+0000","last_clean":"2024-04-06T03:46:45.725999+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:46:45.725999+0000","last_undegraded":"2024-04-06T03:46:45.725999+0000","last_fullsized":"2024-04-06T03:46:45.725999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'46","last_scrub_stamp":"2024-04-06T03:43:07.287843+0000","last_deep_scrub":"291'34","last_deep_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_clean_scrub_stamp":"2024-04-06T03:43:07.287843+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"327'14","reported_seq":1044,"reported_epoch":377,"state":"active+clean","last_fresh":"2024-04-06T03:46:47.416905+0000","last_change":"2024-04-06T03:43:35.224637+0000","last_active":"2024-04-06T03:46:47.416905+0000","last_peered":"2024-04-06T03:46:47.416905+0000","last_clean":"2024-04-06T03:46:47.416905+0000","last_became_active":"2024-04-06T03:42:23.576196+0000","last_became_peered":"2024-04-06T03:42:23.576196+0000","last_unstale":"2024-04-06T03:46:47.416905+0000","last_undegraded":"2024-04-06T03:46:47.416905+0000","last_fullsized":"2024-04-06T03:46:47.416905+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'14","last_scrub_stamp":"2024-04-06T03:43:35.224536+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.224536+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1207566360,"num_objects":18427,"num_object_clones":0,"num_object_copies":55281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18427,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193517,"num_write_kb":6782145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3627233280,"data_stored":3627221064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76617,"ondisk_log_size":76617,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1842,"num_read_kb":1567,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1301,"ondisk_log_size":1301,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1528,"num_read_kb":1523,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743747,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1074896,"kb_used_data":530056,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92670256,"statfs":{"total":95995035648,"available":94894342144,"internally_reserved":0,"allocated":542777344,"data_stored":542407812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":557906378},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136205,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213736,"kb_used_data":659168,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92531416,"statfs":{"total":95995035648,"available":94752169984,"internally_reserved":0,"allocated":674988032,"data_stored":674657626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575234,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738763,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280532,"kb_used_data":645452,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92464620,"statfs":{"total":95995035648,"available":94683770880,"internally_reserved":0,"allocated":660942848,"data_stored":660569012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379000,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972584,"kb_used_data":431776,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92772568,"statfs":{"total":95995035648,"available":94999109632,"internally_reserved":0,"allocated":442138624,"data_stored":441807560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084651,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135264,"kb_used_data":530648,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92609888,"statfs":{"total":95995035648,"available":94832525312,"internally_reserved":0,"allocated":543383552,"data_stored":543060328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855715,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972104,"kb_used_data":435520,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92773048,"statfs":{"total":95995035648,"available":94999601152,"internally_reserved":0,"allocated":445972480,"data_stored":445608735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495961,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949256,"kb_used_data":376640,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92795896,"statfs":{"total":95995035648,"available":95022997504,"internally_reserved":0,"allocated":385679360,"data_stored":385380070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":639172608,"data_stored":639172608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":432410624,"data_stored":432406552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":540409856,"data_stored":540409856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":439750656,"data_stored":439746584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":375521280,"data_stored":375521280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":670633984,"data_stored":670629912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":529334272,"data_stored":529334272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:47:00.887 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-06T03:47:00.888 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:47:00.888 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-04-06T03:47:00.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:47:00.950 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.008150+0000 osd.0 (osd.0) 607 : cluster 0 5.1e scrub starts 2024-04-06T03:47:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.011075+0000 osd.0 (osd.0) 608 : cluster 0 5.1e scrub ok 2024-04-06T03:47:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.039250+0000 osd.7 (osd.7) 609 : cluster 0 1.0 scrub starts 2024-04-06T03:47:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.044945+0000 osd.7 (osd.7) 610 : cluster 0 1.0 scrub ok 2024-04-06T03:47:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.336130+0000 osd.2 (osd.2) 493 : cluster 0 2.e scrub starts 2024-04-06T03:47:00.960 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.344368+0000 osd.2 (osd.2) 494 : cluster 0 2.e scrub ok 2024-04-06T03:47:00.978 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:00.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:01.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:01.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:01.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:01.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:01.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.395 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:01.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:01.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:01.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:01.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:01.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:01.773 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:01.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:01.963 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:01.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:01 smithi153 bash[23852]: audit 2024-04-06T03:47:00.340092+0000 mgr.y (mgr.24368) 956 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:01 smithi153 bash[23852]: cluster 2024-04-06T03:47:00.560306+0000 mgr.y (mgr.24368) 957 : cluster 0 pgmap v1038: 148 pgs: 148 active+clean; 862 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:47:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:01 smithi153 bash[23852]: cluster 2024-04-06T03:47:01.340202+0000 osd.2 (osd.2) 495 : cluster 0 5.6 scrub starts 2024-04-06T03:47:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:01 smithi153 bash[23852]: cluster 2024-04-06T03:47:01.348169+0000 osd.2 (osd.2) 496 : cluster 0 5.6 scrub ok 2024-04-06T03:47:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:01 smithi153 bash[23852]: audit 2024-04-06T03:47:01.778574+0000 mon.a (mon.0) 1757 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:47:02.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.096 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:02.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[21467]: audit 2024-04-06T03:47:00.340092+0000 mgr.y (mgr.24368) 956 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:02.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[21467]: cluster 2024-04-06T03:47:00.560306+0000 mgr.y (mgr.24368) 957 : cluster 0 pgmap v1038: 148 pgs: 148 active+clean; 862 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:47:02.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[21467]: cluster 2024-04-06T03:47:01.340202+0000 osd.2 (osd.2) 495 : cluster 0 5.6 scrub starts 2024-04-06T03:47:02.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[21467]: cluster 2024-04-06T03:47:01.348169+0000 osd.2 (osd.2) 496 : cluster 0 5.6 scrub ok 2024-04-06T03:47:02.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[21467]: audit 2024-04-06T03:47:01.778574+0000 mon.a (mon.0) 1757 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:47:02.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[29221]: audit 2024-04-06T03:47:00.340092+0000 mgr.y (mgr.24368) 956 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:02.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[29221]: cluster 2024-04-06T03:47:00.560306+0000 mgr.y (mgr.24368) 957 : cluster 0 pgmap v1038: 148 pgs: 148 active+clean; 862 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-04-06T03:47:02.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[29221]: cluster 2024-04-06T03:47:01.340202+0000 osd.2 (osd.2) 495 : cluster 0 5.6 scrub starts 2024-04-06T03:47:02.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[29221]: cluster 2024-04-06T03:47:01.348169+0000 osd.2 (osd.2) 496 : cluster 0 5.6 scrub ok 2024-04-06T03:47:02.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:01 smithi050 bash[29221]: audit 2024-04-06T03:47:01.778574+0000 mon.a (mon.0) 1757 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:47:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:02.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:02.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:02.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:02.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.676 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:02.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.818 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:02.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:02.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:02.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:03.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:02 smithi153 bash[23852]: cluster 2024-04-06T03:47:01.067682+0000 osd.3 (osd.3) 603 : cluster 0 6.7 scrub starts 2024-04-06T03:47:03.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:02 smithi153 bash[23852]: cluster 2024-04-06T03:47:01.226362+0000 osd.3 (osd.3) 604 : cluster 0 6.7 scrub ok 2024-04-06T03:47:03.071 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:03.096 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:03.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:03.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:02 smithi050 bash[21467]: cluster 2024-04-06T03:47:01.067682+0000 osd.3 (osd.3) 603 : cluster 0 6.7 scrub starts 2024-04-06T03:47:03.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:02 smithi050 bash[21467]: cluster 2024-04-06T03:47:01.226362+0000 osd.3 (osd.3) 604 : cluster 0 6.7 scrub ok 2024-04-06T03:47:03.209 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:02 smithi050 bash[29221]: cluster 2024-04-06T03:47:01.067682+0000 osd.3 (osd.3) 603 : cluster 0 6.7 scrub starts 2024-04-06T03:47:03.210 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:02 smithi050 bash[29221]: cluster 2024-04-06T03:47:01.226362+0000 osd.3 (osd.3) 604 : cluster 0 6.7 scrub ok 2024-04-06T03:47:03.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:03.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:03.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:03.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:03.501 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:03.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:03.635 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:03.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:03.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:03.870 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:03.887 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:03.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:03.990 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:02.047652+0000 osd.3 (osd.3) 605 : cluster 0 11.9 scrub starts 2024-04-06T03:47:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:02.308074+0000 osd.3 (osd.3) 606 : cluster 0 11.9 scrub ok 2024-04-06T03:47:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:02.560858+0000 mgr.y (mgr.24368) 958 : cluster 0 pgmap v1039: 148 pgs: 148 active+clean; 844 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 822 op/s 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:02.657307+0000 osd.6 (osd.6) 111 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:02.657384+0000 osd.6 (osd.6) 112 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:02.806988+0000 mon.a (mon.0) 1758 : cluster 0 osdmap e378: 8 total, 8 up, 7 in 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:03.002782+0000 osd.7 (osd.7) 611 : cluster 0 2.9 scrub starts 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[21467]: cluster 2024-04-06T03:47:03.007600+0000 osd.7 (osd.7) 612 : cluster 0 2.9 scrub ok 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:02.047652+0000 osd.3 (osd.3) 605 : cluster 0 11.9 scrub starts 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:02.308074+0000 osd.3 (osd.3) 606 : cluster 0 11.9 scrub ok 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:02.560858+0000 mgr.y (mgr.24368) 958 : cluster 0 pgmap v1039: 148 pgs: 148 active+clean; 844 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 822 op/s 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:02.657307+0000 osd.6 (osd.6) 111 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:02.657384+0000 osd.6 (osd.6) 112 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:02.806988+0000 mon.a (mon.0) 1758 : cluster 0 osdmap e378: 8 total, 8 up, 7 in 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:03.002782+0000 osd.7 (osd.7) 611 : cluster 0 2.9 scrub starts 2024-04-06T03:47:03.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:03 smithi050 bash[29221]: cluster 2024-04-06T03:47:03.007600+0000 osd.7 (osd.7) 612 : cluster 0 2.9 scrub ok 2024-04-06T03:47:04.021 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:04.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:04.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:04.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:02.047652+0000 osd.3 (osd.3) 605 : cluster 0 11.9 scrub starts 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:02.308074+0000 osd.3 (osd.3) 606 : cluster 0 11.9 scrub ok 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:02.560858+0000 mgr.y (mgr.24368) 958 : cluster 0 pgmap v1039: 148 pgs: 148 active+clean; 844 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 822 op/s 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:02.657307+0000 osd.6 (osd.6) 111 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:02.657384+0000 osd.6 (osd.6) 112 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:02.806988+0000 mon.a (mon.0) 1758 : cluster 0 osdmap e378: 8 total, 8 up, 7 in 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:03.002782+0000 osd.7 (osd.7) 611 : cluster 0 2.9 scrub starts 2024-04-06T03:47:04.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:03 smithi153 bash[23852]: cluster 2024-04-06T03:47:03.007600+0000 osd.7 (osd.7) 612 : cluster 0 2.9 scrub ok 2024-04-06T03:47:04.320 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:04.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:04.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:04.491 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:47:04.526 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:47:04.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.574 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:47:04.592 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:04.618 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:47:04.658 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:47:04.695 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:47:04.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:04.738 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:47:04.738 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:04.772 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:47:04.777 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:47:04.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:04.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:04.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:47:05.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[21467]: audit 2024-04-06T03:47:04.451195+0000 mon.c (mon.2) 114 : audit 0 from='client.? 172.21.15.50:0/3138851640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:05.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[21467]: cluster 2024-04-06T03:47:04.561455+0000 mgr.y (mgr.24368) 959 : cluster 0 pgmap v1041: 148 pgs: 148 active+clean; 732 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 925 op/s 2024-04-06T03:47:05.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[21467]: audit 2024-04-06T03:47:04.894404+0000 mon.c (mon.2) 115 : audit 1 from='client.? 172.21.15.50:0/759695134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:05.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[21467]: audit 2024-04-06T03:47:04.894856+0000 mon.a (mon.0) 1759 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[29221]: audit 2024-04-06T03:47:04.451195+0000 mon.c (mon.2) 114 : audit 0 from='client.? 172.21.15.50:0/3138851640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[29221]: cluster 2024-04-06T03:47:04.561455+0000 mgr.y (mgr.24368) 959 : cluster 0 pgmap v1041: 148 pgs: 148 active+clean; 732 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 925 op/s 2024-04-06T03:47:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[29221]: audit 2024-04-06T03:47:04.894404+0000 mon.c (mon.2) 115 : audit 1 from='client.? 172.21.15.50:0/759695134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:05 smithi050 bash[29221]: audit 2024-04-06T03:47:04.894856+0000 mon.a (mon.0) 1759 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:05.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:05.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:05 smithi153 bash[23852]: audit 2024-04-06T03:47:04.451195+0000 mon.c (mon.2) 114 : audit 0 from='client.? 172.21.15.50:0/3138851640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:05.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:05 smithi153 bash[23852]: cluster 2024-04-06T03:47:04.561455+0000 mgr.y (mgr.24368) 959 : cluster 0 pgmap v1041: 148 pgs: 148 active+clean; 732 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 925 op/s 2024-04-06T03:47:05.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:05 smithi153 bash[23852]: audit 2024-04-06T03:47:04.894404+0000 mon.c (mon.2) 115 : audit 1 from='client.? 172.21.15.50:0/759695134' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:05.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:05 smithi153 bash[23852]: audit 2024-04-06T03:47:04.894856+0000 mon.a (mon.0) 1759 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:05.144 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:05.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:05.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:05.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:05.396 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:05.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:05.533 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:05.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:05.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:05.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:05.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:05.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:06.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:06.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.004570+0000 mon.a (mon.0) 1760 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:47:06.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.004638+0000 mon.a (mon.0) 1761 : cluster 1 Cluster is now healthy 2024-04-06T03:47:06.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: audit 2024-04-06T03:47:05.017053+0000 mon.a (mon.0) 1762 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:47:06.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.022499+0000 mon.a (mon.0) 1763 : cluster 0 osdmap e379: 8 total, 8 up, 7 in 2024-04-06T03:47:06.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: audit 2024-04-06T03:47:05.642553+0000 mon.a (mon.0) 1764 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: audit 2024-04-06T03:47:05.646081+0000 mon.a (mon.0) 1765 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.921991+0000 osd.3 (osd.3) 607 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.922057+0000 osd.3 (osd.3) 608 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.004570+0000 mon.a (mon.0) 1760 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.004638+0000 mon.a (mon.0) 1761 : cluster 1 Cluster is now healthy 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: audit 2024-04-06T03:47:05.017053+0000 mon.a (mon.0) 1762 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.022499+0000 mon.a (mon.0) 1763 : cluster 0 osdmap e379: 8 total, 8 up, 7 in 2024-04-06T03:47:06.052 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: audit 2024-04-06T03:47:05.642553+0000 mon.a (mon.0) 1764 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:06.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: audit 2024-04-06T03:47:05.646081+0000 mon.a (mon.0) 1765 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:06.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.921991+0000 osd.3 (osd.3) 607 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:06.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:06 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.922057+0000 osd.3 (osd.3) 608 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:06.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:06.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:06.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:06.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.004570+0000 mon.a (mon.0) 1760 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.004638+0000 mon.a (mon.0) 1761 : cluster 1 Cluster is now healthy 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: audit 2024-04-06T03:47:05.017053+0000 mon.a (mon.0) 1762 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.022499+0000 mon.a (mon.0) 1763 : cluster 0 osdmap e379: 8 total, 8 up, 7 in 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: audit 2024-04-06T03:47:05.642553+0000 mon.a (mon.0) 1764 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: audit 2024-04-06T03:47:05.646081+0000 mon.a (mon.0) 1765 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.921991+0000 osd.3 (osd.3) 607 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:06.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:06 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.922057+0000 osd.3 (osd.3) 608 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:06.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:06.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:06.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:06.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:06.577 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:06.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:06.724 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:06.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:06.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:06.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:06.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:07.212 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.524381+0000 osd.4 (osd.4) 583 : cluster 0 5.1c scrub starts 2024-04-06T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[21467]: cluster 2024-04-06T03:47:05.526862+0000 osd.4 (osd.4) 584 : cluster 0 5.1c scrub ok 2024-04-06T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[21467]: cluster 2024-04-06T03:47:06.485563+0000 osd.2 (osd.2) 497 : cluster 0 5.d scrub starts 2024-04-06T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[21467]: cluster 2024-04-06T03:47:06.488846+0000 osd.2 (osd.2) 498 : cluster 0 5.d scrub ok 2024-04-06T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[21467]: cluster 2024-04-06T03:47:06.562133+0000 mgr.y (mgr.24368) 960 : cluster 0 pgmap v1043: 148 pgs: 148 active+clean; 546 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 973 op/s 2024-04-06T03:47:07.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[21467]: cluster 2024-04-06T03:47:06.646342+0000 mon.a (mon.0) 1766 : cluster 0 osdmap e380: 8 total, 8 up, 7 in 2024-04-06T03:47:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.524381+0000 osd.4 (osd.4) 583 : cluster 0 5.1c scrub starts 2024-04-06T03:47:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[29221]: cluster 2024-04-06T03:47:05.526862+0000 osd.4 (osd.4) 584 : cluster 0 5.1c scrub ok 2024-04-06T03:47:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[29221]: cluster 2024-04-06T03:47:06.485563+0000 osd.2 (osd.2) 497 : cluster 0 5.d scrub starts 2024-04-06T03:47:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[29221]: cluster 2024-04-06T03:47:06.488846+0000 osd.2 (osd.2) 498 : cluster 0 5.d scrub ok 2024-04-06T03:47:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[29221]: cluster 2024-04-06T03:47:06.562133+0000 mgr.y (mgr.24368) 960 : cluster 0 pgmap v1043: 148 pgs: 148 active+clean; 546 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 973 op/s 2024-04-06T03:47:07.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:07 smithi050 bash[29221]: cluster 2024-04-06T03:47:06.646342+0000 mon.a (mon.0) 1766 : cluster 0 osdmap e380: 8 total, 8 up, 7 in 2024-04-06T03:47:07.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:07 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.524381+0000 osd.4 (osd.4) 583 : cluster 0 5.1c scrub starts 2024-04-06T03:47:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:07 smithi153 bash[23852]: cluster 2024-04-06T03:47:05.526862+0000 osd.4 (osd.4) 584 : cluster 0 5.1c scrub ok 2024-04-06T03:47:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:07 smithi153 bash[23852]: cluster 2024-04-06T03:47:06.485563+0000 osd.2 (osd.2) 497 : cluster 0 5.d scrub starts 2024-04-06T03:47:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:07 smithi153 bash[23852]: cluster 2024-04-06T03:47:06.488846+0000 osd.2 (osd.2) 498 : cluster 0 5.d scrub ok 2024-04-06T03:47:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:07 smithi153 bash[23852]: cluster 2024-04-06T03:47:06.562133+0000 mgr.y (mgr.24368) 960 : cluster 0 pgmap v1043: 148 pgs: 148 active+clean; 546 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 973 op/s 2024-04-06T03:47:07.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:07 smithi153 bash[23852]: cluster 2024-04-06T03:47:06.646342+0000 mon.a (mon.0) 1766 : cluster 0 osdmap e380: 8 total, 8 up, 7 in 2024-04-06T03:47:07.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:07.341 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:07.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:07.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:07.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:07.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:07.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:07.722 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:07.759 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:07.760 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:07.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:07.852 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:07.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:07.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:07.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:08.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:08.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:08.215 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:08.290 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:08.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:08 smithi153 bash[23852]: cluster 2024-04-06T03:47:07.354276+0000 osd.4 (osd.4) 585 : cluster 0 3.14 scrub starts 2024-04-06T03:47:08.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:08 smithi153 bash[23852]: cluster 2024-04-06T03:47:07.356247+0000 osd.4 (osd.4) 586 : cluster 0 3.14 scrub ok 2024-04-06T03:47:08.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:08.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:08.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.562 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:08.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:08.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:08.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:08.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:08 smithi050 bash[21467]: cluster 2024-04-06T03:47:07.354276+0000 osd.4 (osd.4) 585 : cluster 0 3.14 scrub starts 2024-04-06T03:47:08.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:08 smithi050 bash[21467]: cluster 2024-04-06T03:47:07.356247+0000 osd.4 (osd.4) 586 : cluster 0 3.14 scrub ok 2024-04-06T03:47:08.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:08 smithi050 bash[29221]: cluster 2024-04-06T03:47:07.354276+0000 osd.4 (osd.4) 585 : cluster 0 3.14 scrub starts 2024-04-06T03:47:08.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:08 smithi050 bash[29221]: cluster 2024-04-06T03:47:07.356247+0000 osd.4 (osd.4) 586 : cluster 0 3.14 scrub ok 2024-04-06T03:47:08.792 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:08.832 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:08.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:08.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:08.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:09.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:09.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:09.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.206 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:09.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:47:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:09 smithi153 bash[23852]: cluster 2024-04-06T03:47:07.302366+0000 osd.1 (osd.1) 401 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:09 smithi153 bash[23852]: cluster 2024-04-06T03:47:07.302423+0000 osd.1 (osd.1) 402 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:09 smithi153 bash[23852]: cluster 2024-04-06T03:47:08.041684+0000 mon.a (mon.0) 1767 : cluster 0 osdmap e381: 8 total, 8 up, 7 in 2024-04-06T03:47:09.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:09 smithi153 bash[23852]: cluster 2024-04-06T03:47:08.562738+0000 mgr.y (mgr.24368) 961 : cluster 0 pgmap v1046: 148 pgs: 148 active+clean; 419 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-04-06T03:47:09.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:09.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set noscrub 2024-04-06T03:47:09.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:09.470 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[29221]: cluster 2024-04-06T03:47:07.302366+0000 osd.1 (osd.1) 401 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:09.470 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[29221]: cluster 2024-04-06T03:47:07.302423+0000 osd.1 (osd.1) 402 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:09.470 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[29221]: cluster 2024-04-06T03:47:08.041684+0000 mon.a (mon.0) 1767 : cluster 0 osdmap e381: 8 total, 8 up, 7 in 2024-04-06T03:47:09.470 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[29221]: cluster 2024-04-06T03:47:08.562738+0000 mgr.y (mgr.24368) 961 : cluster 0 pgmap v1046: 148 pgs: 148 active+clean; 419 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-04-06T03:47:09.471 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[21467]: cluster 2024-04-06T03:47:07.302366+0000 osd.1 (osd.1) 401 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:09.471 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[21467]: cluster 2024-04-06T03:47:07.302423+0000 osd.1 (osd.1) 402 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:09.471 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[21467]: cluster 2024-04-06T03:47:08.041684+0000 mon.a (mon.0) 1767 : cluster 0 osdmap e381: 8 total, 8 up, 7 in 2024-04-06T03:47:09.471 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[21467]: cluster 2024-04-06T03:47:08.562738+0000 mgr.y (mgr.24368) 961 : cluster 0 pgmap v1046: 148 pgs: 148 active+clean; 419 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-04-06T03:47:09.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:09.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:09.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:09.730 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:47:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:47:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:47:09.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:09.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:09.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:09.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:10.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:10.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:10.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:10.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:10.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:10.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:10.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:10.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:10.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:10.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:10.647 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:47:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:47:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:10.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:10.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:10.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:10.891 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:11.014 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:10 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.035900+0000 osd.7 (osd.7) 613 : cluster 0 3.16 scrub starts 2024-04-06T03:47:11.014 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:10 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.039023+0000 osd.7 (osd.7) 614 : cluster 0 3.16 scrub ok 2024-04-06T03:47:11.029 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:11.132 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:11.153 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:11.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:10 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.035900+0000 osd.7 (osd.7) 613 : cluster 0 3.16 scrub starts 2024-04-06T03:47:11.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:10 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.039023+0000 osd.7 (osd.7) 614 : cluster 0 3.16 scrub ok 2024-04-06T03:47:11.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:10 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.035900+0000 osd.7 (osd.7) 613 : cluster 0 3.16 scrub starts 2024-04-06T03:47:11.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:10 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.039023+0000 osd.7 (osd.7) 614 : cluster 0 3.16 scrub ok 2024-04-06T03:47:11.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:11.283 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:11.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:11.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:11.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:11.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:11.643 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:11.665 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[21467]: audit 2024-04-06T03:47:10.348126+0000 mgr.y (mgr.24368) 962 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.469868+0000 osd.4 (osd.4) 587 : cluster 0 5.e deep-scrub starts 2024-04-06T03:47:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.480592+0000 osd.4 (osd.4) 588 : cluster 0 5.e deep-scrub ok 2024-04-06T03:47:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.563596+0000 mgr.y (mgr.24368) 963 : cluster 0 pgmap v1047: 148 pgs: 148 active+clean; 267 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 1.24k op/s 2024-04-06T03:47:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.992826+0000 osd.0 (osd.0) 609 : cluster 0 5.3 scrub starts 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[21467]: cluster 2024-04-06T03:47:10.994274+0000 osd.0 (osd.0) 610 : cluster 0 5.3 scrub ok 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[29221]: audit 2024-04-06T03:47:10.348126+0000 mgr.y (mgr.24368) 962 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.469868+0000 osd.4 (osd.4) 587 : cluster 0 5.e deep-scrub starts 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.480592+0000 osd.4 (osd.4) 588 : cluster 0 5.e deep-scrub ok 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.563596+0000 mgr.y (mgr.24368) 963 : cluster 0 pgmap v1047: 148 pgs: 148 active+clean; 267 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 1.24k op/s 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.992826+0000 osd.0 (osd.0) 609 : cluster 0 5.3 scrub starts 2024-04-06T03:47:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:11 smithi050 bash[29221]: cluster 2024-04-06T03:47:10.994274+0000 osd.0 (osd.0) 610 : cluster 0 5.3 scrub ok 2024-04-06T03:47:11.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:11.823 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:11.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:11.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:11.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:12.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:11 smithi153 bash[23852]: audit 2024-04-06T03:47:10.348126+0000 mgr.y (mgr.24368) 962 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:11 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.469868+0000 osd.4 (osd.4) 587 : cluster 0 5.e deep-scrub starts 2024-04-06T03:47:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:11 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.480592+0000 osd.4 (osd.4) 588 : cluster 0 5.e deep-scrub ok 2024-04-06T03:47:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:11 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.563596+0000 mgr.y (mgr.24368) 963 : cluster 0 pgmap v1047: 148 pgs: 148 active+clean; 267 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 1.24k op/s 2024-04-06T03:47:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:11 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.992826+0000 osd.0 (osd.0) 609 : cluster 0 5.3 scrub starts 2024-04-06T03:47:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:11 smithi153 bash[23852]: cluster 2024-04-06T03:47:10.994274+0000 osd.0 (osd.0) 610 : cluster 0 5.3 scrub ok 2024-04-06T03:47:12.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:12.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:12.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:12.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:12.359 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:12.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:12.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:12.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:12.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:12.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:12.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:12.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:13.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:13.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:13.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:13.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:13.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:13.280 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:13.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:13.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:13.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:13.556 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:13.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[21467]: cluster 2024-04-06T03:47:12.564243+0000 mgr.y (mgr.24368) 964 : cluster 0 pgmap v1048: 148 pgs: 148 active+clean; 250 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 542 B/s rd, 0 B/s wr, 844 op/s 2024-04-06T03:47:13.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[21467]: cluster 2024-04-06T03:47:12.578418+0000 osd.2 (osd.2) 499 : cluster 0 6.f scrub starts 2024-04-06T03:47:13.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[21467]: cluster 2024-04-06T03:47:12.770677+0000 osd.2 (osd.2) 500 : cluster 0 6.f scrub ok 2024-04-06T03:47:13.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[21467]: cluster 2024-04-06T03:47:13.129538+0000 osd.7 (osd.7) 615 : cluster 0 6.3 scrub starts 2024-04-06T03:47:13.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[21467]: cluster 2024-04-06T03:47:13.250718+0000 osd.7 (osd.7) 616 : cluster 0 6.3 scrub ok 2024-04-06T03:47:13.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[29221]: cluster 2024-04-06T03:47:12.564243+0000 mgr.y (mgr.24368) 964 : cluster 0 pgmap v1048: 148 pgs: 148 active+clean; 250 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 542 B/s rd, 0 B/s wr, 844 op/s 2024-04-06T03:47:13.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[29221]: cluster 2024-04-06T03:47:12.578418+0000 osd.2 (osd.2) 499 : cluster 0 6.f scrub starts 2024-04-06T03:47:13.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[29221]: cluster 2024-04-06T03:47:12.770677+0000 osd.2 (osd.2) 500 : cluster 0 6.f scrub ok 2024-04-06T03:47:13.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[29221]: cluster 2024-04-06T03:47:13.129538+0000 osd.7 (osd.7) 615 : cluster 0 6.3 scrub starts 2024-04-06T03:47:13.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:13 smithi050 bash[29221]: cluster 2024-04-06T03:47:13.250718+0000 osd.7 (osd.7) 616 : cluster 0 6.3 scrub ok 2024-04-06T03:47:13.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:13.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:13.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:13.695 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Removed 105971 objects 2024-04-06T03:47:13.695 INFO:tasks.radosbench.radosbench.0.smithi050.stdout:Clean up completed and total clean up time :112.437 2024-04-06T03:47:13.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:13.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:13 smithi153 bash[23852]: cluster 2024-04-06T03:47:12.564243+0000 mgr.y (mgr.24368) 964 : cluster 0 pgmap v1048: 148 pgs: 148 active+clean; 250 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 542 B/s rd, 0 B/s wr, 844 op/s 2024-04-06T03:47:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:13 smithi153 bash[23852]: cluster 2024-04-06T03:47:12.578418+0000 osd.2 (osd.2) 499 : cluster 0 6.f scrub starts 2024-04-06T03:47:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:13 smithi153 bash[23852]: cluster 2024-04-06T03:47:12.770677+0000 osd.2 (osd.2) 500 : cluster 0 6.f scrub ok 2024-04-06T03:47:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:13 smithi153 bash[23852]: cluster 2024-04-06T03:47:13.129538+0000 osd.7 (osd.7) 615 : cluster 0 6.3 scrub starts 2024-04-06T03:47:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:13 smithi153 bash[23852]: cluster 2024-04-06T03:47:13.250718+0000 osd.7 (osd.7) 616 : cluster 0 6.3 scrub ok 2024-04-06T03:47:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:13.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:13.949 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:13.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:14.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:14.080 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:14.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:14.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:14.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:14.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:14.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:14.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:14.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:14.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:14.737 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:14.840 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:14.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:14.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:14.970 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:14.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:15.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:15.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:15.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:15.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:15 smithi153 bash[23852]: cluster 2024-04-06T03:47:13.979246+0000 osd.0 (osd.0) 611 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:47:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:15 smithi153 bash[23852]: cluster 2024-04-06T03:47:13.988735+0000 osd.0 (osd.0) 612 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:47:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:15 smithi153 bash[23852]: cluster 2024-04-06T03:47:14.565147+0000 mgr.y (mgr.24368) 965 : cluster 0 pgmap v1049: 148 pgs: 148 active+clean; 143 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 808 op/s 2024-04-06T03:47:15.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:15.388 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:15 smithi050 bash[21467]: cluster 2024-04-06T03:47:13.979246+0000 osd.0 (osd.0) 611 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:47:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:15 smithi050 bash[21467]: cluster 2024-04-06T03:47:13.988735+0000 osd.0 (osd.0) 612 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:47:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:15 smithi050 bash[21467]: cluster 2024-04-06T03:47:14.565147+0000 mgr.y (mgr.24368) 965 : cluster 0 pgmap v1049: 148 pgs: 148 active+clean; 143 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 808 op/s 2024-04-06T03:47:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:15 smithi050 bash[29221]: cluster 2024-04-06T03:47:13.979246+0000 osd.0 (osd.0) 611 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:47:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:15 smithi050 bash[29221]: cluster 2024-04-06T03:47:13.988735+0000 osd.0 (osd.0) 612 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:47:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:15 smithi050 bash[29221]: cluster 2024-04-06T03:47:14.565147+0000 mgr.y (mgr.24368) 965 : cluster 0 pgmap v1049: 148 pgs: 148 active+clean; 143 MiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 808 op/s 2024-04-06T03:47:15.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:15.518 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:15.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:15.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:15.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:15.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:15.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:15.912 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:15.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:16.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:16.047 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:16.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:16.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:16 smithi153 bash[23852]: cluster 2024-04-06T03:47:15.016000+0000 osd.0 (osd.0) 613 : cluster 0 6.10 scrub starts 2024-04-06T03:47:16.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:16 smithi153 bash[23852]: cluster 2024-04-06T03:47:15.099533+0000 osd.0 (osd.0) 614 : cluster 0 6.10 scrub ok 2024-04-06T03:47:16.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:16 smithi153 bash[23852]: cluster 2024-04-06T03:47:15.127184+0000 osd.7 (osd.7) 617 : cluster 0 11.5 scrub starts 2024-04-06T03:47:16.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:16 smithi153 bash[23852]: cluster 2024-04-06T03:47:15.155457+0000 osd.7 (osd.7) 618 : cluster 0 11.5 scrub ok 2024-04-06T03:47:16.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:16.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:16.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[21467]: cluster 2024-04-06T03:47:15.016000+0000 osd.0 (osd.0) 613 : cluster 0 6.10 scrub starts 2024-04-06T03:47:16.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[21467]: cluster 2024-04-06T03:47:15.099533+0000 osd.0 (osd.0) 614 : cluster 0 6.10 scrub ok 2024-04-06T03:47:16.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[21467]: cluster 2024-04-06T03:47:15.127184+0000 osd.7 (osd.7) 617 : cluster 0 11.5 scrub starts 2024-04-06T03:47:16.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[21467]: cluster 2024-04-06T03:47:15.155457+0000 osd.7 (osd.7) 618 : cluster 0 11.5 scrub ok 2024-04-06T03:47:16.292 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[29221]: cluster 2024-04-06T03:47:15.016000+0000 osd.0 (osd.0) 613 : cluster 0 6.10 scrub starts 2024-04-06T03:47:16.292 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[29221]: cluster 2024-04-06T03:47:15.099533+0000 osd.0 (osd.0) 614 : cluster 0 6.10 scrub ok 2024-04-06T03:47:16.292 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[29221]: cluster 2024-04-06T03:47:15.127184+0000 osd.7 (osd.7) 617 : cluster 0 11.5 scrub starts 2024-04-06T03:47:16.292 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:16 smithi050 bash[29221]: cluster 2024-04-06T03:47:15.155457+0000 osd.7 (osd.7) 618 : cluster 0 11.5 scrub ok 2024-04-06T03:47:16.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:16.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:16.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:16.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:16.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:16.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:16.692 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:16.781 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:16.781 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:16.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:16.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:16.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:16.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:16.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:16.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:17.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:17.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:17.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:17.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:17.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:15.581852+0000 osd.2 (osd.2) 501 : cluster 0 3.0 scrub starts 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:15.587252+0000 osd.2 (osd.2) 502 : cluster 0 3.0 scrub ok 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.051473+0000 osd.0 (osd.0) 615 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.053225+0000 osd.0 (osd.0) 616 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.143380+0000 osd.7 (osd.7) 619 : cluster 0 11.3 scrub starts 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.168453+0000 osd.7 (osd.7) 620 : cluster 0 11.3 scrub ok 2024-04-06T03:47:17.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:17 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.565931+0000 mgr.y (mgr.24368) 966 : cluster 0 pgmap v1050: 148 pgs: 148 active+clean; 15 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 928 B/s rd, 0 B/s wr, 857 op/s 2024-04-06T03:47:17.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:17.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:15.581852+0000 osd.2 (osd.2) 501 : cluster 0 3.0 scrub starts 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:15.587252+0000 osd.2 (osd.2) 502 : cluster 0 3.0 scrub ok 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.051473+0000 osd.0 (osd.0) 615 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.053225+0000 osd.0 (osd.0) 616 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.143380+0000 osd.7 (osd.7) 619 : cluster 0 11.3 scrub starts 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.168453+0000 osd.7 (osd.7) 620 : cluster 0 11.3 scrub ok 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.565931+0000 mgr.y (mgr.24368) 966 : cluster 0 pgmap v1050: 148 pgs: 148 active+clean; 15 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 928 B/s rd, 0 B/s wr, 857 op/s 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:15.581852+0000 osd.2 (osd.2) 501 : cluster 0 3.0 scrub starts 2024-04-06T03:47:17.362 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:15.587252+0000 osd.2 (osd.2) 502 : cluster 0 3.0 scrub ok 2024-04-06T03:47:17.363 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.051473+0000 osd.0 (osd.0) 615 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:47:17.363 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.053225+0000 osd.0 (osd.0) 616 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:47:17.363 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.143380+0000 osd.7 (osd.7) 619 : cluster 0 11.3 scrub starts 2024-04-06T03:47:17.363 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.168453+0000 osd.7 (osd.7) 620 : cluster 0 11.3 scrub ok 2024-04-06T03:47:17.363 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:17 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.565931+0000 mgr.y (mgr.24368) 966 : cluster 0 pgmap v1050: 148 pgs: 148 active+clean; 15 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 928 B/s rd, 0 B/s wr, 857 op/s 2024-04-06T03:47:17.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:17.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:17.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:17.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:17.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:17.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:17.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:17.866 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:17.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:17.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:17.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:18.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:18.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:18.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:18.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:18 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.582469+0000 osd.2 (osd.2) 503 : cluster 0 6.1 scrub starts 2024-04-06T03:47:18.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:18 smithi153 bash[23852]: cluster 2024-04-06T03:47:16.620138+0000 osd.2 (osd.2) 504 : cluster 0 6.1 scrub ok 2024-04-06T03:47:18.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:18 smithi153 bash[23852]: cluster 2024-04-06T03:47:17.117618+0000 osd.7 (osd.7) 621 : cluster 0 3.11 scrub starts 2024-04-06T03:47:18.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:18 smithi153 bash[23852]: cluster 2024-04-06T03:47:17.121921+0000 osd.7 (osd.7) 622 : cluster 0 3.11 scrub ok 2024-04-06T03:47:18.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:18 smithi153 bash[23852]: cluster 2024-04-06T03:47:17.363839+0000 osd.4 (osd.4) 589 : cluster 0 11.e deep-scrub starts 2024-04-06T03:47:18.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:18 smithi153 bash[23852]: cluster 2024-04-06T03:47:17.394925+0000 osd.4 (osd.4) 590 : cluster 0 11.e deep-scrub ok 2024-04-06T03:47:18.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:18.295 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.582469+0000 osd.2 (osd.2) 503 : cluster 0 6.1 scrub starts 2024-04-06T03:47:18.295 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[29221]: cluster 2024-04-06T03:47:16.620138+0000 osd.2 (osd.2) 504 : cluster 0 6.1 scrub ok 2024-04-06T03:47:18.295 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[29221]: cluster 2024-04-06T03:47:17.117618+0000 osd.7 (osd.7) 621 : cluster 0 3.11 scrub starts 2024-04-06T03:47:18.295 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[29221]: cluster 2024-04-06T03:47:17.121921+0000 osd.7 (osd.7) 622 : cluster 0 3.11 scrub ok 2024-04-06T03:47:18.295 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[29221]: cluster 2024-04-06T03:47:17.363839+0000 osd.4 (osd.4) 589 : cluster 0 11.e deep-scrub starts 2024-04-06T03:47:18.295 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[29221]: cluster 2024-04-06T03:47:17.394925+0000 osd.4 (osd.4) 590 : cluster 0 11.e deep-scrub ok 2024-04-06T03:47:18.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.582469+0000 osd.2 (osd.2) 503 : cluster 0 6.1 scrub starts 2024-04-06T03:47:18.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[21467]: cluster 2024-04-06T03:47:16.620138+0000 osd.2 (osd.2) 504 : cluster 0 6.1 scrub ok 2024-04-06T03:47:18.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[21467]: cluster 2024-04-06T03:47:17.117618+0000 osd.7 (osd.7) 621 : cluster 0 3.11 scrub starts 2024-04-06T03:47:18.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[21467]: cluster 2024-04-06T03:47:17.121921+0000 osd.7 (osd.7) 622 : cluster 0 3.11 scrub ok 2024-04-06T03:47:18.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[21467]: cluster 2024-04-06T03:47:17.363839+0000 osd.4 (osd.4) 589 : cluster 0 11.e deep-scrub starts 2024-04-06T03:47:18.296 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:18 smithi050 bash[21467]: cluster 2024-04-06T03:47:17.394925+0000 osd.4 (osd.4) 590 : cluster 0 11.e deep-scrub ok 2024-04-06T03:47:18.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:18.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:18.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:18.524 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:18.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:18.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:18.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:18.889 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:18.917 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:18.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:19.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:19.049 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:19.152 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:19.164 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:19 smithi153 bash[23852]: cluster 2024-04-06T03:47:17.567467+0000 osd.2 (osd.2) 505 : cluster 0 11.c scrub starts 2024-04-06T03:47:19.164 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:19 smithi153 bash[23852]: cluster 2024-04-06T03:47:17.601871+0000 osd.2 (osd.2) 506 : cluster 0 11.c scrub ok 2024-04-06T03:47:19.164 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:19 smithi153 bash[23852]: cluster 2024-04-06T03:47:18.395199+0000 osd.4 (osd.4) 591 : cluster 0 3.9 scrub starts 2024-04-06T03:47:19.164 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:19 smithi153 bash[23852]: cluster 2024-04-06T03:47:18.399806+0000 osd.4 (osd.4) 592 : cluster 0 3.9 scrub ok 2024-04-06T03:47:19.164 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:19 smithi153 bash[23852]: cluster 2024-04-06T03:47:18.567031+0000 mgr.y (mgr.24368) 967 : cluster 0 pgmap v1051: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 802 op/s 2024-04-06T03:47:19.179 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:19.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:19.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[21467]: cluster 2024-04-06T03:47:17.567467+0000 osd.2 (osd.2) 505 : cluster 0 11.c scrub starts 2024-04-06T03:47:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[21467]: cluster 2024-04-06T03:47:17.601871+0000 osd.2 (osd.2) 506 : cluster 0 11.c scrub ok 2024-04-06T03:47:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[21467]: cluster 2024-04-06T03:47:18.395199+0000 osd.4 (osd.4) 591 : cluster 0 3.9 scrub starts 2024-04-06T03:47:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[21467]: cluster 2024-04-06T03:47:18.399806+0000 osd.4 (osd.4) 592 : cluster 0 3.9 scrub ok 2024-04-06T03:47:19.304 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[21467]: cluster 2024-04-06T03:47:18.567031+0000 mgr.y (mgr.24368) 967 : cluster 0 pgmap v1051: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 802 op/s 2024-04-06T03:47:19.312 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:19.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:19.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:19.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:19.548 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:19.576 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:19.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:19.706 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:19.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:47:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:47:19.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[29221]: cluster 2024-04-06T03:47:17.567467+0000 osd.2 (osd.2) 505 : cluster 0 11.c scrub starts 2024-04-06T03:47:19.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[29221]: cluster 2024-04-06T03:47:17.601871+0000 osd.2 (osd.2) 506 : cluster 0 11.c scrub ok 2024-04-06T03:47:19.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[29221]: cluster 2024-04-06T03:47:18.395199+0000 osd.4 (osd.4) 591 : cluster 0 3.9 scrub starts 2024-04-06T03:47:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[29221]: cluster 2024-04-06T03:47:18.399806+0000 osd.4 (osd.4) 592 : cluster 0 3.9 scrub ok 2024-04-06T03:47:19.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:19 smithi050 bash[29221]: cluster 2024-04-06T03:47:18.567031+0000 mgr.y (mgr.24368) 967 : cluster 0 pgmap v1051: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 802 op/s 2024-04-06T03:47:19.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:19.838 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:19.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:19.970 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:19.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:20.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:20.102 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:20.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[21467]: cluster 2024-04-06T03:47:18.576058+0000 osd.3 (osd.3) 609 : cluster 0 3.5 scrub starts 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[21467]: cluster 2024-04-06T03:47:18.580243+0000 osd.3 (osd.3) 610 : cluster 0 3.5 scrub ok 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.051256+0000 osd.0 (osd.0) 617 : cluster 0 6.d scrub starts 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.198011+0000 osd.0 (osd.0) 618 : cluster 0 6.d scrub ok 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.507707+0000 osd.2 (osd.2) 507 : cluster 0 11.d scrub starts 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.528361+0000 osd.2 (osd.2) 508 : cluster 0 11.d scrub ok 2024-04-06T03:47:20.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[29221]: cluster 2024-04-06T03:47:18.576058+0000 osd.3 (osd.3) 609 : cluster 0 3.5 scrub starts 2024-04-06T03:47:20.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[29221]: cluster 2024-04-06T03:47:18.580243+0000 osd.3 (osd.3) 610 : cluster 0 3.5 scrub ok 2024-04-06T03:47:20.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.051256+0000 osd.0 (osd.0) 617 : cluster 0 6.d scrub starts 2024-04-06T03:47:20.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.198011+0000 osd.0 (osd.0) 618 : cluster 0 6.d scrub ok 2024-04-06T03:47:20.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.507707+0000 osd.2 (osd.2) 507 : cluster 0 11.d scrub starts 2024-04-06T03:47:20.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:20 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.528361+0000 osd.2 (osd.2) 508 : cluster 0 11.d scrub ok 2024-04-06T03:47:20.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:20.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:20.350 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[23852]: cluster 2024-04-06T03:47:18.576058+0000 osd.3 (osd.3) 609 : cluster 0 3.5 scrub starts 2024-04-06T03:47:20.350 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[23852]: cluster 2024-04-06T03:47:18.580243+0000 osd.3 (osd.3) 610 : cluster 0 3.5 scrub ok 2024-04-06T03:47:20.350 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.051256+0000 osd.0 (osd.0) 617 : cluster 0 6.d scrub starts 2024-04-06T03:47:20.350 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.198011+0000 osd.0 (osd.0) 618 : cluster 0 6.d scrub ok 2024-04-06T03:47:20.351 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.507707+0000 osd.2 (osd.2) 507 : cluster 0 11.d scrub starts 2024-04-06T03:47:20.351 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.528361+0000 osd.2 (osd.2) 508 : cluster 0 11.d scrub ok 2024-04-06T03:47:20.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:20.469 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:20.495 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:20.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:20.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:20.734 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:47:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:47:20.750 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:20.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:20.880 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:20.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:20.983 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:21.010 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:21.113 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:21.141 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:21.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.600109+0000 osd.3 (osd.3) 611 : cluster 0 11.b scrub starts 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.616199+0000 osd.3 (osd.3) 612 : cluster 0 11.b scrub ok 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.668789+0000 osd.4 (osd.4) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.668894+0000 osd.4 (osd.4) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.988680+0000 osd.0 (osd.0) 619 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:19.988762+0000 osd.0 (osd.0) 620 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:20.047132+0000 mon.a (mon.0) 1768 : cluster 0 osdmap e382: 8 total, 8 up, 7 in 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:20.128753+0000 osd.7 (osd.7) 623 : cluster 0 11.f scrub starts 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:20.138555+0000 osd.7 (osd.7) 624 : cluster 0 11.f scrub ok 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: audit 2024-04-06T03:47:20.350129+0000 mgr.y (mgr.24368) 968 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:20.380449+0000 osd.4 (osd.4) 595 : cluster 0 11.7 scrub starts 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:20.393917+0000 osd.4 (osd.4) 596 : cluster 0 11.7 scrub ok 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: cluster 2024-04-06T03:47:20.567664+0000 mgr.y (mgr.24368) 969 : cluster 0 pgmap v1053: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 426 op/s 2024-04-06T03:47:21.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[21467]: audit 2024-04-06T03:47:20.664676+0000 mon.a (mon.0) 1769 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.600109+0000 osd.3 (osd.3) 611 : cluster 0 11.b scrub starts 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.616199+0000 osd.3 (osd.3) 612 : cluster 0 11.b scrub ok 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.668789+0000 osd.4 (osd.4) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.668894+0000 osd.4 (osd.4) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.988680+0000 osd.0 (osd.0) 619 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:19.988762+0000 osd.0 (osd.0) 620 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:20.047132+0000 mon.a (mon.0) 1768 : cluster 0 osdmap e382: 8 total, 8 up, 7 in 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:20.128753+0000 osd.7 (osd.7) 623 : cluster 0 11.f scrub starts 2024-04-06T03:47:21.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:20.138555+0000 osd.7 (osd.7) 624 : cluster 0 11.f scrub ok 2024-04-06T03:47:21.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: audit 2024-04-06T03:47:20.350129+0000 mgr.y (mgr.24368) 968 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:21.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:20.380449+0000 osd.4 (osd.4) 595 : cluster 0 11.7 scrub starts 2024-04-06T03:47:21.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:20.393917+0000 osd.4 (osd.4) 596 : cluster 0 11.7 scrub ok 2024-04-06T03:47:21.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: cluster 2024-04-06T03:47:20.567664+0000 mgr.y (mgr.24368) 969 : cluster 0 pgmap v1053: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 426 op/s 2024-04-06T03:47:21.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:21 smithi050 bash[29221]: audit 2024-04-06T03:47:20.664676+0000 mon.a (mon.0) 1769 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:21.244 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.600109+0000 osd.3 (osd.3) 611 : cluster 0 11.b scrub starts 2024-04-06T03:47:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.616199+0000 osd.3 (osd.3) 612 : cluster 0 11.b scrub ok 2024-04-06T03:47:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.668789+0000 osd.4 (osd.4) 593 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.668894+0000 osd.4 (osd.4) 594 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:21.255 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.988680+0000 osd.0 (osd.0) 619 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:19.988762+0000 osd.0 (osd.0) 620 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:20.047132+0000 mon.a (mon.0) 1768 : cluster 0 osdmap e382: 8 total, 8 up, 7 in 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:20.128753+0000 osd.7 (osd.7) 623 : cluster 0 11.f scrub starts 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:20.138555+0000 osd.7 (osd.7) 624 : cluster 0 11.f scrub ok 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: audit 2024-04-06T03:47:20.350129+0000 mgr.y (mgr.24368) 968 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:20.380449+0000 osd.4 (osd.4) 595 : cluster 0 11.7 scrub starts 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:20.393917+0000 osd.4 (osd.4) 596 : cluster 0 11.7 scrub ok 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: cluster 2024-04-06T03:47:20.567664+0000 mgr.y (mgr.24368) 969 : cluster 0 pgmap v1053: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 426 op/s 2024-04-06T03:47:21.256 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:21 smithi153 bash[23852]: audit 2024-04-06T03:47:20.664676+0000 mon.a (mon.0) 1769 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:21.271 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:21.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.401 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:21.410 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:47:21.410 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:47:21.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:21.630 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:47:21.638 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:47:21.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:21.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:21.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:21.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:21.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:21.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:22.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.043 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:22.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.155 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:22 smithi153 bash[23852]: cluster 2024-04-06T03:47:21.507454+0000 osd.2 (osd.2) 509 : cluster 0 2.c scrub starts 2024-04-06T03:47:22.155 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:22 smithi153 bash[23852]: cluster 2024-04-06T03:47:21.508869+0000 osd.2 (osd.2) 510 : cluster 0 2.c scrub ok 2024-04-06T03:47:22.155 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:22 smithi153 bash[23852]: audit 2024-04-06T03:47:21.599373+0000 mgr.y (mgr.24368) 970 : audit 0 from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:47:22.171 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:22.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:22 smithi050 bash[29221]: cluster 2024-04-06T03:47:21.507454+0000 osd.2 (osd.2) 509 : cluster 0 2.c scrub starts 2024-04-06T03:47:22.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:22 smithi050 bash[29221]: cluster 2024-04-06T03:47:21.508869+0000 osd.2 (osd.2) 510 : cluster 0 2.c scrub ok 2024-04-06T03:47:22.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:22 smithi050 bash[29221]: audit 2024-04-06T03:47:21.599373+0000 mgr.y (mgr.24368) 970 : audit 0 from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:47:22.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:22 smithi050 bash[21467]: cluster 2024-04-06T03:47:21.507454+0000 osd.2 (osd.2) 509 : cluster 0 2.c scrub starts 2024-04-06T03:47:22.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:22 smithi050 bash[21467]: cluster 2024-04-06T03:47:21.508869+0000 osd.2 (osd.2) 510 : cluster 0 2.c scrub ok 2024-04-06T03:47:22.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:22 smithi050 bash[21467]: audit 2024-04-06T03:47:21.599373+0000 mgr.y (mgr.24368) 970 : audit 0 from='client.15618 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:47:22.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:22.301 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:22.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:22.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:22.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:22.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:22.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.693 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:22.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:22.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:22.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:22.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:23.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:23.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:23.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:23.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:23 smithi153 bash[23852]: cluster 2024-04-06T03:47:22.568203+0000 mgr.y (mgr.24368) 971 : cluster 0 pgmap v1054: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 399 op/s 2024-04-06T03:47:23.215 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:23.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:23.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:23 smithi050 bash[21467]: cluster 2024-04-06T03:47:22.568203+0000 mgr.y (mgr.24368) 971 : cluster 0 pgmap v1054: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 399 op/s 2024-04-06T03:47:23.320 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:23 smithi050 bash[29221]: cluster 2024-04-06T03:47:22.568203+0000 mgr.y (mgr.24368) 971 : cluster 0 pgmap v1054: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 399 op/s 2024-04-06T03:47:23.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:23.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:23.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:23.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:23.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:23.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:23.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:23.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:23.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:23.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:23.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:24.000 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:24.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:24.129 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:24.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:24.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:24.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:24 smithi050 bash[21467]: cluster 2024-04-06T03:47:22.977489+0000 osd.0 (osd.0) 621 : cluster 0 3.6 scrub starts 2024-04-06T03:47:24.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:24 smithi050 bash[21467]: cluster 2024-04-06T03:47:22.981241+0000 osd.0 (osd.0) 622 : cluster 0 3.6 scrub ok 2024-04-06T03:47:24.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:24 smithi050 bash[29221]: cluster 2024-04-06T03:47:22.977489+0000 osd.0 (osd.0) 621 : cluster 0 3.6 scrub starts 2024-04-06T03:47:24.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:24 smithi050 bash[29221]: cluster 2024-04-06T03:47:22.981241+0000 osd.0 (osd.0) 622 : cluster 0 3.6 scrub ok 2024-04-06T03:47:24.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:24 smithi153 bash[23852]: cluster 2024-04-06T03:47:22.977489+0000 osd.0 (osd.0) 621 : cluster 0 3.6 scrub starts 2024-04-06T03:47:24.243 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:24 smithi153 bash[23852]: cluster 2024-04-06T03:47:22.981241+0000 osd.0 (osd.0) 622 : cluster 0 3.6 scrub ok 2024-04-06T03:47:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:24.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:24.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:24.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:24.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:24.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:24.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:24.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:24.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:24.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:24.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:24.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:25.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:25.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:25.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:25.150 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.097901+0000 osd.7 (osd.7) 625 : cluster 0 2.0 scrub starts 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.101353+0000 osd.7 (osd.7) 626 : cluster 0 2.0 scrub ok 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.454519+0000 osd.4 (osd.4) 597 : cluster 0 3.3 scrub starts 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.459471+0000 osd.4 (osd.4) 598 : cluster 0 3.3 scrub ok 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.568999+0000 mgr.y (mgr.24368) 972 : cluster 0 pgmap v1055: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 228 op/s 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.097901+0000 osd.7 (osd.7) 625 : cluster 0 2.0 scrub starts 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.101353+0000 osd.7 (osd.7) 626 : cluster 0 2.0 scrub ok 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.454519+0000 osd.4 (osd.4) 597 : cluster 0 3.3 scrub starts 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.459471+0000 osd.4 (osd.4) 598 : cluster 0 3.3 scrub ok 2024-04-06T03:47:25.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:25 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.568999+0000 mgr.y (mgr.24368) 972 : cluster 0 pgmap v1055: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 228 op/s 2024-04-06T03:47:25.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:25.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:25.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:25 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.097901+0000 osd.7 (osd.7) 625 : cluster 0 2.0 scrub starts 2024-04-06T03:47:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:25 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.101353+0000 osd.7 (osd.7) 626 : cluster 0 2.0 scrub ok 2024-04-06T03:47:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:25 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.454519+0000 osd.4 (osd.4) 597 : cluster 0 3.3 scrub starts 2024-04-06T03:47:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:25 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.459471+0000 osd.4 (osd.4) 598 : cluster 0 3.3 scrub ok 2024-04-06T03:47:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:25 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.568999+0000 mgr.y (mgr.24368) 972 : cluster 0 pgmap v1055: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 228 op/s 2024-04-06T03:47:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:25.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:25.429 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:25.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:25.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:25.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_blocked_ops 2024-04-06T03:47:25.659 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:25.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:25.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1053,"stamp":"2024-04-06T03:47:20.567253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":3644,"num_read_kb":3458,"num_write":213323,"num_write_kb":6784606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":2958161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78106,"ondisk_log_size":78106,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4027128,"kb_used_data":7596,"kb_used_omap":10,"kb_used_meta":4019317,"kb_avail":652188936,"statfs":{"total":671965249536,"available":667841470464,"internally_reserved":0,"allocated":7778304,"data_stored":5325561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":4115780715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-279117848,"num_objects":-4260,"num_object_clones":0,"num_object_copies":-12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4260,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004327"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":707,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.048477+0000","last_change":"2024-04-06T03:43:33.261450+0000","last_active":"2024-04-06T03:47:08.048477+0000","last_peered":"2024-04-06T03:47:08.048477+0000","last_clean":"2024-04-06T03:47:08.048477+0000","last_became_active":"2024-04-06T03:42:23.588459+0000","last_became_peered":"2024-04-06T03:42:23.588459+0000","last_unstale":"2024-04-06T03:47:08.048477+0000","last_undegraded":"2024-04-06T03:47:08.048477+0000","last_fullsized":"2024-04-06T03:47:08.048477+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_clean_scrub_stamp":"2024-04-06T03:43:33.261299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":719,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.046857+0000","last_change":"2024-04-06T03:45:17.323934+0000","last_active":"2024-04-06T03:47:20.046857+0000","last_peered":"2024-04-06T03:47:20.046857+0000","last_clean":"2024-04-06T03:47:20.046857+0000","last_became_active":"2024-04-06T03:42:57.110401+0000","last_became_peered":"2024-04-06T03:42:57.110401+0000","last_unstale":"2024-04-06T03:47:20.046857+0000","last_undegraded":"2024-04-06T03:47:20.046857+0000","last_fullsized":"2024-04-06T03:47:20.046857+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:17.323812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:28.764134+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.323812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":735,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051454+0000","last_change":"2024-04-06T03:45:21.991669+0000","last_active":"2024-04-06T03:47:08.051454+0000","last_peered":"2024-04-06T03:47:08.051454+0000","last_clean":"2024-04-06T03:47:08.051454+0000","last_became_active":"2024-04-06T03:29:57.153232+0000","last_became_peered":"2024-04-06T03:29:57.153232+0000","last_unstale":"2024-04-06T03:47:08.051454+0000","last_undegraded":"2024-04-06T03:47:08.051454+0000","last_fullsized":"2024-04-06T03:47:08.051454+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_clean_scrub_stamp":"2024-04-06T03:45:21.990975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"307'42","reported_seq":809,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050473+0000","last_change":"2024-04-06T03:44:58.409103+0000","last_active":"2024-04-06T03:47:08.050473+0000","last_peered":"2024-04-06T03:47:08.050473+0000","last_clean":"2024-04-06T03:47:08.050473+0000","last_became_active":"2024-04-06T03:42:23.588857+0000","last_became_peered":"2024-04-06T03:42:23.588857+0000","last_unstale":"2024-04-06T03:47:08.050473+0000","last_undegraded":"2024-04-06T03:47:08.050473+0000","last_fullsized":"2024-04-06T03:47:08.050473+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'42","last_scrub_stamp":"2024-04-06T03:44:58.408958+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-06T03:24:29.974899+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.408958+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":711,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047541+0000","last_change":"2024-04-06T03:45:38.544714+0000","last_active":"2024-04-06T03:47:20.047541+0000","last_peered":"2024-04-06T03:47:20.047541+0000","last_clean":"2024-04-06T03:47:20.047541+0000","last_became_active":"2024-04-06T03:42:57.091902+0000","last_became_peered":"2024-04-06T03:42:57.091902+0000","last_unstale":"2024-04-06T03:47:20.047541+0000","last_undegraded":"2024-04-06T03:47:20.047541+0000","last_fullsized":"2024-04-06T03:47:20.047541+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:38.544490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:38.544490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":592,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053422+0000","last_change":"2024-04-06T03:45:37.144389+0000","last_active":"2024-04-06T03:47:08.053422+0000","last_peered":"2024-04-06T03:47:08.053422+0000","last_clean":"2024-04-06T03:47:08.053422+0000","last_became_active":"2024-04-06T03:42:57.100816+0000","last_became_peered":"2024-04-06T03:42:57.100816+0000","last_unstale":"2024-04-06T03:47:08.053422+0000","last_undegraded":"2024-04-06T03:47:08.053422+0000","last_fullsized":"2024-04-06T03:47:08.053422+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:37.144305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:37.144305+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.144305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050245+0000","last_change":"2024-04-06T03:44:57.401411+0000","last_active":"2024-04-06T03:47:08.050245+0000","last_peered":"2024-04-06T03:47:08.050245+0000","last_clean":"2024-04-06T03:47:08.050245+0000","last_became_active":"2024-04-06T03:20:51.146608+0000","last_became_peered":"2024-04-06T03:20:51.146608+0000","last_unstale":"2024-04-06T03:47:08.050245+0000","last_undegraded":"2024-04-06T03:47:08.050245+0000","last_fullsized":"2024-04-06T03:47:08.050245+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:57.401286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:52.627780+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.401286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"307'39","reported_seq":780,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051170+0000","last_change":"2024-04-06T03:46:50.085063+0000","last_active":"2024-04-06T03:47:08.051170+0000","last_peered":"2024-04-06T03:47:08.051170+0000","last_clean":"2024-04-06T03:47:08.051170+0000","last_became_active":"2024-04-06T03:20:53.347060+0000","last_became_peered":"2024-04-06T03:20:53.347060+0000","last_unstale":"2024-04-06T03:47:08.051170+0000","last_undegraded":"2024-04-06T03:47:08.051170+0000","last_fullsized":"2024-04-06T03:47:08.051170+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:46:50.084593+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:46:50.084593+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:15.546746+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":111,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.054615+0000","last_change":"2024-04-06T03:45:08.003956+0000","last_active":"2024-04-06T03:47:08.054615+0000","last_peered":"2024-04-06T03:47:08.054615+0000","last_clean":"2024-04-06T03:47:08.054615+0000","last_became_active":"2024-04-06T03:42:57.078887+0000","last_became_peered":"2024-04-06T03:42:57.078887+0000","last_unstale":"2024-04-06T03:47:08.054615+0000","last_undegraded":"2024-04-06T03:47:08.054615+0000","last_fullsized":"2024-04-06T03:47:08.054615+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:08.003828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:45:08.003828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":703,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.052797+0000","last_change":"2024-04-06T03:45:25.274860+0000","last_active":"2024-04-06T03:47:08.052797+0000","last_peered":"2024-04-06T03:47:08.052797+0000","last_clean":"2024-04-06T03:47:08.052797+0000","last_became_active":"2024-04-06T03:42:57.112847+0000","last_became_peered":"2024-04-06T03:42:57.112847+0000","last_unstale":"2024-04-06T03:47:08.052797+0000","last_undegraded":"2024-04-06T03:47:08.052797+0000","last_fullsized":"2024-04-06T03:47:08.052797+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_clean_scrub_stamp":"2024-04-06T03:45:25.274778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":741,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053672+0000","last_change":"2024-04-06T03:45:04.452779+0000","last_active":"2024-04-06T03:47:08.053672+0000","last_peered":"2024-04-06T03:47:08.053672+0000","last_clean":"2024-04-06T03:47:08.053672+0000","last_became_active":"2024-04-06T03:42:23.570089+0000","last_became_peered":"2024-04-06T03:42:23.570089+0000","last_unstale":"2024-04-06T03:47:08.053672+0000","last_undegraded":"2024-04-06T03:47:08.053672+0000","last_fullsized":"2024-04-06T03:47:08.053672+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:45:04.452660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.452660+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"307'46","reported_seq":745,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047792+0000","last_change":"2024-04-06T03:44:57.339512+0000","last_active":"2024-04-06T03:47:20.047792+0000","last_peered":"2024-04-06T03:47:20.047792+0000","last_clean":"2024-04-06T03:47:20.047792+0000","last_became_active":"2024-04-06T03:42:57.110510+0000","last_became_peered":"2024-04-06T03:42:57.110510+0000","last_unstale":"2024-04-06T03:47:20.047792+0000","last_undegraded":"2024-04-06T03:47:20.047792+0000","last_fullsized":"2024-04-06T03:47:20.047792+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'46","last_scrub_stamp":"2024-04-06T03:44:57.339353+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:44:57.339353+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":707,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.047536+0000","last_change":"2024-04-06T03:45:05.101359+0000","last_active":"2024-04-06T03:47:08.047536+0000","last_peered":"2024-04-06T03:47:08.047536+0000","last_clean":"2024-04-06T03:47:08.047536+0000","last_became_active":"2024-04-06T03:42:57.092354+0000","last_became_peered":"2024-04-06T03:42:57.092354+0000","last_unstale":"2024-04-06T03:47:08.047536+0000","last_undegraded":"2024-04-06T03:47:08.047536+0000","last_fullsized":"2024-04-06T03:47:08.047536+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:05.101251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:05.101251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":582,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049766+0000","last_change":"2024-04-06T03:45:02.133616+0000","last_active":"2024-04-06T03:47:08.049766+0000","last_peered":"2024-04-06T03:47:08.049766+0000","last_clean":"2024-04-06T03:47:08.049766+0000","last_became_active":"2024-04-06T03:25:54.496246+0000","last_became_peered":"2024-04-06T03:25:54.496246+0000","last_unstale":"2024-04-06T03:47:08.049766+0000","last_undegraded":"2024-04-06T03:47:08.049766+0000","last_fullsized":"2024-04-06T03:47:08.049766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:02.132904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.132904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.054489+0000","last_change":"2024-04-06T03:46:49.067449+0000","last_active":"2024-04-06T03:47:08.054489+0000","last_peered":"2024-04-06T03:47:08.054489+0000","last_clean":"2024-04-06T03:47:08.054489+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:47:08.054489+0000","last_undegraded":"2024-04-06T03:47:08.054489+0000","last_fullsized":"2024-04-06T03:47:08.054489+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:49.067339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:46:49.067339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:11.262399+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"307'36","reported_seq":732,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.050078+0000","last_change":"2024-04-06T03:45:18.356128+0000","last_active":"2024-04-06T03:47:20.050078+0000","last_peered":"2024-04-06T03:47:20.050078+0000","last_clean":"2024-04-06T03:47:20.050078+0000","last_became_active":"2024-04-06T03:42:23.592623+0000","last_became_peered":"2024-04-06T03:42:23.592623+0000","last_unstale":"2024-04-06T03:47:20.050078+0000","last_undegraded":"2024-04-06T03:47:20.050078+0000","last_fullsized":"2024-04-06T03:47:20.050078+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'36","last_scrub_stamp":"2024-04-06T03:45:18.356021+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:45:18.356021+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":718,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049931+0000","last_change":"2024-04-06T03:46:53.453513+0000","last_active":"2024-04-06T03:47:08.049931+0000","last_peered":"2024-04-06T03:47:08.049931+0000","last_clean":"2024-04-06T03:47:08.049931+0000","last_became_active":"2024-04-06T03:42:23.577258+0000","last_became_peered":"2024-04-06T03:42:23.577258+0000","last_unstale":"2024-04-06T03:47:08.049931+0000","last_undegraded":"2024-04-06T03:47:08.049931+0000","last_fullsized":"2024-04-06T03:47:08.049931+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:53.453158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:46:53.453158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:17.061461+0000","scrub_duration":355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":717,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048229+0000","last_change":"2024-04-06T03:47:05.527153+0000","last_active":"2024-04-06T03:47:20.048229+0000","last_peered":"2024-04-06T03:47:20.048229+0000","last_clean":"2024-04-06T03:47:20.048229+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:47:20.048229+0000","last_undegraded":"2024-04-06T03:47:20.048229+0000","last_fullsized":"2024-04-06T03:47:20.048229+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:05.526885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:26:58.762469+0000","last_clean_scrub_stamp":"2024-04-06T03:47:05.526885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:32.606758+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049886+0000","last_change":"2024-04-06T03:45:01.585396+0000","last_active":"2024-04-06T03:47:08.049886+0000","last_peered":"2024-04-06T03:47:08.049886+0000","last_clean":"2024-04-06T03:47:08.049886+0000","last_became_active":"2024-04-06T03:42:23.588672+0000","last_became_peered":"2024-04-06T03:42:23.588672+0000","last_unstale":"2024-04-06T03:47:08.049886+0000","last_undegraded":"2024-04-06T03:47:08.049886+0000","last_fullsized":"2024-04-06T03:47:08.049886+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.585125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.585125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"307'39","reported_seq":797,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048152+0000","last_change":"2024-04-06T03:45:04.428891+0000","last_active":"2024-04-06T03:47:20.048152+0000","last_peered":"2024-04-06T03:47:20.048152+0000","last_clean":"2024-04-06T03:47:20.048152+0000","last_became_active":"2024-04-06T03:42:57.096631+0000","last_became_peered":"2024-04-06T03:42:57.096631+0000","last_unstale":"2024-04-06T03:47:20.048152+0000","last_undegraded":"2024-04-06T03:47:20.048152+0000","last_fullsized":"2024-04-06T03:47:20.048152+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:45:04.428726+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.428726+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":712,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049562+0000","last_change":"2024-04-06T03:43:35.230118+0000","last_active":"2024-04-06T03:47:20.049562+0000","last_peered":"2024-04-06T03:47:20.049562+0000","last_clean":"2024-04-06T03:47:20.049562+0000","last_became_active":"2024-04-06T03:42:23.594112+0000","last_became_peered":"2024-04-06T03:42:23.594112+0000","last_unstale":"2024-04-06T03:47:20.049562+0000","last_undegraded":"2024-04-06T03:47:20.049562+0000","last_fullsized":"2024-04-06T03:47:20.049562+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:35.230016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:07.110775+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.230016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":110,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.050260+0000","last_change":"2024-04-06T03:43:36.185334+0000","last_active":"2024-04-06T03:47:20.050260+0000","last_peered":"2024-04-06T03:47:20.050260+0000","last_clean":"2024-04-06T03:47:20.050260+0000","last_became_active":"2024-04-06T03:42:57.103063+0000","last_became_peered":"2024-04-06T03:42:57.103063+0000","last_unstale":"2024-04-06T03:47:20.050260+0000","last_undegraded":"2024-04-06T03:47:20.050260+0000","last_fullsized":"2024-04-06T03:47:20.050260+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:36.185042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:43:36.185042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":582,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048440+0000","last_change":"2024-04-06T03:45:26.399149+0000","last_active":"2024-04-06T03:47:20.048440+0000","last_peered":"2024-04-06T03:47:20.048440+0000","last_clean":"2024-04-06T03:47:20.048440+0000","last_became_active":"2024-04-06T03:42:23.593721+0000","last_became_peered":"2024-04-06T03:42:23.593721+0000","last_unstale":"2024-04-06T03:47:20.048440+0000","last_undegraded":"2024-04-06T03:47:20.048440+0000","last_fullsized":"2024-04-06T03:47:20.048440+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:26.399024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:26.399024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"307'18","reported_seq":748,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049962+0000","last_change":"2024-04-06T03:45:37.208824+0000","last_active":"2024-04-06T03:47:08.049962+0000","last_peered":"2024-04-06T03:47:08.049962+0000","last_clean":"2024-04-06T03:47:08.049962+0000","last_became_active":"2024-04-06T03:20:53.335032+0000","last_became_peered":"2024-04-06T03:20:53.335032+0000","last_unstale":"2024-04-06T03:47:08.049962+0000","last_undegraded":"2024-04-06T03:47:08.049962+0000","last_fullsized":"2024-04-06T03:47:08.049962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'18","last_scrub_stamp":"2024-04-06T03:45:37.208678+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.208678+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":585,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048501+0000","last_change":"2024-04-06T03:43:44.166907+0000","last_active":"2024-04-06T03:47:20.048501+0000","last_peered":"2024-04-06T03:47:20.048501+0000","last_clean":"2024-04-06T03:47:20.048501+0000","last_became_active":"2024-04-06T03:42:57.096442+0000","last_became_peered":"2024-04-06T03:42:57.096442+0000","last_unstale":"2024-04-06T03:47:20.048501+0000","last_undegraded":"2024-04-06T03:47:20.048501+0000","last_fullsized":"2024-04-06T03:47:20.048501+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:44.166672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:44.166672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":716,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049428+0000","last_change":"2024-04-06T03:47:00.011223+0000","last_active":"2024-04-06T03:47:08.049428+0000","last_peered":"2024-04-06T03:47:08.049428+0000","last_clean":"2024-04-06T03:47:08.049428+0000","last_became_active":"2024-04-06T03:20:55.765480+0000","last_became_peered":"2024-04-06T03:20:55.765480+0000","last_unstale":"2024-04-06T03:47:08.049428+0000","last_undegraded":"2024-04-06T03:47:08.049428+0000","last_fullsized":"2024-04-06T03:47:08.049428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:00.011082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_clean_scrub_stamp":"2024-04-06T03:47:00.011082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.388770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"78'1","reported_seq":720,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.054895+0000","last_change":"2024-04-06T03:43:28.171388+0000","last_active":"2024-04-06T03:47:08.054895+0000","last_peered":"2024-04-06T03:47:08.054895+0000","last_clean":"2024-04-06T03:47:08.054895+0000","last_became_active":"2024-04-06T03:42:23.577120+0000","last_became_peered":"2024-04-06T03:42:23.577120+0000","last_unstale":"2024-04-06T03:47:08.054895+0000","last_undegraded":"2024-04-06T03:47:08.054895+0000","last_fullsized":"2024-04-06T03:47:08.054895+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:43:28.171183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:28.171183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"307'32","reported_seq":764,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051078+0000","last_change":"2024-04-06T03:45:17.921718+0000","last_active":"2024-04-06T03:47:08.051078+0000","last_peered":"2024-04-06T03:47:08.051078+0000","last_clean":"2024-04-06T03:47:08.051078+0000","last_became_active":"2024-04-06T03:42:57.070724+0000","last_became_peered":"2024-04-06T03:42:57.070724+0000","last_unstale":"2024-04-06T03:47:08.051078+0000","last_undegraded":"2024-04-06T03:47:08.051078+0000","last_fullsized":"2024-04-06T03:47:08.051078+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'32","last_scrub_stamp":"2024-04-06T03:45:17.921596+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.921596+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":696,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.054726+0000","last_change":"2024-04-06T03:45:11.392248+0000","last_active":"2024-04-06T03:47:08.054726+0000","last_peered":"2024-04-06T03:47:08.054726+0000","last_clean":"2024-04-06T03:47:08.054726+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:47:08.054726+0000","last_undegraded":"2024-04-06T03:47:08.054726+0000","last_fullsized":"2024-04-06T03:47:08.054726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:45:11.392164+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:41:29.127643+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.392164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"327'15","reported_seq":929,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.050307+0000","last_change":"2024-04-06T03:45:36.499577+0000","last_active":"2024-04-06T03:47:20.050307+0000","last_peered":"2024-04-06T03:47:20.050307+0000","last_clean":"2024-04-06T03:47:20.050307+0000","last_became_active":"2024-04-06T03:42:23.594397+0000","last_became_peered":"2024-04-06T03:42:23.594397+0000","last_unstale":"2024-04-06T03:47:20.050307+0000","last_undegraded":"2024-04-06T03:47:20.050307+0000","last_fullsized":"2024-04-06T03:47:20.050307+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'15","last_scrub_stamp":"2024-04-06T03:45:36.499448+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-06T03:22:05.612834+0000","last_clean_scrub_stamp":"2024-04-06T03:45:36.499448+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":584,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047064+0000","last_change":"2024-04-06T03:45:31.513869+0000","last_active":"2024-04-06T03:47:20.047064+0000","last_peered":"2024-04-06T03:47:20.047064+0000","last_clean":"2024-04-06T03:47:20.047064+0000","last_became_active":"2024-04-06T03:25:54.492631+0000","last_became_peered":"2024-04-06T03:25:54.492631+0000","last_unstale":"2024-04-06T03:47:20.047064+0000","last_undegraded":"2024-04-06T03:47:20.047064+0000","last_fullsized":"2024-04-06T03:47:20.047064+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:31.513718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:46.961637+0000","last_clean_scrub_stamp":"2024-04-06T03:45:31.513718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"307'74","reported_seq":808,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055242+0000","last_change":"2024-04-06T03:45:13.928026+0000","last_active":"2024-04-06T03:47:08.055242+0000","last_peered":"2024-04-06T03:47:08.055242+0000","last_clean":"2024-04-06T03:47:08.055242+0000","last_became_active":"2024-04-06T03:42:57.093866+0000","last_became_peered":"2024-04-06T03:42:57.093866+0000","last_unstale":"2024-04-06T03:47:08.055242+0000","last_undegraded":"2024-04-06T03:47:08.055242+0000","last_fullsized":"2024-04-06T03:47:08.055242+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'74","last_scrub_stamp":"2024-04-06T03:45:13.927878+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:45:13.927878+0000","objects_scrubbed":11,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":97,"num_read_kb":84,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":700,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050256+0000","last_change":"2024-04-06T03:43:34.209777+0000","last_active":"2024-04-06T03:47:08.050256+0000","last_peered":"2024-04-06T03:47:08.050256+0000","last_clean":"2024-04-06T03:47:08.050256+0000","last_became_active":"2024-04-06T03:42:23.588998+0000","last_became_peered":"2024-04-06T03:42:23.588998+0000","last_unstale":"2024-04-06T03:47:08.050256+0000","last_undegraded":"2024-04-06T03:47:08.050256+0000","last_fullsized":"2024-04-06T03:47:08.050256+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:34.209617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:43:34.209617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":718,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055718+0000","last_change":"2024-04-06T03:46:54.004908+0000","last_active":"2024-04-06T03:47:08.055718+0000","last_peered":"2024-04-06T03:47:08.055718+0000","last_clean":"2024-04-06T03:47:08.055718+0000","last_became_active":"2024-04-06T03:42:23.577555+0000","last_became_peered":"2024-04-06T03:42:23.577555+0000","last_unstale":"2024-04-06T03:47:08.055718+0000","last_undegraded":"2024-04-06T03:47:08.055718+0000","last_fullsized":"2024-04-06T03:47:08.055718+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:54.004800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:46:54.004800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:13.380811+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":606,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044592+0000","last_change":"2024-04-06T03:46:51.174954+0000","last_active":"2024-04-06T03:47:20.044592+0000","last_peered":"2024-04-06T03:47:20.044592+0000","last_clean":"2024-04-06T03:47:20.044592+0000","last_became_active":"2024-04-06T03:42:23.582180+0000","last_became_peered":"2024-04-06T03:42:23.582180+0000","last_unstale":"2024-04-06T03:47:20.044592+0000","last_undegraded":"2024-04-06T03:47:20.044592+0000","last_fullsized":"2024-04-06T03:47:20.044592+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:51.174158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:26.863691+0000","last_clean_scrub_stamp":"2024-04-06T03:46:51.174158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:13.918372+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"344'39","reported_seq":685,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:10.039124+0000","last_change":"2024-04-06T03:47:10.039178+0000","last_active":"2024-04-06T03:47:10.039124+0000","last_peered":"2024-04-06T03:47:10.039124+0000","last_clean":"2024-04-06T03:47:10.039124+0000","last_became_active":"2024-04-06T03:42:57.093955+0000","last_became_peered":"2024-04-06T03:42:57.093955+0000","last_unstale":"2024-04-06T03:47:10.039124+0000","last_undegraded":"2024-04-06T03:47:10.039124+0000","last_fullsized":"2024-04-06T03:47:10.039124+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'39","last_scrub_stamp":"2024-04-06T03:47:10.039032+0000","last_deep_scrub":"344'39","last_deep_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_clean_scrub_stamp":"2024-04-06T03:47:10.039032+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:30.542161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":44,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":702,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.052931+0000","last_change":"2024-04-06T03:46:47.027083+0000","last_active":"2024-04-06T03:47:08.052931+0000","last_peered":"2024-04-06T03:47:08.052931+0000","last_clean":"2024-04-06T03:47:08.052931+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:47:08.052931+0000","last_undegraded":"2024-04-06T03:47:08.052931+0000","last_fullsized":"2024-04-06T03:47:08.052931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:47.026665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:42.005342+0000","last_clean_scrub_stamp":"2024-04-06T03:46:47.026665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:00.658796+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":584,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047102+0000","last_change":"2024-04-06T03:45:28.454818+0000","last_active":"2024-04-06T03:47:20.047102+0000","last_peered":"2024-04-06T03:47:20.047102+0000","last_clean":"2024-04-06T03:47:20.047102+0000","last_became_active":"2024-04-06T03:42:23.580807+0000","last_became_peered":"2024-04-06T03:42:23.580807+0000","last_unstale":"2024-04-06T03:47:20.047102+0000","last_undegraded":"2024-04-06T03:47:20.047102+0000","last_fullsized":"2024-04-06T03:47:20.047102+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:28.454731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:28.454731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":609,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044814+0000","last_change":"2024-04-06T03:46:47.217038+0000","last_active":"2024-04-06T03:47:20.044814+0000","last_peered":"2024-04-06T03:47:20.044814+0000","last_clean":"2024-04-06T03:47:20.044814+0000","last_became_active":"2024-04-06T03:42:23.578658+0000","last_became_peered":"2024-04-06T03:42:23.578658+0000","last_unstale":"2024-04-06T03:47:20.044814+0000","last_undegraded":"2024-04-06T03:47:20.044814+0000","last_fullsized":"2024-04-06T03:47:20.044814+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:47.216929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:46:47.216929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:49.286842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"307'28","reported_seq":776,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051069+0000","last_change":"2024-04-06T03:46:44.002540+0000","last_active":"2024-04-06T03:47:08.051069+0000","last_peered":"2024-04-06T03:47:08.051069+0000","last_clean":"2024-04-06T03:47:08.051069+0000","last_became_active":"2024-04-06T03:20:53.346548+0000","last_became_peered":"2024-04-06T03:20:53.346548+0000","last_unstale":"2024-04-06T03:47:08.051069+0000","last_undegraded":"2024-04-06T03:47:08.051069+0000","last_fullsized":"2024-04-06T03:47:08.051069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'28","last_scrub_stamp":"2024-04-06T03:46:44.002421+0000","last_deep_scrub":"121'16","last_deep_scrub_stamp":"2024-04-06T03:24:46.160706+0000","last_clean_scrub_stamp":"2024-04-06T03:46:44.002421+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:04.666967+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":32,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":688,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.054712+0000","last_change":"2024-04-06T03:43:29.215763+0000","last_active":"2024-04-06T03:47:08.054712+0000","last_peered":"2024-04-06T03:47:08.054712+0000","last_clean":"2024-04-06T03:47:08.054712+0000","last_became_active":"2024-04-06T03:20:57.826734+0000","last_became_peered":"2024-04-06T03:20:57.826734+0000","last_unstale":"2024-04-06T03:47:08.054712+0000","last_undegraded":"2024-04-06T03:47:08.054712+0000","last_fullsized":"2024-04-06T03:47:08.054712+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:29.215509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:43:29.215509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":108,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051226+0000","last_change":"2024-04-06T03:45:15.944386+0000","last_active":"2024-04-06T03:47:08.051226+0000","last_peered":"2024-04-06T03:47:08.051226+0000","last_clean":"2024-04-06T03:47:08.051226+0000","last_became_active":"2024-04-06T03:42:57.085102+0000","last_became_peered":"2024-04-06T03:42:57.085102+0000","last_unstale":"2024-04-06T03:47:08.051226+0000","last_undegraded":"2024-04-06T03:47:08.051226+0000","last_fullsized":"2024-04-06T03:47:08.051226+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:15.944262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:15.944262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":115,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049608+0000","last_change":"2024-04-06T03:46:56.048292+0000","last_active":"2024-04-06T03:47:08.049608+0000","last_peered":"2024-04-06T03:47:08.049608+0000","last_clean":"2024-04-06T03:47:08.049608+0000","last_became_active":"2024-04-06T03:42:57.091715+0000","last_became_peered":"2024-04-06T03:42:57.091715+0000","last_unstale":"2024-04-06T03:47:08.049608+0000","last_undegraded":"2024-04-06T03:47:08.049608+0000","last_fullsized":"2024-04-06T03:47:08.049608+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:56.048194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:46:56.048194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:17.598971+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"307'43","reported_seq":836,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047936+0000","last_change":"2024-04-06T03:47:07.356358+0000","last_active":"2024-04-06T03:47:20.047936+0000","last_peered":"2024-04-06T03:47:20.047936+0000","last_clean":"2024-04-06T03:47:20.047936+0000","last_became_active":"2024-04-06T03:42:23.592079+0000","last_became_peered":"2024-04-06T03:42:23.592079+0000","last_unstale":"2024-04-06T03:47:20.047936+0000","last_undegraded":"2024-04-06T03:47:20.047936+0000","last_fullsized":"2024-04-06T03:47:20.047936+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'43","last_scrub_stamp":"2024-04-06T03:47:07.356251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:47:07.356251+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:21.058327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":716,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:15.099673+0000","last_change":"2024-04-06T03:47:15.099716+0000","last_active":"2024-04-06T03:47:15.099673+0000","last_peered":"2024-04-06T03:47:15.099673+0000","last_clean":"2024-04-06T03:47:15.099673+0000","last_became_active":"2024-04-06T03:42:57.089541+0000","last_became_peered":"2024-04-06T03:42:57.089541+0000","last_unstale":"2024-04-06T03:47:15.099673+0000","last_undegraded":"2024-04-06T03:47:15.099673+0000","last_fullsized":"2024-04-06T03:47:15.099673+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:15.099541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_clean_scrub_stamp":"2024-04-06T03:47:15.099541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:29.143899+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":698,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050094+0000","last_change":"2024-04-06T03:43:26.163518+0000","last_active":"2024-04-06T03:47:08.050094+0000","last_peered":"2024-04-06T03:47:08.050094+0000","last_clean":"2024-04-06T03:47:08.050094+0000","last_became_active":"2024-04-06T03:42:57.079386+0000","last_became_peered":"2024-04-06T03:42:57.079386+0000","last_unstale":"2024-04-06T03:47:08.050094+0000","last_undegraded":"2024-04-06T03:47:08.050094+0000","last_fullsized":"2024-04-06T03:47:08.050094+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:26.163320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.163320+0000","last_clean_scrub_stamp":"2024-04-06T03:43:26.163320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":572,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.048666+0000","last_change":"2024-04-06T03:43:41.985445+0000","last_active":"2024-04-06T03:47:08.048666+0000","last_peered":"2024-04-06T03:47:08.048666+0000","last_clean":"2024-04-06T03:47:08.048666+0000","last_became_active":"2024-04-06T03:42:23.589224+0000","last_became_peered":"2024-04-06T03:42:23.589224+0000","last_unstale":"2024-04-06T03:47:08.048666+0000","last_undegraded":"2024-04-06T03:47:08.048666+0000","last_fullsized":"2024-04-06T03:47:08.048666+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:41.985333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:08.589840+0000","last_clean_scrub_stamp":"2024-04-06T03:43:41.985333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"307'27","reported_seq":747,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053871+0000","last_change":"2024-04-06T03:43:38.248493+0000","last_active":"2024-04-06T03:47:08.053871+0000","last_peered":"2024-04-06T03:47:08.053871+0000","last_clean":"2024-04-06T03:47:08.053871+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:47:08.053871+0000","last_undegraded":"2024-04-06T03:47:08.053871+0000","last_fullsized":"2024-04-06T03:47:08.053871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:43:38.248258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:38.248258+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":705,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.050568+0000","last_change":"2024-04-06T03:44:58.475184+0000","last_active":"2024-04-06T03:47:20.050568+0000","last_peered":"2024-04-06T03:47:20.050568+0000","last_clean":"2024-04-06T03:47:20.050568+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:47:20.050568+0000","last_undegraded":"2024-04-06T03:47:20.050568+0000","last_fullsized":"2024-04-06T03:47:20.050568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:58.475083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.475083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":1029,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:12.544850+0000","last_change":"2024-04-06T03:45:41.030902+0000","last_active":"2024-04-06T03:47:12.544850+0000","last_peered":"2024-04-06T03:47:12.544850+0000","last_clean":"2024-04-06T03:47:12.544850+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:47:12.544850+0000","last_undegraded":"2024-04-06T03:47:12.544850+0000","last_fullsized":"2024-04-06T03:47:12.544850+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:45:41.030592+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:22:59.658761+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.030592+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050116+0000","last_change":"2024-04-06T03:45:34.135652+0000","last_active":"2024-04-06T03:47:08.050116+0000","last_peered":"2024-04-06T03:47:08.050116+0000","last_clean":"2024-04-06T03:47:08.050116+0000","last_became_active":"2024-04-06T03:20:51.164107+0000","last_became_peered":"2024-04-06T03:20:51.164107+0000","last_unstale":"2024-04-06T03:47:08.050116+0000","last_undegraded":"2024-04-06T03:47:08.050116+0000","last_fullsized":"2024-04-06T03:47:08.050116+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.135479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.135479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"381'25","reported_seq":756,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.989001+0000","last_change":"2024-04-06T03:47:13.989179+0000","last_active":"2024-04-06T03:47:13.989001+0000","last_peered":"2024-04-06T03:47:13.989001+0000","last_clean":"2024-04-06T03:47:13.989001+0000","last_became_active":"2024-04-06T03:20:53.334782+0000","last_became_peered":"2024-04-06T03:20:53.334782+0000","last_unstale":"2024-04-06T03:47:13.989001+0000","last_undegraded":"2024-04-06T03:47:13.989001+0000","last_fullsized":"2024-04-06T03:47:13.989001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'25","last_scrub_stamp":"2024-04-06T03:47:13.988764+0000","last_deep_scrub":"381'25","last_deep_scrub_stamp":"2024-04-06T03:47:13.988764+0000","last_clean_scrub_stamp":"2024-04-06T03:47:13.988764+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:30.619958+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":695,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049616+0000","last_change":"2024-04-06T03:46:53.122510+0000","last_active":"2024-04-06T03:47:08.049616+0000","last_peered":"2024-04-06T03:47:08.049616+0000","last_clean":"2024-04-06T03:47:08.049616+0000","last_became_active":"2024-04-06T03:29:18.566539+0000","last_became_peered":"2024-04-06T03:29:18.566539+0000","last_unstale":"2024-04-06T03:47:08.049616+0000","last_undegraded":"2024-04-06T03:47:08.049616+0000","last_fullsized":"2024-04-06T03:47:08.049616+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:53.122063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:46:53.122063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:17.731751+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"338'15","reported_seq":826,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:12.436753+0000","last_change":"2024-04-06T03:46:44.991251+0000","last_active":"2024-04-06T03:47:12.436753+0000","last_peered":"2024-04-06T03:47:12.436753+0000","last_clean":"2024-04-06T03:47:12.436753+0000","last_became_active":"2024-04-06T03:42:57.087361+0000","last_became_peered":"2024-04-06T03:42:57.087361+0000","last_unstale":"2024-04-06T03:47:12.436753+0000","last_undegraded":"2024-04-06T03:47:12.436753+0000","last_fullsized":"2024-04-06T03:47:12.436753+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'15","last_scrub_stamp":"2024-04-06T03:46:44.991131+0000","last_deep_scrub":"309'14","last_deep_scrub_stamp":"2024-04-06T03:41:21.025392+0000","last_clean_scrub_stamp":"2024-04-06T03:46:44.991131+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:57.162775+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":573,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051142+0000","last_change":"2024-04-06T03:45:35.931813+0000","last_active":"2024-04-06T03:47:08.051142+0000","last_peered":"2024-04-06T03:47:08.051142+0000","last_clean":"2024-04-06T03:47:08.051142+0000","last_became_active":"2024-04-06T03:30:49.560819+0000","last_became_peered":"2024-04-06T03:30:49.560819+0000","last_unstale":"2024-04-06T03:47:08.051142+0000","last_undegraded":"2024-04-06T03:47:08.051142+0000","last_fullsized":"2024-04-06T03:47:08.051142+0000","mapping_epoch":196,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:35.931720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:21.406614+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.931720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"307'32","reported_seq":758,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053303+0000","last_change":"2024-04-06T03:46:59.394155+0000","last_active":"2024-04-06T03:47:08.053303+0000","last_peered":"2024-04-06T03:47:08.053303+0000","last_clean":"2024-04-06T03:47:08.053303+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:47:08.053303+0000","last_undegraded":"2024-04-06T03:47:08.053303+0000","last_fullsized":"2024-04-06T03:47:08.053303+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'32","last_scrub_stamp":"2024-04-06T03:46:59.393878+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:46:59.393878+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:15.525294+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053185+0000","last_change":"2024-04-06T03:44:59.520843+0000","last_active":"2024-04-06T03:47:08.053185+0000","last_peered":"2024-04-06T03:47:08.053185+0000","last_clean":"2024-04-06T03:47:08.053185+0000","last_became_active":"2024-04-06T03:42:23.574848+0000","last_became_peered":"2024-04-06T03:42:23.574848+0000","last_unstale":"2024-04-06T03:47:08.053185+0000","last_undegraded":"2024-04-06T03:47:08.053185+0000","last_fullsized":"2024-04-06T03:47:08.053185+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:44:59.520731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:36.947232+0000","last_clean_scrub_stamp":"2024-04-06T03:44:59.520731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":572,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050157+0000","last_change":"2024-04-06T03:45:11.925553+0000","last_active":"2024-04-06T03:47:08.050157+0000","last_peered":"2024-04-06T03:47:08.050157+0000","last_clean":"2024-04-06T03:47:08.050157+0000","last_became_active":"2024-04-06T03:42:57.092555+0000","last_became_peered":"2024-04-06T03:42:57.092555+0000","last_unstale":"2024-04-06T03:47:08.050157+0000","last_undegraded":"2024-04-06T03:47:08.050157+0000","last_fullsized":"2024-04-06T03:47:08.050157+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:11.925422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.925422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":589,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048010+0000","last_change":"2024-04-06T03:45:29.469379+0000","last_active":"2024-04-06T03:47:20.048010+0000","last_peered":"2024-04-06T03:47:20.048010+0000","last_clean":"2024-04-06T03:47:20.048010+0000","last_became_active":"2024-04-06T03:42:57.109758+0000","last_became_peered":"2024-04-06T03:42:57.109758+0000","last_unstale":"2024-04-06T03:47:20.048010+0000","last_undegraded":"2024-04-06T03:47:20.048010+0000","last_fullsized":"2024-04-06T03:47:20.048010+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.469254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.469254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"307'14","reported_seq":701,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.047710+0000","last_change":"2024-04-06T03:45:35.162259+0000","last_active":"2024-04-06T03:47:08.047710+0000","last_peered":"2024-04-06T03:47:08.047710+0000","last_clean":"2024-04-06T03:47:08.047710+0000","last_became_active":"2024-04-06T03:42:23.581903+0000","last_became_peered":"2024-04-06T03:42:23.581903+0000","last_unstale":"2024-04-06T03:47:08.047710+0000","last_undegraded":"2024-04-06T03:47:08.047710+0000","last_fullsized":"2024-04-06T03:47:08.047710+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'14","last_scrub_stamp":"2024-04-06T03:45:35.162112+0000","last_deep_scrub":"111'6","last_deep_scrub_stamp":"2024-04-06T03:23:21.875237+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.162112+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":732,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044639+0000","last_change":"2024-04-06T03:46:41.301282+0000","last_active":"2024-04-06T03:47:20.044639+0000","last_peered":"2024-04-06T03:47:20.044639+0000","last_clean":"2024-04-06T03:47:20.044639+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:47:20.044639+0000","last_undegraded":"2024-04-06T03:47:20.044639+0000","last_fullsized":"2024-04-06T03:47:20.044639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:46:41.301153+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:46:41.301153+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:55.802820+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049969+0000","last_change":"2024-04-06T03:46:49.107917+0000","last_active":"2024-04-06T03:47:08.049969+0000","last_peered":"2024-04-06T03:47:08.049969+0000","last_clean":"2024-04-06T03:47:08.049969+0000","last_became_active":"2024-04-06T03:42:57.084986+0000","last_became_peered":"2024-04-06T03:42:57.084986+0000","last_unstale":"2024-04-06T03:47:08.049969+0000","last_undegraded":"2024-04-06T03:47:08.049969+0000","last_fullsized":"2024-04-06T03:47:08.049969+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:49.107837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:46:49.107837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:01.195456+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":748,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.043899+0000","last_change":"2024-04-06T03:45:29.043189+0000","last_active":"2024-04-06T03:47:20.043899+0000","last_peered":"2024-04-06T03:47:20.043899+0000","last_clean":"2024-04-06T03:47:20.043899+0000","last_became_active":"2024-04-06T03:42:57.071688+0000","last_became_peered":"2024-04-06T03:42:57.071688+0000","last_unstale":"2024-04-06T03:47:20.043899+0000","last_undegraded":"2024-04-06T03:47:20.043899+0000","last_fullsized":"2024-04-06T03:47:20.043899+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.043086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.043086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"307'39","reported_seq":819,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:17.122105+0000","last_change":"2024-04-06T03:47:17.122193+0000","last_active":"2024-04-06T03:47:17.122105+0000","last_peered":"2024-04-06T03:47:17.122105+0000","last_clean":"2024-04-06T03:47:17.122105+0000","last_became_active":"2024-04-06T03:42:23.574886+0000","last_became_peered":"2024-04-06T03:42:23.574886+0000","last_unstale":"2024-04-06T03:47:17.122105+0000","last_undegraded":"2024-04-06T03:47:17.122105+0000","last_fullsized":"2024-04-06T03:47:17.122105+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:47:17.121935+0000","last_deep_scrub":"111'23","last_deep_scrub_stamp":"2024-04-06T03:30:14.264595+0000","last_clean_scrub_stamp":"2024-04-06T03:47:17.121935+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:46.049771+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"381'13430","reported_seq":13577,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.618801+0000","last_change":"2024-04-06T03:45:41.170863+0000","last_active":"2024-04-06T03:47:13.618801+0000","last_peered":"2024-04-06T03:47:13.618801+0000","last_clean":"2024-04-06T03:47:13.618801+0000","last_became_active":"2024-04-06T03:43:25.018581+0000","last_became_peered":"2024-04-06T03:43:25.018581+0000","last_unstale":"2024-04-06T03:47:13.618801+0000","last_undegraded":"2024-04-06T03:47:13.618801+0000","last_fullsized":"2024-04-06T03:47:13.618801+0000","mapping_epoch":345,"log_start":"370'8900","ondisk_log_start":"370'8900","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'8005","last_scrub_stamp":"2024-04-06T03:45:41.170736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.170736+0000","objects_scrubbed":5473,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13430,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":703,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050172+0000","last_change":"2024-04-06T03:45:30.126413+0000","last_active":"2024-04-06T03:47:08.050172+0000","last_peered":"2024-04-06T03:47:08.050172+0000","last_clean":"2024-04-06T03:47:08.050172+0000","last_became_active":"2024-04-06T03:42:57.093752+0000","last_became_peered":"2024-04-06T03:42:57.093752+0000","last_unstale":"2024-04-06T03:47:08.050172+0000","last_undegraded":"2024-04-06T03:47:08.050172+0000","last_fullsized":"2024-04-06T03:47:08.050172+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.126246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.126246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":736,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049535+0000","last_change":"2024-04-06T03:44:59.372004+0000","last_active":"2024-04-06T03:47:20.049535+0000","last_peered":"2024-04-06T03:47:20.049535+0000","last_clean":"2024-04-06T03:47:20.049535+0000","last_became_active":"2024-04-06T03:20:51.143239+0000","last_became_peered":"2024-04-06T03:20:51.143239+0000","last_unstale":"2024-04-06T03:47:20.049535+0000","last_undegraded":"2024-04-06T03:47:20.049535+0000","last_fullsized":"2024-04-06T03:47:20.049535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:44:59.371923+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:44:59.371923+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"307'39","reported_seq":799,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055225+0000","last_change":"2024-04-06T03:45:38.163802+0000","last_active":"2024-04-06T03:47:08.055225+0000","last_peered":"2024-04-06T03:47:08.055225+0000","last_clean":"2024-04-06T03:47:08.055225+0000","last_became_active":"2024-04-06T03:42:57.113147+0000","last_became_peered":"2024-04-06T03:42:57.113147+0000","last_unstale":"2024-04-06T03:47:08.055225+0000","last_undegraded":"2024-04-06T03:47:08.055225+0000","last_fullsized":"2024-04-06T03:47:08.055225+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'39","last_scrub_stamp":"2024-04-06T03:45:38.163676+0000","last_deep_scrub":"207'31","last_deep_scrub_stamp":"2024-04-06T03:31:56.847237+0000","last_clean_scrub_stamp":"2024-04-06T03:45:38.163676+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":741,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044706+0000","last_change":"2024-04-06T03:46:58.015638+0000","last_active":"2024-04-06T03:47:20.044706+0000","last_peered":"2024-04-06T03:47:20.044706+0000","last_clean":"2024-04-06T03:47:20.044706+0000","last_became_active":"2024-04-06T03:42:57.074804+0000","last_became_peered":"2024-04-06T03:42:57.074804+0000","last_unstale":"2024-04-06T03:47:20.044706+0000","last_undegraded":"2024-04-06T03:47:20.044706+0000","last_fullsized":"2024-04-06T03:47:20.044706+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:58.015512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:46:58.015512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:16.748002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"381'13472","reported_seq":13638,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044785+0000","last_change":"2024-04-06T03:47:19.528655+0000","last_active":"2024-04-06T03:47:20.044785+0000","last_peered":"2024-04-06T03:47:20.044785+0000","last_clean":"2024-04-06T03:47:20.044785+0000","last_became_active":"2024-04-06T03:43:25.130333+0000","last_became_peered":"2024-04-06T03:43:25.130333+0000","last_unstale":"2024-04-06T03:47:20.044785+0000","last_undegraded":"2024-04-06T03:47:20.044785+0000","last_fullsized":"2024-04-06T03:47:20.044785+0000","mapping_epoch":345,"log_start":"367'8000","ondisk_log_start":"367'8000","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'13472","last_scrub_stamp":"2024-04-06T03:47:19.528375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:47:19.528375+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:45.314653+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13472,"num_write_kb":431041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":712,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.047477+0000","last_change":"2024-04-06T03:45:28.217075+0000","last_active":"2024-04-06T03:47:08.047477+0000","last_peered":"2024-04-06T03:47:08.047477+0000","last_clean":"2024-04-06T03:47:08.047477+0000","last_became_active":"2024-04-06T03:20:57.821457+0000","last_became_peered":"2024-04-06T03:20:57.821457+0000","last_unstale":"2024-04-06T03:47:08.047477+0000","last_undegraded":"2024-04-06T03:47:08.047477+0000","last_fullsized":"2024-04-06T03:47:08.047477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:28.216763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:44.386020+0000","last_clean_scrub_stamp":"2024-04-06T03:45:28.216763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":681,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049340+0000","last_change":"2024-04-06T03:46:54.030498+0000","last_active":"2024-04-06T03:47:08.049340+0000","last_peered":"2024-04-06T03:47:08.049340+0000","last_clean":"2024-04-06T03:47:08.049340+0000","last_became_active":"2024-04-06T03:42:57.101420+0000","last_became_peered":"2024-04-06T03:42:57.101420+0000","last_unstale":"2024-04-06T03:47:08.049340+0000","last_undegraded":"2024-04-06T03:47:08.049340+0000","last_fullsized":"2024-04-06T03:47:08.049340+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:46:54.030391+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:46:54.030391+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:17.967714+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"307'49","reported_seq":912,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:12.546024+0000","last_change":"2024-04-06T03:43:10.099251+0000","last_active":"2024-04-06T03:47:12.546024+0000","last_peered":"2024-04-06T03:47:12.546024+0000","last_clean":"2024-04-06T03:47:12.546024+0000","last_became_active":"2024-04-06T03:25:54.485518+0000","last_became_peered":"2024-04-06T03:25:54.485518+0000","last_unstale":"2024-04-06T03:47:12.546024+0000","last_undegraded":"2024-04-06T03:47:12.546024+0000","last_fullsized":"2024-04-06T03:47:12.546024+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'49","last_scrub_stamp":"2024-04-06T03:43:10.099011+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:43:10.099011+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":184,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049371+0000","last_change":"2024-04-06T03:43:32.211974+0000","last_active":"2024-04-06T03:47:20.049371+0000","last_peered":"2024-04-06T03:47:20.049371+0000","last_clean":"2024-04-06T03:47:20.049371+0000","last_became_active":"2024-04-06T03:42:57.119712+0000","last_became_peered":"2024-04-06T03:42:57.119712+0000","last_unstale":"2024-04-06T03:47:20.049371+0000","last_undegraded":"2024-04-06T03:47:20.049371+0000","last_fullsized":"2024-04-06T03:47:20.049371+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:43:32.211738+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:22:20.722147+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.211738+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.3","version":"327'13","reported_seq":1065,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:12.436739+0000","last_change":"2024-04-06T03:47:10.994403+0000","last_active":"2024-04-06T03:47:12.436739+0000","last_peered":"2024-04-06T03:47:12.436739+0000","last_clean":"2024-04-06T03:47:12.436739+0000","last_became_active":"2024-04-06T03:42:23.582156+0000","last_became_peered":"2024-04-06T03:42:23.582156+0000","last_unstale":"2024-04-06T03:47:12.436739+0000","last_undegraded":"2024-04-06T03:47:12.436739+0000","last_fullsized":"2024-04-06T03:47:12.436739+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'13","last_scrub_stamp":"2024-04-06T03:47:10.994280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:47:10.994280+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:14.453972+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"381'13074","reported_seq":13157,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.639861+0000","last_change":"2024-04-06T03:43:25.029317+0000","last_active":"2024-04-06T03:47:13.639861+0000","last_peered":"2024-04-06T03:47:13.639861+0000","last_clean":"2024-04-06T03:47:13.639861+0000","last_became_active":"2024-04-06T03:43:25.028550+0000","last_became_peered":"2024-04-06T03:43:25.028550+0000","last_unstale":"2024-04-06T03:47:13.639861+0000","last_undegraded":"2024-04-06T03:47:13.639861+0000","last_fullsized":"2024-04-06T03:47:13.639861+0000","mapping_epoch":345,"log_start":"369'8200","ondisk_log_start":"369'8200","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4874,"log_dups_size":0,"ondisk_log_size":4874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13074,"num_write_kb":418368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":697,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050330+0000","last_change":"2024-04-06T03:45:29.035614+0000","last_active":"2024-04-06T03:47:08.050330+0000","last_peered":"2024-04-06T03:47:08.050330+0000","last_clean":"2024-04-06T03:47:08.050330+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:47:08.050330+0000","last_undegraded":"2024-04-06T03:47:08.050330+0000","last_fullsized":"2024-04-06T03:47:08.050330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:29.035484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:45:29.035484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":115,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.048916+0000","last_change":"2024-04-06T03:46:52.016829+0000","last_active":"2024-04-06T03:47:08.048916+0000","last_peered":"2024-04-06T03:47:08.048916+0000","last_clean":"2024-04-06T03:47:08.048916+0000","last_became_active":"2024-04-06T03:42:57.089778+0000","last_became_peered":"2024-04-06T03:42:57.089778+0000","last_unstale":"2024-04-06T03:47:08.048916+0000","last_undegraded":"2024-04-06T03:47:08.048916+0000","last_fullsized":"2024-04-06T03:47:08.048916+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:52.016719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:46:52.016719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:53.366936+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"307'54","reported_seq":846,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049069+0000","last_change":"2024-04-06T03:43:39.143819+0000","last_active":"2024-04-06T03:47:20.049069+0000","last_peered":"2024-04-06T03:47:20.049069+0000","last_clean":"2024-04-06T03:47:20.049069+0000","last_became_active":"2024-04-06T03:42:23.591518+0000","last_became_peered":"2024-04-06T03:42:23.591518+0000","last_unstale":"2024-04-06T03:47:20.049069+0000","last_undegraded":"2024-04-06T03:47:20.049069+0000","last_fullsized":"2024-04-06T03:47:20.049069+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'54","last_scrub_stamp":"2024-04-06T03:43:39.143539+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:43:39.143539+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":721,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:16.053334+0000","last_change":"2024-04-06T03:47:16.053363+0000","last_active":"2024-04-06T03:47:16.053334+0000","last_peered":"2024-04-06T03:47:16.053334+0000","last_clean":"2024-04-06T03:47:16.053334+0000","last_became_active":"2024-04-06T03:42:57.097491+0000","last_became_peered":"2024-04-06T03:42:57.097491+0000","last_unstale":"2024-04-06T03:47:16.053334+0000","last_undegraded":"2024-04-06T03:47:16.053334+0000","last_fullsized":"2024-04-06T03:47:16.053334+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:16.053230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:47:16.053230+0000","last_clean_scrub_stamp":"2024-04-06T03:47:16.053230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:42.139825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"381'13518","reported_seq":13601,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.654574+0000","last_change":"2024-04-06T03:43:25.131438+0000","last_active":"2024-04-06T03:47:13.654574+0000","last_peered":"2024-04-06T03:47:13.654574+0000","last_clean":"2024-04-06T03:47:13.654574+0000","last_became_active":"2024-04-06T03:43:25.130484+0000","last_became_peered":"2024-04-06T03:43:25.130484+0000","last_unstale":"2024-04-06T03:47:13.654574+0000","last_undegraded":"2024-04-06T03:47:13.654574+0000","last_fullsized":"2024-04-06T03:47:13.654574+0000","mapping_epoch":345,"log_start":"369'8600","ondisk_log_start":"369'8600","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13518,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":715,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053418+0000","last_change":"2024-04-06T03:45:23.445396+0000","last_active":"2024-04-06T03:47:08.053418+0000","last_peered":"2024-04-06T03:47:08.053418+0000","last_clean":"2024-04-06T03:47:08.053418+0000","last_became_active":"2024-04-06T03:42:23.569974+0000","last_became_peered":"2024-04-06T03:42:23.569974+0000","last_unstale":"2024-04-06T03:47:08.053418+0000","last_undegraded":"2024-04-06T03:47:08.053418+0000","last_fullsized":"2024-04-06T03:47:08.053418+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:23.445295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:45:23.445295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"307'54","reported_seq":761,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.043739+0000","last_change":"2024-04-06T03:47:15.587391+0000","last_active":"2024-04-06T03:47:20.043739+0000","last_peered":"2024-04-06T03:47:20.043739+0000","last_clean":"2024-04-06T03:47:20.043739+0000","last_became_active":"2024-04-06T03:42:57.096119+0000","last_became_peered":"2024-04-06T03:42:57.096119+0000","last_unstale":"2024-04-06T03:47:20.043739+0000","last_undegraded":"2024-04-06T03:47:20.043739+0000","last_fullsized":"2024-04-06T03:47:20.043739+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'54","last_scrub_stamp":"2024-04-06T03:47:15.587258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:47:15.587258+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:34.951438+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":73,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":743,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044117+0000","last_change":"2024-04-06T03:46:49.154499+0000","last_active":"2024-04-06T03:47:20.044117+0000","last_peered":"2024-04-06T03:47:20.044117+0000","last_clean":"2024-04-06T03:47:20.044117+0000","last_became_active":"2024-04-06T03:20:51.150158+0000","last_became_peered":"2024-04-06T03:20:51.150158+0000","last_unstale":"2024-04-06T03:47:20.044117+0000","last_undegraded":"2024-04-06T03:47:20.044117+0000","last_fullsized":"2024-04-06T03:47:20.044117+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:49.152620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:46:49.152620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.417526+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":743,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044826+0000","last_change":"2024-04-06T03:47:01.350298+0000","last_active":"2024-04-06T03:47:20.044826+0000","last_peered":"2024-04-06T03:47:20.044826+0000","last_clean":"2024-04-06T03:47:20.044826+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:47:20.044826+0000","last_undegraded":"2024-04-06T03:47:20.044826+0000","last_fullsized":"2024-04-06T03:47:20.044826+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:01.348175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:47:01.348175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:12.956864+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"381'13096","reported_seq":13179,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.658140+0000","last_change":"2024-04-06T03:43:25.131247+0000","last_active":"2024-04-06T03:47:13.658140+0000","last_peered":"2024-04-06T03:47:13.658140+0000","last_clean":"2024-04-06T03:47:13.658140+0000","last_became_active":"2024-04-06T03:43:25.130333+0000","last_became_peered":"2024-04-06T03:43:25.130333+0000","last_unstale":"2024-04-06T03:47:13.658140+0000","last_undegraded":"2024-04-06T03:47:13.658140+0000","last_fullsized":"2024-04-06T03:47:13.658140+0000","mapping_epoch":345,"log_start":"369'8200","ondisk_log_start":"369'8200","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13096,"num_write_kb":419072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":568,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050038+0000","last_change":"2024-04-06T03:47:01.227868+0000","last_active":"2024-04-06T03:47:08.050038+0000","last_peered":"2024-04-06T03:47:08.050038+0000","last_clean":"2024-04-06T03:47:08.050038+0000","last_became_active":"2024-04-06T03:25:54.487690+0000","last_became_peered":"2024-04-06T03:25:54.487690+0000","last_unstale":"2024-04-06T03:47:08.050038+0000","last_undegraded":"2024-04-06T03:47:08.050038+0000","last_fullsized":"2024-04-06T03:47:08.050038+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:01.226368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:21.171511+0000","last_clean_scrub_stamp":"2024-04-06T03:47:01.226368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:23.762618+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"307'42","reported_seq":808,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051021+0000","last_change":"2024-04-06T03:45:33.957089+0000","last_active":"2024-04-06T03:47:08.051021+0000","last_peered":"2024-04-06T03:47:08.051021+0000","last_clean":"2024-04-06T03:47:08.051021+0000","last_became_active":"2024-04-06T03:42:23.588643+0000","last_became_peered":"2024-04-06T03:42:23.588643+0000","last_unstale":"2024-04-06T03:47:08.051021+0000","last_undegraded":"2024-04-06T03:47:08.051021+0000","last_fullsized":"2024-04-06T03:47:08.051021+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'42","last_scrub_stamp":"2024-04-06T03:45:33.956933+0000","last_deep_scrub":"307'42","last_deep_scrub_stamp":"2024-04-06T03:41:26.470096+0000","last_clean_scrub_stamp":"2024-04-06T03:45:33.956933+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":602,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.043982+0000","last_change":"2024-04-06T03:45:30.075970+0000","last_active":"2024-04-06T03:47:20.043982+0000","last_peered":"2024-04-06T03:47:20.043982+0000","last_clean":"2024-04-06T03:47:20.043982+0000","last_became_active":"2024-04-06T03:25:54.496866+0000","last_became_peered":"2024-04-06T03:25:54.496866+0000","last_unstale":"2024-04-06T03:47:20.043982+0000","last_undegraded":"2024-04-06T03:47:20.043982+0000","last_fullsized":"2024-04-06T03:47:20.043982+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.075882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.075882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":1005,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055373+0000","last_change":"2024-04-06T03:47:00.045120+0000","last_active":"2024-04-06T03:47:08.055373+0000","last_peered":"2024-04-06T03:47:08.055373+0000","last_clean":"2024-04-06T03:47:08.055373+0000","last_became_active":"2024-04-06T03:42:26.582173+0000","last_became_peered":"2024-04-06T03:42:26.582173+0000","last_unstale":"2024-04-06T03:47:08.055373+0000","last_undegraded":"2024-04-06T03:47:08.055373+0000","last_fullsized":"2024-04-06T03:47:08.055373+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:47:00.044955+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:22:36.461578+0000","last_clean_scrub_stamp":"2024-04-06T03:47:00.044955+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:11.276239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":707,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.054645+0000","last_change":"2024-04-06T03:45:12.385460+0000","last_active":"2024-04-06T03:47:08.054645+0000","last_peered":"2024-04-06T03:47:08.054645+0000","last_clean":"2024-04-06T03:47:08.054645+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:47:08.054645+0000","last_undegraded":"2024-04-06T03:47:08.054645+0000","last_fullsized":"2024-04-06T03:47:08.054645+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:12.385092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:45:12.385092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"381'13500","reported_seq":13598,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.693304+0000","last_change":"2024-04-06T03:47:02.308281+0000","last_active":"2024-04-06T03:47:13.693304+0000","last_peered":"2024-04-06T03:47:13.693304+0000","last_clean":"2024-04-06T03:47:13.693304+0000","last_became_active":"2024-04-06T03:43:25.130714+0000","last_became_peered":"2024-04-06T03:43:25.130714+0000","last_unstale":"2024-04-06T03:47:13.693304+0000","last_undegraded":"2024-04-06T03:47:13.693304+0000","last_fullsized":"2024-04-06T03:47:13.693304+0000","mapping_epoch":345,"log_start":"369'8600","ondisk_log_start":"369'8600","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'12828","last_scrub_stamp":"2024-04-06T03:47:02.308081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:47:02.308081+0000","objects_scrubbed":676,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.096592+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13500,"num_write_kb":432000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":108,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049287+0000","last_change":"2024-04-06T03:43:42.965111+0000","last_active":"2024-04-06T03:47:08.049287+0000","last_peered":"2024-04-06T03:47:08.049287+0000","last_clean":"2024-04-06T03:47:08.049287+0000","last_became_active":"2024-04-06T03:42:57.093548+0000","last_became_peered":"2024-04-06T03:42:57.093548+0000","last_unstale":"2024-04-06T03:47:08.049287+0000","last_undegraded":"2024-04-06T03:47:08.049287+0000","last_fullsized":"2024-04-06T03:47:08.049287+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.964733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:37.513299+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.964733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"307'49","reported_seq":795,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051113+0000","last_change":"2024-04-06T03:45:17.013250+0000","last_active":"2024-04-06T03:47:08.051113+0000","last_peered":"2024-04-06T03:47:08.051113+0000","last_clean":"2024-04-06T03:47:08.051113+0000","last_became_active":"2024-04-06T03:20:53.333088+0000","last_became_peered":"2024-04-06T03:20:53.333088+0000","last_unstale":"2024-04-06T03:47:08.051113+0000","last_undegraded":"2024-04-06T03:47:08.051113+0000","last_fullsized":"2024-04-06T03:47:08.051113+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'49","last_scrub_stamp":"2024-04-06T03:45:17.013155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:17.013155+0000","objects_scrubbed":8,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":714,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055292+0000","last_change":"2024-04-06T03:43:42.231771+0000","last_active":"2024-04-06T03:47:08.055292+0000","last_peered":"2024-04-06T03:47:08.055292+0000","last_clean":"2024-04-06T03:47:08.055292+0000","last_became_active":"2024-04-06T03:42:57.093654+0000","last_became_peered":"2024-04-06T03:42:57.093654+0000","last_unstale":"2024-04-06T03:47:08.055292+0000","last_undegraded":"2024-04-06T03:47:08.055292+0000","last_fullsized":"2024-04-06T03:47:08.055292+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.231565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.231565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":119,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050950+0000","last_change":"2024-04-06T03:45:32.117873+0000","last_active":"2024-04-06T03:47:08.050950+0000","last_peered":"2024-04-06T03:47:08.050950+0000","last_clean":"2024-04-06T03:47:08.050950+0000","last_became_active":"2024-04-06T03:42:57.098171+0000","last_became_peered":"2024-04-06T03:42:57.098171+0000","last_unstale":"2024-04-06T03:47:08.050950+0000","last_undegraded":"2024-04-06T03:47:08.050950+0000","last_fullsized":"2024-04-06T03:47:08.050950+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_clean_scrub_stamp":"2024-04-06T03:45:32.117767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.c","version":"381'13166","reported_seq":13322,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044324+0000","last_change":"2024-04-06T03:47:17.602011+0000","last_active":"2024-04-06T03:47:20.044324+0000","last_peered":"2024-04-06T03:47:20.044324+0000","last_clean":"2024-04-06T03:47:20.044324+0000","last_became_active":"2024-04-06T03:43:25.027401+0000","last_became_peered":"2024-04-06T03:43:25.027401+0000","last_unstale":"2024-04-06T03:47:20.044324+0000","last_undegraded":"2024-04-06T03:47:20.044324+0000","last_fullsized":"2024-04-06T03:47:20.044324+0000","mapping_epoch":345,"log_start":"365'7700","ondisk_log_start":"365'7700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'13166","last_scrub_stamp":"2024-04-06T03:47:17.601878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:47:17.601878+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:40.872063+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13166,"num_write_kb":421312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":722,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044340+0000","last_change":"2024-04-06T03:47:16.620263+0000","last_active":"2024-04-06T03:47:20.044340+0000","last_peered":"2024-04-06T03:47:20.044340+0000","last_clean":"2024-04-06T03:47:20.044340+0000","last_became_active":"2024-04-06T03:42:57.095020+0000","last_became_peered":"2024-04-06T03:42:57.095020+0000","last_unstale":"2024-04-06T03:47:20.044340+0000","last_undegraded":"2024-04-06T03:47:20.044340+0000","last_fullsized":"2024-04-06T03:47:20.044340+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:16.620144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:56.770310+0000","last_clean_scrub_stamp":"2024-04-06T03:47:16.620144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:43.515968+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":715,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053581+0000","last_change":"2024-04-06T03:45:34.168883+0000","last_active":"2024-04-06T03:47:08.053581+0000","last_peered":"2024-04-06T03:47:08.053581+0000","last_clean":"2024-04-06T03:47:08.053581+0000","last_became_active":"2024-04-06T03:20:51.140837+0000","last_became_peered":"2024-04-06T03:20:51.140837+0000","last_unstale":"2024-04-06T03:47:08.053581+0000","last_undegraded":"2024-04-06T03:47:08.053581+0000","last_fullsized":"2024-04-06T03:47:08.053581+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.165081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.165081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"338'64","reported_seq":1065,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044508+0000","last_change":"2024-04-06T03:46:53.395723+0000","last_active":"2024-04-06T03:47:20.044508+0000","last_peered":"2024-04-06T03:47:20.044508+0000","last_clean":"2024-04-06T03:47:20.044508+0000","last_became_active":"2024-04-06T03:42:57.101642+0000","last_became_peered":"2024-04-06T03:42:57.101642+0000","last_unstale":"2024-04-06T03:47:20.044508+0000","last_undegraded":"2024-04-06T03:47:20.044508+0000","last_fullsized":"2024-04-06T03:47:20.044508+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'64","last_scrub_stamp":"2024-04-06T03:46:53.395305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:46:53.395305+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:01.452463+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":80,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":591,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.048588+0000","last_change":"2024-04-06T03:45:07.981010+0000","last_active":"2024-04-06T03:47:08.048588+0000","last_peered":"2024-04-06T03:47:08.048588+0000","last_clean":"2024-04-06T03:47:08.048588+0000","last_became_active":"2024-04-06T03:42:23.569142+0000","last_became_peered":"2024-04-06T03:42:23.569142+0000","last_unstale":"2024-04-06T03:47:08.048588+0000","last_undegraded":"2024-04-06T03:47:08.048588+0000","last_fullsized":"2024-04-06T03:47:08.048588+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:07.980779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:07.980779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"381'13140","reported_seq":13223,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.676118+0000","last_change":"2024-04-06T03:43:25.029080+0000","last_active":"2024-04-06T03:47:13.676118+0000","last_peered":"2024-04-06T03:47:13.676118+0000","last_clean":"2024-04-06T03:47:13.676118+0000","last_became_active":"2024-04-06T03:43:25.028236+0000","last_became_peered":"2024-04-06T03:43:25.028236+0000","last_unstale":"2024-04-06T03:47:13.676118+0000","last_undegraded":"2024-04-06T03:47:13.676118+0000","last_fullsized":"2024-04-06T03:47:13.676118+0000","mapping_epoch":345,"log_start":"370'8900","ondisk_log_start":"370'8900","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13140,"num_write_kb":420480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":707,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047829+0000","last_change":"2024-04-06T03:45:41.425674+0000","last_active":"2024-04-06T03:47:20.047829+0000","last_peered":"2024-04-06T03:47:20.047829+0000","last_clean":"2024-04-06T03:47:20.047829+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:47:20.047829+0000","last_undegraded":"2024-04-06T03:47:20.047829+0000","last_fullsized":"2024-04-06T03:47:20.047829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:41.425546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:45:41.425546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"78'1","reported_seq":130,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048595+0000","last_change":"2024-04-06T03:46:51.289627+0000","last_active":"2024-04-06T03:47:20.048595+0000","last_peered":"2024-04-06T03:47:20.048595+0000","last_clean":"2024-04-06T03:47:20.048595+0000","last_became_active":"2024-04-06T03:42:57.116773+0000","last_became_peered":"2024-04-06T03:42:57.116773+0000","last_unstale":"2024-04-06T03:47:20.048595+0000","last_undegraded":"2024-04-06T03:47:20.048595+0000","last_fullsized":"2024-04-06T03:47:20.048595+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:46:51.289384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:46:51.289384+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:13.202157+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"307'27","reported_seq":756,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049835+0000","last_change":"2024-04-06T03:45:36.917456+0000","last_active":"2024-04-06T03:47:08.049835+0000","last_peered":"2024-04-06T03:47:08.049835+0000","last_clean":"2024-04-06T03:47:08.049835+0000","last_became_active":"2024-04-06T03:20:53.334820+0000","last_became_peered":"2024-04-06T03:20:53.334820+0000","last_unstale":"2024-04-06T03:47:08.049835+0000","last_undegraded":"2024-04-06T03:47:08.049835+0000","last_fullsized":"2024-04-06T03:47:08.049835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:45:36.917321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:36.917321+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"90'5","reported_seq":2266,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:16.558802+0000","last_change":"2024-04-06T03:46:46.090765+0000","last_active":"2024-04-06T03:47:16.558802+0000","last_peered":"2024-04-06T03:47:16.558802+0000","last_clean":"2024-04-06T03:47:16.558802+0000","last_became_active":"2024-04-06T03:20:54.738196+0000","last_became_peered":"2024-04-06T03:20:54.738196+0000","last_unstale":"2024-04-06T03:47:16.558802+0000","last_undegraded":"2024-04-06T03:47:16.558802+0000","last_fullsized":"2024-04-06T03:47:16.558802+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'5","last_scrub_stamp":"2024-04-06T03:46:46.090642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:53.304114+0000","last_clean_scrub_stamp":"2024-04-06T03:46:46.090642+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:11.641079+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1552,"num_read_kb":1547,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":711,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047225+0000","last_change":"2024-04-06T03:46:56.536025+0000","last_active":"2024-04-06T03:47:20.047225+0000","last_peered":"2024-04-06T03:47:20.047225+0000","last_clean":"2024-04-06T03:47:20.047225+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:47:20.047225+0000","last_undegraded":"2024-04-06T03:47:20.047225+0000","last_fullsized":"2024-04-06T03:47:20.047225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:56.535903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:35.943511+0000","last_clean_scrub_stamp":"2024-04-06T03:46:56.535903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.021227+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"381'13210","reported_seq":13303,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048939+0000","last_change":"2024-04-06T03:47:17.395237+0000","last_active":"2024-04-06T03:47:20.048939+0000","last_peered":"2024-04-06T03:47:20.048939+0000","last_clean":"2024-04-06T03:47:20.048939+0000","last_became_active":"2024-04-06T03:43:25.024471+0000","last_became_peered":"2024-04-06T03:43:25.024471+0000","last_unstale":"2024-04-06T03:47:20.048939+0000","last_undegraded":"2024-04-06T03:47:20.048939+0000","last_fullsized":"2024-04-06T03:47:20.048939+0000","mapping_epoch":345,"log_start":"370'8600","ondisk_log_start":"370'8600","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'13210","last_scrub_stamp":"2024-04-06T03:47:17.394938+0000","last_deep_scrub":"381'13210","last_deep_scrub_stamp":"2024-04-06T03:47:17.394938+0000","last_clean_scrub_stamp":"2024-04-06T03:47:17.394938+0000","objects_scrubbed":0,"log_size":4610,"log_dups_size":0,"ondisk_log_size":4610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:21.528614+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13210,"num_write_kb":422720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":715,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.250799+0000","last_change":"2024-04-06T03:47:13.250839+0000","last_active":"2024-04-06T03:47:13.250799+0000","last_peered":"2024-04-06T03:47:13.250799+0000","last_clean":"2024-04-06T03:47:13.250799+0000","last_became_active":"2024-04-06T03:42:23.562715+0000","last_became_peered":"2024-04-06T03:42:23.562715+0000","last_unstale":"2024-04-06T03:47:13.250799+0000","last_undegraded":"2024-04-06T03:47:13.250799+0000","last_fullsized":"2024-04-06T03:47:13.250799+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:13.250722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:47:13.250722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:18.435168+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":589,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053633+0000","last_change":"2024-04-06T03:46:56.994566+0000","last_active":"2024-04-06T03:47:08.053633+0000","last_peered":"2024-04-06T03:47:08.053633+0000","last_clean":"2024-04-06T03:47:08.053633+0000","last_became_active":"2024-04-06T03:42:23.567776+0000","last_became_peered":"2024-04-06T03:42:23.567776+0000","last_unstale":"2024-04-06T03:47:08.053633+0000","last_undegraded":"2024-04-06T03:47:08.053633+0000","last_fullsized":"2024-04-06T03:47:08.053633+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:56.994448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:46:56.994448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:07.052225+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"312'42","reported_seq":799,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049323+0000","last_change":"2024-04-06T03:45:04.006917+0000","last_active":"2024-04-06T03:47:08.049323+0000","last_peered":"2024-04-06T03:47:08.049323+0000","last_clean":"2024-04-06T03:47:08.049323+0000","last_became_active":"2024-04-06T03:42:57.102077+0000","last_became_peered":"2024-04-06T03:42:57.102077+0000","last_unstale":"2024-04-06T03:47:08.049323+0000","last_undegraded":"2024-04-06T03:47:08.049323+0000","last_fullsized":"2024-04-06T03:47:08.049323+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'42","last_scrub_stamp":"2024-04-06T03:45:04.006799+0000","last_deep_scrub":"312'42","last_deep_scrub_stamp":"2024-04-06T03:41:44.156017+0000","last_clean_scrub_stamp":"2024-04-06T03:45:04.006799+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":732,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047539+0000","last_change":"2024-04-06T03:46:46.343119+0000","last_active":"2024-04-06T03:47:20.047539+0000","last_peered":"2024-04-06T03:47:20.047539+0000","last_clean":"2024-04-06T03:47:20.047539+0000","last_became_active":"2024-04-06T03:42:23.576890+0000","last_became_peered":"2024-04-06T03:42:23.576890+0000","last_unstale":"2024-04-06T03:47:20.047539+0000","last_undegraded":"2024-04-06T03:47:20.047539+0000","last_fullsized":"2024-04-06T03:47:20.047539+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:46.343009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:19.191417+0000","last_clean_scrub_stamp":"2024-04-06T03:46:46.343009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:58.257816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"338'13","reported_seq":1041,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:12.548058+0000","last_change":"2024-04-06T03:45:20.921396+0000","last_active":"2024-04-06T03:47:12.548058+0000","last_peered":"2024-04-06T03:47:12.548058+0000","last_clean":"2024-04-06T03:47:12.548058+0000","last_became_active":"2024-04-06T03:42:57.094766+0000","last_became_peered":"2024-04-06T03:42:57.094766+0000","last_unstale":"2024-04-06T03:47:12.548058+0000","last_undegraded":"2024-04-06T03:47:12.548058+0000","last_fullsized":"2024-04-06T03:47:12.548058+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'13","last_scrub_stamp":"2024-04-06T03:45:20.921243+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:41:18.436223+0000","last_clean_scrub_stamp":"2024-04-06T03:45:20.921243+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1","version":"381'13296","reported_seq":13442,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.696382+0000","last_change":"2024-04-06T03:44:58.331410+0000","last_active":"2024-04-06T03:47:13.696382+0000","last_peered":"2024-04-06T03:47:13.696382+0000","last_clean":"2024-04-06T03:47:13.696382+0000","last_became_active":"2024-04-06T03:43:25.130935+0000","last_became_peered":"2024-04-06T03:43:25.130935+0000","last_unstale":"2024-04-06T03:47:13.696382+0000","last_undegraded":"2024-04-06T03:47:13.696382+0000","last_fullsized":"2024-04-06T03:47:13.696382+0000","mapping_epoch":345,"log_start":"369'8400","ondisk_log_start":"369'8400","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"359'5388","last_scrub_stamp":"2024-04-06T03:44:58.331222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:44:58.331222+0000","objects_scrubbed":5324,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13296,"num_write_kb":425472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":708,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049091+0000","last_change":"2024-04-06T03:45:42.012146+0000","last_active":"2024-04-06T03:47:08.049091+0000","last_peered":"2024-04-06T03:47:08.049091+0000","last_clean":"2024-04-06T03:47:08.049091+0000","last_became_active":"2024-04-06T03:42:23.590773+0000","last_became_peered":"2024-04-06T03:42:23.590773+0000","last_unstale":"2024-04-06T03:47:08.049091+0000","last_undegraded":"2024-04-06T03:47:08.049091+0000","last_fullsized":"2024-04-06T03:47:08.049091+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:42.012040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:45:42.012040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":589,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.050344+0000","last_change":"2024-04-06T03:45:35.465666+0000","last_active":"2024-04-06T03:47:20.050344+0000","last_peered":"2024-04-06T03:47:20.050344+0000","last_clean":"2024-04-06T03:47:20.050344+0000","last_became_active":"2024-04-06T03:42:23.595618+0000","last_became_peered":"2024-04-06T03:42:23.595618+0000","last_unstale":"2024-04-06T03:47:20.050344+0000","last_undegraded":"2024-04-06T03:47:20.050344+0000","last_fullsized":"2024-04-06T03:47:20.050344+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:35.465468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.966446+0000","last_clean_scrub_stamp":"2024-04-06T03:45:35.465468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":713,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055466+0000","last_change":"2024-04-06T03:45:30.229007+0000","last_active":"2024-04-06T03:47:08.055466+0000","last_peered":"2024-04-06T03:47:08.055466+0000","last_clean":"2024-04-06T03:47:08.055466+0000","last_became_active":"2024-04-06T03:42:57.115680+0000","last_became_peered":"2024-04-06T03:42:57.115680+0000","last_unstale":"2024-04-06T03:47:08.055466+0000","last_undegraded":"2024-04-06T03:47:08.055466+0000","last_fullsized":"2024-04-06T03:47:08.055466+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:30.228890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:45:30.228890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"307'43","reported_seq":787,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.047315+0000","last_change":"2024-04-06T03:47:18.400075+0000","last_active":"2024-04-06T03:47:20.047315+0000","last_peered":"2024-04-06T03:47:20.047315+0000","last_clean":"2024-04-06T03:47:20.047315+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:47:20.047315+0000","last_undegraded":"2024-04-06T03:47:20.047315+0000","last_fullsized":"2024-04-06T03:47:20.047315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'43","last_scrub_stamp":"2024-04-06T03:47:18.399821+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:47:18.399821+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:46.377446+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"381'13154","reported_seq":13237,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.654694+0000","last_change":"2024-04-06T03:43:25.131506+0000","last_active":"2024-04-06T03:47:13.654694+0000","last_peered":"2024-04-06T03:47:13.654694+0000","last_clean":"2024-04-06T03:47:13.654694+0000","last_became_active":"2024-04-06T03:43:25.130566+0000","last_became_peered":"2024-04-06T03:43:25.130566+0000","last_unstale":"2024-04-06T03:47:13.654694+0000","last_undegraded":"2024-04-06T03:47:13.654694+0000","last_fullsized":"2024-04-06T03:47:13.654694+0000","mapping_epoch":345,"log_start":"369'8300","ondisk_log_start":"369'8300","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13154,"num_write_kb":420928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":540,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049298+0000","last_change":"2024-04-06T03:43:42.121160+0000","last_active":"2024-04-06T03:47:08.049298+0000","last_peered":"2024-04-06T03:47:08.049298+0000","last_clean":"2024-04-06T03:47:08.049298+0000","last_became_active":"2024-04-06T03:42:57.078094+0000","last_became_peered":"2024-04-06T03:42:57.078094+0000","last_unstale":"2024-04-06T03:47:08.049298+0000","last_undegraded":"2024-04-06T03:47:08.049298+0000","last_fullsized":"2024-04-06T03:47:08.049298+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:42.121037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:38:58.987355+0000","last_clean_scrub_stamp":"2024-04-06T03:43:42.121037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"188'12","reported_seq":1047,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048554+0000","last_change":"2024-04-06T03:47:10.480723+0000","last_active":"2024-04-06T03:47:20.048554+0000","last_peered":"2024-04-06T03:47:20.048554+0000","last_clean":"2024-04-06T03:47:20.048554+0000","last_became_active":"2024-04-06T03:20:55.763626+0000","last_became_peered":"2024-04-06T03:20:55.763626+0000","last_unstale":"2024-04-06T03:47:20.048554+0000","last_undegraded":"2024-04-06T03:47:20.048554+0000","last_fullsized":"2024-04-06T03:47:20.048554+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'12","last_scrub_stamp":"2024-04-06T03:47:10.480600+0000","last_deep_scrub":"188'12","last_deep_scrub_stamp":"2024-04-06T03:47:10.480600+0000","last_clean_scrub_stamp":"2024-04-06T03:47:10.480600+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:28.233914+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":116,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053491+0000","last_change":"2024-04-06T03:47:03.007757+0000","last_active":"2024-04-06T03:47:08.053491+0000","last_peered":"2024-04-06T03:47:08.053491+0000","last_clean":"2024-04-06T03:47:08.053491+0000","last_became_active":"2024-04-06T03:42:57.092193+0000","last_became_peered":"2024-04-06T03:42:57.092193+0000","last_unstale":"2024-04-06T03:47:08.053491+0000","last_undegraded":"2024-04-06T03:47:08.053491+0000","last_fullsized":"2024-04-06T03:47:08.053491+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:03.007608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:47:03.007608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:29.040460+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"307'51","reported_seq":816,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.051357+0000","last_change":"2024-04-06T03:46:44.064699+0000","last_active":"2024-04-06T03:47:08.051357+0000","last_peered":"2024-04-06T03:47:08.051357+0000","last_clean":"2024-04-06T03:47:08.051357+0000","last_became_active":"2024-04-06T03:42:57.113030+0000","last_became_peered":"2024-04-06T03:42:57.113030+0000","last_unstale":"2024-04-06T03:47:08.051357+0000","last_undegraded":"2024-04-06T03:47:08.051357+0000","last_fullsized":"2024-04-06T03:47:08.051357+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'51","last_scrub_stamp":"2024-04-06T03:46:44.064524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:46:44.064524+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.715096+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"381'13358","reported_seq":13449,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:16.168649+0000","last_change":"2024-04-06T03:47:16.168715+0000","last_active":"2024-04-06T03:47:16.168649+0000","last_peered":"2024-04-06T03:47:16.168649+0000","last_clean":"2024-04-06T03:47:16.168649+0000","last_became_active":"2024-04-06T03:43:25.129076+0000","last_became_peered":"2024-04-06T03:43:25.129076+0000","last_unstale":"2024-04-06T03:47:16.168649+0000","last_undegraded":"2024-04-06T03:47:16.168649+0000","last_fullsized":"2024-04-06T03:47:16.168649+0000","mapping_epoch":345,"log_start":"370'9100","ondisk_log_start":"370'9100","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'13358","last_scrub_stamp":"2024-04-06T03:47:16.168466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:47:16.168466+0000","objects_scrubbed":0,"log_size":4258,"log_dups_size":0,"ondisk_log_size":4258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:16.314771+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13358,"num_write_kb":427456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":712,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049490+0000","last_change":"2024-04-06T03:45:37.526151+0000","last_active":"2024-04-06T03:47:20.049490+0000","last_peered":"2024-04-06T03:47:20.049490+0000","last_clean":"2024-04-06T03:47:20.049490+0000","last_became_active":"2024-04-06T03:42:57.102442+0000","last_became_peered":"2024-04-06T03:42:57.102442+0000","last_unstale":"2024-04-06T03:47:20.049490+0000","last_undegraded":"2024-04-06T03:47:20.049490+0000","last_fullsized":"2024-04-06T03:47:20.049490+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:37.525986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:37.525986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":1071,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.045177+0000","last_change":"2024-04-06T03:47:06.489362+0000","last_active":"2024-04-06T03:47:20.045177+0000","last_peered":"2024-04-06T03:47:20.045177+0000","last_clean":"2024-04-06T03:47:20.045177+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:47:20.045177+0000","last_undegraded":"2024-04-06T03:47:20.045177+0000","last_fullsized":"2024-04-06T03:47:20.045177+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:47:06.488857+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:47:06.488857+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:29.099872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":592,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050693+0000","last_change":"2024-04-06T03:43:32.152585+0000","last_active":"2024-04-06T03:47:08.050693+0000","last_peered":"2024-04-06T03:47:08.050693+0000","last_clean":"2024-04-06T03:47:08.050693+0000","last_became_active":"2024-04-06T03:42:57.078347+0000","last_became_peered":"2024-04-06T03:42:57.078347+0000","last_unstale":"2024-04-06T03:47:08.050693+0000","last_undegraded":"2024-04-06T03:47:08.050693+0000","last_fullsized":"2024-04-06T03:47:08.050693+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:32.152478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:35:23.908290+0000","last_clean_scrub_stamp":"2024-04-06T03:43:32.152478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"307'27","reported_seq":750,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049994+0000","last_change":"2024-04-06T03:45:10.955262+0000","last_active":"2024-04-06T03:47:08.049994+0000","last_peered":"2024-04-06T03:47:08.049994+0000","last_clean":"2024-04-06T03:47:08.049994+0000","last_became_active":"2024-04-06T03:20:53.355354+0000","last_became_peered":"2024-04-06T03:20:53.355354+0000","last_unstale":"2024-04-06T03:47:08.049994+0000","last_undegraded":"2024-04-06T03:47:08.049994+0000","last_fullsized":"2024-04-06T03:47:08.049994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'27","last_scrub_stamp":"2024-04-06T03:45:10.955163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:45:10.955163+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"381'12972","reported_seq":13084,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.043784+0000","last_change":"2024-04-06T03:46:45.928415+0000","last_active":"2024-04-06T03:47:20.043784+0000","last_peered":"2024-04-06T03:47:20.043784+0000","last_clean":"2024-04-06T03:47:20.043784+0000","last_became_active":"2024-04-06T03:43:25.028867+0000","last_became_peered":"2024-04-06T03:43:25.028867+0000","last_unstale":"2024-04-06T03:47:20.043784+0000","last_undegraded":"2024-04-06T03:47:20.043784+0000","last_fullsized":"2024-04-06T03:47:20.043784+0000","mapping_epoch":345,"log_start":"365'7500","ondisk_log_start":"365'7500","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'11351","last_scrub_stamp":"2024-04-06T03:46:45.928172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:46:45.928172+0000","objects_scrubbed":1639,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:15.367617+0000","scrub_duration":707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12972,"num_write_kb":415104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":738,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044963+0000","last_change":"2024-04-06T03:47:12.770800+0000","last_active":"2024-04-06T03:47:20.044963+0000","last_peered":"2024-04-06T03:47:20.044963+0000","last_clean":"2024-04-06T03:47:20.044963+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:47:20.044963+0000","last_undegraded":"2024-04-06T03:47:20.044963+0000","last_fullsized":"2024-04-06T03:47:20.044963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:12.770684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:55.616541+0000","last_clean_scrub_stamp":"2024-04-06T03:47:12.770684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:25.669602+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":539,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048078+0000","last_change":"2024-04-06T03:45:02.458926+0000","last_active":"2024-04-06T03:47:20.048078+0000","last_peered":"2024-04-06T03:47:20.048078+0000","last_clean":"2024-04-06T03:47:20.048078+0000","last_became_active":"2024-04-06T03:42:57.110057+0000","last_became_peered":"2024-04-06T03:42:57.110057+0000","last_unstale":"2024-04-06T03:47:20.048078+0000","last_undegraded":"2024-04-06T03:47:20.048078+0000","last_fullsized":"2024-04-06T03:47:20.048078+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:02.458841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.458841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":714,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053249+0000","last_change":"2024-04-06T03:45:24.308229+0000","last_active":"2024-04-06T03:47:08.053249+0000","last_peered":"2024-04-06T03:47:08.053249+0000","last_clean":"2024-04-06T03:47:08.053249+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:47:08.053249+0000","last_undegraded":"2024-04-06T03:47:08.053249+0000","last_fullsized":"2024-04-06T03:47:08.053249+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:24.308115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:45:24.308115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"307'57","reported_seq":773,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.048965+0000","last_change":"2024-04-06T03:43:43.053137+0000","last_active":"2024-04-06T03:47:20.048965+0000","last_peered":"2024-04-06T03:47:20.048965+0000","last_clean":"2024-04-06T03:47:20.048965+0000","last_became_active":"2024-04-06T03:42:57.110277+0000","last_became_peered":"2024-04-06T03:42:57.110277+0000","last_unstale":"2024-04-06T03:47:20.048965+0000","last_undegraded":"2024-04-06T03:47:20.048965+0000","last_fullsized":"2024-04-06T03:47:20.048965+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'57","last_scrub_stamp":"2024-04-06T03:43:43.052879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:43:43.052879+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"381'13184","reported_seq":13275,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:15.155643+0000","last_change":"2024-04-06T03:47:15.155710+0000","last_active":"2024-04-06T03:47:15.155643+0000","last_peered":"2024-04-06T03:47:15.155643+0000","last_clean":"2024-04-06T03:47:15.155643+0000","last_became_active":"2024-04-06T03:43:25.131461+0000","last_became_peered":"2024-04-06T03:43:25.131461+0000","last_unstale":"2024-04-06T03:47:15.155643+0000","last_undegraded":"2024-04-06T03:47:15.155643+0000","last_fullsized":"2024-04-06T03:47:15.155643+0000","mapping_epoch":345,"log_start":"370'8900","ondisk_log_start":"370'8900","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'13184","last_scrub_stamp":"2024-04-06T03:47:15.155470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:47:15.155470+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:32.594820+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13184,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":695,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.055201+0000","last_change":"2024-04-06T03:45:14.363616+0000","last_active":"2024-04-06T03:47:08.055201+0000","last_peered":"2024-04-06T03:47:08.055201+0000","last_clean":"2024-04-06T03:47:08.055201+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:47:08.055201+0000","last_undegraded":"2024-04-06T03:47:08.055201+0000","last_fullsized":"2024-04-06T03:47:08.055201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:14.363468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:45:14.363468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":734,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.043945+0000","last_change":"2024-04-06T03:45:34.011396+0000","last_active":"2024-04-06T03:47:20.043945+0000","last_peered":"2024-04-06T03:47:20.043945+0000","last_clean":"2024-04-06T03:47:20.043945+0000","last_became_active":"2024-04-06T03:20:55.760940+0000","last_became_peered":"2024-04-06T03:20:55.760940+0000","last_unstale":"2024-04-06T03:47:20.043945+0000","last_undegraded":"2024-04-06T03:47:20.043945+0000","last_fullsized":"2024-04-06T03:47:20.043945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:34.011285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:33.770813+0000","last_clean_scrub_stamp":"2024-04-06T03:45:34.011285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":734,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044211+0000","last_change":"2024-04-06T03:45:10.895775+0000","last_active":"2024-04-06T03:47:20.044211+0000","last_peered":"2024-04-06T03:47:20.044211+0000","last_clean":"2024-04-06T03:47:20.044211+0000","last_became_active":"2024-04-06T03:20:51.147709+0000","last_became_peered":"2024-04-06T03:20:51.147709+0000","last_unstale":"2024-04-06T03:47:20.044211+0000","last_undegraded":"2024-04-06T03:47:20.044211+0000","last_fullsized":"2024-04-06T03:47:20.044211+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:10.895208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:56.575091+0000","last_clean_scrub_stamp":"2024-04-06T03:45:10.895208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"307'53","reported_seq":828,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053768+0000","last_change":"2024-04-06T03:45:02.412015+0000","last_active":"2024-04-06T03:47:08.053768+0000","last_peered":"2024-04-06T03:47:08.053768+0000","last_clean":"2024-04-06T03:47:08.053768+0000","last_became_active":"2024-04-06T03:42:23.570628+0000","last_became_peered":"2024-04-06T03:42:23.570628+0000","last_unstale":"2024-04-06T03:47:08.053768+0000","last_undegraded":"2024-04-06T03:47:08.053768+0000","last_fullsized":"2024-04-06T03:47:08.053768+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'53","last_scrub_stamp":"2024-04-06T03:45:02.411569+0000","last_deep_scrub":"307'53","last_deep_scrub_stamp":"2024-04-06T03:41:26.233369+0000","last_clean_scrub_stamp":"2024-04-06T03:45:02.411569+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"381'13062","reported_seq":13163,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:13.675836+0000","last_change":"2024-04-06T03:46:58.463110+0000","last_active":"2024-04-06T03:47:13.675836+0000","last_peered":"2024-04-06T03:47:13.675836+0000","last_clean":"2024-04-06T03:47:13.675836+0000","last_became_active":"2024-04-06T03:43:25.129488+0000","last_became_peered":"2024-04-06T03:43:25.129488+0000","last_unstale":"2024-04-06T03:47:13.675836+0000","last_undegraded":"2024-04-06T03:47:13.675836+0000","last_fullsized":"2024-04-06T03:47:13.675836+0000","mapping_epoch":345,"log_start":"370'8800","ondisk_log_start":"370'8800","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'12198","last_scrub_stamp":"2024-04-06T03:46:58.460285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:46:58.460285+0000","objects_scrubbed":878,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:09.627511+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13062,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.050989+0000","last_change":"2024-04-06T03:46:59.382867+0000","last_active":"2024-04-06T03:47:08.050989+0000","last_peered":"2024-04-06T03:47:08.050989+0000","last_clean":"2024-04-06T03:47:08.050989+0000","last_became_active":"2024-04-06T03:20:57.834673+0000","last_became_peered":"2024-04-06T03:20:57.834673+0000","last_unstale":"2024-04-06T03:47:08.050989+0000","last_undegraded":"2024-04-06T03:47:08.050989+0000","last_fullsized":"2024-04-06T03:47:08.050989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:59.382579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:46:59.382579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:08.458784+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":735,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.043965+0000","last_change":"2024-04-06T03:46:42.275126+0000","last_active":"2024-04-06T03:47:20.043965+0000","last_peered":"2024-04-06T03:47:20.043965+0000","last_clean":"2024-04-06T03:47:20.043965+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:47:20.043965+0000","last_undegraded":"2024-04-06T03:47:20.043965+0000","last_fullsized":"2024-04-06T03:47:20.043965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:46:42.275016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.097314+0000","last_clean_scrub_stamp":"2024-04-06T03:46:42.275016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:00.582617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":109,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049723+0000","last_change":"2024-04-06T03:45:01.559313+0000","last_active":"2024-04-06T03:47:20.049723+0000","last_peered":"2024-04-06T03:47:20.049723+0000","last_clean":"2024-04-06T03:47:20.049723+0000","last_became_active":"2024-04-06T03:42:57.102050+0000","last_became_peered":"2024-04-06T03:42:57.102050+0000","last_unstale":"2024-04-06T03:47:20.049723+0000","last_undegraded":"2024-04-06T03:47:20.049723+0000","last_fullsized":"2024-04-06T03:47:20.049723+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:01.559226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:45:01.559226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"307'35","reported_seq":760,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.048548+0000","last_change":"2024-04-06T03:43:19.078582+0000","last_active":"2024-04-06T03:47:08.048548+0000","last_peered":"2024-04-06T03:47:08.048548+0000","last_clean":"2024-04-06T03:47:08.048548+0000","last_became_active":"2024-04-06T03:42:23.590160+0000","last_became_peered":"2024-04-06T03:42:23.590160+0000","last_unstale":"2024-04-06T03:47:08.048548+0000","last_undegraded":"2024-04-06T03:47:08.048548+0000","last_fullsized":"2024-04-06T03:47:08.048548+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'35","last_scrub_stamp":"2024-04-06T03:43:19.078336+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:43:19.078336+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"381'13312","reported_seq":13397,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.049833+0000","last_change":"2024-04-06T03:43:25.130562+0000","last_active":"2024-04-06T03:47:20.049833+0000","last_peered":"2024-04-06T03:47:20.049833+0000","last_clean":"2024-04-06T03:47:20.049833+0000","last_became_active":"2024-04-06T03:43:25.129636+0000","last_became_peered":"2024-04-06T03:43:25.129636+0000","last_unstale":"2024-04-06T03:47:20.049833+0000","last_undegraded":"2024-04-06T03:47:20.049833+0000","last_fullsized":"2024-04-06T03:47:20.049833+0000","mapping_epoch":345,"log_start":"370'8700","ondisk_log_start":"370'8700","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:23.989755+0000","last_clean_scrub_stamp":"2024-04-06T03:43:23.989755+0000","objects_scrubbed":0,"log_size":4612,"log_dups_size":0,"ondisk_log_size":4612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13312,"num_write_kb":425984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":585,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.049123+0000","last_change":"2024-04-06T03:45:11.156530+0000","last_active":"2024-04-06T03:47:08.049123+0000","last_peered":"2024-04-06T03:47:08.049123+0000","last_clean":"2024-04-06T03:47:08.049123+0000","last_became_active":"2024-04-06T03:42:23.569567+0000","last_became_peered":"2024-04-06T03:42:23.569567+0000","last_unstale":"2024-04-06T03:47:08.049123+0000","last_undegraded":"2024-04-06T03:47:08.049123+0000","last_fullsized":"2024-04-06T03:47:08.049123+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:45:11.156227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:45:11.156227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":750,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-04-06T03:47:20.044495+0000","last_change":"2024-04-06T03:47:00.344508+0000","last_active":"2024-04-06T03:47:20.044495+0000","last_peered":"2024-04-06T03:47:20.044495+0000","last_clean":"2024-04-06T03:47:20.044495+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:47:20.044495+0000","last_undegraded":"2024-04-06T03:47:20.044495+0000","last_fullsized":"2024-04-06T03:47:20.044495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:47:00.344375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:21:56.412874+0000","last_clean_scrub_stamp":"2024-04-06T03:47:00.344375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:48:14.375951+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"307'46","reported_seq":778,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:08.053770+0000","last_change":"2024-04-06T03:46:55.027247+0000","last_active":"2024-04-06T03:47:08.053770+0000","last_peered":"2024-04-06T03:47:08.053770+0000","last_clean":"2024-04-06T03:47:08.053770+0000","last_became_active":"2024-04-06T03:20:53.357792+0000","last_became_peered":"2024-04-06T03:20:53.357792+0000","last_unstale":"2024-04-06T03:47:08.053770+0000","last_undegraded":"2024-04-06T03:47:08.053770+0000","last_fullsized":"2024-04-06T03:47:08.053770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'46","last_scrub_stamp":"2024-04-06T03:46:55.024886+0000","last_deep_scrub":"291'34","last_deep_scrub_stamp":"2024-04-06T03:39:26.934752+0000","last_clean_scrub_stamp":"2024-04-06T03:46:55.024886+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:47:58.916504+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"327'14","reported_seq":1058,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-04-06T03:47:12.436155+0000","last_change":"2024-04-06T03:43:35.224637+0000","last_active":"2024-04-06T03:47:12.436155+0000","last_peered":"2024-04-06T03:47:12.436155+0000","last_clean":"2024-04-06T03:47:12.436155+0000","last_became_active":"2024-04-06T03:42:23.576196+0000","last_became_peered":"2024-04-06T03:42:23.576196+0000","last_unstale":"2024-04-06T03:47:12.436155+0000","last_undegraded":"2024-04-06T03:47:12.436155+0000","last_fullsized":"2024-04-06T03:47:12.436155+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'14","last_scrub_stamp":"2024-04-06T03:43:35.224536+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:43:35.224536+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211944,"num_write_kb":6782145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76544,"ondisk_log_size":76544,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2286,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1842,"num_read_kb":1567,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":4016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1305,"ondisk_log_size":1305,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1552,"num_read_kb":1547,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743752,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":542144,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93203008,"statfs":{"total":95995035648,"available":95439880192,"internally_reserved":0,"allocated":1499136,"data_stored":1112906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136210,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578196,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93166956,"statfs":{"total":95995035648,"available":95402962944,"internally_reserved":0,"allocated":1454080,"data_stored":1111048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575239,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738768,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":624068,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":93121084,"statfs":{"total":95995035648,"available":95355990016,"internally_reserved":0,"allocated":1503232,"data_stored":1112698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379005,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533220,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211932,"statfs":{"total":95995035648,"available":95449018368,"internally_reserved":0,"allocated":729088,"data_stored":428422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084656,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":617872,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93127280,"statfs":{"total":95995035648,"available":95362334720,"internally_reserved":0,"allocated":860160,"data_stored":520238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855720,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549876,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":93195276,"statfs":{"total":95995035648,"available":95431962624,"internally_reserved":0,"allocated":897024,"data_stored":520653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568495966,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581752,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163400,"statfs":{"total":95995035648,"available":95399321600,"internally_reserved":0,"allocated":835584,"data_stored":519596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:47:25.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-04-06T03:47:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:25.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:25.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:25.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:26.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:26.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:26.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:26.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:26.311 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:26.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:26 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.937264+0000 osd.0 (osd.0) 623 : cluster 0 5.1b deep-scrub starts 2024-04-06T03:47:26.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:26 smithi153 bash[23852]: cluster 2024-04-06T03:47:24.939180+0000 osd.0 (osd.0) 624 : cluster 0 5.1b deep-scrub ok 2024-04-06T03:47:26.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:26 smithi153 bash[23852]: cluster 2024-04-06T03:47:25.476308+0000 osd.2 (osd.2) 511 : cluster 0 2.10 scrub starts 2024-04-06T03:47:26.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:26 smithi153 bash[23852]: cluster 2024-04-06T03:47:25.477593+0000 osd.2 (osd.2) 512 : cluster 0 2.10 scrub ok 2024-04-06T03:47:26.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:26 smithi153 bash[23852]: cluster 2024-04-06T03:47:25.664023+0000 osd.3 (osd.3) 613 : cluster 0 2.19 scrub starts 2024-04-06T03:47:26.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:26 smithi153 bash[23852]: cluster 2024-04-06T03:47:25.665763+0000 osd.3 (osd.3) 614 : cluster 0 2.19 scrub ok 2024-04-06T03:47:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.937264+0000 osd.0 (osd.0) 623 : cluster 0 5.1b deep-scrub starts 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[21467]: cluster 2024-04-06T03:47:24.939180+0000 osd.0 (osd.0) 624 : cluster 0 5.1b deep-scrub ok 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[21467]: cluster 2024-04-06T03:47:25.476308+0000 osd.2 (osd.2) 511 : cluster 0 2.10 scrub starts 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[21467]: cluster 2024-04-06T03:47:25.477593+0000 osd.2 (osd.2) 512 : cluster 0 2.10 scrub ok 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[21467]: cluster 2024-04-06T03:47:25.664023+0000 osd.3 (osd.3) 613 : cluster 0 2.19 scrub starts 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[21467]: cluster 2024-04-06T03:47:25.665763+0000 osd.3 (osd.3) 614 : cluster 0 2.19 scrub ok 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.937264+0000 osd.0 (osd.0) 623 : cluster 0 5.1b deep-scrub starts 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[29221]: cluster 2024-04-06T03:47:24.939180+0000 osd.0 (osd.0) 624 : cluster 0 5.1b deep-scrub ok 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[29221]: cluster 2024-04-06T03:47:25.476308+0000 osd.2 (osd.2) 511 : cluster 0 2.10 scrub starts 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[29221]: cluster 2024-04-06T03:47:25.477593+0000 osd.2 (osd.2) 512 : cluster 0 2.10 scrub ok 2024-04-06T03:47:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[29221]: cluster 2024-04-06T03:47:25.664023+0000 osd.3 (osd.3) 613 : cluster 0 2.19 scrub starts 2024-04-06T03:47:26.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:26 smithi050 bash[29221]: cluster 2024-04-06T03:47:25.665763+0000 osd.3 (osd.3) 614 : cluster 0 2.19 scrub ok 2024-04-06T03:47:26.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:26.467 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:26.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:26.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:26.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:26.732 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:26.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:26.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:26.863 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:26.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:26.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:26.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:27.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:27.126 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:27.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:27 smithi050 bash[21467]: cluster 2024-04-06T03:47:26.111699+0000 osd.7 (osd.7) 627 : cluster 0 5.9 scrub starts 2024-04-06T03:47:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:27 smithi050 bash[21467]: cluster 2024-04-06T03:47:26.113386+0000 osd.7 (osd.7) 628 : cluster 0 5.9 scrub ok 2024-04-06T03:47:27.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:27 smithi050 bash[21467]: cluster 2024-04-06T03:47:26.570311+0000 mgr.y (mgr.24368) 973 : cluster 0 pgmap v1056: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 24 op/s 2024-04-06T03:47:27.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:27 smithi050 bash[29221]: cluster 2024-04-06T03:47:26.111699+0000 osd.7 (osd.7) 627 : cluster 0 5.9 scrub starts 2024-04-06T03:47:27.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:27 smithi050 bash[29221]: cluster 2024-04-06T03:47:26.113386+0000 osd.7 (osd.7) 628 : cluster 0 5.9 scrub ok 2024-04-06T03:47:27.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:27 smithi050 bash[29221]: cluster 2024-04-06T03:47:26.570311+0000 mgr.y (mgr.24368) 973 : cluster 0 pgmap v1056: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 24 op/s 2024-04-06T03:47:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:27.361 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:27.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:27 smithi153 bash[23852]: cluster 2024-04-06T03:47:26.111699+0000 osd.7 (osd.7) 627 : cluster 0 5.9 scrub starts 2024-04-06T03:47:27.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:27 smithi153 bash[23852]: cluster 2024-04-06T03:47:26.113386+0000 osd.7 (osd.7) 628 : cluster 0 5.9 scrub ok 2024-04-06T03:47:27.373 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:27 smithi153 bash[23852]: cluster 2024-04-06T03:47:26.570311+0000 mgr.y (mgr.24368) 973 : cluster 0 pgmap v1056: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 24 op/s 2024-04-06T03:47:27.387 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:27.491 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:27.519 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:27.622 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:27.652 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:27.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:27.784 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:27.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:27.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:27.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:28.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:28.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:28.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:28.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: cluster 2024-04-06T03:47:27.416203+0000 osd.2 (osd.2) 513 : cluster 0 2.3 scrub starts 2024-04-06T03:47:28.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: cluster 2024-04-06T03:47:27.418662+0000 osd.2 (osd.2) 514 : cluster 0 2.3 scrub ok 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: audit 2024-04-06T03:47:28.014879+0000 mon.a (mon.0) 1770 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: audit 2024-04-06T03:47:28.021504+0000 mon.a (mon.0) 1771 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: audit 2024-04-06T03:47:28.023112+0000 mon.a (mon.0) 1772 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: audit 2024-04-06T03:47:28.024876+0000 mon.a (mon.0) 1773 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[21467]: audit 2024-04-06T03:47:28.032270+0000 mon.a (mon.0) 1774 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: cluster 2024-04-06T03:47:27.416203+0000 osd.2 (osd.2) 513 : cluster 0 2.3 scrub starts 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: cluster 2024-04-06T03:47:27.418662+0000 osd.2 (osd.2) 514 : cluster 0 2.3 scrub ok 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: audit 2024-04-06T03:47:28.014879+0000 mon.a (mon.0) 1770 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: audit 2024-04-06T03:47:28.021504+0000 mon.a (mon.0) 1771 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: audit 2024-04-06T03:47:28.023112+0000 mon.a (mon.0) 1772 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:47:28.158 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: audit 2024-04-06T03:47:28.024876+0000 mon.a (mon.0) 1773 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:47:28.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:28 smithi050 bash[29221]: audit 2024-04-06T03:47:28.032270+0000 mon.a (mon.0) 1774 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:28.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:28.303 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:28.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: cluster 2024-04-06T03:47:27.416203+0000 osd.2 (osd.2) 513 : cluster 0 2.3 scrub starts 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: cluster 2024-04-06T03:47:27.418662+0000 osd.2 (osd.2) 514 : cluster 0 2.3 scrub ok 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: audit 2024-04-06T03:47:28.014879+0000 mon.a (mon.0) 1770 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: audit 2024-04-06T03:47:28.021504+0000 mon.a (mon.0) 1771 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: audit 2024-04-06T03:47:28.023112+0000 mon.a (mon.0) 1772 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: audit 2024-04-06T03:47:28.024876+0000 mon.a (mon.0) 1773 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:47:28.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:28 smithi153 bash[23852]: audit 2024-04-06T03:47:28.032270+0000 mon.a (mon.0) 1774 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:28.431 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:28.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:28.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:28.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:28.665 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:28.691 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:28.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:28.820 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:28.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:28.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:28.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:29.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:29.077 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:29.124 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:47:29.147 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:47:29.168 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:47:29.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:29.189 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:47:29.207 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:29.212 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:47:29.234 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:47:29.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[21467]: cluster 2024-04-06T03:47:28.571040+0000 mgr.y (mgr.24368) 974 : cluster 0 pgmap v1057: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:29.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[21467]: cluster 2024-04-06T03:47:28.731615+0000 osd.3 (osd.3) 615 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:47:29.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[21467]: cluster 2024-04-06T03:47:28.733606+0000 osd.3 (osd.3) 616 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:47:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[29221]: cluster 2024-04-06T03:47:28.571040+0000 mgr.y (mgr.24368) 974 : cluster 0 pgmap v1057: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[29221]: cluster 2024-04-06T03:47:28.731615+0000 osd.3 (osd.3) 615 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:47:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[29221]: cluster 2024-04-06T03:47:28.733606+0000 osd.3 (osd.3) 616 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:47:29.259 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:47:29.281 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:47:29.282 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:47:29.282 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:47:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:47:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:47:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:47:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:47:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:47:29.283 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:47:29.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:29.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:29.356 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:47:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:47:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:47:29.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:29.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:29 smithi153 bash[23852]: cluster 2024-04-06T03:47:28.571040+0000 mgr.y (mgr.24368) 974 : cluster 0 pgmap v1057: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:29.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:29 smithi153 bash[23852]: cluster 2024-04-06T03:47:28.731615+0000 osd.3 (osd.3) 615 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:47:29.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:29 smithi153 bash[23852]: cluster 2024-04-06T03:47:28.733606+0000 osd.3 (osd.3) 616 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:47:29.460 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:29.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:29.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:29.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:29.725 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:29.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:29.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:29.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:29.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:29.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:30.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:30.113 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-06T03:47:30.122 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:30.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[29221]: audit 2024-04-06T03:47:29.091509+0000 mon.a (mon.0) 1775 : audit 0 from='client.? 172.21.15.50:0/1449190600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:30.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[29221]: audit 2024-04-06T03:47:29.352575+0000 mon.a (mon.0) 1776 : audit 1 from='client.? 172.21.15.50:0/698908512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:47:30.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[29221]: cluster 2024-04-06T03:47:29.383672+0000 osd.4 (osd.4) 599 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:47:30.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[29221]: cluster 2024-04-06T03:47:29.416078+0000 osd.2 (osd.2) 515 : cluster 0 5.b scrub starts 2024-04-06T03:47:30.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[29221]: cluster 2024-04-06T03:47:29.417501+0000 osd.2 (osd.2) 516 : cluster 0 5.b scrub ok 2024-04-06T03:47:30.168 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[29221]: cluster 2024-04-06T03:47:29.452510+0000 osd.4 (osd.4) 600 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:47:30.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[21467]: audit 2024-04-06T03:47:29.091509+0000 mon.a (mon.0) 1775 : audit 0 from='client.? 172.21.15.50:0/1449190600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:30.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[21467]: audit 2024-04-06T03:47:29.352575+0000 mon.a (mon.0) 1776 : audit 1 from='client.? 172.21.15.50:0/698908512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:47:30.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[21467]: cluster 2024-04-06T03:47:29.383672+0000 osd.4 (osd.4) 599 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:47:30.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[21467]: cluster 2024-04-06T03:47:29.416078+0000 osd.2 (osd.2) 515 : cluster 0 5.b scrub starts 2024-04-06T03:47:30.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[21467]: cluster 2024-04-06T03:47:29.417501+0000 osd.2 (osd.2) 516 : cluster 0 5.b scrub ok 2024-04-06T03:47:30.169 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:30 smithi050 bash[21467]: cluster 2024-04-06T03:47:29.452510+0000 osd.4 (osd.4) 600 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:47:30.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:30.249 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:30.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:30.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[23852]: audit 2024-04-06T03:47:29.091509+0000 mon.a (mon.0) 1775 : audit 0 from='client.? 172.21.15.50:0/1449190600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:30.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[23852]: audit 2024-04-06T03:47:29.352575+0000 mon.a (mon.0) 1776 : audit 1 from='client.? 172.21.15.50:0/698908512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-06T03:47:30.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[23852]: cluster 2024-04-06T03:47:29.383672+0000 osd.4 (osd.4) 599 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:47:30.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[23852]: cluster 2024-04-06T03:47:29.416078+0000 osd.2 (osd.2) 515 : cluster 0 5.b scrub starts 2024-04-06T03:47:30.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[23852]: cluster 2024-04-06T03:47:29.417501+0000 osd.2 (osd.2) 516 : cluster 0 5.b scrub ok 2024-04-06T03:47:30.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[23852]: cluster 2024-04-06T03:47:29.452510+0000 osd.4 (osd.4) 600 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:47:30.380 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:30.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:30.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:30.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:30.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:30.747 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:30.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:47:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:47:30.775 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:30.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:30.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:30.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:31.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:31.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:31.055 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:31.056 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:31.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:31.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[21467]: cluster 2024-04-06T03:47:30.098249+0000 mon.a (mon.0) 1777 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:47:31.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[21467]: audit 2024-04-06T03:47:30.106374+0000 mon.a (mon.0) 1778 : audit 1 from='client.? 172.21.15.50:0/698908512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:47:31.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[21467]: cluster 2024-04-06T03:47:30.110877+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e383: 8 total, 8 up, 7 in 2024-04-06T03:47:31.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[21467]: audit 2024-04-06T03:47:30.361581+0000 mgr.y (mgr.24368) 975 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:31.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[21467]: cluster 2024-04-06T03:47:30.571651+0000 mgr.y (mgr.24368) 976 : cluster 0 pgmap v1059: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:31.153 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[29221]: cluster 2024-04-06T03:47:30.098249+0000 mon.a (mon.0) 1777 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:47:31.154 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[29221]: audit 2024-04-06T03:47:30.106374+0000 mon.a (mon.0) 1778 : audit 1 from='client.? 172.21.15.50:0/698908512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:47:31.154 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[29221]: cluster 2024-04-06T03:47:30.110877+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e383: 8 total, 8 up, 7 in 2024-04-06T03:47:31.154 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[29221]: audit 2024-04-06T03:47:30.361581+0000 mgr.y (mgr.24368) 975 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:31.154 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:31 smithi050 bash[29221]: cluster 2024-04-06T03:47:30.571651+0000 mgr.y (mgr.24368) 976 : cluster 0 pgmap v1059: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:31.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:31.271 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:31.298 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:31.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:31.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:31 smithi153 bash[23852]: cluster 2024-04-06T03:47:30.098249+0000 mon.a (mon.0) 1777 : cluster 3 Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:47:31.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:31 smithi153 bash[23852]: audit 2024-04-06T03:47:30.106374+0000 mon.a (mon.0) 1778 : audit 1 from='client.? 172.21.15.50:0/698908512' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-06T03:47:31.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:31 smithi153 bash[23852]: cluster 2024-04-06T03:47:30.110877+0000 mon.a (mon.0) 1779 : cluster 0 osdmap e383: 8 total, 8 up, 7 in 2024-04-06T03:47:31.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:31 smithi153 bash[23852]: audit 2024-04-06T03:47:30.361581+0000 mgr.y (mgr.24368) 975 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:31.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:31 smithi153 bash[23852]: cluster 2024-04-06T03:47:30.571651+0000 mgr.y (mgr.24368) 976 : cluster 0 pgmap v1059: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:31.427 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:31.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:31.558 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:31.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:31.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:31.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:31.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:31.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:31.948 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:31.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:32.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:32.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:32.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:32 smithi153 bash[23852]: cluster 2024-04-06T03:47:31.036338+0000 osd.0 (osd.0) 625 : cluster 0 5.2 scrub starts 2024-04-06T03:47:32.209 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:32.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:32.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:32.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:32.472 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:32.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:32 smithi050 bash[21467]: cluster 2024-04-06T03:47:31.036338+0000 osd.0 (osd.0) 625 : cluster 0 5.2 scrub starts 2024-04-06T03:47:32.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:32 smithi050 bash[29221]: cluster 2024-04-06T03:47:31.036338+0000 osd.0 (osd.0) 625 : cluster 0 5.2 scrub starts 2024-04-06T03:47:32.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:32.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-06T03:47:32.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:32.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:32.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:32.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:32.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:32.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:32.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:32.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:33.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:33.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:33.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:33.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:33.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:33.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:33.364 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:33 smithi050 bash[21467]: cluster 2024-04-06T03:47:32.572508+0000 mgr.y (mgr.24368) 977 : cluster 0 pgmap v1060: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:33.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:33 smithi050 bash[29221]: cluster 2024-04-06T03:47:32.572508+0000 mgr.y (mgr.24368) 977 : cluster 0 pgmap v1060: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:33.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:33.480 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:33.491 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:33 smithi153 bash[23852]: cluster 2024-04-06T03:47:32.572508+0000 mgr.y (mgr.24368) 977 : cluster 0 pgmap v1060: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:33.509 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:33.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:33.641 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:33.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd set nodeep-scrub 2024-04-06T03:47:33.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:33.772 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:33.875 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:33.903 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:33.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:34.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:34.036 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:34.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:34.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:34.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:34.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:34.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:34.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:34.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:34.564 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:34.667 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:34.694 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:34.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:34.819 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:34.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:34.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:34.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:35.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:35.079 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:35.182 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:35.210 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:35.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:35 smithi050 bash[21467]: cluster 2024-04-06T03:47:34.573405+0000 mgr.y (mgr.24368) 978 : cluster 0 pgmap v1061: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:35.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:35 smithi050 bash[29221]: cluster 2024-04-06T03:47:34.573405+0000 mgr.y (mgr.24368) 978 : cluster 0 pgmap v1061: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:35.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:35 smithi153 bash[23852]: cluster 2024-04-06T03:47:34.573405+0000 mgr.y (mgr.24368) 978 : cluster 0 pgmap v1061: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:35.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:35.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:35.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:35.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:35.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:35.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:35.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:35.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:35.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:35.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:35.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:35.996 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:35.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:36.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:36.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[21467]: cluster 2024-04-06T03:47:34.590145+0000 osd.7 (osd.7) 629 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:36.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[21467]: cluster 2024-04-06T03:47:34.590275+0000 osd.7 (osd.7) 630 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:36.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[21467]: cluster 2024-04-06T03:47:35.033623+0000 mon.a (mon.0) 1780 : cluster 0 osdmap e384: 8 total, 8 up, 7 in 2024-04-06T03:47:36.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[21467]: audit 2024-04-06T03:47:35.684812+0000 mon.a (mon.0) 1781 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:36.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[21467]: audit 2024-04-06T03:47:35.686611+0000 mon.a (mon.0) 1782 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:36.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[29221]: cluster 2024-04-06T03:47:34.590145+0000 osd.7 (osd.7) 629 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:36.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[29221]: cluster 2024-04-06T03:47:34.590275+0000 osd.7 (osd.7) 630 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:36.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[29221]: cluster 2024-04-06T03:47:35.033623+0000 mon.a (mon.0) 1780 : cluster 0 osdmap e384: 8 total, 8 up, 7 in 2024-04-06T03:47:36.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[29221]: audit 2024-04-06T03:47:35.684812+0000 mon.a (mon.0) 1781 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:36.112 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:36 smithi050 bash[29221]: audit 2024-04-06T03:47:35.686611+0000 mon.a (mon.0) 1782 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:36.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:36.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:36.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:36 smithi153 bash[23852]: cluster 2024-04-06T03:47:34.590145+0000 osd.7 (osd.7) 629 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:36 smithi153 bash[23852]: cluster 2024-04-06T03:47:34.590275+0000 osd.7 (osd.7) 630 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:36 smithi153 bash[23852]: cluster 2024-04-06T03:47:35.033623+0000 mon.a (mon.0) 1780 : cluster 0 osdmap e384: 8 total, 8 up, 7 in 2024-04-06T03:47:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:36 smithi153 bash[23852]: audit 2024-04-06T03:47:35.684812+0000 mon.a (mon.0) 1781 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:47:36.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:36 smithi153 bash[23852]: audit 2024-04-06T03:47:35.686611+0000 mon.a (mon.0) 1782 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:36.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:36.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:36.446 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:47:36.446 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:47:36.446 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:47:36.446 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:47:36.446 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:47:36.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:36.515 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:36.618 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:36.647 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:36.750 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:36.776 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:36.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:36.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:36.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:37.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:37.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:set pool 11 pgp_num to 16 2024-04-06T03:47:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:37.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:37.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:37 smithi050 bash[21467]: cluster 2024-04-06T03:47:36.574409+0000 mgr.y (mgr.24368) 979 : cluster 0 pgmap v1063: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:37.154 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:37 smithi050 bash[21467]: audit 2024-04-06T03:47:36.699760+0000 mon.a (mon.0) 1783 : audit 1 from='client.? 172.21.15.50:0/2561859058' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-06T03:47:37.154 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:37 smithi050 bash[29221]: cluster 2024-04-06T03:47:36.574409+0000 mgr.y (mgr.24368) 979 : cluster 0 pgmap v1063: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:37.154 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:37 smithi050 bash[29221]: audit 2024-04-06T03:47:36.699760+0000 mon.a (mon.0) 1783 : audit 1 from='client.? 172.21.15.50:0/2561859058' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-06T03:47:37.165 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:37.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:37 smithi153 bash[23852]: cluster 2024-04-06T03:47:36.574409+0000 mgr.y (mgr.24368) 979 : cluster 0 pgmap v1063: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:37.279 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:37 smithi153 bash[23852]: audit 2024-04-06T03:47:36.699760+0000 mon.a (mon.0) 1783 : audit 1 from='client.? 172.21.15.50:0/2561859058' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-04-06T03:47:37.291 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:37.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:37.422 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:37.525 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:37.552 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:37.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.0 dump_historic_ops 2024-04-06T03:47:37.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:37.682 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:37.786 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:37.811 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:37.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:37.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:37.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:38.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:38.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:38 smithi153 bash[23852]: audit 2024-04-06T03:47:37.037082+0000 mon.a (mon.0) 1784 : audit 1 from='client.? 172.21.15.50:0/2561859058' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-04-06T03:47:38.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:38 smithi153 bash[23852]: cluster 2024-04-06T03:47:37.043256+0000 mon.a (mon.0) 1785 : cluster 0 osdmap e385: 8 total, 8 up, 7 in 2024-04-06T03:47:38.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:38.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:38 smithi050 bash[21467]: audit 2024-04-06T03:47:37.037082+0000 mon.a (mon.0) 1784 : audit 1 from='client.? 172.21.15.50:0/2561859058' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-04-06T03:47:38.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:38 smithi050 bash[21467]: cluster 2024-04-06T03:47:37.043256+0000 mon.a (mon.0) 1785 : cluster 0 osdmap e385: 8 total, 8 up, 7 in 2024-04-06T03:47:38.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:38 smithi050 bash[29221]: audit 2024-04-06T03:47:37.037082+0000 mon.a (mon.0) 1784 : audit 1 from='client.? 172.21.15.50:0/2561859058' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-04-06T03:47:38.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:38 smithi050 bash[29221]: cluster 2024-04-06T03:47:37.043256+0000 mon.a (mon.0) 1785 : cluster 0 osdmap e385: 8 total, 8 up, 7 in 2024-04-06T03:47:38.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:38.262 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-04-06T03:47:38.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-04-06T03:47:38.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:38.333 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:38.436 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:38.464 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:38.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:38.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:38.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.727 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:38.740 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:38.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:38.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:38.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:38.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:39.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:39 smithi050 bash[21467]: cluster 2024-04-06T03:47:38.575636+0000 mgr.y (mgr.24368) 980 : cluster 0 pgmap v1065: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:39.103 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:39 smithi050 bash[29221]: cluster 2024-04-06T03:47:38.575636+0000 mgr.y (mgr.24368) 980 : cluster 0 pgmap v1065: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:39.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:39.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:39.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:39.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:39.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.368 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:47:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:47:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:47:39.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:39.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:39 smithi153 bash[23852]: cluster 2024-04-06T03:47:38.575636+0000 mgr.y (mgr.24368) 980 : cluster 0 pgmap v1065: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:39.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:39.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.758 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:39.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:39.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:39.911 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:39.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:40.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:40.042 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:40.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:40.278 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:40.304 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:40.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:40.436 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:40.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:40.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:47:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:47:40.569 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:40.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:40.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:40.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:40.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:40.929 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:40.956 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:40.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:41.059 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:41.088 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:41.191 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:41.218 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:41.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:41.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:41.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:41.448 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:41.475 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:41.578 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:41.604 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:41.707 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:41.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:41 smithi153 bash[23852]: audit 2024-04-06T03:47:40.372582+0000 mgr.y (mgr.24368) 981 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:41.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:41 smithi153 bash[23852]: cluster 2024-04-06T03:47:40.576428+0000 mgr.y (mgr.24368) 982 : cluster 0 pgmap v1066: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:41.732 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:41.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:41.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:41 smithi050 bash[21467]: audit 2024-04-06T03:47:40.372582+0000 mgr.y (mgr.24368) 981 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:41.845 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:41 smithi050 bash[21467]: cluster 2024-04-06T03:47:40.576428+0000 mgr.y (mgr.24368) 982 : cluster 0 pgmap v1066: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:41.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:41 smithi050 bash[29221]: audit 2024-04-06T03:47:40.372582+0000 mgr.y (mgr.24368) 981 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:41.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:41 smithi050 bash[29221]: cluster 2024-04-06T03:47:40.576428+0000 mgr.y (mgr.24368) 982 : cluster 0 pgmap v1066: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:41.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:41.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:41.961 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:41.990 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:41.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:42.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:42.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:42.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:42.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:42.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:42.481 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:42.508 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:42.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:42.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:42.744 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:42.771 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:42.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:42.900 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:42.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:43.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:43.032 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:43.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:43.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:43.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 7, 6] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-06T03:47:43.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-06T03:47:43.263 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 3, 4, 5, 7, 6] 2024-04-06T03:47:43.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd out 0 2024-04-06T03:47:43.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:43.338 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:43.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:43.468 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:43.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:43.600 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:43.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:43.714 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:43 smithi153 bash[23852]: cluster 2024-04-06T03:47:42.576963+0000 mgr.y (mgr.24368) 983 : cluster 0 pgmap v1067: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:43.729 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:43.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:43 smithi050 bash[21467]: cluster 2024-04-06T03:47:42.576963+0000 mgr.y (mgr.24368) 983 : cluster 0 pgmap v1067: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:43.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:43 smithi050 bash[29221]: cluster 2024-04-06T03:47:42.576963+0000 mgr.y (mgr.24368) 983 : cluster 0 pgmap v1067: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:43.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:43.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:43.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:43.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:43.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:44.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:44.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:44.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:44.248 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:44.295 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:44.373 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:44.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:44.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:44.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:44.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:44.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:44.665 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:44.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:44.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:44.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:44.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:44.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:45.026 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:45.053 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:45.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:45.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:45.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:45.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:45 smithi153 bash[23852]: cluster 2024-04-06T03:47:44.577944+0000 mgr.y (mgr.24368) 984 : cluster 0 pgmap v1068: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:47:45.315 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:45.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:45.446 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:45.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:45.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:45 smithi050 bash[21467]: cluster 2024-04-06T03:47:44.577944+0000 mgr.y (mgr.24368) 984 : cluster 0 pgmap v1068: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:47:45.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:45 smithi050 bash[29221]: cluster 2024-04-06T03:47:44.577944+0000 mgr.y (mgr.24368) 984 : cluster 0 pgmap v1068: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-06T03:47:45.577 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:45.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:45.707 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:45.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:45.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:45.886 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:47:45.907 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:47:45.928 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:47:45.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:45.951 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:47:45.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:45.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:45.978 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:47:46.003 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:47:46.029 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:47:46.055 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:47:46.056 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:47:46.057 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:47:46.064 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:46.090 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:46.193 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:46.220 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:46.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:46 smithi050 bash[21467]: audit 2024-04-06T03:47:45.865572+0000 mon.c (mon.2) 116 : audit 0 from='client.? 172.21.15.50:0/3599914579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:46.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:46 smithi050 bash[29221]: audit 2024-04-06T03:47:45.865572+0000 mon.c (mon.2) 116 : audit 0 from='client.? 172.21.15.50:0/3599914579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:46.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:46 smithi153 bash[23852]: audit 2024-04-06T03:47:45.865572+0000 mon.c (mon.2) 116 : audit 0 from='client.? 172.21.15.50:0/3599914579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:47:46.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:46.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:46.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:46.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:46.476 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:46.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:46.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:46.712 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:46.739 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:46.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:46.867 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:46.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:46.971 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:46.999 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:47.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:47.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:47.129 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:47.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:47.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:47 smithi153 bash[23852]: cluster 2024-04-06T03:47:46.579234+0000 mgr.y (mgr.24368) 985 : cluster 0 pgmap v1069: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:47.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:47 smithi050 bash[21467]: cluster 2024-04-06T03:47:46.579234+0000 mgr.y (mgr.24368) 985 : cluster 0 pgmap v1069: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:47.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:47 smithi050 bash[29221]: cluster 2024-04-06T03:47:46.579234+0000 mgr.y (mgr.24368) 985 : cluster 0 pgmap v1069: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:47.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:47.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:47.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:47.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:47.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:47.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:47.785 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:47.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:47.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:47.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:48.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:48.048 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:48.151 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:48.176 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:48.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:48.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:48.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:48.438 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:48.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:48.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:48.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:48.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:48.831 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:48.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:48.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:48.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:49.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[21467]: cluster 2024-04-06T03:47:47.573671+0000 osd.2 (osd.2) 517 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[21467]: cluster 2024-04-06T03:47:47.573749+0000 osd.2 (osd.2) 518 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[21467]: cluster 2024-04-06T03:47:48.047415+0000 mon.a (mon.0) 1786 : cluster 0 osdmap e386: 8 total, 8 up, 7 in 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[21467]: cluster 2024-04-06T03:47:48.579843+0000 mgr.y (mgr.24368) 986 : cluster 0 pgmap v1071: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[29221]: cluster 2024-04-06T03:47:47.573671+0000 osd.2 (osd.2) 517 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[29221]: cluster 2024-04-06T03:47:47.573749+0000 osd.2 (osd.2) 518 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[29221]: cluster 2024-04-06T03:47:48.047415+0000 mon.a (mon.0) 1786 : cluster 0 osdmap e386: 8 total, 8 up, 7 in 2024-04-06T03:47:49.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[29221]: cluster 2024-04-06T03:47:48.579843+0000 mgr.y (mgr.24368) 986 : cluster 0 pgmap v1071: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:49.090 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:49.177 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:49.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:49.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:49.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:49.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:49.465 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:47:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:47:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:47:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:49 smithi153 bash[23852]: cluster 2024-04-06T03:47:47.573671+0000 osd.2 (osd.2) 517 : cluster 0 purged_snaps scrub starts 2024-04-06T03:47:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:49 smithi153 bash[23852]: cluster 2024-04-06T03:47:47.573749+0000 osd.2 (osd.2) 518 : cluster 0 purged_snaps scrub ok 2024-04-06T03:47:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:49 smithi153 bash[23852]: cluster 2024-04-06T03:47:48.047415+0000 mon.a (mon.0) 1786 : cluster 0 osdmap e386: 8 total, 8 up, 7 in 2024-04-06T03:47:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:49 smithi153 bash[23852]: cluster 2024-04-06T03:47:48.579843+0000 mgr.y (mgr.24368) 986 : cluster 0 pgmap v1071: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:49.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:49.587 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:49.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:49.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:49.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:49.851 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:49.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:49.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:49.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:50.065 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-06T03:47:50.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:50.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:50 smithi050 bash[21467]: audit 2024-04-06T03:47:49.487554+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.50:0/3148167620' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:50.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:50 smithi050 bash[21467]: audit 2024-04-06T03:47:49.487860+0000 mon.a (mon.0) 1787 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:50.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:50 smithi050 bash[29221]: audit 2024-04-06T03:47:49.487554+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.50:0/3148167620' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:50.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:50 smithi050 bash[29221]: audit 2024-04-06T03:47:49.487860+0000 mon.a (mon.0) 1787 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:50.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:50.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:50.219 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:50 smithi153 bash[23852]: audit 2024-04-06T03:47:49.487554+0000 mon.c (mon.2) 117 : audit 1 from='client.? 172.21.15.50:0/3148167620' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:50.220 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:50 smithi153 bash[23852]: audit 2024-04-06T03:47:49.487860+0000 mon.a (mon.0) 1787 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:47:50.234 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:50.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:50.362 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:50.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:50.489 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:50.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:50.548 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:47:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:47:50.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:50.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:50.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:50.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-06T03:47:50.750 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:50.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:50.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:50.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:50.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:51.008 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:51.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[29221]: cluster 2024-04-06T03:47:50.049750+0000 mon.a (mon.0) 1788 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[29221]: audit 2024-04-06T03:47:50.057285+0000 mon.a (mon.0) 1789 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[29221]: cluster 2024-04-06T03:47:50.060961+0000 mon.a (mon.0) 1790 : cluster 0 osdmap e387: 8 total, 8 up, 7 in 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[29221]: audit 2024-04-06T03:47:50.380578+0000 mgr.y (mgr.24368) 987 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[29221]: cluster 2024-04-06T03:47:50.580599+0000 mgr.y (mgr.24368) 988 : cluster 0 pgmap v1073: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[29221]: audit 2024-04-06T03:47:50.717479+0000 mon.a (mon.0) 1791 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[21467]: cluster 2024-04-06T03:47:50.049750+0000 mon.a (mon.0) 1788 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[21467]: audit 2024-04-06T03:47:50.057285+0000 mon.a (mon.0) 1789 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[21467]: cluster 2024-04-06T03:47:50.060961+0000 mon.a (mon.0) 1790 : cluster 0 osdmap e387: 8 total, 8 up, 7 in 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[21467]: audit 2024-04-06T03:47:50.380578+0000 mgr.y (mgr.24368) 987 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:51.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[21467]: cluster 2024-04-06T03:47:50.580599+0000 mgr.y (mgr.24368) 988 : cluster 0 pgmap v1073: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:51.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:51 smithi050 bash[21467]: audit 2024-04-06T03:47:50.717479+0000 mon.a (mon.0) 1791 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:51.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:51.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:51.272 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:51 smithi153 bash[23852]: cluster 2024-04-06T03:47:50.049750+0000 mon.a (mon.0) 1788 : cluster 3 Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:51 smithi153 bash[23852]: audit 2024-04-06T03:47:50.057285+0000 mon.a (mon.0) 1789 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-06T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:51 smithi153 bash[23852]: cluster 2024-04-06T03:47:50.060961+0000 mon.a (mon.0) 1790 : cluster 0 osdmap e387: 8 total, 8 up, 7 in 2024-04-06T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:51 smithi153 bash[23852]: audit 2024-04-06T03:47:50.380578+0000 mgr.y (mgr.24368) 987 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:51 smithi153 bash[23852]: cluster 2024-04-06T03:47:50.580599+0000 mgr.y (mgr.24368) 988 : cluster 0 pgmap v1073: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:51.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:51 smithi153 bash[23852]: audit 2024-04-06T03:47:50.717479+0000 mon.a (mon.0) 1791 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:47:51.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:51.403 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:51.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:51.532 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:51.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:51.663 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:51.766 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:51.794 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:51.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:51.926 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:51.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:52.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:52.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:52.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:52.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:52.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:52.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.319 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:52.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:52.420 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:52.447 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:52.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:52.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:52.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:52.709 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:52.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:52.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:52.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:52.971 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:52.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:53.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:53.103 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:53.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:53.231 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:53.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:53.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:53.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:53.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:53.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:53.626 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:53.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:53.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:53.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:53.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:53.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:53 smithi050 bash[21467]: cluster 2024-04-06T03:47:52.581130+0000 mgr.y (mgr.24368) 989 : cluster 0 pgmap v1074: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:53.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:53 smithi050 bash[29221]: cluster 2024-04-06T03:47:52.581130+0000 mgr.y (mgr.24368) 989 : cluster 0 pgmap v1074: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:53.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:53.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:54.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:53 smithi153 bash[23852]: cluster 2024-04-06T03:47:52.581130+0000 mgr.y (mgr.24368) 989 : cluster 0 pgmap v1074: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:47:54.022 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:54.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:54.153 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:54.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:54.282 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:54.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:54.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:54.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:54.516 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:54.543 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:54.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:54.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:54.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:54.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:54.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:54.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:54.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:55.043 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:55.070 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:55.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:55.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:55.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:55 smithi153 bash[23852]: cluster 2024-04-06T03:47:54.582104+0000 mgr.y (mgr.24368) 990 : cluster 0 pgmap v1075: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:55.299 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:55.327 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:55.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:55 smithi050 bash[21467]: cluster 2024-04-06T03:47:54.582104+0000 mgr.y (mgr.24368) 990 : cluster 0 pgmap v1075: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:55.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:55 smithi050 bash[29221]: cluster 2024-04-06T03:47:54.582104+0000 mgr.y (mgr.24368) 990 : cluster 0 pgmap v1075: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:55.430 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:55.457 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:55.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:55.588 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:55.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:55.716 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:55.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:55.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:55.980 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:55.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:56.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:56.111 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:56.214 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:56.239 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:56.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:56.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:56.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:56.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:56.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:56.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:56.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:56.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:56.893 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:56.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:56.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:57.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:57.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:57.152 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:57.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:57.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:57.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:57.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:57.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:57.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:47:57.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:57.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0 2024-04-06T03:47:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.0' 2024-04-06T03:47:57.779 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:57.791 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:57 smithi153 bash[23852]: cluster 2024-04-06T03:47:56.583367+0000 mgr.y (mgr.24368) 991 : cluster 0 pgmap v1076: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:57.807 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:57.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:57.938 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:57.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:57.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:57.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:57 smithi050 bash[21467]: cluster 2024-04-06T03:47:56.583367+0000 mgr.y (mgr.24368) 991 : cluster 0 pgmap v1076: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:57.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:57 smithi050 bash[29221]: cluster 2024-04-06T03:47:56.583367+0000 mgr.y (mgr.24368) 991 : cluster 0 pgmap v1076: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:47:58.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:58.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:58.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:58.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:47:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:47:58.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:58.330 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:58.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:58.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:58.558 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4 2024-04-06T03:47:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.4' 2024-04-06T03:47:58.610 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-06T03:47:58.611 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-06T03:47:58.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:58.714 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:58.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:58.826 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:47:58.843 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:58.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:58.974 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:47:58.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:58.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:47:59.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:59.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:59.232 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:47:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:47:59.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.346 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:47:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:47:59.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:59.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:59.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:47:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:47:59.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unique_pool_4' removed 2024-04-06T03:47:59.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:47:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:47:59.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[21467]: cluster 2024-04-06T03:47:58.584097+0000 mgr.y (mgr.24368) 992 : cluster 0 pgmap v1077: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[21467]: audit 2024-04-06T03:47:59.367570+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.50:0/98089582' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:47:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[21467]: audit 2024-04-06T03:47:59.367976+0000 mon.a (mon.0) 1792 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:47:59.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[29221]: cluster 2024-04-06T03:47:58.584097+0000 mgr.y (mgr.24368) 992 : cluster 0 pgmap v1077: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:47:59.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[29221]: audit 2024-04-06T03:47:59.367570+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.50:0/98089582' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:47:59.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:47:59 smithi050 bash[29221]: audit 2024-04-06T03:47:59.367976+0000 mon.a (mon.0) 1792 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:47:59.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:47:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:47:59.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:47:59.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:48:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:59 smithi153 bash[23852]: cluster 2024-04-06T03:47:58.584097+0000 mgr.y (mgr.24368) 992 : cluster 0 pgmap v1077: 148 pgs: 148 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:59 smithi153 bash[23852]: audit 2024-04-06T03:47:59.367570+0000 mon.c (mon.2) 118 : audit 1 from='client.? 172.21.15.50:0/98089582' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:48:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:47:59 smithi153 bash[23852]: audit 2024-04-06T03:47:59.367976+0000 mon.a (mon.0) 1792 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-06T03:48:00.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:48:00.156 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:48:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:48:00.259 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:00.281 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:48:00.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:00.411 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:48:00.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:48:00.540 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7 2024-04-06T03:48:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.7' 2024-04-06T03:48:00.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:00.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:48:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:48:00.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:48:00.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:00.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:48:00.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:00.917 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:00 smithi050 bash[29221]: audit 2024-04-06T03:47:59.642424+0000 mon.a (mon.0) 1793 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:48:00.917 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:00 smithi050 bash[29221]: cluster 2024-04-06T03:47:59.646812+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e388: 8 total, 8 up, 7 in 2024-04-06T03:48:00.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:00 smithi050 bash[21467]: audit 2024-04-06T03:47:59.642424+0000 mon.a (mon.0) 1793 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:48:00.917 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:00 smithi050 bash[21467]: cluster 2024-04-06T03:47:59.646812+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e388: 8 total, 8 up, 7 in 2024-04-06T03:48:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:00.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:48:01.036 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:01.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:00 smithi153 bash[23852]: audit 2024-04-06T03:47:59.642424+0000 mon.a (mon.0) 1793 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-04-06T03:48:01.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:00 smithi153 bash[23852]: cluster 2024-04-06T03:47:59.646812+0000 mon.a (mon.0) 1794 : cluster 0 osdmap e388: 8 total, 8 up, 7 in 2024-04-06T03:48:01.064 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:48:01.168 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:01.195 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:48:01.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:48:01.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:48:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:48:01.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:48:01.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3 2024-04-06T03:48:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.3' 2024-04-06T03:48:01.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:01.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:48:01.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:01.724 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:48:01.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:01.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:01 smithi050 bash[29221]: audit 2024-04-06T03:48:00.383441+0000 mgr.y (mgr.24368) 993 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:01.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:01 smithi050 bash[29221]: cluster 2024-04-06T03:48:00.585183+0000 mgr.y (mgr.24368) 994 : cluster 0 pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:48:01.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:01 smithi050 bash[21467]: audit 2024-04-06T03:48:00.383441+0000 mgr.y (mgr.24368) 993 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:01.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:01 smithi050 bash[21467]: cluster 2024-04-06T03:48:00.585183+0000 mgr.y (mgr.24368) 994 : cluster 0 pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:48:01.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:48:01.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:01.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:01 smithi153 bash[23852]: audit 2024-04-06T03:48:00.383441+0000 mgr.y (mgr.24368) 993 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:01.971 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:01 smithi153 bash[23852]: cluster 2024-04-06T03:48:00.585183+0000 mgr.y (mgr.24368) 994 : cluster 0 pgmap v1079: 132 pgs: 132 active+clean; 582 KiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:48:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:01.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:48:02.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1 2024-04-06T03:48:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:02.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.1' 2024-04-06T03:48:02.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:02.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2 2024-04-06T03:48:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:02.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.2' 2024-04-06T03:48:02.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:02.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-06T03:48:02.384 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5 2024-04-06T03:48:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:02.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.5' 2024-04-06T03:48:02.487 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:48:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:48:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:02.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:48:02.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:48:02.648 INFO:teuthology.orchestra.run.smithi153.stderr:Error response from daemon: Cannot kill container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6 2024-04-06T03:48:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T03:48:02.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi153 with status 1: 'sudo docker kill -s 1 ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd.6' 2024-04-06T03:48:02.734 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-06T03:48:02.743 INFO:tasks.thrashosds:joining thrashosds 2024-04-06T03:48:03.706 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.0. 2024-04-06T03:48:03.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[21467]: debug 2024-04-06T03:48:03.497+0000 7f702b621700 -1 mon.a@0(leader).osd e388 definitely_dead 0 2024-04-06T03:48:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[29221]: cluster 2024-04-06T03:48:02.585868+0000 mgr.y (mgr.24368) 995 : cluster 0 pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:48:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[29221]: audit 2024-04-06T03:48:03.503710+0000 mon.c (mon.2) 119 : audit 1 from='client.? 172.21.15.50:0/1141010938' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-06T03:48:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[29221]: audit 2024-04-06T03:48:03.504120+0000 mon.a (mon.0) 1795 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-06T03:48:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[29221]: cluster 2024-04-06T03:48:03.504216+0000 mon.a (mon.0) 1796 : cluster 1 Client client.admin marked osd.0 out, while it was still marked up 2024-04-06T03:48:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:03 smithi153 bash[23852]: cluster 2024-04-06T03:48:02.585868+0000 mgr.y (mgr.24368) 995 : cluster 0 pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:48:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:03 smithi153 bash[23852]: audit 2024-04-06T03:48:03.503710+0000 mon.c (mon.2) 119 : audit 1 from='client.? 172.21.15.50:0/1141010938' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-06T03:48:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:03 smithi153 bash[23852]: audit 2024-04-06T03:48:03.504120+0000 mon.a (mon.0) 1795 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-06T03:48:04.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:03 smithi153 bash[23852]: cluster 2024-04-06T03:48:03.504216+0000 mon.a (mon.0) 1796 : cluster 1 Client client.admin marked osd.0 out, while it was still marked up 2024-04-06T03:48:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[21467]: cluster 2024-04-06T03:48:02.585868+0000 mgr.y (mgr.24368) 995 : cluster 0 pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:48:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[21467]: audit 2024-04-06T03:48:03.503710+0000 mon.c (mon.2) 119 : audit 1 from='client.? 172.21.15.50:0/1141010938' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-06T03:48:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[21467]: audit 2024-04-06T03:48:03.504120+0000 mon.a (mon.0) 1795 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-04-06T03:48:04.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:03 smithi050 bash[21467]: cluster 2024-04-06T03:48:03.504216+0000 mon.a (mon.0) 1796 : cluster 1 Client client.admin marked osd.0 out, while it was still marked up 2024-04-06T03:48:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:04 smithi050 bash[21467]: audit 2024-04-06T03:48:03.671473+0000 mon.a (mon.0) 1797 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-06T03:48:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:04 smithi050 bash[21467]: cluster 2024-04-06T03:48:03.677901+0000 mon.a (mon.0) 1798 : cluster 0 osdmap e389: 8 total, 8 up, 6 in 2024-04-06T03:48:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:04 smithi050 bash[21467]: cluster 2024-04-06T03:48:04.033705+0000 mon.a (mon.0) 1799 : cluster 0 osdmap e390: 8 total, 8 up, 6 in 2024-04-06T03:48:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:04 smithi050 bash[29221]: audit 2024-04-06T03:48:03.671473+0000 mon.a (mon.0) 1797 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-06T03:48:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:04 smithi050 bash[29221]: cluster 2024-04-06T03:48:03.677901+0000 mon.a (mon.0) 1798 : cluster 0 osdmap e389: 8 total, 8 up, 6 in 2024-04-06T03:48:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:04 smithi050 bash[29221]: cluster 2024-04-06T03:48:04.033705+0000 mon.a (mon.0) 1799 : cluster 0 osdmap e390: 8 total, 8 up, 6 in 2024-04-06T03:48:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:04 smithi153 bash[23852]: audit 2024-04-06T03:48:03.671473+0000 mon.a (mon.0) 1797 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-04-06T03:48:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:04 smithi153 bash[23852]: cluster 2024-04-06T03:48:03.677901+0000 mon.a (mon.0) 1798 : cluster 0 osdmap e389: 8 total, 8 up, 6 in 2024-04-06T03:48:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:04 smithi153 bash[23852]: cluster 2024-04-06T03:48:04.033705+0000 mon.a (mon.0) 1799 : cluster 0 osdmap e390: 8 total, 8 up, 6 in 2024-04-06T03:48:05.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:05 smithi050 bash[21467]: cluster 2024-04-06T03:48:04.586345+0000 mgr.y (mgr.24368) 996 : cluster 0 pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T03:48:05.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:05 smithi050 bash[21467]: cluster 2024-04-06T03:48:05.051618+0000 mon.a (mon.0) 1800 : cluster 0 osdmap e391: 8 total, 8 up, 6 in 2024-04-06T03:48:05.908 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:05 smithi050 bash[29221]: cluster 2024-04-06T03:48:04.586345+0000 mgr.y (mgr.24368) 996 : cluster 0 pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T03:48:05.908 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:05 smithi050 bash[29221]: cluster 2024-04-06T03:48:05.051618+0000 mon.a (mon.0) 1800 : cluster 0 osdmap e391: 8 total, 8 up, 6 in 2024-04-06T03:48:05.961 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:05 smithi153 bash[23852]: cluster 2024-04-06T03:48:04.586345+0000 mgr.y (mgr.24368) 996 : cluster 0 pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-04-06T03:48:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:05 smithi153 bash[23852]: cluster 2024-04-06T03:48:05.051618+0000 mon.a (mon.0) 1800 : cluster 0 osdmap e391: 8 total, 8 up, 6 in 2024-04-06T03:48:06.096 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-06T03:48:06.117 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-06T03:48:06.138 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-06T03:48:06.159 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-06T03:48:06.185 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-06T03:48:06.210 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-06T03:48:06.234 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-06T03:48:06.259 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-06T03:48:06.261 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-06T03:48:06.261 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-06T03:48:06.261 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-06T03:48:06.261 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-06T03:48:06.261 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-06T03:48:06.262 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-06T03:48:06.262 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-06T03:48:06.262 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-06T03:48:06.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:06 smithi050 bash[21467]: audit 2024-04-06T03:48:05.750724+0000 mon.a (mon.0) 1801 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:06.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:06 smithi050 bash[21467]: cluster 2024-04-06T03:48:06.053385+0000 mon.a (mon.0) 1802 : cluster 0 osdmap e392: 8 total, 8 up, 6 in 2024-04-06T03:48:06.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:06 smithi050 bash[21467]: audit 2024-04-06T03:48:06.075788+0000 mon.a (mon.0) 1803 : audit 0 from='client.? 172.21.15.50:0/3738094462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:48:06.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:06 smithi050 bash[29221]: audit 2024-04-06T03:48:05.750724+0000 mon.a (mon.0) 1801 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:06.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:06 smithi050 bash[29221]: cluster 2024-04-06T03:48:06.053385+0000 mon.a (mon.0) 1802 : cluster 0 osdmap e392: 8 total, 8 up, 6 in 2024-04-06T03:48:06.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:06 smithi050 bash[29221]: audit 2024-04-06T03:48:06.075788+0000 mon.a (mon.0) 1803 : audit 0 from='client.? 172.21.15.50:0/3738094462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:48:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:06 smithi153 bash[23852]: audit 2024-04-06T03:48:05.750724+0000 mon.a (mon.0) 1801 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:06 smithi153 bash[23852]: cluster 2024-04-06T03:48:06.053385+0000 mon.a (mon.0) 1802 : cluster 0 osdmap e392: 8 total, 8 up, 6 in 2024-04-06T03:48:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:06 smithi153 bash[23852]: audit 2024-04-06T03:48:06.075788+0000 mon.a (mon.0) 1803 : audit 0 from='client.? 172.21.15.50:0/3738094462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-06T03:48:07.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:07 smithi050 bash[21467]: cluster 2024-04-06T03:48:06.587076+0000 mgr.y (mgr.24368) 997 : cluster 0 pgmap v1086: 132 pgs: 1 remapped+peering, 131 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 638 B/s, 18 objects/s recovering 2024-04-06T03:48:07.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:07 smithi050 bash[21467]: cluster 2024-04-06T03:48:07.058295+0000 mon.a (mon.0) 1804 : cluster 0 osdmap e393: 8 total, 8 up, 6 in 2024-04-06T03:48:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:07 smithi050 bash[29221]: cluster 2024-04-06T03:48:06.587076+0000 mgr.y (mgr.24368) 997 : cluster 0 pgmap v1086: 132 pgs: 1 remapped+peering, 131 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 638 B/s, 18 objects/s recovering 2024-04-06T03:48:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:07 smithi050 bash[29221]: cluster 2024-04-06T03:48:07.058295+0000 mon.a (mon.0) 1804 : cluster 0 osdmap e393: 8 total, 8 up, 6 in 2024-04-06T03:48:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:07 smithi153 bash[23852]: cluster 2024-04-06T03:48:06.587076+0000 mgr.y (mgr.24368) 997 : cluster 0 pgmap v1086: 132 pgs: 1 remapped+peering, 131 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 638 B/s, 18 objects/s recovering 2024-04-06T03:48:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:07 smithi153 bash[23852]: cluster 2024-04-06T03:48:07.058295+0000 mon.a (mon.0) 1804 : cluster 0 osdmap e393: 8 total, 8 up, 6 in 2024-04-06T03:48:09.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:48:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:48:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:48:09.594 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:09.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:09 smithi050 bash[21467]: cluster 2024-04-06T03:48:08.588060+0000 mgr.y (mgr.24368) 998 : cluster 0 pgmap v1088: 132 pgs: 1 remapped+peering, 131 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 775 B/s, 22 objects/s recovering 2024-04-06T03:48:09.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:09 smithi050 bash[29221]: cluster 2024-04-06T03:48:08.588060+0000 mgr.y (mgr.24368) 998 : cluster 0 pgmap v1088: 132 pgs: 1 remapped+peering, 131 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 775 B/s, 22 objects/s recovering 2024-04-06T03:48:10.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:09 smithi153 bash[23852]: cluster 2024-04-06T03:48:08.588060+0000 mgr.y (mgr.24368) 998 : cluster 0 pgmap v1088: 132 pgs: 1 remapped+peering, 131 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 775 B/s, 22 objects/s recovering 2024-04-06T03:48:10.495 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-06T03:48:10.495 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-04-06T03:48:10.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd in 1 2024-04-06T03:48:10.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:48:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:48:11.710 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:48:12.003 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:11 smithi050 bash[21467]: audit 2024-04-06T03:48:10.394074+0000 mgr.y (mgr.24368) 999 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:12.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:11 smithi050 bash[21467]: cluster 2024-04-06T03:48:10.589367+0000 mgr.y (mgr.24368) 1000 : cluster 0 pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 97 KiB/s, 17 objects/s recovering 2024-04-06T03:48:12.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:11 smithi050 bash[21467]: audit 2024-04-06T03:48:11.252694+0000 mon.a (mon.0) 1805 : audit 1 from='client.? 172.21.15.50:0/1181211142' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:12.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:11 smithi050 bash[29221]: audit 2024-04-06T03:48:10.394074+0000 mgr.y (mgr.24368) 999 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:12.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:11 smithi050 bash[29221]: cluster 2024-04-06T03:48:10.589367+0000 mgr.y (mgr.24368) 1000 : cluster 0 pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 97 KiB/s, 17 objects/s recovering 2024-04-06T03:48:12.004 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:11 smithi050 bash[29221]: audit 2024-04-06T03:48:11.252694+0000 mon.a (mon.0) 1805 : audit 1 from='client.? 172.21.15.50:0/1181211142' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:11 smithi153 bash[23852]: audit 2024-04-06T03:48:10.394074+0000 mgr.y (mgr.24368) 999 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:11 smithi153 bash[23852]: cluster 2024-04-06T03:48:10.589367+0000 mgr.y (mgr.24368) 1000 : cluster 0 pgmap v1089: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 97 KiB/s, 17 objects/s recovering 2024-04-06T03:48:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:11 smithi153 bash[23852]: audit 2024-04-06T03:48:11.252694+0000 mon.a (mon.0) 1805 : audit 1 from='client.? 172.21.15.50:0/1181211142' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:12.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:12 smithi050 bash[21467]: cluster 2024-04-06T03:48:11.700848+0000 mon.a (mon.0) 1806 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:48:12.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:12 smithi050 bash[21467]: audit 2024-04-06T03:48:11.703181+0000 mon.a (mon.0) 1807 : audit 1 from='client.? 172.21.15.50:0/1181211142' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:48:12.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:12 smithi050 bash[21467]: cluster 2024-04-06T03:48:11.711512+0000 mon.a (mon.0) 1808 : cluster 0 osdmap e394: 8 total, 8 up, 6 in 2024-04-06T03:48:12.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:12 smithi050 bash[29221]: cluster 2024-04-06T03:48:11.700848+0000 mon.a (mon.0) 1806 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:48:12.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:12 smithi050 bash[29221]: audit 2024-04-06T03:48:11.703181+0000 mon.a (mon.0) 1807 : audit 1 from='client.? 172.21.15.50:0/1181211142' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:48:12.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:12 smithi050 bash[29221]: cluster 2024-04-06T03:48:11.711512+0000 mon.a (mon.0) 1808 : cluster 0 osdmap e394: 8 total, 8 up, 6 in 2024-04-06T03:48:13.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:12 smithi153 bash[23852]: cluster 2024-04-06T03:48:11.700848+0000 mon.a (mon.0) 1806 : cluster 3 Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-06T03:48:13.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:12 smithi153 bash[23852]: audit 2024-04-06T03:48:11.703181+0000 mon.a (mon.0) 1807 : audit 1 from='client.? 172.21.15.50:0/1181211142' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:48:13.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:12 smithi153 bash[23852]: cluster 2024-04-06T03:48:11.711512+0000 mon.a (mon.0) 1808 : cluster 0 osdmap e394: 8 total, 8 up, 6 in 2024-04-06T03:48:13.832 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:48:13.832 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:48:13.832 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-06T03:48:13.832 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:48:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:13 smithi050 bash[21467]: cluster 2024-04-06T03:48:12.590010+0000 mgr.y (mgr.24368) 1001 : cluster 0 pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 88 KiB/s, 4 objects/s recovering 2024-04-06T03:48:13.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:13 smithi050 bash[29221]: cluster 2024-04-06T03:48:12.590010+0000 mgr.y (mgr.24368) 1001 : cluster 0 pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 88 KiB/s, 4 objects/s recovering 2024-04-06T03:48:14.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:13 smithi153 bash[23852]: cluster 2024-04-06T03:48:12.590010+0000 mgr.y (mgr.24368) 1001 : cluster 0 pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 88 KiB/s, 4 objects/s recovering 2024-04-06T03:48:14.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_blocked_ops 2024-04-06T03:48:15.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:15 smithi050 bash[21467]: cluster 2024-04-06T03:48:14.590988+0000 mgr.y (mgr.24368) 1002 : cluster 0 pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 3 objects/s recovering 2024-04-06T03:48:15.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:15 smithi050 bash[29221]: cluster 2024-04-06T03:48:14.590988+0000 mgr.y (mgr.24368) 1002 : cluster 0 pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 3 objects/s recovering 2024-04-06T03:48:15.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:15 smithi153 bash[23852]: cluster 2024-04-06T03:48:14.590988+0000 mgr.y (mgr.24368) 1002 : cluster 0 pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 3 objects/s recovering 2024-04-06T03:48:15.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset noscrub 2024-04-06T03:48:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:16 smithi153 bash[23852]: cluster 2024-04-06T03:48:14.634862+0000 osd.5 (osd.5) 155 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:16 smithi153 bash[23852]: cluster 2024-04-06T03:48:14.635029+0000 osd.5 (osd.5) 156 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:16 smithi153 bash[23852]: cluster 2024-04-06T03:48:15.050457+0000 mon.a (mon.0) 1809 : cluster 0 osdmap e395: 8 total, 8 up, 6 in 2024-04-06T03:48:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:16 smithi153 bash[23852]: cluster 2024-04-06T03:48:15.139453+0000 osd.7 (osd.7) 631 : cluster 0 3.15 scrub starts 2024-04-06T03:48:16.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:16 smithi153 bash[23852]: cluster 2024-04-06T03:48:15.148520+0000 osd.7 (osd.7) 632 : cluster 0 3.15 scrub ok 2024-04-06T03:48:16.388 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:16.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[21467]: cluster 2024-04-06T03:48:14.634862+0000 osd.5 (osd.5) 155 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:16.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[21467]: cluster 2024-04-06T03:48:14.635029+0000 osd.5 (osd.5) 156 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:16.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[21467]: cluster 2024-04-06T03:48:15.050457+0000 mon.a (mon.0) 1809 : cluster 0 osdmap e395: 8 total, 8 up, 6 in 2024-04-06T03:48:16.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[21467]: cluster 2024-04-06T03:48:15.139453+0000 osd.7 (osd.7) 631 : cluster 0 3.15 scrub starts 2024-04-06T03:48:16.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[21467]: cluster 2024-04-06T03:48:15.148520+0000 osd.7 (osd.7) 632 : cluster 0 3.15 scrub ok 2024-04-06T03:48:16.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[29221]: cluster 2024-04-06T03:48:14.634862+0000 osd.5 (osd.5) 155 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:16.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[29221]: cluster 2024-04-06T03:48:14.635029+0000 osd.5 (osd.5) 156 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:16.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[29221]: cluster 2024-04-06T03:48:15.050457+0000 mon.a (mon.0) 1809 : cluster 0 osdmap e395: 8 total, 8 up, 6 in 2024-04-06T03:48:16.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[29221]: cluster 2024-04-06T03:48:15.139453+0000 osd.7 (osd.7) 631 : cluster 0 3.15 scrub starts 2024-04-06T03:48:16.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:16 smithi050 bash[29221]: cluster 2024-04-06T03:48:15.148520+0000 osd.7 (osd.7) 632 : cluster 0 3.15 scrub ok 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[29221]: cluster 2024-04-06T03:48:15.589760+0000 osd.2 (osd.2) 519 : cluster 0 6.a scrub starts 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[29221]: cluster 2024-04-06T03:48:15.592187+0000 osd.2 (osd.2) 520 : cluster 0 6.a scrub ok 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[29221]: cluster 2024-04-06T03:48:15.917666+0000 osd.3 (osd.3) 617 : cluster 0 6.13 scrub starts 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[29221]: cluster 2024-04-06T03:48:16.032214+0000 osd.3 (osd.3) 618 : cluster 0 6.13 scrub ok 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[29221]: cluster 2024-04-06T03:48:16.592113+0000 mgr.y (mgr.24368) 1003 : cluster 0 pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[21467]: cluster 2024-04-06T03:48:15.589760+0000 osd.2 (osd.2) 519 : cluster 0 6.a scrub starts 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[21467]: cluster 2024-04-06T03:48:15.592187+0000 osd.2 (osd.2) 520 : cluster 0 6.a scrub ok 2024-04-06T03:48:17.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[21467]: cluster 2024-04-06T03:48:15.917666+0000 osd.3 (osd.3) 617 : cluster 0 6.13 scrub starts 2024-04-06T03:48:17.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[21467]: cluster 2024-04-06T03:48:16.032214+0000 osd.3 (osd.3) 618 : cluster 0 6.13 scrub ok 2024-04-06T03:48:17.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:17 smithi050 bash[21467]: cluster 2024-04-06T03:48:16.592113+0000 mgr.y (mgr.24368) 1003 : cluster 0 pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:48:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:17 smithi153 bash[23852]: cluster 2024-04-06T03:48:15.589760+0000 osd.2 (osd.2) 519 : cluster 0 6.a scrub starts 2024-04-06T03:48:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:17 smithi153 bash[23852]: cluster 2024-04-06T03:48:15.592187+0000 osd.2 (osd.2) 520 : cluster 0 6.a scrub ok 2024-04-06T03:48:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:17 smithi153 bash[23852]: cluster 2024-04-06T03:48:15.917666+0000 osd.3 (osd.3) 617 : cluster 0 6.13 scrub starts 2024-04-06T03:48:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:17 smithi153 bash[23852]: cluster 2024-04-06T03:48:16.032214+0000 osd.3 (osd.3) 618 : cluster 0 6.13 scrub ok 2024-04-06T03:48:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:17 smithi153 bash[23852]: cluster 2024-04-06T03:48:16.592113+0000 mgr.y (mgr.24368) 1003 : cluster 0 pgmap v1094: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 0 objects/s recovering 2024-04-06T03:48:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:18 smithi050 bash[21467]: cluster 2024-04-06T03:48:16.612056+0000 osd.2 (osd.2) 521 : cluster 0 5.2 scrub starts 2024-04-06T03:48:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:18 smithi050 bash[21467]: cluster 2024-04-06T03:48:16.614609+0000 osd.2 (osd.2) 522 : cluster 0 5.2 scrub ok 2024-04-06T03:48:18.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:18 smithi050 bash[21467]: cluster 2024-04-06T03:48:16.902528+0000 osd.3 (osd.3) 619 : cluster 0 6.1b scrub starts 2024-04-06T03:48:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:18 smithi050 bash[29221]: cluster 2024-04-06T03:48:16.612056+0000 osd.2 (osd.2) 521 : cluster 0 5.2 scrub starts 2024-04-06T03:48:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:18 smithi050 bash[29221]: cluster 2024-04-06T03:48:16.614609+0000 osd.2 (osd.2) 522 : cluster 0 5.2 scrub ok 2024-04-06T03:48:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:18 smithi050 bash[29221]: cluster 2024-04-06T03:48:16.902528+0000 osd.3 (osd.3) 619 : cluster 0 6.1b scrub starts 2024-04-06T03:48:18.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:18 smithi153 bash[23852]: cluster 2024-04-06T03:48:16.612056+0000 osd.2 (osd.2) 521 : cluster 0 5.2 scrub starts 2024-04-06T03:48:18.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:18 smithi153 bash[23852]: cluster 2024-04-06T03:48:16.614609+0000 osd.2 (osd.2) 522 : cluster 0 5.2 scrub ok 2024-04-06T03:48:18.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:18 smithi153 bash[23852]: cluster 2024-04-06T03:48:16.902528+0000 osd.3 (osd.3) 619 : cluster 0 6.1b scrub starts 2024-04-06T03:48:19.463 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:19.464 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:19.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[21467]: cluster 2024-04-06T03:48:17.091826+0000 osd.3 (osd.3) 620 : cluster 0 6.1b scrub ok 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[21467]: cluster 2024-04-06T03:48:17.908901+0000 osd.3 (osd.3) 621 : cluster 0 3.c scrub starts 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[21467]: cluster 2024-04-06T03:48:17.913473+0000 osd.3 (osd.3) 622 : cluster 0 3.c scrub ok 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[21467]: cluster 2024-04-06T03:48:18.592777+0000 mgr.y (mgr.24368) 1004 : cluster 0 pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:48:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[29221]: cluster 2024-04-06T03:48:17.091826+0000 osd.3 (osd.3) 620 : cluster 0 6.1b scrub ok 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[29221]: cluster 2024-04-06T03:48:17.908901+0000 osd.3 (osd.3) 621 : cluster 0 3.c scrub starts 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[29221]: cluster 2024-04-06T03:48:17.913473+0000 osd.3 (osd.3) 622 : cluster 0 3.c scrub ok 2024-04-06T03:48:19.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:19 smithi050 bash[29221]: cluster 2024-04-06T03:48:18.592777+0000 mgr.y (mgr.24368) 1004 : cluster 0 pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:19 smithi153 bash[23852]: cluster 2024-04-06T03:48:17.091826+0000 osd.3 (osd.3) 620 : cluster 0 6.1b scrub ok 2024-04-06T03:48:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:19 smithi153 bash[23852]: cluster 2024-04-06T03:48:17.908901+0000 osd.3 (osd.3) 621 : cluster 0 3.c scrub starts 2024-04-06T03:48:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:19 smithi153 bash[23852]: cluster 2024-04-06T03:48:17.913473+0000 osd.3 (osd.3) 622 : cluster 0 3.c scrub ok 2024-04-06T03:48:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:19 smithi153 bash[23852]: cluster 2024-04-06T03:48:18.592777+0000 mgr.y (mgr.24368) 1004 : cluster 0 pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:20.272 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.1. 2024-04-06T03:48:20.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:20 smithi050 bash[29221]: audit 2024-04-06T03:48:20.199393+0000 mon.c (mon.2) 120 : audit 0 from='client.? 172.21.15.50:0/3735530950' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-06T03:48:20.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:20 smithi050 bash[29221]: audit 2024-04-06T03:48:20.200068+0000 mon.a (mon.0) 1810 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-06T03:48:20.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:20 smithi050 bash[21467]: debug 2024-04-06T03:48:20.193+0000 7f702b621700 -1 mon.a@0(leader).osd e395 definitely_dead 0 2024-04-06T03:48:20.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:20 smithi050 bash[21467]: audit 2024-04-06T03:48:20.199393+0000 mon.c (mon.2) 120 : audit 0 from='client.? 172.21.15.50:0/3735530950' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-06T03:48:20.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:20 smithi050 bash[21467]: audit 2024-04-06T03:48:20.200068+0000 mon.a (mon.0) 1810 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-06T03:48:20.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:48:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:48:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:20 smithi153 bash[23852]: audit 2024-04-06T03:48:20.199393+0000 mon.c (mon.2) 120 : audit 0 from='client.? 172.21.15.50:0/3735530950' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-06T03:48:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:20 smithi153 bash[23852]: audit 2024-04-06T03:48:20.200068+0000 mon.a (mon.0) 1810 : audit 0 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-04-06T03:48:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:21 smithi153 bash[23852]: audit 2024-04-06T03:48:20.253113+0000 mon.a (mon.0) 1811 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-06T03:48:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:21 smithi153 bash[23852]: cluster 2024-04-06T03:48:20.264787+0000 mon.a (mon.0) 1812 : cluster 0 osdmap e396: 8 total, 8 up, 7 in 2024-04-06T03:48:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:21 smithi153 bash[23852]: audit 2024-04-06T03:48:20.397742+0000 mgr.y (mgr.24368) 1005 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:21 smithi153 bash[23852]: cluster 2024-04-06T03:48:20.593676+0000 mgr.y (mgr.24368) 1006 : cluster 0 pgmap v1097: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:21 smithi153 bash[23852]: audit 2024-04-06T03:48:20.773666+0000 mon.a (mon.0) 1813 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:21.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:21 smithi153 bash[23852]: cluster 2024-04-06T03:48:21.268756+0000 mon.a (mon.0) 1814 : cluster 0 osdmap e397: 8 total, 8 up, 7 in 2024-04-06T03:48:21.765 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-04-06T03:48:21.765 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-06T03:48:21.765 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-06T03:48:21.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd in 0 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[21467]: audit 2024-04-06T03:48:20.253113+0000 mon.a (mon.0) 1811 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[21467]: cluster 2024-04-06T03:48:20.264787+0000 mon.a (mon.0) 1812 : cluster 0 osdmap e396: 8 total, 8 up, 7 in 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[21467]: audit 2024-04-06T03:48:20.397742+0000 mgr.y (mgr.24368) 1005 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[21467]: cluster 2024-04-06T03:48:20.593676+0000 mgr.y (mgr.24368) 1006 : cluster 0 pgmap v1097: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[21467]: audit 2024-04-06T03:48:20.773666+0000 mon.a (mon.0) 1813 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[21467]: cluster 2024-04-06T03:48:21.268756+0000 mon.a (mon.0) 1814 : cluster 0 osdmap e397: 8 total, 8 up, 7 in 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[29221]: audit 2024-04-06T03:48:20.253113+0000 mon.a (mon.0) 1811 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[29221]: cluster 2024-04-06T03:48:20.264787+0000 mon.a (mon.0) 1812 : cluster 0 osdmap e396: 8 total, 8 up, 7 in 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[29221]: audit 2024-04-06T03:48:20.397742+0000 mgr.y (mgr.24368) 1005 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:21.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[29221]: cluster 2024-04-06T03:48:20.593676+0000 mgr.y (mgr.24368) 1006 : cluster 0 pgmap v1097: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:21.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[29221]: audit 2024-04-06T03:48:20.773666+0000 mon.a (mon.0) 1813 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:21.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:21 smithi050 bash[29221]: cluster 2024-04-06T03:48:21.268756+0000 mon.a (mon.0) 1814 : cluster 0 osdmap e397: 8 total, 8 up, 7 in 2024-04-06T03:48:22.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:22 smithi153 bash[23852]: cluster 2024-04-06T03:48:20.594800+0000 osd.2 (osd.2) 523 : cluster 0 2.16 scrub starts 2024-04-06T03:48:22.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:22 smithi153 bash[23852]: cluster 2024-04-06T03:48:20.597480+0000 osd.2 (osd.2) 524 : cluster 0 2.16 scrub ok 2024-04-06T03:48:22.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:22 smithi153 bash[23852]: cluster 2024-04-06T03:48:20.743035+0000 osd.4 (osd.4) 601 : cluster 0 2.f scrub starts 2024-04-06T03:48:22.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:22 smithi153 bash[23852]: cluster 2024-04-06T03:48:20.746485+0000 osd.4 (osd.4) 602 : cluster 0 2.f scrub ok 2024-04-06T03:48:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[21467]: cluster 2024-04-06T03:48:20.594800+0000 osd.2 (osd.2) 523 : cluster 0 2.16 scrub starts 2024-04-06T03:48:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[21467]: cluster 2024-04-06T03:48:20.597480+0000 osd.2 (osd.2) 524 : cluster 0 2.16 scrub ok 2024-04-06T03:48:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[21467]: cluster 2024-04-06T03:48:20.743035+0000 osd.4 (osd.4) 601 : cluster 0 2.f scrub starts 2024-04-06T03:48:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[21467]: cluster 2024-04-06T03:48:20.746485+0000 osd.4 (osd.4) 602 : cluster 0 2.f scrub ok 2024-04-06T03:48:22.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[29221]: cluster 2024-04-06T03:48:20.594800+0000 osd.2 (osd.2) 523 : cluster 0 2.16 scrub starts 2024-04-06T03:48:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[29221]: cluster 2024-04-06T03:48:20.597480+0000 osd.2 (osd.2) 524 : cluster 0 2.16 scrub ok 2024-04-06T03:48:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[29221]: cluster 2024-04-06T03:48:20.743035+0000 osd.4 (osd.4) 601 : cluster 0 2.f scrub starts 2024-04-06T03:48:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:22 smithi050 bash[29221]: cluster 2024-04-06T03:48:20.746485+0000 osd.4 (osd.4) 602 : cluster 0 2.f scrub ok 2024-04-06T03:48:23.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[21467]: cluster 2024-04-06T03:48:22.311293+0000 mon.a (mon.0) 1815 : cluster 0 osdmap e398: 8 total, 8 up, 7 in 2024-04-06T03:48:23.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[21467]: cluster 2024-04-06T03:48:22.594375+0000 mgr.y (mgr.24368) 1007 : cluster 0 pgmap v1100: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:48:23.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[21467]: cluster 2024-04-06T03:48:22.952386+0000 osd.3 (osd.3) 623 : cluster 0 2.a scrub starts 2024-04-06T03:48:23.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[21467]: cluster 2024-04-06T03:48:22.954886+0000 osd.3 (osd.3) 624 : cluster 0 2.a scrub ok 2024-04-06T03:48:23.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[29221]: cluster 2024-04-06T03:48:22.311293+0000 mon.a (mon.0) 1815 : cluster 0 osdmap e398: 8 total, 8 up, 7 in 2024-04-06T03:48:23.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[29221]: cluster 2024-04-06T03:48:22.594375+0000 mgr.y (mgr.24368) 1007 : cluster 0 pgmap v1100: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:48:23.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[29221]: cluster 2024-04-06T03:48:22.952386+0000 osd.3 (osd.3) 623 : cluster 0 2.a scrub starts 2024-04-06T03:48:23.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:23 smithi050 bash[29221]: cluster 2024-04-06T03:48:22.954886+0000 osd.3 (osd.3) 624 : cluster 0 2.a scrub ok 2024-04-06T03:48:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:23 smithi153 bash[23852]: cluster 2024-04-06T03:48:22.311293+0000 mon.a (mon.0) 1815 : cluster 0 osdmap e398: 8 total, 8 up, 7 in 2024-04-06T03:48:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:23 smithi153 bash[23852]: cluster 2024-04-06T03:48:22.594375+0000 mgr.y (mgr.24368) 1007 : cluster 0 pgmap v1100: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:48:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:23 smithi153 bash[23852]: cluster 2024-04-06T03:48:22.952386+0000 osd.3 (osd.3) 623 : cluster 0 2.a scrub starts 2024-04-06T03:48:23.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:23 smithi153 bash[23852]: cluster 2024-04-06T03:48:22.954886+0000 osd.3 (osd.3) 624 : cluster 0 2.a scrub ok 2024-04-06T03:48:24.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:22.752420+0000 osd.4 (osd.4) 603 : cluster 0 3.a scrub starts 2024-04-06T03:48:24.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:22.757847+0000 osd.4 (osd.4) 604 : cluster 0 3.a scrub ok 2024-04-06T03:48:24.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.196348+0000 osd.7 (osd.7) 633 : cluster 0 3.d scrub starts 2024-04-06T03:48:24.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.200761+0000 osd.7 (osd.7) 634 : cluster 0 3.d scrub ok 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.324896+0000 mon.a (mon.0) 1816 : cluster 0 osdmap e399: 8 total, 8 up, 7 in 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.700523+0000 osd.1 (osd.1) 403 : cluster 0 4.2 scrub starts 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.703802+0000 osd.1 (osd.1) 404 : cluster 0 4.2 scrub ok 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:22.752420+0000 osd.4 (osd.4) 603 : cluster 0 3.a scrub starts 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:22.757847+0000 osd.4 (osd.4) 604 : cluster 0 3.a scrub ok 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.196348+0000 osd.7 (osd.7) 633 : cluster 0 3.d scrub starts 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.200761+0000 osd.7 (osd.7) 634 : cluster 0 3.d scrub ok 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.324896+0000 mon.a (mon.0) 1816 : cluster 0 osdmap e399: 8 total, 8 up, 7 in 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.700523+0000 osd.1 (osd.1) 403 : cluster 0 4.2 scrub starts 2024-04-06T03:48:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:24 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.703802+0000 osd.1 (osd.1) 404 : cluster 0 4.2 scrub ok 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:22.752420+0000 osd.4 (osd.4) 603 : cluster 0 3.a scrub starts 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:22.757847+0000 osd.4 (osd.4) 604 : cluster 0 3.a scrub ok 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.196348+0000 osd.7 (osd.7) 633 : cluster 0 3.d scrub starts 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.200761+0000 osd.7 (osd.7) 634 : cluster 0 3.d scrub ok 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.324896+0000 mon.a (mon.0) 1816 : cluster 0 osdmap e399: 8 total, 8 up, 7 in 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.700523+0000 osd.1 (osd.1) 403 : cluster 0 4.2 scrub starts 2024-04-06T03:48:24.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:24 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.703802+0000 osd.1 (osd.1) 404 : cluster 0 4.2 scrub ok 2024-04-06T03:48:25.657 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:48:25.657 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:48:25.657 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:48:25.657 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:48:25.657 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.676888+0000 osd.4 (osd.4) 605 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.677032+0000 osd.4 (osd.4) 606 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.765456+0000 osd.4 (osd.4) 607 : cluster 0 6.1d scrub starts 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:23.879388+0000 osd.4 (osd.4) 608 : cluster 0 6.1d scrub ok 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.332117+0000 mon.a (mon.0) 1817 : cluster 0 osdmap e400: 8 total, 8 up, 7 in 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.594956+0000 mgr.y (mgr.24368) 1008 : cluster 0 pgmap v1103: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 207 B/s, 9 objects/s recovering 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.687471+0000 osd.1 (osd.1) 405 : cluster 0 5.1d scrub starts 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.689962+0000 osd.1 (osd.1) 406 : cluster 0 5.1d scrub ok 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.676888+0000 osd.4 (osd.4) 605 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:25.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.677032+0000 osd.4 (osd.4) 606 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:25.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.765456+0000 osd.4 (osd.4) 607 : cluster 0 6.1d scrub starts 2024-04-06T03:48:25.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:23.879388+0000 osd.4 (osd.4) 608 : cluster 0 6.1d scrub ok 2024-04-06T03:48:25.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.332117+0000 mon.a (mon.0) 1817 : cluster 0 osdmap e400: 8 total, 8 up, 7 in 2024-04-06T03:48:25.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.594956+0000 mgr.y (mgr.24368) 1008 : cluster 0 pgmap v1103: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 207 B/s, 9 objects/s recovering 2024-04-06T03:48:25.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.687471+0000 osd.1 (osd.1) 405 : cluster 0 5.1d scrub starts 2024-04-06T03:48:25.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:25 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.689962+0000 osd.1 (osd.1) 406 : cluster 0 5.1d scrub ok 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.676888+0000 osd.4 (osd.4) 605 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.677032+0000 osd.4 (osd.4) 606 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.765456+0000 osd.4 (osd.4) 607 : cluster 0 6.1d scrub starts 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:23.879388+0000 osd.4 (osd.4) 608 : cluster 0 6.1d scrub ok 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.332117+0000 mon.a (mon.0) 1817 : cluster 0 osdmap e400: 8 total, 8 up, 7 in 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.594956+0000 mgr.y (mgr.24368) 1008 : cluster 0 pgmap v1103: 132 pgs: 7 peering, 125 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 207 B/s, 9 objects/s recovering 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.687471+0000 osd.1 (osd.1) 405 : cluster 0 5.1d scrub starts 2024-04-06T03:48:25.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:25 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.689962+0000 osd.1 (osd.1) 406 : cluster 0 5.1d scrub ok 2024-04-06T03:48:26.362 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-06T03:48:26.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.1 dump_historic_ops 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.588329+0000 osd.2 (osd.2) 525 : cluster 0 6.14 scrub starts 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.593287+0000 osd.2 (osd.2) 526 : cluster 0 6.14 scrub ok 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.806143+0000 osd.4 (osd.4) 609 : cluster 0 3.1c scrub starts 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: cluster 2024-04-06T03:48:24.810527+0000 osd.4 (osd.4) 610 : cluster 0 3.1c scrub ok 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: audit 2024-04-06T03:48:25.924291+0000 mon.c (mon.2) 121 : audit 1 from='client.? 172.21.15.50:0/3313462206' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: audit 2024-04-06T03:48:25.925103+0000 mon.a (mon.0) 1818 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:26.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: cluster 2024-04-06T03:48:25.992221+0000 osd.3 (osd.3) 625 : cluster 0 6.11 scrub starts 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[21467]: cluster 2024-04-06T03:48:25.994597+0000 osd.3 (osd.3) 626 : cluster 0 6.11 scrub ok 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.588329+0000 osd.2 (osd.2) 525 : cluster 0 6.14 scrub starts 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.593287+0000 osd.2 (osd.2) 526 : cluster 0 6.14 scrub ok 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.806143+0000 osd.4 (osd.4) 609 : cluster 0 3.1c scrub starts 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: cluster 2024-04-06T03:48:24.810527+0000 osd.4 (osd.4) 610 : cluster 0 3.1c scrub ok 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: audit 2024-04-06T03:48:25.924291+0000 mon.c (mon.2) 121 : audit 1 from='client.? 172.21.15.50:0/3313462206' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: audit 2024-04-06T03:48:25.925103+0000 mon.a (mon.0) 1818 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: cluster 2024-04-06T03:48:25.992221+0000 osd.3 (osd.3) 625 : cluster 0 6.11 scrub starts 2024-04-06T03:48:26.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:26 smithi050 bash[29221]: cluster 2024-04-06T03:48:25.994597+0000 osd.3 (osd.3) 626 : cluster 0 6.11 scrub ok 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.588329+0000 osd.2 (osd.2) 525 : cluster 0 6.14 scrub starts 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.593287+0000 osd.2 (osd.2) 526 : cluster 0 6.14 scrub ok 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.806143+0000 osd.4 (osd.4) 609 : cluster 0 3.1c scrub starts 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: cluster 2024-04-06T03:48:24.810527+0000 osd.4 (osd.4) 610 : cluster 0 3.1c scrub ok 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: audit 2024-04-06T03:48:25.924291+0000 mon.c (mon.2) 121 : audit 1 from='client.? 172.21.15.50:0/3313462206' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: audit 2024-04-06T03:48:25.925103+0000 mon.a (mon.0) 1818 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: cluster 2024-04-06T03:48:25.992221+0000 osd.3 (osd.3) 625 : cluster 0 6.11 scrub starts 2024-04-06T03:48:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:26 smithi153 bash[23852]: cluster 2024-04-06T03:48:25.994597+0000 osd.3 (osd.3) 626 : cluster 0 6.11 scrub ok 2024-04-06T03:48:27.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd unset nodeep-scrub 2024-04-06T03:48:27.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:25.556628+0000 osd.2 (osd.2) 527 : cluster 0 3.1f scrub starts 2024-04-06T03:48:27.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:25.559595+0000 osd.2 (osd.2) 528 : cluster 0 3.1f scrub ok 2024-04-06T03:48:27.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:25.793667+0000 osd.4 (osd.4) 611 : cluster 0 6.17 scrub starts 2024-04-06T03:48:27.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:26.005350+0000 osd.4 (osd.4) 612 : cluster 0 6.17 scrub ok 2024-04-06T03:48:27.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: audit 2024-04-06T03:48:26.347098+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:26.354787+0000 mon.a (mon.0) 1820 : cluster 0 osdmap e401: 8 total, 8 up, 7 in 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:26.595365+0000 mgr.y (mgr.24368) 1009 : cluster 0 pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 954 B/s rd, 0 op/s; 135 KiB/s, 28 objects/s recovering 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:26.625067+0000 osd.1 (osd.1) 407 : cluster 0 6.4 scrub starts 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[29221]: cluster 2024-04-06T03:48:26.628589+0000 osd.1 (osd.1) 408 : cluster 0 6.4 scrub ok 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:25.556628+0000 osd.2 (osd.2) 527 : cluster 0 3.1f scrub starts 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:25.559595+0000 osd.2 (osd.2) 528 : cluster 0 3.1f scrub ok 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:25.793667+0000 osd.4 (osd.4) 611 : cluster 0 6.17 scrub starts 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:26.005350+0000 osd.4 (osd.4) 612 : cluster 0 6.17 scrub ok 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: audit 2024-04-06T03:48:26.347098+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:26.354787+0000 mon.a (mon.0) 1820 : cluster 0 osdmap e401: 8 total, 8 up, 7 in 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:26.595365+0000 mgr.y (mgr.24368) 1009 : cluster 0 pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 954 B/s rd, 0 op/s; 135 KiB/s, 28 objects/s recovering 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:26.625067+0000 osd.1 (osd.1) 407 : cluster 0 6.4 scrub starts 2024-04-06T03:48:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:27 smithi050 bash[21467]: cluster 2024-04-06T03:48:26.628589+0000 osd.1 (osd.1) 408 : cluster 0 6.4 scrub ok 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:25.556628+0000 osd.2 (osd.2) 527 : cluster 0 3.1f scrub starts 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:25.559595+0000 osd.2 (osd.2) 528 : cluster 0 3.1f scrub ok 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:25.793667+0000 osd.4 (osd.4) 611 : cluster 0 6.17 scrub starts 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:26.005350+0000 osd.4 (osd.4) 612 : cluster 0 6.17 scrub ok 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: audit 2024-04-06T03:48:26.347098+0000 mon.a (mon.0) 1819 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:26.354787+0000 mon.a (mon.0) 1820 : cluster 0 osdmap e401: 8 total, 8 up, 7 in 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:26.595365+0000 mgr.y (mgr.24368) 1009 : cluster 0 pgmap v1105: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 954 B/s rd, 0 op/s; 135 KiB/s, 28 objects/s recovering 2024-04-06T03:48:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:26.625067+0000 osd.1 (osd.1) 407 : cluster 0 6.4 scrub starts 2024-04-06T03:48:27.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:27 smithi153 bash[23852]: cluster 2024-04-06T03:48:26.628589+0000 osd.1 (osd.1) 408 : cluster 0 6.4 scrub ok 2024-04-06T03:48:27.812 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:28 smithi050 bash[21467]: audit 2024-04-06T03:48:28.105587+0000 mon.a (mon.0) 1821 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:48:28.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:28 smithi050 bash[29221]: audit 2024-04-06T03:48:28.105587+0000 mon.a (mon.0) 1821 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:48:28.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:28 smithi153 bash[23852]: audit 2024-04-06T03:48:28.105587+0000 mon.a (mon.0) 1821 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:48:29.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:48:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:48:29.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: cluster 2024-04-06T03:48:28.526824+0000 osd.2 (osd.2) 529 : cluster 0 5.a scrub starts 2024-04-06T03:48:29.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: cluster 2024-04-06T03:48:28.528623+0000 osd.2 (osd.2) 530 : cluster 0 5.a scrub ok 2024-04-06T03:48:29.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: cluster 2024-04-06T03:48:28.595952+0000 mgr.y (mgr.24368) 1010 : cluster 0 pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 97 KiB/s, 20 objects/s recovering 2024-04-06T03:48:29.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: audit 2024-04-06T03:48:28.832620+0000 mon.a (mon.0) 1822 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: audit 2024-04-06T03:48:28.834208+0000 mon.a (mon.0) 1823 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: audit 2024-04-06T03:48:28.844259+0000 mon.a (mon.0) 1824 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: cluster 2024-04-06T03:48:28.982397+0000 osd.3 (osd.3) 627 : cluster 0 2.14 scrub starts 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[29221]: cluster 2024-04-06T03:48:28.984090+0000 osd.3 (osd.3) 628 : cluster 0 2.14 scrub ok 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: cluster 2024-04-06T03:48:28.526824+0000 osd.2 (osd.2) 529 : cluster 0 5.a scrub starts 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: cluster 2024-04-06T03:48:28.528623+0000 osd.2 (osd.2) 530 : cluster 0 5.a scrub ok 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: cluster 2024-04-06T03:48:28.595952+0000 mgr.y (mgr.24368) 1010 : cluster 0 pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 97 KiB/s, 20 objects/s recovering 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: audit 2024-04-06T03:48:28.832620+0000 mon.a (mon.0) 1822 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: audit 2024-04-06T03:48:28.834208+0000 mon.a (mon.0) 1823 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: audit 2024-04-06T03:48:28.844259+0000 mon.a (mon.0) 1824 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: cluster 2024-04-06T03:48:28.982397+0000 osd.3 (osd.3) 627 : cluster 0 2.14 scrub starts 2024-04-06T03:48:29.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:29 smithi050 bash[21467]: cluster 2024-04-06T03:48:28.984090+0000 osd.3 (osd.3) 628 : cluster 0 2.14 scrub ok 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: cluster 2024-04-06T03:48:28.526824+0000 osd.2 (osd.2) 529 : cluster 0 5.a scrub starts 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: cluster 2024-04-06T03:48:28.528623+0000 osd.2 (osd.2) 530 : cluster 0 5.a scrub ok 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: cluster 2024-04-06T03:48:28.595952+0000 mgr.y (mgr.24368) 1010 : cluster 0 pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 97 KiB/s, 20 objects/s recovering 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: audit 2024-04-06T03:48:28.832620+0000 mon.a (mon.0) 1822 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: audit 2024-04-06T03:48:28.834208+0000 mon.a (mon.0) 1823 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: audit 2024-04-06T03:48:28.844259+0000 mon.a (mon.0) 1824 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:48:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: cluster 2024-04-06T03:48:28.982397+0000 osd.3 (osd.3) 627 : cluster 0 2.14 scrub starts 2024-04-06T03:48:29.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:29 smithi153 bash[23852]: cluster 2024-04-06T03:48:28.984090+0000 osd.3 (osd.3) 628 : cluster 0 2.14 scrub ok 2024-04-06T03:48:30.798 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:48:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:48:31.407 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.0. 2024-04-06T03:48:31.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.182888+0000 osd.6 (osd.6) 113 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.183053+0000 osd.6 (osd.6) 114 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.374247+0000 mon.a (mon.0) 1825 : cluster 0 osdmap e402: 8 total, 8 up, 7 in 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: audit 2024-04-06T03:48:30.400788+0000 mgr.y (mgr.24368) 1011 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.531095+0000 osd.2 (osd.2) 531 : cluster 0 5.6 scrub starts 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.533232+0000 osd.2 (osd.2) 532 : cluster 0 5.6 scrub ok 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.596904+0000 mgr.y (mgr.24368) 1012 : cluster 0 pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 816 B/s rd, 0 op/s; 92 KiB/s, 13 objects/s recovering 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.623682+0000 osd.1 (osd.1) 409 : cluster 0 2.d scrub starts 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: cluster 2024-04-06T03:48:30.626220+0000 osd.1 (osd.1) 410 : cluster 0 2.d scrub ok 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[29221]: audit 2024-04-06T03:48:31.286551+0000 mon.a (mon.0) 1826 : audit 0 from='client.? 172.21.15.50:0/572885770' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: debug 2024-04-06T03:48:31.280+0000 7f702b621700 -1 mon.a@0(leader).osd e402 definitely_dead 0 2024-04-06T03:48:31.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.182888+0000 osd.6 (osd.6) 113 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.183053+0000 osd.6 (osd.6) 114 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.374247+0000 mon.a (mon.0) 1825 : cluster 0 osdmap e402: 8 total, 8 up, 7 in 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: audit 2024-04-06T03:48:30.400788+0000 mgr.y (mgr.24368) 1011 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.531095+0000 osd.2 (osd.2) 531 : cluster 0 5.6 scrub starts 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.533232+0000 osd.2 (osd.2) 532 : cluster 0 5.6 scrub ok 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.596904+0000 mgr.y (mgr.24368) 1012 : cluster 0 pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 816 B/s rd, 0 op/s; 92 KiB/s, 13 objects/s recovering 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.623682+0000 osd.1 (osd.1) 409 : cluster 0 2.d scrub starts 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: cluster 2024-04-06T03:48:30.626220+0000 osd.1 (osd.1) 410 : cluster 0 2.d scrub ok 2024-04-06T03:48:31.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:31 smithi050 bash[21467]: audit 2024-04-06T03:48:31.286551+0000 mon.a (mon.0) 1826 : audit 0 from='client.? 172.21.15.50:0/572885770' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-06T03:48:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.182888+0000 osd.6 (osd.6) 113 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.183053+0000 osd.6 (osd.6) 114 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.374247+0000 mon.a (mon.0) 1825 : cluster 0 osdmap e402: 8 total, 8 up, 7 in 2024-04-06T03:48:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: audit 2024-04-06T03:48:30.400788+0000 mgr.y (mgr.24368) 1011 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.531095+0000 osd.2 (osd.2) 531 : cluster 0 5.6 scrub starts 2024-04-06T03:48:31.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.533232+0000 osd.2 (osd.2) 532 : cluster 0 5.6 scrub ok 2024-04-06T03:48:31.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.596904+0000 mgr.y (mgr.24368) 1012 : cluster 0 pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 816 B/s rd, 0 op/s; 92 KiB/s, 13 objects/s recovering 2024-04-06T03:48:31.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.623682+0000 osd.1 (osd.1) 409 : cluster 0 2.d scrub starts 2024-04-06T03:48:31.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: cluster 2024-04-06T03:48:30.626220+0000 osd.1 (osd.1) 410 : cluster 0 2.d scrub ok 2024-04-06T03:48:31.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:31 smithi153 bash[23852]: audit 2024-04-06T03:48:31.286551+0000 mon.a (mon.0) 1826 : audit 0 from='client.? 172.21.15.50:0/572885770' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-04-06T03:48:32.340 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-06T03:48:32.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-06T03:48:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:32 smithi050 bash[21467]: audit 2024-04-06T03:48:31.382479+0000 mon.a (mon.0) 1827 : audit 1 from='client.? 172.21.15.50:0/572885770' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-06T03:48:32.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:32 smithi050 bash[21467]: cluster 2024-04-06T03:48:31.412040+0000 mon.a (mon.0) 1828 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-04-06T03:48:32.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:32 smithi050 bash[29221]: audit 2024-04-06T03:48:31.382479+0000 mon.a (mon.0) 1827 : audit 1 from='client.? 172.21.15.50:0/572885770' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-06T03:48:32.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:32 smithi050 bash[29221]: cluster 2024-04-06T03:48:31.412040+0000 mon.a (mon.0) 1828 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-04-06T03:48:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:32 smithi153 bash[23852]: audit 2024-04-06T03:48:31.382479+0000 mon.a (mon.0) 1827 : audit 1 from='client.? 172.21.15.50:0/572885770' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-04-06T03:48:32.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:32 smithi153 bash[23852]: cluster 2024-04-06T03:48:31.412040+0000 mon.a (mon.0) 1828 : cluster 0 osdmap e403: 8 total, 8 up, 8 in 2024-04-06T03:48:32.937 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:32.938 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:33.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:33 smithi050 bash[29221]: cluster 2024-04-06T03:48:32.389136+0000 mon.a (mon.0) 1829 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-04-06T03:48:33.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:33 smithi050 bash[29221]: cluster 2024-04-06T03:48:32.597471+0000 mgr.y (mgr.24368) 1013 : cluster 0 pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 3 objects/s recovering 2024-04-06T03:48:33.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:33 smithi050 bash[21467]: cluster 2024-04-06T03:48:32.389136+0000 mon.a (mon.0) 1829 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-04-06T03:48:33.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:33 smithi050 bash[21467]: cluster 2024-04-06T03:48:32.597471+0000 mgr.y (mgr.24368) 1013 : cluster 0 pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 3 objects/s recovering 2024-04-06T03:48:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:33 smithi153 bash[23852]: cluster 2024-04-06T03:48:32.389136+0000 mon.a (mon.0) 1829 : cluster 0 osdmap e404: 8 total, 8 up, 8 in 2024-04-06T03:48:33.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:33 smithi153 bash[23852]: cluster 2024-04-06T03:48:32.597471+0000 mgr.y (mgr.24368) 1013 : cluster 0 pgmap v1111: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 3 objects/s recovering 2024-04-06T03:48:34.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.116239+0000 osd.7 (osd.7) 635 : cluster 0 6.1c scrub starts 2024-04-06T03:48:34.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.119529+0000 osd.7 (osd.7) 636 : cluster 0 6.1c scrub ok 2024-04-06T03:48:34.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.412685+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-04-06T03:48:34.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.426988+0000 osd.3 (osd.3) 629 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.427139+0000 osd.3 (osd.3) 630 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.687738+0000 osd.1 (osd.1) 411 : cluster 0 6.19 scrub starts 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:33.787995+0000 osd.1 (osd.1) 412 : cluster 0 6.19 scrub ok 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[21467]: cluster 2024-04-06T03:48:34.039972+0000 mon.a (mon.0) 1831 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.116239+0000 osd.7 (osd.7) 635 : cluster 0 6.1c scrub starts 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.119529+0000 osd.7 (osd.7) 636 : cluster 0 6.1c scrub ok 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.412685+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.426988+0000 osd.3 (osd.3) 629 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.427139+0000 osd.3 (osd.3) 630 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.687738+0000 osd.1 (osd.1) 411 : cluster 0 6.19 scrub starts 2024-04-06T03:48:34.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:33.787995+0000 osd.1 (osd.1) 412 : cluster 0 6.19 scrub ok 2024-04-06T03:48:34.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:34 smithi050 bash[29221]: cluster 2024-04-06T03:48:34.039972+0000 mon.a (mon.0) 1831 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.116239+0000 osd.7 (osd.7) 635 : cluster 0 6.1c scrub starts 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.119529+0000 osd.7 (osd.7) 636 : cluster 0 6.1c scrub ok 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.412685+0000 mon.a (mon.0) 1830 : cluster 0 osdmap e405: 8 total, 8 up, 8 in 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.426988+0000 osd.3 (osd.3) 629 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.427139+0000 osd.3 (osd.3) 630 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.687738+0000 osd.1 (osd.1) 411 : cluster 0 6.19 scrub starts 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:33.787995+0000 osd.1 (osd.1) 412 : cluster 0 6.19 scrub ok 2024-04-06T03:48:34.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:34 smithi153 bash[23852]: cluster 2024-04-06T03:48:34.039972+0000 mon.a (mon.0) 1831 : cluster 0 osdmap e406: 8 total, 8 up, 8 in 2024-04-06T03:48:35.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[21467]: cluster 2024-04-06T03:48:34.598158+0000 mgr.y (mgr.24368) 1014 : cluster 0 pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 273 B/s, 9 objects/s recovering 2024-04-06T03:48:35.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[21467]: cluster 2024-04-06T03:48:34.991861+0000 osd.1 (osd.1) 413 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:35.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[21467]: cluster 2024-04-06T03:48:34.991986+0000 osd.1 (osd.1) 414 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:35.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[21467]: cluster 2024-04-06T03:48:35.049932+0000 mon.a (mon.0) 1832 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-04-06T03:48:35.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[29221]: cluster 2024-04-06T03:48:34.598158+0000 mgr.y (mgr.24368) 1014 : cluster 0 pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 273 B/s, 9 objects/s recovering 2024-04-06T03:48:35.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[29221]: cluster 2024-04-06T03:48:34.991861+0000 osd.1 (osd.1) 413 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:35.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[29221]: cluster 2024-04-06T03:48:34.991986+0000 osd.1 (osd.1) 414 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:35.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:35 smithi050 bash[29221]: cluster 2024-04-06T03:48:35.049932+0000 mon.a (mon.0) 1832 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-04-06T03:48:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:35 smithi153 bash[23852]: cluster 2024-04-06T03:48:34.598158+0000 mgr.y (mgr.24368) 1014 : cluster 0 pgmap v1114: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 273 B/s, 9 objects/s recovering 2024-04-06T03:48:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:35 smithi153 bash[23852]: cluster 2024-04-06T03:48:34.991861+0000 osd.1 (osd.1) 413 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:35 smithi153 bash[23852]: cluster 2024-04-06T03:48:34.991986+0000 osd.1 (osd.1) 414 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:35.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:35 smithi153 bash[23852]: cluster 2024-04-06T03:48:35.049932+0000 mon.a (mon.0) 1832 : cluster 0 osdmap e407: 8 total, 8 up, 8 in 2024-04-06T03:48:36.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:36 smithi050 bash[29221]: cluster 2024-04-06T03:48:35.607656+0000 osd.1 (osd.1) 415 : cluster 0 5.12 scrub starts 2024-04-06T03:48:36.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:36 smithi050 bash[29221]: cluster 2024-04-06T03:48:35.610064+0000 osd.1 (osd.1) 416 : cluster 0 5.12 scrub ok 2024-04-06T03:48:36.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:36 smithi050 bash[29221]: audit 2024-04-06T03:48:35.796983+0000 mon.a (mon.0) 1833 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:36.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:36 smithi050 bash[21467]: cluster 2024-04-06T03:48:35.607656+0000 osd.1 (osd.1) 415 : cluster 0 5.12 scrub starts 2024-04-06T03:48:36.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:36 smithi050 bash[21467]: cluster 2024-04-06T03:48:35.610064+0000 osd.1 (osd.1) 416 : cluster 0 5.12 scrub ok 2024-04-06T03:48:36.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:36 smithi050 bash[21467]: audit 2024-04-06T03:48:35.796983+0000 mon.a (mon.0) 1833 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:36.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:36 smithi153 bash[23852]: cluster 2024-04-06T03:48:35.607656+0000 osd.1 (osd.1) 415 : cluster 0 5.12 scrub starts 2024-04-06T03:48:36.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:36 smithi153 bash[23852]: cluster 2024-04-06T03:48:35.610064+0000 osd.1 (osd.1) 416 : cluster 0 5.12 scrub ok 2024-04-06T03:48:36.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:36 smithi153 bash[23852]: audit 2024-04-06T03:48:35.796983+0000 mon.a (mon.0) 1833 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:37.151 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.485580+0000 osd.2 (osd.2) 533 : cluster 0 2.1 scrub starts 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.488064+0000 osd.2 (osd.2) 534 : cluster 0 2.1 scrub ok 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.599283+0000 mgr.y (mgr.24368) 1015 : cluster 0 pgmap v1116: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 137 KiB/s, 14 objects/s recovering 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.733672+0000 osd.3 (osd.3) 631 : cluster 0 2.1e scrub starts 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.735309+0000 osd.3 (osd.3) 632 : cluster 0 2.1e scrub ok 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.485580+0000 osd.2 (osd.2) 533 : cluster 0 2.1 scrub starts 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.488064+0000 osd.2 (osd.2) 534 : cluster 0 2.1 scrub ok 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.599283+0000 mgr.y (mgr.24368) 1015 : cluster 0 pgmap v1116: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 137 KiB/s, 14 objects/s recovering 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.733672+0000 osd.3 (osd.3) 631 : cluster 0 2.1e scrub starts 2024-04-06T03:48:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:37 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.735309+0000 osd.3 (osd.3) 632 : cluster 0 2.1e scrub ok 2024-04-06T03:48:37.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:37 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.485580+0000 osd.2 (osd.2) 533 : cluster 0 2.1 scrub starts 2024-04-06T03:48:37.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:37 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.488064+0000 osd.2 (osd.2) 534 : cluster 0 2.1 scrub ok 2024-04-06T03:48:37.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:37 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.599283+0000 mgr.y (mgr.24368) 1015 : cluster 0 pgmap v1116: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 137 KiB/s, 14 objects/s recovering 2024-04-06T03:48:37.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:37 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.733672+0000 osd.3 (osd.3) 631 : cluster 0 2.1e scrub starts 2024-04-06T03:48:37.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:37 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.735309+0000 osd.3 (osd.3) 632 : cluster 0 2.1e scrub ok 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.755766+0000 osd.4 (osd.4) 613 : cluster 0 5.c scrub starts 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[29221]: cluster 2024-04-06T03:48:36.757717+0000 osd.4 (osd.4) 614 : cluster 0 5.c scrub ok 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[29221]: cluster 2024-04-06T03:48:37.124953+0000 osd.7 (osd.7) 637 : cluster 0 2.1d scrub starts 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[29221]: cluster 2024-04-06T03:48:37.128256+0000 osd.7 (osd.7) 638 : cluster 0 2.1d scrub ok 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[29221]: cluster 2024-04-06T03:48:37.670572+0000 osd.1 (osd.1) 417 : cluster 0 3.19 scrub starts 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[29221]: cluster 2024-04-06T03:48:37.676700+0000 osd.1 (osd.1) 418 : cluster 0 3.19 scrub ok 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.755766+0000 osd.4 (osd.4) 613 : cluster 0 5.c scrub starts 2024-04-06T03:48:38.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[21467]: cluster 2024-04-06T03:48:36.757717+0000 osd.4 (osd.4) 614 : cluster 0 5.c scrub ok 2024-04-06T03:48:38.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[21467]: cluster 2024-04-06T03:48:37.124953+0000 osd.7 (osd.7) 637 : cluster 0 2.1d scrub starts 2024-04-06T03:48:38.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[21467]: cluster 2024-04-06T03:48:37.128256+0000 osd.7 (osd.7) 638 : cluster 0 2.1d scrub ok 2024-04-06T03:48:38.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[21467]: cluster 2024-04-06T03:48:37.670572+0000 osd.1 (osd.1) 417 : cluster 0 3.19 scrub starts 2024-04-06T03:48:38.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:38 smithi050 bash[21467]: cluster 2024-04-06T03:48:37.676700+0000 osd.1 (osd.1) 418 : cluster 0 3.19 scrub ok 2024-04-06T03:48:38.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:38 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.755766+0000 osd.4 (osd.4) 613 : cluster 0 5.c scrub starts 2024-04-06T03:48:38.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:38 smithi153 bash[23852]: cluster 2024-04-06T03:48:36.757717+0000 osd.4 (osd.4) 614 : cluster 0 5.c scrub ok 2024-04-06T03:48:38.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:38 smithi153 bash[23852]: cluster 2024-04-06T03:48:37.124953+0000 osd.7 (osd.7) 637 : cluster 0 2.1d scrub starts 2024-04-06T03:48:38.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:38 smithi153 bash[23852]: cluster 2024-04-06T03:48:37.128256+0000 osd.7 (osd.7) 638 : cluster 0 2.1d scrub ok 2024-04-06T03:48:38.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:38 smithi153 bash[23852]: cluster 2024-04-06T03:48:37.670572+0000 osd.1 (osd.1) 417 : cluster 0 3.19 scrub starts 2024-04-06T03:48:38.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:38 smithi153 bash[23852]: cluster 2024-04-06T03:48:37.676700+0000 osd.1 (osd.1) 418 : cluster 0 3.19 scrub ok 2024-04-06T03:48:39.116 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:48:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:48:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:48:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:48:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.116 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84323 10.b 10:d7ae9671:::benchmark_data_smithi050_121293_object84322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.285021+0000", 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 501.82816490099998, 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53370567599999996, 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84323 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285021+0000", 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285021+0000", 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285023+0000", 2024-04-06T03:48:39.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9099999999999999e-06 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285052+0000", 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8946e-05 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285074+0000", 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2099999999999998e-05 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285086+0000", 2024-04-06T03:48:39.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1498000000000001e-05 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285148+0000", 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2232999999999999e-05 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285313+0000", 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000164747 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285479+0000", 2024-04-06T03:48:39.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016641900000000001 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.291176+0000", 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056968820000000003 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.291269+0000", 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2799000000000004e-05 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.818569+0000", 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.527300247 2024-04-06T03:48:39.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.818661+0000", 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1887999999999998e-05 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.818727+0000", 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6006999999999993e-05 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84326 10.b 10:daf47e2e:::benchmark_data_smithi050_121293_object84325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.285851+0000", 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 501.82733550500001, 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.533052156, 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84326 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285851+0000", 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285851+0000", 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285851+0000", 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0100000000000004e-07 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285885+0000", 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3649000000000003e-05 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285893+0000", 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2020000000000004e-06 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285907+0000", 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3427999999999999e-05 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288470+0000", 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025631880000000001 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288619+0000", 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014897900000000001 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288783+0000", 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016445699999999999 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.291258+0000", 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024746659999999999 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.291278+0000", 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0040000000000001e-05 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.818786+0000", 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.527507854 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.818854+0000", 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7933000000000002e-05 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.818903+0000", 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9058999999999999e-05 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:166972 10.b 10:de987f93:::benchmark_data_smithi050_121293_object68155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.572919+0000", 2024-04-06T03:48:39.128 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.54026720799999, 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98569403099999997, 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 166972 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.572919+0000", 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.572919+0000", 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.572920+0000", 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.395e-06 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.572926+0000", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5049999999999998e-06 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.572940+0000", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4177e-05 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.572949+0000", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1980000000000007e-06 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.573010+0000", 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0958999999999997e-05 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.573019+0000", 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2939999999999994e-06 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.554896+0000", 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98187640600000003 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.555028+0000", 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013244 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.555149+0000", 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120463 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558435+0000", 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032861869999999999 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558450+0000", 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5390999999999999e-05 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558497+0000", 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6932999999999999e-05 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558559+0000", 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1395000000000004e-05 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558613+0000", 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4287999999999999e-05 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:166980 10.b 10:dfd9dd73:::benchmark_data_smithi050_121293_object68163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.576335+0000", 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.53685090099998, 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98930651700000005, 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 166980 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576335+0000", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576335+0000", 2024-04-06T03:48:39.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576337+0000", 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4249999999999999e-06 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576343+0000", 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3709999999999996e-06 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576361+0000", 2024-04-06T03:48:39.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7479999999999999e-05 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576369+0000", 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9460000000000004e-06 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576449+0000", 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9794000000000002e-05 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.576458+0000", 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8459999999999997e-06 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558669+0000", 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98221085299999999 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558790+0000", 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012116999999999999 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.558914+0000", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012364900000000001 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565287+0000", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063735670000000001 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565437+0000", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149362 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565529+0000", 2024-04-06T03:48:39.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2769999999999994e-05 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565593+0000", 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3149000000000001e-05 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565642+0000", 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9135000000000003e-05 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.141 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:166985 10.b 10:d83937e6:::benchmark_data_smithi050_121293_object68168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.589337+0000", 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.52384888099999, 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97868156699999997, 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 166985 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.142 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589337+0000", 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589337+0000", 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589339+0000", 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.454e-06 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589345+0000", 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5479999999999998e-06 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589367+0000", 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1724000000000001e-05 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589377+0000", 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-06 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589441+0000", 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4653000000000006e-05 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.589452+0000", 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.006e-05 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.562746+0000", 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.973294039 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.562843+0000", 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7329000000000003e-05 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.562913+0000", 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9727000000000005e-05 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565512+0000", 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002599882 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565649+0000", 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136701 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.567897+0000", 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022481649999999999 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.567973+0000", 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5578e-05 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.568019+0000", 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5906999999999999e-05 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167013 10.b 10:d93d19ed:::benchmark_data_smithi050_121293_object68196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.611094+0000", 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.50209224700001, 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96409966800000002, 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.148 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167013 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611094+0000", 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611094+0000", 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611095+0000", 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.172e-06 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611101+0000", 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4909999999999996e-06 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611117+0000", 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6042e-05 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611126+0000", 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6409999999999997e-06 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611180+0000", 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3912000000000002e-05 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.611187+0000", 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9920000000000001e-06 2024-04-06T03:48:39.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565678+0000", 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95449078499999995 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565774+0000", 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6368999999999998e-05 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.565897+0000", 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012254599999999999 2024-04-06T03:48:39.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.568050+0000", 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021527389999999999 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.574998+0000", 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069482909999999997 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575043+0000", 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5510000000000003e-05 2024-04-06T03:48:39.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575147+0000", 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010387699999999999 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575194+0000", 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6301000000000003e-05 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167022 10.b 10:d3b7ec3e:::benchmark_data_smithi050_121293_object68205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.621240+0000", 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.491945896, 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95885330099999999, 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167022 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621240+0000", 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621240+0000", 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621241+0000", 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5199999999999997e-07 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621245+0000", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2810000000000002e-06 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621253+0000", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1419999999999993e-06 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621260+0000", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1790000000000002e-06 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621315+0000", 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4716999999999998e-05 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.621325+0000", 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4690000000000003e-06 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.568067+0000", 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94674274800000002 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.568169+0000", 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010148499999999999 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.568392+0000", 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022281300000000001 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575003+0000", 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066117160000000001 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575240+0000", 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023659899999999999 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.579975+0000", 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047353630000000002 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580046+0000", 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0714000000000001e-05 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580094+0000", 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7522999999999999e-05 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167027 10.b 10:d2eaedff:::benchmark_data_smithi050_121293_object68210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.626112+0000", 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.48707389899999, 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95421582900000002, 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.161 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167027 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626112+0000", 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626112+0000", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626114+0000", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6640000000000001e-06 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626120+0000", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2979999999999997e-06 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626138+0000", 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7580000000000001e-05 2024-04-06T03:48:39.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626148+0000", 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0216e-05 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626204+0000", 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5784e-05 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.626214+0000", 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0176999999999999e-05 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570223+0000", 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.944008925 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570335+0000", 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112205 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570438+0000", 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103387 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575203+0000", 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047649479999999998 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575281+0000", 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7668000000000002e-05 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580226+0000", 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0049447800000000002 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580288+0000", 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2444e-05 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580328+0000", 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9752999999999997e-05 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167030 10.b 10:d4674b61:::benchmark_data_smithi050_121293_object68213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.628857+0000", 2024-04-06T03:48:39.167 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.48432917999997, 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95167843200000002, 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167030 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628857+0000", 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628857+0000", 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628858+0000", 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8400000000000002e-07 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628864+0000", 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.541e-06 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628875+0000", 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1117e-05 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628883+0000", 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8629999999999997e-06 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628950+0000", 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6854999999999996e-05 2024-04-06T03:48:39.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.628960+0000", 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3770000000000002e-06 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.572676+0000", 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94371667999999997 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.572784+0000", 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010784400000000001 2024-04-06T03:48:39.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.572903+0000", 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011851899999999999 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575250+0000", 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023475340000000001 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575323+0000", 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2591999999999999e-05 2024-04-06T03:48:39.172 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580387+0000", 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050639769999999999 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580494+0000", 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010685 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580535+0000", 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1699000000000002e-05 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167032 10.b 10:d520d534:::benchmark_data_smithi050_121293_object68215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.629495+0000", 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.48369083099999, 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95965309200000004, 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.174 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167032 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629495+0000", 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629495+0000", 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629496+0000", 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9899999999999999e-07 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629501+0000", 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0010000000000001e-06 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629516+0000", 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4775e-05 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629524+0000", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0539999999999995e-06 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629571+0000", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7005999999999997e-05 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.629579+0000", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9070000000000007e-06 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575359+0000", 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94578034 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575458+0000", 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8937999999999994e-05 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.575566+0000", 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107979 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580165+0000", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004599105 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580341+0000", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017574100000000001 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589051+0000", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087095790000000003 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589104+0000", 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3779000000000001e-05 2024-04-06T03:48:39.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589148+0000", 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3989000000000002e-05 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167038 10.b 10:dff4ac18:::benchmark_data_smithi050_121293_object68221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.636823+0000", 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.47636326399999, 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95249477000000005, 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.180 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167038 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636823+0000", 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636823+0000", 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636824+0000", 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3060000000000001e-06 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636829+0000", 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0730000000000004e-06 2024-04-06T03:48:39.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636842+0000", 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2639999999999999e-05 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636850+0000", 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9529999999999999e-06 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636909+0000", 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9250999999999999e-05 2024-04-06T03:48:39.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.636920+0000", 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0462000000000001e-05 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.577879+0000", 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94095943900000001 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.578009+0000", 2024-04-06T03:48:39.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012991899999999999 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.578144+0000", 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013541200000000001 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580639+0000", 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024951449999999998 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580649+0000", 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1330000000000005e-06 2024-04-06T03:48:39.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589239+0000", 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085906249999999993 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589282+0000", 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3238999999999997e-05 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589318+0000", 2024-04-06T03:48:39.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5172999999999997e-05 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167055 10.b 10:d617d971:::benchmark_data_smithi050_121293_object68238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.655106+0000", 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.45808002199999, 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93435809299999995, 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167055 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655106+0000", 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655106+0000", 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.188 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655108+0000", 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5990000000000001e-06 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655115+0000", 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3499999999999999e-06 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655133+0000", 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7571000000000002e-05 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.189 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655143+0000", 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0448999999999999e-05 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655212+0000", 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8441000000000004e-05 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.655222+0000", 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0241000000000001e-05 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.190 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580676+0000", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92545392400000004 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580781+0000", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105086 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.580884+0000", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103282 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.191 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589159+0000", 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082748849999999992 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589200+0000", 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0544999999999999e-05 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589389+0000", 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018912000000000001 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.192 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589430+0000", 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1362000000000001e-05 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589464+0000", 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4238e-05 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167073 10.b 10:d7423740:::benchmark_data_smithi050_121293_object68256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.677652+0000", 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.43553378500002, 2024-04-06T03:48:39.193 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.91192474800000001, 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167073 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.194 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677652+0000", 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677652+0000", 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.195 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677654+0000", 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5400000000000001e-06 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677661+0000", 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.272e-06 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677678+0000", 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6719e-05 2024-04-06T03:48:39.196 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677687+0000", 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9930000000000001e-06 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677757+0000", 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0494999999999995e-05 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.677770+0000", 2024-04-06T03:48:39.197 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2394e-05 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.583272+0000", 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.90550209800000003 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.583382+0000", 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011049599999999999 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.198 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.583518+0000", 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013589099999999999 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589164+0000", 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056460190000000004 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589354+0000", 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000189431 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.199 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589532+0000", 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017780300000000001 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589556+0000", 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4641999999999999e-05 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589577+0000", 2024-04-06T03:48:39.200 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0954999999999998e-05 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167091 10.b 10:da7356c1:::benchmark_data_smithi050_121293_object68274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.704996+0000", 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.40818977200001, 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89540225600000001, 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.201 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167091 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.704996+0000", 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.704996+0000", 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.202 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.704998+0000", 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4109999999999999e-06 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.705005+0000", 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8739999999999997e-06 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.705022+0000", 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7705000000000001e-05 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.203 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.705032+0000", 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1470000000000007e-06 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.705099+0000", 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7350999999999999e-05 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.705112+0000", 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3217000000000001e-05 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.204 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.585955+0000", 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.880843233 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.586068+0000", 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011236899999999999 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.586188+0000", 2024-04-06T03:48:39.205 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012029 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589504+0000", 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033161330000000002 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600222+0000", 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010718294 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600319+0000", 2024-04-06T03:48:39.206 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6544000000000003e-05 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600358+0000", 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8971999999999999e-05 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600399+0000", 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0716000000000001e-05 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.207 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167114 10.b 10:d81f69ad:::benchmark_data_smithi050_121293_object68297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.744063+0000", 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.36912335599999, 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85927676500000005, 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167114 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.208 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744063+0000", 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744063+0000", 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744063+0000", 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1900000000000002e-07 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.209 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744067+0000", 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5020000000000001e-06 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744078+0000", 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0959e-05 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.210 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744086+0000", 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7209999999999994e-06 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744155+0000", 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9173999999999999e-05 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.744163+0000", 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3129999999999999e-06 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.211 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589594+0000", 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.84543109100000002 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589664+0000", 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9876000000000007e-05 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.589754+0000", 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9908999999999996e-05 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.212 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.593962+0000", 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042083609999999999 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600228+0000", 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0062654140000000004 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603234+0000", 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030061189999999998 2024-04-06T03:48:39.213 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603294+0000", 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0007999999999999e-05 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603340+0000", 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5698999999999997e-05 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.214 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167116 10.b 10:d3352893:::benchmark_data_smithi050_121293_object68299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.747897+0000", 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.36528946499999, 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.85558650300000005, 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167116 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.215 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747897+0000", 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747897+0000", 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747898+0000", 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0270000000000001e-06 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.216 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747902+0000", 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5600000000000004e-06 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747912+0000", 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8649999999999996e-06 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747926+0000", 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3576e-05 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.217 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747978+0000", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2289e-05 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.747988+0000", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0033e-05 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.591642+0000", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.843653816 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.218 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.591732+0000", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0163999999999999e-05 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.591844+0000", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011154000000000001 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600209+0000", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0083654179999999995 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600234+0000", 2024-04-06T03:48:39.219 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4584e-05 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603388+0000", 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031544569999999998 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603439+0000", 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0757000000000002e-05 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603483+0000", 2024-04-06T03:48:39.220 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4416999999999998e-05 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167139 10.b 10:d9be74ee:::benchmark_data_smithi050_121293_object68322:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.804718+0000", 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.30846807500001, 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.79896374699999995, 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.221 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167139 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804718+0000", 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804718+0000", 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.222 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804720+0000", 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.888e-06 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804729+0000", 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9789999999999999e-06 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804750+0000", 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1041999999999999e-05 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.223 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804762+0000", 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1698999999999999e-05 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804829+0000", 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6834000000000004e-05 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.804840+0000", 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1884e-05 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.594001+0000", 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.78916057799999995 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.594071+0000", 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9895999999999996e-05 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.594147+0000", 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6169999999999997e-05 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.225 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600238+0000", 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0060910139999999996 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600277+0000", 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8936999999999997e-05 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603565+0000", 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032877100000000001 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.226 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603634+0000", 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8839999999999998e-05 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603682+0000", 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8276000000000003e-05 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.227 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167151 10.b 10:d8337179:::benchmark_data_smithi050_121293_object68334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:01.856737+0000", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 397.25644887499999, 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.74708349299999999, 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167151 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856737+0000", 2024-04-06T03:48:39.228 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856737+0000", 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856738+0000", 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1880000000000001e-06 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856744+0000", 2024-04-06T03:48:39.229 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9170000000000003e-06 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856761+0000", 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6164e-05 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856769+0000", 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9190000000000005e-06 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.230 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856840+0000", 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0838000000000002e-05 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:01.856851+0000", 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0920999999999999e-05 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.597721+0000", 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.740870205 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.231 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.597803+0000", 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1607000000000007e-05 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.597875+0000", 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2260000000000003e-05 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.600291+0000", 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024154459999999999 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.232 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.602209+0000", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019177910000000001 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603725+0000", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015164359999999999 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603780+0000", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5058e-05 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.603821+0000", 2024-04-06T03:48:39.233 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0742999999999999e-05 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(3.4 397/396 [PushOp(3:22d26bf9:::data_loggenerations_metadata:head, version: 85'37, data_included: [0~28], data_size: 28, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:22d26bf9:::data_loggenerations_metadata:head@85'37, size: 28, copy_subset: [(0, 28)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 28, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:48:21.306123+0000", 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.807063653, 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024626230000000002, 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.234 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306123+0000", 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306123+0000", 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306123+0000", 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6000000000000003e-07 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.235 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306154+0000", 2024-04-06T03:48:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0457e-05 2024-04-06T03:48:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306178+0000", 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4020999999999999e-05 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306197+0000", 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8950999999999999e-05 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.306325+0000", 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012851 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.308190+0000", 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018651519999999999 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.308585+0000", 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000394872 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPush(3.0 397/396 [PushOp(3:013d6c33:gc::gc.30:head, version: 85'10, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:013d6c33:gc::gc.30:head@85'10, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:48:21.322058+0000", 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.791128638, 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047318380000000004, 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.322058+0000", 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:48:39.240 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.322058+0000", 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.322058+0000", 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6499999999999999e-07 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.322075+0000", 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7272999999999999e-05 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:48:39.241 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.322096+0000", 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0933000000000001e-05 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.322116+0000", 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9143e-05 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.323529+0000", 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001413533 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.242 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.323546+0000", 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6875e-05 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.326789+0000", 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032434159999999998 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:48:39.243 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:48:39.425 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:48:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:37.831655+0000 osd.0 (osd.0) 626 : cluster 0 6.18 scrub starts 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:37.926122+0000 osd.0 (osd.0) 627 : cluster 0 6.18 scrub ok 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:38.104816+0000 osd.7 (osd.7) 639 : cluster 0 6.5 scrub starts 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:38.205693+0000 osd.7 (osd.7) 640 : cluster 0 6.5 scrub ok 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:38.600451+0000 mgr.y (mgr.24368) 1016 : cluster 0 pgmap v1117: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 96 KiB/s, 6 objects/s recovering 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:38.668999+0000 osd.1 (osd.1) 419 : cluster 0 5.7 scrub starts 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: cluster 2024-04-06T03:48:38.723323+0000 osd.1 (osd.1) 420 : cluster 0 5.7 scrub ok 2024-04-06T03:48:39.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[29221]: audit 2024-04-06T03:48:39.141312+0000 mon.a (mon.0) 1834 : audit 1 from='client.? 172.21.15.50:0/3390430548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:37.831655+0000 osd.0 (osd.0) 626 : cluster 0 6.18 scrub starts 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:37.926122+0000 osd.0 (osd.0) 627 : cluster 0 6.18 scrub ok 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:38.104816+0000 osd.7 (osd.7) 639 : cluster 0 6.5 scrub starts 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:38.205693+0000 osd.7 (osd.7) 640 : cluster 0 6.5 scrub ok 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:38.600451+0000 mgr.y (mgr.24368) 1016 : cluster 0 pgmap v1117: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 96 KiB/s, 6 objects/s recovering 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:38.668999+0000 osd.1 (osd.1) 419 : cluster 0 5.7 scrub starts 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: cluster 2024-04-06T03:48:38.723323+0000 osd.1 (osd.1) 420 : cluster 0 5.7 scrub ok 2024-04-06T03:48:39.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:39 smithi050 bash[21467]: audit 2024-04-06T03:48:39.141312+0000 mon.a (mon.0) 1834 : audit 1 from='client.? 172.21.15.50:0/3390430548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:37.831655+0000 osd.0 (osd.0) 626 : cluster 0 6.18 scrub starts 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:37.926122+0000 osd.0 (osd.0) 627 : cluster 0 6.18 scrub ok 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:38.104816+0000 osd.7 (osd.7) 639 : cluster 0 6.5 scrub starts 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:38.205693+0000 osd.7 (osd.7) 640 : cluster 0 6.5 scrub ok 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:38.600451+0000 mgr.y (mgr.24368) 1016 : cluster 0 pgmap v1117: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 96 KiB/s, 6 objects/s recovering 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:38.668999+0000 osd.1 (osd.1) 419 : cluster 0 5.7 scrub starts 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: cluster 2024-04-06T03:48:38.723323+0000 osd.1 (osd.1) 420 : cluster 0 5.7 scrub ok 2024-04-06T03:48:39.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:39 smithi153 bash[23852]: audit 2024-04-06T03:48:39.141312+0000 mon.a (mon.0) 1834 : audit 1 from='client.? 172.21.15.50:0/3390430548' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-06T03:48:40.045 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-06T03:48:40.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[21467]: cluster 2024-04-06T03:48:39.705842+0000 osd.1 (osd.1) 421 : cluster 0 2.6 scrub starts 2024-04-06T03:48:40.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[21467]: cluster 2024-04-06T03:48:39.708895+0000 osd.1 (osd.1) 422 : cluster 0 2.6 scrub ok 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.037877+0000 mon.a (mon.0) 1835 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.037933+0000 mon.a (mon.0) 1836 : cluster 1 Cluster is now healthy 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[21467]: audit 2024-04-06T03:48:40.040213+0000 mon.a (mon.0) 1837 : audit 1 from='client.? 172.21.15.50:0/3390430548' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.047066+0000 mon.a (mon.0) 1838 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[29221]: cluster 2024-04-06T03:48:39.705842+0000 osd.1 (osd.1) 421 : cluster 0 2.6 scrub starts 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[29221]: cluster 2024-04-06T03:48:39.708895+0000 osd.1 (osd.1) 422 : cluster 0 2.6 scrub ok 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.037877+0000 mon.a (mon.0) 1835 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.037933+0000 mon.a (mon.0) 1836 : cluster 1 Cluster is now healthy 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[29221]: audit 2024-04-06T03:48:40.040213+0000 mon.a (mon.0) 1837 : audit 1 from='client.? 172.21.15.50:0/3390430548' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:48:40.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:40 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.047066+0000 mon.a (mon.0) 1838 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-04-06T03:48:40.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:48:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[23852]: cluster 2024-04-06T03:48:39.705842+0000 osd.1 (osd.1) 421 : cluster 0 2.6 scrub starts 2024-04-06T03:48:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[23852]: cluster 2024-04-06T03:48:39.708895+0000 osd.1 (osd.1) 422 : cluster 0 2.6 scrub ok 2024-04-06T03:48:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.037877+0000 mon.a (mon.0) 1835 : cluster 1 Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-06T03:48:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.037933+0000 mon.a (mon.0) 1836 : cluster 1 Cluster is now healthy 2024-04-06T03:48:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[23852]: audit 2024-04-06T03:48:40.040213+0000 mon.a (mon.0) 1837 : audit 1 from='client.? 172.21.15.50:0/3390430548' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-06T03:48:40.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:40 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.047066+0000 mon.a (mon.0) 1838 : cluster 0 osdmap e408: 8 total, 8 up, 8 in 2024-04-06T03:48:40.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-06T03:48:41.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.407795+0000 osd.2 (osd.2) 535 : cluster 0 2.17 scrub starts 2024-04-06T03:48:41.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: audit 2024-04-06T03:48:40.408733+0000 mgr.y (mgr.24368) 1017 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:41.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.410587+0000 osd.2 (osd.2) 536 : cluster 0 2.17 scrub ok 2024-04-06T03:48:41.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.601401+0000 mgr.y (mgr.24368) 1018 : cluster 0 pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 780 B/s rd, 0 op/s; 88 KiB/s, 3 objects/s recovering 2024-04-06T03:48:41.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.683972+0000 osd.3 (osd.3) 633 : cluster 0 2.1b scrub starts 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.686706+0000 osd.3 (osd.3) 634 : cluster 0 2.1b scrub ok 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.707984+0000 osd.1 (osd.1) 423 : cluster 0 2.2 scrub starts 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[21467]: cluster 2024-04-06T03:48:40.710492+0000 osd.1 (osd.1) 424 : cluster 0 2.2 scrub ok 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.407795+0000 osd.2 (osd.2) 535 : cluster 0 2.17 scrub starts 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: audit 2024-04-06T03:48:40.408733+0000 mgr.y (mgr.24368) 1017 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.410587+0000 osd.2 (osd.2) 536 : cluster 0 2.17 scrub ok 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.601401+0000 mgr.y (mgr.24368) 1018 : cluster 0 pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 780 B/s rd, 0 op/s; 88 KiB/s, 3 objects/s recovering 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.683972+0000 osd.3 (osd.3) 633 : cluster 0 2.1b scrub starts 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.686706+0000 osd.3 (osd.3) 634 : cluster 0 2.1b scrub ok 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.707984+0000 osd.1 (osd.1) 423 : cluster 0 2.2 scrub starts 2024-04-06T03:48:41.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:41 smithi050 bash[29221]: cluster 2024-04-06T03:48:40.710492+0000 osd.1 (osd.1) 424 : cluster 0 2.2 scrub ok 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.407795+0000 osd.2 (osd.2) 535 : cluster 0 2.17 scrub starts 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: audit 2024-04-06T03:48:40.408733+0000 mgr.y (mgr.24368) 1017 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.410587+0000 osd.2 (osd.2) 536 : cluster 0 2.17 scrub ok 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.601401+0000 mgr.y (mgr.24368) 1018 : cluster 0 pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 780 B/s rd, 0 op/s; 88 KiB/s, 3 objects/s recovering 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.683972+0000 osd.3 (osd.3) 633 : cluster 0 2.1b scrub starts 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.686706+0000 osd.3 (osd.3) 634 : cluster 0 2.1b scrub ok 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.707984+0000 osd.1 (osd.1) 423 : cluster 0 2.2 scrub starts 2024-04-06T03:48:41.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:41 smithi153 bash[23852]: cluster 2024-04-06T03:48:40.710492+0000 osd.1 (osd.1) 424 : cluster 0 2.2 scrub ok 2024-04-06T03:48:42.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:42 smithi050 bash[21467]: cluster 2024-04-06T03:48:41.009405+0000 osd.7 (osd.7) 641 : cluster 0 6.15 scrub starts 2024-04-06T03:48:42.669 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:42 smithi050 bash[21467]: cluster 2024-04-06T03:48:41.103140+0000 osd.7 (osd.7) 642 : cluster 0 6.15 scrub ok 2024-04-06T03:48:42.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:42 smithi050 bash[29221]: cluster 2024-04-06T03:48:41.009405+0000 osd.7 (osd.7) 641 : cluster 0 6.15 scrub starts 2024-04-06T03:48:42.669 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:42 smithi050 bash[29221]: cluster 2024-04-06T03:48:41.103140+0000 osd.7 (osd.7) 642 : cluster 0 6.15 scrub ok 2024-04-06T03:48:42.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:42 smithi153 bash[23852]: cluster 2024-04-06T03:48:41.009405+0000 osd.7 (osd.7) 641 : cluster 0 6.15 scrub starts 2024-04-06T03:48:42.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:42 smithi153 bash[23852]: cluster 2024-04-06T03:48:41.103140+0000 osd.7 (osd.7) 642 : cluster 0 6.15 scrub ok 2024-04-06T03:48:42.840 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a: {} 2024-04-06T03:48:42.856 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b: {} 2024-04-06T03:48:42.867 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c: {} 2024-04-06T03:48:42.868 INFO:teuthology.orchestra.run.smithi050.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-06T03:48:42.868 INFO:teuthology.orchestra.run.smithi050.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-06T03:48:42.868 INFO:teuthology.orchestra.run.smithi050.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-06T03:48:43.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[21467]: cluster 2024-04-06T03:48:41.849459+0000 osd.4 (osd.4) 615 : cluster 0 3.1d scrub starts 2024-04-06T03:48:43.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[21467]: cluster 2024-04-06T03:48:41.854161+0000 osd.4 (osd.4) 616 : cluster 0 3.1d scrub ok 2024-04-06T03:48:43.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[21467]: cluster 2024-04-06T03:48:42.602371+0000 mgr.y (mgr.24368) 1019 : cluster 0 pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 2 objects/s recovering 2024-04-06T03:48:43.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[21467]: audit 2024-04-06T03:48:42.830519+0000 mon.c (mon.2) 122 : audit 0 from='client.? 172.21.15.50:0/3993636566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:48:43.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[29221]: cluster 2024-04-06T03:48:41.849459+0000 osd.4 (osd.4) 615 : cluster 0 3.1d scrub starts 2024-04-06T03:48:43.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[29221]: cluster 2024-04-06T03:48:41.854161+0000 osd.4 (osd.4) 616 : cluster 0 3.1d scrub ok 2024-04-06T03:48:43.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[29221]: cluster 2024-04-06T03:48:42.602371+0000 mgr.y (mgr.24368) 1019 : cluster 0 pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 2 objects/s recovering 2024-04-06T03:48:43.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:43 smithi050 bash[29221]: audit 2024-04-06T03:48:42.830519+0000 mon.c (mon.2) 122 : audit 0 from='client.? 172.21.15.50:0/3993636566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:48:43.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:43 smithi153 bash[23852]: cluster 2024-04-06T03:48:41.849459+0000 osd.4 (osd.4) 615 : cluster 0 3.1d scrub starts 2024-04-06T03:48:43.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:43 smithi153 bash[23852]: cluster 2024-04-06T03:48:41.854161+0000 osd.4 (osd.4) 616 : cluster 0 3.1d scrub ok 2024-04-06T03:48:43.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:43 smithi153 bash[23852]: cluster 2024-04-06T03:48:42.602371+0000 mgr.y (mgr.24368) 1019 : cluster 0 pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 72 KiB/s, 2 objects/s recovering 2024-04-06T03:48:43.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:43 smithi153 bash[23852]: audit 2024-04-06T03:48:42.830519+0000 mon.c (mon.2) 122 : audit 0 from='client.? 172.21.15.50:0/3993636566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-06T03:48:43.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 0 1 2024-04-06T03:48:44.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[21467]: cluster 2024-04-06T03:48:42.851199+0000 osd.0 (osd.0) 628 : cluster 0 3.1 scrub starts 2024-04-06T03:48:44.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[21467]: cluster 2024-04-06T03:48:42.854163+0000 osd.0 (osd.0) 629 : cluster 0 3.1 scrub ok 2024-04-06T03:48:44.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[21467]: cluster 2024-04-06T03:48:43.427635+0000 osd.2 (osd.2) 537 : cluster 0 3.4 deep-scrub starts 2024-04-06T03:48:44.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[21467]: cluster 2024-04-06T03:48:43.450044+0000 osd.2 (osd.2) 538 : cluster 0 3.4 deep-scrub ok 2024-04-06T03:48:44.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[29221]: cluster 2024-04-06T03:48:42.851199+0000 osd.0 (osd.0) 628 : cluster 0 3.1 scrub starts 2024-04-06T03:48:44.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[29221]: cluster 2024-04-06T03:48:42.854163+0000 osd.0 (osd.0) 629 : cluster 0 3.1 scrub ok 2024-04-06T03:48:44.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[29221]: cluster 2024-04-06T03:48:43.427635+0000 osd.2 (osd.2) 537 : cluster 0 3.4 deep-scrub starts 2024-04-06T03:48:44.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:44 smithi050 bash[29221]: cluster 2024-04-06T03:48:43.450044+0000 osd.2 (osd.2) 538 : cluster 0 3.4 deep-scrub ok 2024-04-06T03:48:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:44 smithi153 bash[23852]: cluster 2024-04-06T03:48:42.851199+0000 osd.0 (osd.0) 628 : cluster 0 3.1 scrub starts 2024-04-06T03:48:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:44 smithi153 bash[23852]: cluster 2024-04-06T03:48:42.854163+0000 osd.0 (osd.0) 629 : cluster 0 3.1 scrub ok 2024-04-06T03:48:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:44 smithi153 bash[23852]: cluster 2024-04-06T03:48:43.427635+0000 osd.2 (osd.2) 537 : cluster 0 3.4 deep-scrub starts 2024-04-06T03:48:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:44 smithi153 bash[23852]: cluster 2024-04-06T03:48:43.450044+0000 osd.2 (osd.2) 538 : cluster 0 3.4 deep-scrub ok 2024-04-06T03:48:45.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:45 smithi050 bash[21467]: cluster 2024-04-06T03:48:43.850318+0000 osd.4 (osd.4) 617 : cluster 0 3.1a scrub starts 2024-04-06T03:48:45.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:45 smithi050 bash[21467]: cluster 2024-04-06T03:48:43.854728+0000 osd.4 (osd.4) 618 : cluster 0 3.1a scrub ok 2024-04-06T03:48:45.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:45 smithi050 bash[21467]: cluster 2024-04-06T03:48:44.603404+0000 mgr.y (mgr.24368) 1020 : cluster 0 pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:48:45.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:45 smithi050 bash[29221]: cluster 2024-04-06T03:48:43.850318+0000 osd.4 (osd.4) 617 : cluster 0 3.1a scrub starts 2024-04-06T03:48:45.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:45 smithi050 bash[29221]: cluster 2024-04-06T03:48:43.854728+0000 osd.4 (osd.4) 618 : cluster 0 3.1a scrub ok 2024-04-06T03:48:45.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:45 smithi050 bash[29221]: cluster 2024-04-06T03:48:44.603404+0000 mgr.y (mgr.24368) 1020 : cluster 0 pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:48:45.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:45 smithi153 bash[23852]: cluster 2024-04-06T03:48:43.850318+0000 osd.4 (osd.4) 617 : cluster 0 3.1a scrub starts 2024-04-06T03:48:45.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:45 smithi153 bash[23852]: cluster 2024-04-06T03:48:43.854728+0000 osd.4 (osd.4) 618 : cluster 0 3.1a scrub ok 2024-04-06T03:48:45.824 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:45 smithi153 bash[23852]: cluster 2024-04-06T03:48:44.603404+0000 mgr.y (mgr.24368) 1020 : cluster 0 pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 44 B/s, 1 objects/s recovering 2024-04-06T03:48:46.389 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[21467]: cluster 2024-04-06T03:48:45.012680+0000 osd.7 (osd.7) 643 : cluster 0 5.4 scrub starts 2024-04-06T03:48:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[21467]: cluster 2024-04-06T03:48:45.015408+0000 osd.7 (osd.7) 644 : cluster 0 5.4 scrub ok 2024-04-06T03:48:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[21467]: cluster 2024-04-06T03:48:45.386141+0000 osd.2 (osd.2) 539 : cluster 0 2.e deep-scrub starts 2024-04-06T03:48:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[21467]: cluster 2024-04-06T03:48:45.388484+0000 osd.2 (osd.2) 540 : cluster 0 2.e deep-scrub ok 2024-04-06T03:48:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[29221]: cluster 2024-04-06T03:48:45.012680+0000 osd.7 (osd.7) 643 : cluster 0 5.4 scrub starts 2024-04-06T03:48:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[29221]: cluster 2024-04-06T03:48:45.015408+0000 osd.7 (osd.7) 644 : cluster 0 5.4 scrub ok 2024-04-06T03:48:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[29221]: cluster 2024-04-06T03:48:45.386141+0000 osd.2 (osd.2) 539 : cluster 0 2.e deep-scrub starts 2024-04-06T03:48:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:46 smithi050 bash[29221]: cluster 2024-04-06T03:48:45.388484+0000 osd.2 (osd.2) 540 : cluster 0 2.e deep-scrub ok 2024-04-06T03:48:46.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:46 smithi153 bash[23852]: cluster 2024-04-06T03:48:45.012680+0000 osd.7 (osd.7) 643 : cluster 0 5.4 scrub starts 2024-04-06T03:48:46.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:46 smithi153 bash[23852]: cluster 2024-04-06T03:48:45.015408+0000 osd.7 (osd.7) 644 : cluster 0 5.4 scrub ok 2024-04-06T03:48:46.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:46 smithi153 bash[23852]: cluster 2024-04-06T03:48:45.386141+0000 osd.2 (osd.2) 539 : cluster 0 2.e deep-scrub starts 2024-04-06T03:48:46.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:46 smithi153 bash[23852]: cluster 2024-04-06T03:48:45.388484+0000 osd.2 (osd.2) 540 : cluster 0 2.e deep-scrub ok 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:45.887278+0000 osd.0 (osd.0) 630 : cluster 0 2.1f scrub starts 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:45.889305+0000 osd.0 (osd.0) 631 : cluster 0 2.1f scrub ok 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:46.604557+0000 mgr.y (mgr.24368) 1021 : cluster 0 pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:46.789808+0000 osd.1 (osd.1) 425 : cluster 0 2.15 scrub starts 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:46.792242+0000 osd.1 (osd.1) 426 : cluster 0 2.15 scrub ok 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:46.806967+0000 osd.3 (osd.3) 635 : cluster 0 3.18 scrub starts 2024-04-06T03:48:47.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[29221]: cluster 2024-04-06T03:48:46.809359+0000 osd.3 (osd.3) 636 : cluster 0 3.18 scrub ok 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:45.887278+0000 osd.0 (osd.0) 630 : cluster 0 2.1f scrub starts 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:45.889305+0000 osd.0 (osd.0) 631 : cluster 0 2.1f scrub ok 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:46.604557+0000 mgr.y (mgr.24368) 1021 : cluster 0 pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:46.789808+0000 osd.1 (osd.1) 425 : cluster 0 2.15 scrub starts 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:46.792242+0000 osd.1 (osd.1) 426 : cluster 0 2.15 scrub ok 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:46.806967+0000 osd.3 (osd.3) 635 : cluster 0 3.18 scrub starts 2024-04-06T03:48:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:47 smithi050 bash[21467]: cluster 2024-04-06T03:48:46.809359+0000 osd.3 (osd.3) 636 : cluster 0 3.18 scrub ok 2024-04-06T03:48:47.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:45.887278+0000 osd.0 (osd.0) 630 : cluster 0 2.1f scrub starts 2024-04-06T03:48:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:45.889305+0000 osd.0 (osd.0) 631 : cluster 0 2.1f scrub ok 2024-04-06T03:48:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:46.604557+0000 mgr.y (mgr.24368) 1021 : cluster 0 pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:46.789808+0000 osd.1 (osd.1) 425 : cluster 0 2.15 scrub starts 2024-04-06T03:48:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:46.792242+0000 osd.1 (osd.1) 426 : cluster 0 2.15 scrub ok 2024-04-06T03:48:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:46.806967+0000 osd.3 (osd.3) 635 : cluster 0 3.18 scrub starts 2024-04-06T03:48:47.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:47 smithi153 bash[23852]: cluster 2024-04-06T03:48:46.809359+0000 osd.3 (osd.3) 636 : cluster 0 3.18 scrub ok 2024-04-06T03:48:48.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.385657+0000 osd.0 (osd.0) 632 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:48.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.385775+0000 osd.0 (osd.0) 633 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:48.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.427136+0000 osd.2 (osd.2) 541 : cluster 0 6.f deep-scrub starts 2024-04-06T03:48:48.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.491245+0000 mon.a (mon.0) 1839 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-04-06T03:48:48.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.771485+0000 osd.3 (osd.3) 637 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:48:48.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.774615+0000 osd.3 (osd.3) 638 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:48:48.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.385657+0000 osd.0 (osd.0) 632 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:48.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.385775+0000 osd.0 (osd.0) 633 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:48.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.427136+0000 osd.2 (osd.2) 541 : cluster 0 6.f deep-scrub starts 2024-04-06T03:48:48.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.491245+0000 mon.a (mon.0) 1839 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-04-06T03:48:48.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.771485+0000 osd.3 (osd.3) 637 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:48:48.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:48 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.774615+0000 osd.3 (osd.3) 638 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:48:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:48 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.385657+0000 osd.0 (osd.0) 632 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:48 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.385775+0000 osd.0 (osd.0) 633 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:48 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.427136+0000 osd.2 (osd.2) 541 : cluster 0 6.f deep-scrub starts 2024-04-06T03:48:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:48 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.491245+0000 mon.a (mon.0) 1839 : cluster 0 osdmap e409: 8 total, 8 up, 8 in 2024-04-06T03:48:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:48 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.771485+0000 osd.3 (osd.3) 637 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:48:48.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:48 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.774615+0000 osd.3 (osd.3) 638 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:48:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:49.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[29221]: cluster 2024-04-06T03:48:47.654190+0000 osd.2 (osd.2) 542 : cluster 0 6.f deep-scrub ok 2024-04-06T03:48:49.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[29221]: cluster 2024-04-06T03:48:48.036357+0000 osd.7 (osd.7) 645 : cluster 0 2.8 scrub starts 2024-04-06T03:48:49.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[29221]: cluster 2024-04-06T03:48:48.039113+0000 osd.7 (osd.7) 646 : cluster 0 2.8 scrub ok 2024-04-06T03:48:49.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[29221]: cluster 2024-04-06T03:48:48.605454+0000 mgr.y (mgr.24368) 1022 : cluster 0 pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:49.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[21467]: cluster 2024-04-06T03:48:47.654190+0000 osd.2 (osd.2) 542 : cluster 0 6.f deep-scrub ok 2024-04-06T03:48:49.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[21467]: cluster 2024-04-06T03:48:48.036357+0000 osd.7 (osd.7) 645 : cluster 0 2.8 scrub starts 2024-04-06T03:48:49.621 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[21467]: cluster 2024-04-06T03:48:48.039113+0000 osd.7 (osd.7) 646 : cluster 0 2.8 scrub ok 2024-04-06T03:48:49.621 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[21467]: cluster 2024-04-06T03:48:48.605454+0000 mgr.y (mgr.24368) 1022 : cluster 0 pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:49.621 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:48:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:48:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:48:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:49 smithi153 bash[23852]: cluster 2024-04-06T03:48:47.654190+0000 osd.2 (osd.2) 542 : cluster 0 6.f deep-scrub ok 2024-04-06T03:48:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:49 smithi153 bash[23852]: cluster 2024-04-06T03:48:48.036357+0000 osd.7 (osd.7) 645 : cluster 0 2.8 scrub starts 2024-04-06T03:48:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:49 smithi153 bash[23852]: cluster 2024-04-06T03:48:48.039113+0000 osd.7 (osd.7) 646 : cluster 0 2.8 scrub ok 2024-04-06T03:48:49.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:49 smithi153 bash[23852]: cluster 2024-04-06T03:48:48.605454+0000 mgr.y (mgr.24368) 1022 : cluster 0 pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:50.216 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:48:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:48:50.216 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-06T03:48:50.216 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:48.894718+0000 osd.0 (osd.0) 634 : cluster 0 6.0 scrub starts 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:48.988268+0000 osd.0 (osd.0) 635 : cluster 0 6.0 scrub ok 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.040758+0000 osd.7 (osd.7) 647 : cluster 0 6.8 scrub starts 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.042755+0000 osd.7 (osd.7) 648 : cluster 0 6.8 scrub ok 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.423870+0000 osd.2 (osd.2) 543 : cluster 0 5.8 scrub starts 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.426531+0000 osd.2 (osd.2) 544 : cluster 0 5.8 scrub ok 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.796218+0000 osd.4 (osd.4) 619 : cluster 0 5.18 scrub starts 2024-04-06T03:48:50.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.798790+0000 osd.4 (osd.4) 620 : cluster 0 5.18 scrub ok 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.811962+0000 osd.3 (osd.3) 639 : cluster 0 5.16 scrub starts 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[29221]: cluster 2024-04-06T03:48:49.814313+0000 osd.3 (osd.3) 640 : cluster 0 5.16 scrub ok 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:48.894718+0000 osd.0 (osd.0) 634 : cluster 0 6.0 scrub starts 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:48.988268+0000 osd.0 (osd.0) 635 : cluster 0 6.0 scrub ok 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.040758+0000 osd.7 (osd.7) 647 : cluster 0 6.8 scrub starts 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.042755+0000 osd.7 (osd.7) 648 : cluster 0 6.8 scrub ok 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.423870+0000 osd.2 (osd.2) 543 : cluster 0 5.8 scrub starts 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.426531+0000 osd.2 (osd.2) 544 : cluster 0 5.8 scrub ok 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.796218+0000 osd.4 (osd.4) 619 : cluster 0 5.18 scrub starts 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.798790+0000 osd.4 (osd.4) 620 : cluster 0 5.18 scrub ok 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.811962+0000 osd.3 (osd.3) 639 : cluster 0 5.16 scrub starts 2024-04-06T03:48:50.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:50 smithi050 bash[21467]: cluster 2024-04-06T03:48:49.814313+0000 osd.3 (osd.3) 640 : cluster 0 5.16 scrub ok 2024-04-06T03:48:50.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:48.894718+0000 osd.0 (osd.0) 634 : cluster 0 6.0 scrub starts 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:48.988268+0000 osd.0 (osd.0) 635 : cluster 0 6.0 scrub ok 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.040758+0000 osd.7 (osd.7) 647 : cluster 0 6.8 scrub starts 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.042755+0000 osd.7 (osd.7) 648 : cluster 0 6.8 scrub ok 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.423870+0000 osd.2 (osd.2) 543 : cluster 0 5.8 scrub starts 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.426531+0000 osd.2 (osd.2) 544 : cluster 0 5.8 scrub ok 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.796218+0000 osd.4 (osd.4) 619 : cluster 0 5.18 scrub starts 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.798790+0000 osd.4 (osd.4) 620 : cluster 0 5.18 scrub ok 2024-04-06T03:48:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.811962+0000 osd.3 (osd.3) 639 : cluster 0 5.16 scrub starts 2024-04-06T03:48:50.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:50 smithi153 bash[23852]: cluster 2024-04-06T03:48:49.814313+0000 osd.3 (osd.3) 640 : cluster 0 5.16 scrub ok 2024-04-06T03:48:51.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.036885+0000 osd.7 (osd.7) 649 : cluster 0 5.1a scrub starts 2024-04-06T03:48:51.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.039275+0000 osd.7 (osd.7) 650 : cluster 0 5.1a scrub ok 2024-04-06T03:48:51.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: audit 2024-04-06T03:48:50.411367+0000 mgr.y (mgr.24368) 1023 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.431097+0000 osd.2 (osd.2) 545 : cluster 0 2.c scrub starts 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.433484+0000 osd.2 (osd.2) 546 : cluster 0 2.c scrub ok 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.606675+0000 mgr.y (mgr.24368) 1024 : cluster 0 pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.814800+0000 osd.3 (osd.3) 641 : cluster 0 3.b scrub starts 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.820638+0000 osd.3 (osd.3) 642 : cluster 0 3.b scrub ok 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: audit 2024-04-06T03:48:50.824367+0000 mon.a (mon.0) 1840 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:48:51.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[29221]: audit 2024-04-06T03:48:50.826587+0000 mon.a (mon.0) 1841 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:51.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_blocked_ops 2024-04-06T03:48:51.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.036885+0000 osd.7 (osd.7) 649 : cluster 0 5.1a scrub starts 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.039275+0000 osd.7 (osd.7) 650 : cluster 0 5.1a scrub ok 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: audit 2024-04-06T03:48:50.411367+0000 mgr.y (mgr.24368) 1023 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.431097+0000 osd.2 (osd.2) 545 : cluster 0 2.c scrub starts 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.433484+0000 osd.2 (osd.2) 546 : cluster 0 2.c scrub ok 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.606675+0000 mgr.y (mgr.24368) 1024 : cluster 0 pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.814800+0000 osd.3 (osd.3) 641 : cluster 0 3.b scrub starts 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.820638+0000 osd.3 (osd.3) 642 : cluster 0 3.b scrub ok 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: audit 2024-04-06T03:48:50.824367+0000 mon.a (mon.0) 1840 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:48:51.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:51 smithi050 bash[21467]: audit 2024-04-06T03:48:50.826587+0000 mon.a (mon.0) 1841 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.036885+0000 osd.7 (osd.7) 649 : cluster 0 5.1a scrub starts 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.039275+0000 osd.7 (osd.7) 650 : cluster 0 5.1a scrub ok 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: audit 2024-04-06T03:48:50.411367+0000 mgr.y (mgr.24368) 1023 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.431097+0000 osd.2 (osd.2) 545 : cluster 0 2.c scrub starts 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.433484+0000 osd.2 (osd.2) 546 : cluster 0 2.c scrub ok 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.606675+0000 mgr.y (mgr.24368) 1024 : cluster 0 pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.814800+0000 osd.3 (osd.3) 641 : cluster 0 3.b scrub starts 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.820638+0000 osd.3 (osd.3) 642 : cluster 0 3.b scrub ok 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: audit 2024-04-06T03:48:50.824367+0000 mon.a (mon.0) 1840 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:48:51.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:51 smithi153 bash[23852]: audit 2024-04-06T03:48:50.826587+0000 mon.a (mon.0) 1841 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.922107+0000 osd.0 (osd.0) 636 : cluster 0 2.13 scrub starts 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:50.924980+0000 osd.0 (osd.0) 637 : cluster 0 2.13 scrub ok 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.039276+0000 osd.7 (osd.7) 651 : cluster 0 2.b scrub starts 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.041570+0000 osd.7 (osd.7) 652 : cluster 0 2.b scrub ok 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.835491+0000 osd.3 (osd.3) 643 : cluster 0 5.0 scrub starts 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.837720+0000 osd.3 (osd.3) 644 : cluster 0 5.0 scrub ok 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.848918+0000 osd.4 (osd.4) 621 : cluster 0 2.11 scrub starts 2024-04-06T03:48:52.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:52 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.850444+0000 osd.4 (osd.4) 622 : cluster 0 2.11 scrub ok 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.922107+0000 osd.0 (osd.0) 636 : cluster 0 2.13 scrub starts 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:50.924980+0000 osd.0 (osd.0) 637 : cluster 0 2.13 scrub ok 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.039276+0000 osd.7 (osd.7) 651 : cluster 0 2.b scrub starts 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.041570+0000 osd.7 (osd.7) 652 : cluster 0 2.b scrub ok 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.835491+0000 osd.3 (osd.3) 643 : cluster 0 5.0 scrub starts 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.837720+0000 osd.3 (osd.3) 644 : cluster 0 5.0 scrub ok 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.848918+0000 osd.4 (osd.4) 621 : cluster 0 2.11 scrub starts 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.850444+0000 osd.4 (osd.4) 622 : cluster 0 2.11 scrub ok 2024-04-06T03:48:52.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.922107+0000 osd.0 (osd.0) 636 : cluster 0 2.13 scrub starts 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:50.924980+0000 osd.0 (osd.0) 637 : cluster 0 2.13 scrub ok 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.039276+0000 osd.7 (osd.7) 651 : cluster 0 2.b scrub starts 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.041570+0000 osd.7 (osd.7) 652 : cluster 0 2.b scrub ok 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.835491+0000 osd.3 (osd.3) 643 : cluster 0 5.0 scrub starts 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.837720+0000 osd.3 (osd.3) 644 : cluster 0 5.0 scrub ok 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.848918+0000 osd.4 (osd.4) 621 : cluster 0 2.11 scrub starts 2024-04-06T03:48:52.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:52 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.850444+0000 osd.4 (osd.4) 622 : cluster 0 2.11 scrub ok 2024-04-06T03:48:53.522 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:51.996461+0000 osd.7 (osd.7) 653 : cluster 0 3.e scrub starts 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:52.000735+0000 osd.7 (osd.7) 654 : cluster 0 3.e scrub ok 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:52.360400+0000 osd.2 (osd.2) 547 : cluster 0 5.d scrub starts 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:52.363976+0000 osd.2 (osd.2) 548 : cluster 0 5.d scrub ok 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:52.607595+0000 mgr.y (mgr.24368) 1025 : cluster 0 pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:52.880866+0000 osd.3 (osd.3) 645 : cluster 0 3.2 scrub starts 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: cluster 2024-04-06T03:48:52.884320+0000 osd.3 (osd.3) 646 : cluster 0 3.2 scrub ok 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: audit 2024-04-06T03:48:53.291539+0000 mon.c (mon.2) 123 : audit 1 from='client.? 172.21.15.50:0/48265155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:48:53.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:53 smithi153 bash[23852]: audit 2024-04-06T03:48:53.291945+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:48:53.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:51.996461+0000 osd.7 (osd.7) 653 : cluster 0 3.e scrub starts 2024-04-06T03:48:53.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:52.000735+0000 osd.7 (osd.7) 654 : cluster 0 3.e scrub ok 2024-04-06T03:48:53.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:52.360400+0000 osd.2 (osd.2) 547 : cluster 0 5.d scrub starts 2024-04-06T03:48:53.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:52.363976+0000 osd.2 (osd.2) 548 : cluster 0 5.d scrub ok 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:52.607595+0000 mgr.y (mgr.24368) 1025 : cluster 0 pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:52.880866+0000 osd.3 (osd.3) 645 : cluster 0 3.2 scrub starts 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: cluster 2024-04-06T03:48:52.884320+0000 osd.3 (osd.3) 646 : cluster 0 3.2 scrub ok 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: audit 2024-04-06T03:48:53.291539+0000 mon.c (mon.2) 123 : audit 1 from='client.? 172.21.15.50:0/48265155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[21467]: audit 2024-04-06T03:48:53.291945+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:51.996461+0000 osd.7 (osd.7) 653 : cluster 0 3.e scrub starts 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:52.000735+0000 osd.7 (osd.7) 654 : cluster 0 3.e scrub ok 2024-04-06T03:48:53.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:52.360400+0000 osd.2 (osd.2) 547 : cluster 0 5.d scrub starts 2024-04-06T03:48:53.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:52.363976+0000 osd.2 (osd.2) 548 : cluster 0 5.d scrub ok 2024-04-06T03:48:53.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:52.607595+0000 mgr.y (mgr.24368) 1025 : cluster 0 pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:53.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:52.880866+0000 osd.3 (osd.3) 645 : cluster 0 3.2 scrub starts 2024-04-06T03:48:53.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: cluster 2024-04-06T03:48:52.884320+0000 osd.3 (osd.3) 646 : cluster 0 3.2 scrub ok 2024-04-06T03:48:53.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: audit 2024-04-06T03:48:53.291539+0000 mon.c (mon.2) 123 : audit 1 from='client.? 172.21.15.50:0/48265155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:48:53.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:53 smithi050 bash[29221]: audit 2024-04-06T03:48:53.291945+0000 mon.a (mon.0) 1842 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:48:54.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 0 1 2024-04-06T03:48:54.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.467449+0000 osd.7 (osd.7) 655 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:54.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.467560+0000 osd.7 (osd.7) 656 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:54.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: audit 2024-04-06T03:48:53.508613+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:48:54.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.516068+0000 mon.a (mon.0) 1844 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.824119+0000 osd.4 (osd.4) 623 : cluster 0 5.11 scrub starts 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.826722+0000 osd.4 (osd.4) 624 : cluster 0 5.11 scrub ok 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.917824+0000 osd.3 (osd.3) 647 : cluster 0 6.c scrub starts 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.920527+0000 osd.3 (osd.3) 648 : cluster 0 6.c scrub ok 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.933185+0000 osd.0 (osd.0) 638 : cluster 0 3.1b scrub starts 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.936979+0000 osd.0 (osd.0) 639 : cluster 0 3.1b scrub ok 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.467449+0000 osd.7 (osd.7) 655 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.467560+0000 osd.7 (osd.7) 656 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: audit 2024-04-06T03:48:53.508613+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.516068+0000 mon.a (mon.0) 1844 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.824119+0000 osd.4 (osd.4) 623 : cluster 0 5.11 scrub starts 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.826722+0000 osd.4 (osd.4) 624 : cluster 0 5.11 scrub ok 2024-04-06T03:48:54.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.917824+0000 osd.3 (osd.3) 647 : cluster 0 6.c scrub starts 2024-04-06T03:48:54.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.920527+0000 osd.3 (osd.3) 648 : cluster 0 6.c scrub ok 2024-04-06T03:48:54.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.933185+0000 osd.0 (osd.0) 638 : cluster 0 3.1b scrub starts 2024-04-06T03:48:54.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:54 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.936979+0000 osd.0 (osd.0) 639 : cluster 0 3.1b scrub ok 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.467449+0000 osd.7 (osd.7) 655 : cluster 0 purged_snaps scrub starts 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.467560+0000 osd.7 (osd.7) 656 : cluster 0 purged_snaps scrub ok 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: audit 2024-04-06T03:48:53.508613+0000 mon.a (mon.0) 1843 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.516068+0000 mon.a (mon.0) 1844 : cluster 0 osdmap e410: 8 total, 8 up, 8 in 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.824119+0000 osd.4 (osd.4) 623 : cluster 0 5.11 scrub starts 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.826722+0000 osd.4 (osd.4) 624 : cluster 0 5.11 scrub ok 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.917824+0000 osd.3 (osd.3) 647 : cluster 0 6.c scrub starts 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.920527+0000 osd.3 (osd.3) 648 : cluster 0 6.c scrub ok 2024-04-06T03:48:54.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.933185+0000 osd.0 (osd.0) 638 : cluster 0 3.1b scrub starts 2024-04-06T03:48:54.799 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:54 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.936979+0000 osd.0 (osd.0) 639 : cluster 0 3.1b scrub ok 2024-04-06T03:48:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:55 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.980049+0000 osd.7 (osd.7) 657 : cluster 0 2.5 scrub starts 2024-04-06T03:48:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:55 smithi153 bash[23852]: cluster 2024-04-06T03:48:53.982348+0000 osd.7 (osd.7) 658 : cluster 0 2.5 scrub ok 2024-04-06T03:48:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:55 smithi153 bash[23852]: cluster 2024-04-06T03:48:54.608378+0000 mgr.y (mgr.24368) 1026 : cluster 0 pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:55 smithi153 bash[23852]: cluster 2024-04-06T03:48:54.888659+0000 osd.3 (osd.3) 649 : cluster 0 3.7 scrub starts 2024-04-06T03:48:55.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:55 smithi153 bash[23852]: cluster 2024-04-06T03:48:54.891426+0000 osd.3 (osd.3) 650 : cluster 0 3.7 scrub ok 2024-04-06T03:48:55.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.980049+0000 osd.7 (osd.7) 657 : cluster 0 2.5 scrub starts 2024-04-06T03:48:55.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[21467]: cluster 2024-04-06T03:48:53.982348+0000 osd.7 (osd.7) 658 : cluster 0 2.5 scrub ok 2024-04-06T03:48:55.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[21467]: cluster 2024-04-06T03:48:54.608378+0000 mgr.y (mgr.24368) 1026 : cluster 0 pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:55.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[21467]: cluster 2024-04-06T03:48:54.888659+0000 osd.3 (osd.3) 649 : cluster 0 3.7 scrub starts 2024-04-06T03:48:55.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[21467]: cluster 2024-04-06T03:48:54.891426+0000 osd.3 (osd.3) 650 : cluster 0 3.7 scrub ok 2024-04-06T03:48:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.980049+0000 osd.7 (osd.7) 657 : cluster 0 2.5 scrub starts 2024-04-06T03:48:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[29221]: cluster 2024-04-06T03:48:53.982348+0000 osd.7 (osd.7) 658 : cluster 0 2.5 scrub ok 2024-04-06T03:48:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[29221]: cluster 2024-04-06T03:48:54.608378+0000 mgr.y (mgr.24368) 1026 : cluster 0 pgmap v1128: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:48:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[29221]: cluster 2024-04-06T03:48:54.888659+0000 osd.3 (osd.3) 649 : cluster 0 3.7 scrub starts 2024-04-06T03:48:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:55 smithi050 bash[29221]: cluster 2024-04-06T03:48:54.891426+0000 osd.3 (osd.3) 650 : cluster 0 3.7 scrub ok 2024-04-06T03:48:56.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:56 smithi153 bash[23852]: cluster 2024-04-06T03:48:55.360379+0000 osd.2 (osd.2) 549 : cluster 0 5.b scrub starts 2024-04-06T03:48:56.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:56 smithi153 bash[23852]: cluster 2024-04-06T03:48:55.362869+0000 osd.2 (osd.2) 550 : cluster 0 5.b scrub ok 2024-04-06T03:48:56.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:56 smithi153 bash[23852]: cluster 2024-04-06T03:48:55.848110+0000 osd.3 (osd.3) 651 : cluster 0 6.6 scrub starts 2024-04-06T03:48:56.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:56 smithi153 bash[23852]: cluster 2024-04-06T03:48:55.892110+0000 osd.3 (osd.3) 652 : cluster 0 6.6 scrub ok 2024-04-06T03:48:56.856 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[21467]: cluster 2024-04-06T03:48:55.360379+0000 osd.2 (osd.2) 549 : cluster 0 5.b scrub starts 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[21467]: cluster 2024-04-06T03:48:55.362869+0000 osd.2 (osd.2) 550 : cluster 0 5.b scrub ok 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[21467]: cluster 2024-04-06T03:48:55.848110+0000 osd.3 (osd.3) 651 : cluster 0 6.6 scrub starts 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[21467]: cluster 2024-04-06T03:48:55.892110+0000 osd.3 (osd.3) 652 : cluster 0 6.6 scrub ok 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[29221]: cluster 2024-04-06T03:48:55.360379+0000 osd.2 (osd.2) 549 : cluster 0 5.b scrub starts 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[29221]: cluster 2024-04-06T03:48:55.362869+0000 osd.2 (osd.2) 550 : cluster 0 5.b scrub ok 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[29221]: cluster 2024-04-06T03:48:55.848110+0000 osd.3 (osd.3) 651 : cluster 0 6.6 scrub starts 2024-04-06T03:48:56.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:56 smithi050 bash[29221]: cluster 2024-04-06T03:48:55.892110+0000 osd.3 (osd.3) 652 : cluster 0 6.6 scrub ok 2024-04-06T03:48:57.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:57 smithi153 bash[23852]: cluster 2024-04-06T03:48:56.609557+0000 mgr.y (mgr.24368) 1027 : cluster 0 pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:48:57.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:57 smithi153 bash[23852]: cluster 2024-04-06T03:48:56.877112+0000 osd.3 (osd.3) 653 : cluster 0 2.12 scrub starts 2024-04-06T03:48:57.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:57 smithi153 bash[23852]: cluster 2024-04-06T03:48:56.879135+0000 osd.3 (osd.3) 654 : cluster 0 2.12 scrub ok 2024-04-06T03:48:57.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:57 smithi050 bash[29221]: cluster 2024-04-06T03:48:56.609557+0000 mgr.y (mgr.24368) 1027 : cluster 0 pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:48:57.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:57 smithi050 bash[29221]: cluster 2024-04-06T03:48:56.877112+0000 osd.3 (osd.3) 653 : cluster 0 2.12 scrub starts 2024-04-06T03:48:57.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:57 smithi050 bash[29221]: cluster 2024-04-06T03:48:56.879135+0000 osd.3 (osd.3) 654 : cluster 0 2.12 scrub ok 2024-04-06T03:48:57.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:57 smithi050 bash[21467]: cluster 2024-04-06T03:48:56.609557+0000 mgr.y (mgr.24368) 1027 : cluster 0 pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:48:57.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:57 smithi050 bash[21467]: cluster 2024-04-06T03:48:56.877112+0000 osd.3 (osd.3) 653 : cluster 0 2.12 scrub starts 2024-04-06T03:48:57.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:57 smithi050 bash[21467]: cluster 2024-04-06T03:48:56.879135+0000 osd.3 (osd.3) 654 : cluster 0 2.12 scrub ok 2024-04-06T03:48:58.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:58 smithi153 bash[23852]: cluster 2024-04-06T03:48:57.938583+0000 osd.0 (osd.0) 640 : cluster 0 3.10 scrub starts 2024-04-06T03:48:58.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:58 smithi153 bash[23852]: cluster 2024-04-06T03:48:57.941128+0000 osd.0 (osd.0) 641 : cluster 0 3.10 scrub ok 2024-04-06T03:48:58.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:58 smithi050 bash[21467]: cluster 2024-04-06T03:48:57.938583+0000 osd.0 (osd.0) 640 : cluster 0 3.10 scrub starts 2024-04-06T03:48:58.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:58 smithi050 bash[21467]: cluster 2024-04-06T03:48:57.941128+0000 osd.0 (osd.0) 641 : cluster 0 3.10 scrub ok 2024-04-06T03:48:58.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:58 smithi050 bash[29221]: cluster 2024-04-06T03:48:57.938583+0000 osd.0 (osd.0) 640 : cluster 0 3.10 scrub starts 2024-04-06T03:48:58.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:58 smithi050 bash[29221]: cluster 2024-04-06T03:48:57.941128+0000 osd.0 (osd.0) 641 : cluster 0 3.10 scrub ok 2024-04-06T03:48:59.296 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:48:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.067754+0000 osd.7 (osd.7) 659 : cluster 0 5.19 scrub starts 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.070038+0000 osd.7 (osd.7) 660 : cluster 0 5.19 scrub ok 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.351380+0000 osd.2 (osd.2) 551 : cluster 0 3.0 scrub starts 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.359124+0000 osd.2 (osd.2) 552 : cluster 0 3.0 scrub ok 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.610612+0000 mgr.y (mgr.24368) 1028 : cluster 0 pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.987227+0000 osd.0 (osd.0) 642 : cluster 0 5.15 scrub starts 2024-04-06T03:48:59.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[21467]: cluster 2024-04-06T03:48:58.990358+0000 osd.0 (osd.0) 643 : cluster 0 5.15 scrub ok 2024-04-06T03:48:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.067754+0000 osd.7 (osd.7) 659 : cluster 0 5.19 scrub starts 2024-04-06T03:48:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.070038+0000 osd.7 (osd.7) 660 : cluster 0 5.19 scrub ok 2024-04-06T03:48:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.351380+0000 osd.2 (osd.2) 551 : cluster 0 3.0 scrub starts 2024-04-06T03:48:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.359124+0000 osd.2 (osd.2) 552 : cluster 0 3.0 scrub ok 2024-04-06T03:48:59.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.610612+0000 mgr.y (mgr.24368) 1028 : cluster 0 pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:48:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.987227+0000 osd.0 (osd.0) 642 : cluster 0 5.15 scrub starts 2024-04-06T03:48:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:48:59 smithi050 bash[29221]: cluster 2024-04-06T03:48:58.990358+0000 osd.0 (osd.0) 643 : cluster 0 5.15 scrub ok 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.067754+0000 osd.7 (osd.7) 659 : cluster 0 5.19 scrub starts 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.070038+0000 osd.7 (osd.7) 660 : cluster 0 5.19 scrub ok 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.351380+0000 osd.2 (osd.2) 551 : cluster 0 3.0 scrub starts 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.359124+0000 osd.2 (osd.2) 552 : cluster 0 3.0 scrub ok 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.610612+0000 mgr.y (mgr.24368) 1028 : cluster 0 pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.987227+0000 osd.0 (osd.0) 642 : cluster 0 5.15 scrub starts 2024-04-06T03:49:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:48:59 smithi153 bash[23852]: cluster 2024-04-06T03:48:58.990358+0000 osd.0 (osd.0) 643 : cluster 0 5.15 scrub ok 2024-04-06T03:49:00.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.102069+0000 osd.7 (osd.7) 661 : cluster 0 6.12 scrub starts 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.144357+0000 osd.7 (osd.7) 662 : cluster 0 6.12 scrub ok 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.393369+0000 osd.2 (osd.2) 553 : cluster 0 2.10 scrub starts 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.395878+0000 osd.2 (osd.2) 554 : cluster 0 2.10 scrub ok 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.813402+0000 osd.1 (osd.1) 427 : cluster 0 2.9 scrub starts 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.815515+0000 osd.1 (osd.1) 428 : cluster 0 2.9 scrub ok 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.960959+0000 osd.4 (osd.4) 625 : cluster 0 5.1f scrub starts 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.964368+0000 osd.4 (osd.4) 626 : cluster 0 5.1f scrub ok 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:48:59.990777+0000 osd.0 (osd.0) 644 : cluster 0 6.16 scrub starts 2024-04-06T03:49:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:00 smithi153 bash[23852]: cluster 2024-04-06T03:49:00.084792+0000 osd.0 (osd.0) 645 : cluster 0 6.16 scrub ok 2024-04-06T03:49:00.874 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:49:00.874 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:49:00.874 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:49:00.874 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:49:00.874 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:49:00.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.102069+0000 osd.7 (osd.7) 661 : cluster 0 6.12 scrub starts 2024-04-06T03:49:00.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.144357+0000 osd.7 (osd.7) 662 : cluster 0 6.12 scrub ok 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.393369+0000 osd.2 (osd.2) 553 : cluster 0 2.10 scrub starts 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.395878+0000 osd.2 (osd.2) 554 : cluster 0 2.10 scrub ok 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.813402+0000 osd.1 (osd.1) 427 : cluster 0 2.9 scrub starts 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.815515+0000 osd.1 (osd.1) 428 : cluster 0 2.9 scrub ok 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.960959+0000 osd.4 (osd.4) 625 : cluster 0 5.1f scrub starts 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.964368+0000 osd.4 (osd.4) 626 : cluster 0 5.1f scrub ok 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:48:59.990777+0000 osd.0 (osd.0) 644 : cluster 0 6.16 scrub starts 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[21467]: cluster 2024-04-06T03:49:00.084792+0000 osd.0 (osd.0) 645 : cluster 0 6.16 scrub ok 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.102069+0000 osd.7 (osd.7) 661 : cluster 0 6.12 scrub starts 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.144357+0000 osd.7 (osd.7) 662 : cluster 0 6.12 scrub ok 2024-04-06T03:49:00.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.393369+0000 osd.2 (osd.2) 553 : cluster 0 2.10 scrub starts 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.395878+0000 osd.2 (osd.2) 554 : cluster 0 2.10 scrub ok 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.813402+0000 osd.1 (osd.1) 427 : cluster 0 2.9 scrub starts 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.815515+0000 osd.1 (osd.1) 428 : cluster 0 2.9 scrub ok 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.960959+0000 osd.4 (osd.4) 625 : cluster 0 5.1f scrub starts 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.964368+0000 osd.4 (osd.4) 626 : cluster 0 5.1f scrub ok 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:48:59.990777+0000 osd.0 (osd.0) 644 : cluster 0 6.16 scrub starts 2024-04-06T03:49:00.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:00 smithi050 bash[29221]: cluster 2024-04-06T03:49:00.084792+0000 osd.0 (osd.0) 645 : cluster 0 6.16 scrub ok 2024-04-06T03:49:01.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[21467]: cluster 2024-04-06T03:49:00.139201+0000 osd.7 (osd.7) 663 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:49:01.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[21467]: cluster 2024-04-06T03:49:00.146145+0000 osd.7 (osd.7) 664 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[21467]: audit 2024-04-06T03:49:00.412835+0000 mgr.y (mgr.24368) 1029 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[21467]: cluster 2024-04-06T03:49:00.611878+0000 mgr.y (mgr.24368) 1030 : cluster 0 pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[21467]: cluster 2024-04-06T03:49:00.978104+0000 osd.0 (osd.0) 646 : cluster 0 3.17 scrub starts 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[21467]: cluster 2024-04-06T03:49:00.980543+0000 osd.0 (osd.0) 647 : cluster 0 3.17 scrub ok 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[29221]: cluster 2024-04-06T03:49:00.139201+0000 osd.7 (osd.7) 663 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[29221]: cluster 2024-04-06T03:49:00.146145+0000 osd.7 (osd.7) 664 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[29221]: audit 2024-04-06T03:49:00.412835+0000 mgr.y (mgr.24368) 1029 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[29221]: cluster 2024-04-06T03:49:00.611878+0000 mgr.y (mgr.24368) 1030 : cluster 0 pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[29221]: cluster 2024-04-06T03:49:00.978104+0000 osd.0 (osd.0) 646 : cluster 0 3.17 scrub starts 2024-04-06T03:49:01.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:01 smithi050 bash[29221]: cluster 2024-04-06T03:49:00.980543+0000 osd.0 (osd.0) 647 : cluster 0 3.17 scrub ok 2024-04-06T03:49:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:01 smithi153 bash[23852]: cluster 2024-04-06T03:49:00.139201+0000 osd.7 (osd.7) 663 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:49:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:01 smithi153 bash[23852]: cluster 2024-04-06T03:49:00.146145+0000 osd.7 (osd.7) 664 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:49:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:01 smithi153 bash[23852]: audit 2024-04-06T03:49:00.412835+0000 mgr.y (mgr.24368) 1029 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:01 smithi153 bash[23852]: cluster 2024-04-06T03:49:00.611878+0000 mgr.y (mgr.24368) 1030 : cluster 0 pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:01 smithi153 bash[23852]: cluster 2024-04-06T03:49:00.978104+0000 osd.0 (osd.0) 646 : cluster 0 3.17 scrub starts 2024-04-06T03:49:02.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:01 smithi153 bash[23852]: cluster 2024-04-06T03:49:00.980543+0000 osd.0 (osd.0) 647 : cluster 0 3.17 scrub ok 2024-04-06T03:49:02.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.2 dump_historic_ops 2024-04-06T03:49:02.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:02 smithi050 bash[29221]: cluster 2024-04-06T03:49:02.392597+0000 osd.2 (osd.2) 555 : cluster 0 2.3 scrub starts 2024-04-06T03:49:02.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:02 smithi050 bash[29221]: cluster 2024-04-06T03:49:02.394456+0000 osd.2 (osd.2) 556 : cluster 0 2.3 scrub ok 2024-04-06T03:49:02.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:02 smithi050 bash[21467]: cluster 2024-04-06T03:49:02.392597+0000 osd.2 (osd.2) 555 : cluster 0 2.3 scrub starts 2024-04-06T03:49:02.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:02 smithi050 bash[21467]: cluster 2024-04-06T03:49:02.394456+0000 osd.2 (osd.2) 556 : cluster 0 2.3 scrub ok 2024-04-06T03:49:03.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:02 smithi153 bash[23852]: cluster 2024-04-06T03:49:02.392597+0000 osd.2 (osd.2) 555 : cluster 0 2.3 scrub starts 2024-04-06T03:49:03.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:02 smithi153 bash[23852]: cluster 2024-04-06T03:49:02.394456+0000 osd.2 (osd.2) 556 : cluster 0 2.3 scrub ok 2024-04-06T03:49:03.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[29221]: cluster 2024-04-06T03:49:02.151865+0000 osd.7 (osd.7) 665 : cluster 0 2.7 scrub starts 2024-04-06T03:49:03.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[29221]: cluster 2024-04-06T03:49:02.154349+0000 osd.7 (osd.7) 666 : cluster 0 2.7 scrub ok 2024-04-06T03:49:03.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[29221]: cluster 2024-04-06T03:49:02.612739+0000 mgr.y (mgr.24368) 1031 : cluster 0 pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:03.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[29221]: cluster 2024-04-06T03:49:02.877674+0000 osd.3 (osd.3) 655 : cluster 0 6.b scrub starts 2024-04-06T03:49:03.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[21467]: cluster 2024-04-06T03:49:02.151865+0000 osd.7 (osd.7) 665 : cluster 0 2.7 scrub starts 2024-04-06T03:49:03.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[21467]: cluster 2024-04-06T03:49:02.154349+0000 osd.7 (osd.7) 666 : cluster 0 2.7 scrub ok 2024-04-06T03:49:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[21467]: cluster 2024-04-06T03:49:02.612739+0000 mgr.y (mgr.24368) 1031 : cluster 0 pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:03 smithi050 bash[21467]: cluster 2024-04-06T03:49:02.877674+0000 osd.3 (osd.3) 655 : cluster 0 6.b scrub starts 2024-04-06T03:49:04.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:03 smithi153 bash[23852]: cluster 2024-04-06T03:49:02.151865+0000 osd.7 (osd.7) 665 : cluster 0 2.7 scrub starts 2024-04-06T03:49:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:03 smithi153 bash[23852]: cluster 2024-04-06T03:49:02.154349+0000 osd.7 (osd.7) 666 : cluster 0 2.7 scrub ok 2024-04-06T03:49:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:03 smithi153 bash[23852]: cluster 2024-04-06T03:49:02.612739+0000 mgr.y (mgr.24368) 1031 : cluster 0 pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:04.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:03 smithi153 bash[23852]: cluster 2024-04-06T03:49:02.877674+0000 osd.3 (osd.3) 655 : cluster 0 6.b scrub starts 2024-04-06T03:49:04.594 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-06T03:49:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[21467]: cluster 2024-04-06T03:49:03.084778+0000 osd.3 (osd.3) 656 : cluster 0 6.b scrub ok 2024-04-06T03:49:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[21467]: cluster 2024-04-06T03:49:03.826302+0000 osd.2 (osd.2) 557 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[21467]: cluster 2024-04-06T03:49:03.826405+0000 osd.2 (osd.2) 558 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[21467]: audit 2024-04-06T03:49:03.938319+0000 mon.c (mon.2) 124 : audit 1 from='client.? 172.21.15.50:0/3407217215' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[21467]: audit 2024-04-06T03:49:03.938964+0000 mon.a (mon.0) 1845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[29221]: cluster 2024-04-06T03:49:03.084778+0000 osd.3 (osd.3) 656 : cluster 0 6.b scrub ok 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[29221]: cluster 2024-04-06T03:49:03.826302+0000 osd.2 (osd.2) 557 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[29221]: cluster 2024-04-06T03:49:03.826405+0000 osd.2 (osd.2) 558 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[29221]: audit 2024-04-06T03:49:03.938319+0000 mon.c (mon.2) 124 : audit 1 from='client.? 172.21.15.50:0/3407217215' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:49:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:04 smithi050 bash[29221]: audit 2024-04-06T03:49:03.938964+0000 mon.a (mon.0) 1845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:49:05.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:04 smithi153 bash[23852]: cluster 2024-04-06T03:49:03.084778+0000 osd.3 (osd.3) 656 : cluster 0 6.b scrub ok 2024-04-06T03:49:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:04 smithi153 bash[23852]: cluster 2024-04-06T03:49:03.826302+0000 osd.2 (osd.2) 557 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:04 smithi153 bash[23852]: cluster 2024-04-06T03:49:03.826405+0000 osd.2 (osd.2) 558 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:04 smithi153 bash[23852]: audit 2024-04-06T03:49:03.938319+0000 mon.c (mon.2) 124 : audit 1 from='client.? 172.21.15.50:0/3407217215' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:49:05.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:04 smithi153 bash[23852]: audit 2024-04-06T03:49:03.938964+0000 mon.a (mon.0) 1845 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-06T03:49:05.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 1 1 2024-04-06T03:49:05.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: audit 2024-04-06T03:49:04.583241+0000 mon.a (mon.0) 1846 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:49:05.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: cluster 2024-04-06T03:49:04.589977+0000 mon.a (mon.0) 1847 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-04-06T03:49:05.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: cluster 2024-04-06T03:49:04.613359+0000 mgr.y (mgr.24368) 1032 : cluster 0 pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: cluster 2024-04-06T03:49:04.743699+0000 osd.1 (osd.1) 429 : cluster 0 6.1 scrub starts 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: cluster 2024-04-06T03:49:04.815014+0000 osd.1 (osd.1) 430 : cluster 0 6.1 scrub ok 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: cluster 2024-04-06T03:49:04.851734+0000 osd.3 (osd.3) 657 : cluster 0 5.14 scrub starts 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[21467]: cluster 2024-04-06T03:49:04.855219+0000 osd.3 (osd.3) 658 : cluster 0 5.14 scrub ok 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: audit 2024-04-06T03:49:04.583241+0000 mon.a (mon.0) 1846 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: cluster 2024-04-06T03:49:04.589977+0000 mon.a (mon.0) 1847 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: cluster 2024-04-06T03:49:04.613359+0000 mgr.y (mgr.24368) 1032 : cluster 0 pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: cluster 2024-04-06T03:49:04.743699+0000 osd.1 (osd.1) 429 : cluster 0 6.1 scrub starts 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: cluster 2024-04-06T03:49:04.815014+0000 osd.1 (osd.1) 430 : cluster 0 6.1 scrub ok 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: cluster 2024-04-06T03:49:04.851734+0000 osd.3 (osd.3) 657 : cluster 0 5.14 scrub starts 2024-04-06T03:49:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:05 smithi050 bash[29221]: cluster 2024-04-06T03:49:04.855219+0000 osd.3 (osd.3) 658 : cluster 0 5.14 scrub ok 2024-04-06T03:49:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: audit 2024-04-06T03:49:04.583241+0000 mon.a (mon.0) 1846 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-06T03:49:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: cluster 2024-04-06T03:49:04.589977+0000 mon.a (mon.0) 1847 : cluster 0 osdmap e411: 8 total, 8 up, 8 in 2024-04-06T03:49:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: cluster 2024-04-06T03:49:04.613359+0000 mgr.y (mgr.24368) 1032 : cluster 0 pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: cluster 2024-04-06T03:49:04.743699+0000 osd.1 (osd.1) 429 : cluster 0 6.1 scrub starts 2024-04-06T03:49:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: cluster 2024-04-06T03:49:04.815014+0000 osd.1 (osd.1) 430 : cluster 0 6.1 scrub ok 2024-04-06T03:49:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: cluster 2024-04-06T03:49:04.851734+0000 osd.3 (osd.3) 657 : cluster 0 5.14 scrub starts 2024-04-06T03:49:06.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:05 smithi153 bash[23852]: cluster 2024-04-06T03:49:04.855219+0000 osd.3 (osd.3) 658 : cluster 0 5.14 scrub ok 2024-04-06T03:49:06.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:06 smithi050 bash[21467]: audit 2024-04-06T03:49:05.857818+0000 mon.a (mon.0) 1848 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:06.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:06 smithi050 bash[29221]: audit 2024-04-06T03:49:05.857818+0000 mon.a (mon.0) 1848 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:06 smithi153 bash[23852]: audit 2024-04-06T03:49:05.857818+0000 mon.a (mon.0) 1848 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:07.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[21467]: cluster 2024-04-06T03:49:06.614486+0000 mgr.y (mgr.24368) 1033 : cluster 0 pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:07.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[21467]: cluster 2024-04-06T03:49:07.069396+0000 osd.4 (osd.4) 627 : cluster 0 5.f scrub starts 2024-04-06T03:49:07.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[21467]: cluster 2024-04-06T03:49:07.071946+0000 osd.4 (osd.4) 628 : cluster 0 5.f scrub ok 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[21467]: cluster 2024-04-06T03:49:07.239726+0000 osd.7 (osd.7) 667 : cluster 0 3.f deep-scrub starts 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[21467]: cluster 2024-04-06T03:49:07.245111+0000 osd.7 (osd.7) 668 : cluster 0 3.f deep-scrub ok 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[29221]: cluster 2024-04-06T03:49:06.614486+0000 mgr.y (mgr.24368) 1033 : cluster 0 pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[29221]: cluster 2024-04-06T03:49:07.069396+0000 osd.4 (osd.4) 627 : cluster 0 5.f scrub starts 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[29221]: cluster 2024-04-06T03:49:07.071946+0000 osd.4 (osd.4) 628 : cluster 0 5.f scrub ok 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[29221]: cluster 2024-04-06T03:49:07.239726+0000 osd.7 (osd.7) 667 : cluster 0 3.f deep-scrub starts 2024-04-06T03:49:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:07 smithi050 bash[29221]: cluster 2024-04-06T03:49:07.245111+0000 osd.7 (osd.7) 668 : cluster 0 3.f deep-scrub ok 2024-04-06T03:49:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:07 smithi153 bash[23852]: cluster 2024-04-06T03:49:06.614486+0000 mgr.y (mgr.24368) 1033 : cluster 0 pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:07 smithi153 bash[23852]: cluster 2024-04-06T03:49:07.069396+0000 osd.4 (osd.4) 627 : cluster 0 5.f scrub starts 2024-04-06T03:49:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:07 smithi153 bash[23852]: cluster 2024-04-06T03:49:07.071946+0000 osd.4 (osd.4) 628 : cluster 0 5.f scrub ok 2024-04-06T03:49:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:07 smithi153 bash[23852]: cluster 2024-04-06T03:49:07.239726+0000 osd.7 (osd.7) 667 : cluster 0 3.f deep-scrub starts 2024-04-06T03:49:08.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:07 smithi153 bash[23852]: cluster 2024-04-06T03:49:07.245111+0000 osd.7 (osd.7) 668 : cluster 0 3.f deep-scrub ok 2024-04-06T03:49:08.167 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:08.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:08 smithi050 bash[29221]: cluster 2024-04-06T03:49:08.260243+0000 osd.7 (osd.7) 669 : cluster 0 2.1c scrub starts 2024-04-06T03:49:08.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:08 smithi050 bash[29221]: cluster 2024-04-06T03:49:08.262684+0000 osd.7 (osd.7) 670 : cluster 0 2.1c scrub ok 2024-04-06T03:49:08.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:08 smithi050 bash[21467]: cluster 2024-04-06T03:49:08.260243+0000 osd.7 (osd.7) 669 : cluster 0 2.1c scrub starts 2024-04-06T03:49:08.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:08 smithi050 bash[21467]: cluster 2024-04-06T03:49:08.262684+0000 osd.7 (osd.7) 670 : cluster 0 2.1c scrub ok 2024-04-06T03:49:09.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:08 smithi153 bash[23852]: cluster 2024-04-06T03:49:08.260243+0000 osd.7 (osd.7) 669 : cluster 0 2.1c scrub starts 2024-04-06T03:49:09.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:08 smithi153 bash[23852]: cluster 2024-04-06T03:49:08.262684+0000 osd.7 (osd.7) 670 : cluster 0 2.1c scrub ok 2024-04-06T03:49:09.587 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:49:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:49:09.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[21467]: cluster 2024-04-06T03:49:08.615553+0000 mgr.y (mgr.24368) 1034 : cluster 0 pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:09.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[21467]: cluster 2024-04-06T03:49:09.039112+0000 osd.0 (osd.0) 648 : cluster 0 6.9 scrub starts 2024-04-06T03:49:09.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[21467]: cluster 2024-04-06T03:49:09.168687+0000 osd.0 (osd.0) 649 : cluster 0 6.9 scrub ok 2024-04-06T03:49:09.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[21467]: cluster 2024-04-06T03:49:09.270023+0000 osd.7 (osd.7) 671 : cluster 0 5.10 scrub starts 2024-04-06T03:49:09.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[21467]: cluster 2024-04-06T03:49:09.272638+0000 osd.7 (osd.7) 672 : cluster 0 5.10 scrub ok 2024-04-06T03:49:09.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[29221]: cluster 2024-04-06T03:49:08.615553+0000 mgr.y (mgr.24368) 1034 : cluster 0 pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:09.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[29221]: cluster 2024-04-06T03:49:09.039112+0000 osd.0 (osd.0) 648 : cluster 0 6.9 scrub starts 2024-04-06T03:49:09.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[29221]: cluster 2024-04-06T03:49:09.168687+0000 osd.0 (osd.0) 649 : cluster 0 6.9 scrub ok 2024-04-06T03:49:09.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[29221]: cluster 2024-04-06T03:49:09.270023+0000 osd.7 (osd.7) 671 : cluster 0 5.10 scrub starts 2024-04-06T03:49:09.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:09 smithi050 bash[29221]: cluster 2024-04-06T03:49:09.272638+0000 osd.7 (osd.7) 672 : cluster 0 5.10 scrub ok 2024-04-06T03:49:10.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:09 smithi153 bash[23852]: cluster 2024-04-06T03:49:08.615553+0000 mgr.y (mgr.24368) 1034 : cluster 0 pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:10.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:09 smithi153 bash[23852]: cluster 2024-04-06T03:49:09.039112+0000 osd.0 (osd.0) 648 : cluster 0 6.9 scrub starts 2024-04-06T03:49:10.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:09 smithi153 bash[23852]: cluster 2024-04-06T03:49:09.168687+0000 osd.0 (osd.0) 649 : cluster 0 6.9 scrub ok 2024-04-06T03:49:10.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:09 smithi153 bash[23852]: cluster 2024-04-06T03:49:09.270023+0000 osd.7 (osd.7) 671 : cluster 0 5.10 scrub starts 2024-04-06T03:49:10.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:09 smithi153 bash[23852]: cluster 2024-04-06T03:49:09.272638+0000 osd.7 (osd.7) 672 : cluster 0 5.10 scrub ok 2024-04-06T03:49:10.593 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:10.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:49:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:49:11.871 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:49:11.871 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84284 10.5 10:ab1670b1:::benchmark_data_smithi050_121293_object84283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.251785+0000", 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 534.61841910199996, 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.553444669, 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.872 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84284 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251785+0000", 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251785+0000", 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251787+0000", 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.654e-06 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251806+0000", 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8978999999999999e-05 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251821+0000", 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4772e-05 2024-04-06T03:49:11.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251828+0000", 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7729999999999999e-06 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.251896+0000", 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7665999999999999e-05 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.252160+0000", 2024-04-06T03:49:11.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026426100000000001 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.252567+0000", 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040631400000000001 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.256697+0000", 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041303599999999996 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.259869+0000", 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031716029999999998 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.804993+0000", 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54512430300000003 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805108+0000", 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115173 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805230+0000", 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121811 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84296 10.5 10:a00a2d50:::benchmark_data_smithi050_121293_object84295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.260163+0000", 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 534.61004118000005, 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54529861700000004, 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.879 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84296 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260163+0000", 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260163+0000", 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260165+0000", 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.353e-06 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260185+0000", 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0602e-05 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260198+0000", 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3125999999999999e-05 2024-04-06T03:49:11.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260206+0000", 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.038e-06 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260262+0000", 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5720999999999997e-05 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260386+0000", 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012419699999999999 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.260536+0000", 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014956799999999999 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.263569+0000", 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003033277 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.264076+0000", 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050717899999999996 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805305+0000", 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54122838200000001 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805395+0000", 2024-04-06T03:49:11.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0693999999999996e-05 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.805462+0000", 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.648e-05 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84317 10.5 10:a61baf96:::benchmark_data_smithi050_121293_object84316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.281228+0000", 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 534.58897677799996, 2024-04-06T03:49:11.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53607865799999999, 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84317 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281228+0000", 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281228+0000", 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281229+0000", 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.077e-06 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281252+0000", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3617000000000001e-05 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281265+0000", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3216e-05 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281273+0000", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9289999999999998e-06 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281333+0000", 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9434000000000001e-05 2024-04-06T03:49:11.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281465+0000", 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013207599999999999 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.281609+0000", 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014411299999999999 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288194+0000", 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006585105 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.289260+0000", 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010654449999999999 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817035+0000", 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52777511399999999 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817192+0000", 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000156973 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817306+0000", 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114559 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:84322 10.5 10:a7999a3b:::benchmark_data_smithi050_121293_object84321:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:40:17.285007+0000", 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 534.58519724600001, 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53267631500000001, 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.891 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84322 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285007+0000", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285007+0000", 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285008+0000", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.392e-06 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285029+0000", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0820999999999999e-05 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285041+0000", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1715e-05 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285050+0000", 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8109999999999992e-06 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285128+0000", 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8628999999999995e-05 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285273+0000", 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000144752 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.285417+0000", 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014338400000000001 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.288378+0000", 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029612089999999998 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.291212+0000", 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002834016 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817432+0000", 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52621990600000002 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817580+0000", 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148221 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:40:17.817683+0000", 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103459 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:125694 10.5 10:a235a722:::benchmark_data_smithi050_121293_object26877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:05.968814+0000", 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 485.90138999999999, 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.517048438, 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 125694 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968814+0000", 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968814+0000", 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968815+0000", 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8799999999999998e-07 2024-04-06T03:49:11.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968817+0000", 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8899999999999999e-06 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968821+0000", 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.25e-06 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.968825+0000", 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8619999999999996e-06 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.971804+0000", 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029791119999999999 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.971917+0000", 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112909 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.971989+0000", 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2778e-05 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.975226+0000", 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032362430000000002 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:05.979463+0000", 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042373840000000003 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485763+0000", 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50629953500000002 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485834+0000", 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1685000000000002e-05 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:06.485863+0000", 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8501999999999999e-05 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.902 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:132557 10.5 10:ab2329a2:::benchmark_data_smithi050_121293_object33740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:15.033163+0000", 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 476.83704084800002, 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.50213324599999998, 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 132557 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033163+0000", 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033163+0000", 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033165+0000", 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.122e-06 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033170+0000", 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7370000000000003e-06 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033181+0000", 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0574000000000001e-05 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033187+0000", 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1659999999999999e-06 2024-04-06T03:49:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033245+0000", 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7890999999999999e-05 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033383+0000", 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137961 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.033511+0000", 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012850700000000001 2024-04-06T03:49:11.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.039189+0000", 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056775660000000002 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.044714+0000", 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0055248290000000002 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535199+0000", 2024-04-06T03:49:11.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49048494599999998 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535250+0000", 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0898e-05 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:15.535297+0000", 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7049000000000003e-05 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:165868 10.5 10:afb65d11:::benchmark_data_smithi050_121293_object67051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:41:58.597166+0000", 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 433.273037934, 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49994584800000003, 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 165868 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597166+0000", 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597166+0000", 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597168+0000", 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2139999999999999e-06 2024-04-06T03:49:11.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597172+0000", 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8280000000000002e-06 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597182+0000", 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1109999999999997e-06 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597187+0000", 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3159999999999996e-06 2024-04-06T03:49:11.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597232+0000", 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4932000000000003e-05 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597310+0000", 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8149e-05 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.597421+0000", 2024-04-06T03:49:11.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011133699999999999 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:58.606930+0000", 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095090890000000001 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.080788+0000", 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47385771799999998 2024-04-06T03:49:11.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.096979+0000", 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016190623000000001 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.097055+0000", 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6276e-05 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:41:59.097112+0000", 2024-04-06T03:49:11.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7255e-05 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:171114 10.3 10:c26c9224:::benchmark_data_smithi050_121293_object72297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:09.004447+0000", 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 422.86575749500003, 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.58491213500000006, 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:11.915 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 171114 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004447+0000", 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004447+0000", 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004448+0000", 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0860000000000001e-06 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004454+0000", 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7080000000000002e-06 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004466+0000", 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2658e-05 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.004475+0000", 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4630000000000001e-06 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.008731+0000", 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004256592 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.008822+0000", 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0902999999999993e-05 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.008908+0000", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5931000000000001e-05 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.010809+0000", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019007760000000001 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.011786+0000", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00097677300000000001 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.459408+0000", 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.44762270700000001 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.589303+0000", 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.12989435099999999 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:09.589359+0000", 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6187000000000003e-05 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:84073 11.2 11:4984dc81:::benchmark_data_smithi050_138698_object84072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:56.078953+0000", 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 255.791251343, 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.49957127899999998, 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84073 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.921 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.078953+0000", 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.078953+0000", 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.078954+0000", 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.178e-06 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.078984+0000", 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0301999999999999e-05 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.078997+0000", 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3019e-05 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.079005+0000", 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0879999999999997e-06 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.079050+0000", 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5522000000000002e-05 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.079146+0000", 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6321000000000002e-05 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.079236+0000", 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9903999999999998e-05 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086063+0000", 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068265039999999997 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088682+0000", 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002619003 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.578393+0000", 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48971152899999998 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.578467+0000", 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3973000000000005e-05 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.578524+0000", 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6935999999999998e-05 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.926 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:84083 11.2 11:47e4a055:::benchmark_data_smithi050_138698_object84082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:56.086355+0000", 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 255.78384926000001, 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.52089983200000001, 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84083 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086355+0000", 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086355+0000", 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086356+0000", 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5700000000000001e-07 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086387+0000", 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.078e-05 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086396+0000", 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5170000000000005e-06 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.086405+0000", 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4200000000000007e-06 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088716+0000", 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023114630000000001 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.571097+0000", 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48238078299999998 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.571298+0000", 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000201065 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.578637+0000", 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073391259999999996 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.585469+0000", 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068319710000000001 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607137+0000", 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021667829 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607200+0000", 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3008000000000004e-05 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607255+0000", 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5013000000000003e-05 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:84086 11.2 11:4ede86a9:::benchmark_data_smithi050_138698_object84085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:56.088462+0000", 2024-04-06T03:49:11.932 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 255.78174227599999, 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.51892547700000002, 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84086 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088462+0000", 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088462+0000", 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088463+0000", 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1620000000000001e-06 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088484+0000", 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0506999999999999e-05 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088494+0000", 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.059e-05 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.088502+0000", 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2570000000000001e-06 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.574659+0000", 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.48615761200000002 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.574875+0000", 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021544500000000001 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.575295+0000", 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042063000000000002 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.582247+0000", 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069521549999999998 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.593185+0000", 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010937573000000001 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607298+0000", 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014112693000000001 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607349+0000", 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1206999999999999e-05 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607388+0000", 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8646000000000002e-05 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:84088 11.2 11:467dcd0f:::benchmark_data_smithi050_138698_object84087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:56.090941+0000", 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 255.77926365299999, 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.51656917300000005, 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 84088 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.090941+0000", 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.090941+0000", 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.090941+0000", 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0800000000000004e-07 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.090959+0000", 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7470999999999999e-05 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.570454+0000", 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.47949539600000002 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.570481+0000", 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6444e-05 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.578884+0000", 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084036330000000006 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.579055+0000", 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017063399999999999 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.579197+0000", 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014219699999999999 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.593198+0000", 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014000670999999999 2024-04-06T03:49:11.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.593252+0000", 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4484999999999998e-05 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607425+0000", 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014172736999999999 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607474+0000", 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8393999999999998e-05 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:56.607510+0000", 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6303000000000001e-05 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85919 11.2 11:4ccbde0b:::benchmark_data_smithi050_138698_object85918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.404357+0000", 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 253.46584689400001, 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55055180500000001, 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85919 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404357+0000", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404357+0000", 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404421+0000", 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3175000000000004e-05 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404439+0000", 2024-04-06T03:49:11.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8661999999999999e-05 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404450+0000", 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1161e-05 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404459+0000", 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5080000000000005e-06 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404589+0000", 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013033599999999999 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404683+0000", 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4073999999999994e-05 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404774+0000", 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0787999999999999e-05 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.409179+0000", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044049969999999999 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.409218+0000", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9029999999999997e-05 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.954778+0000", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54555958999999998 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.954850+0000", 2024-04-06T03:49:11.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1743999999999996e-05 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.954909+0000", 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9740000000000001e-05 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85922 11.2 11:4ed627a6:::benchmark_data_smithi050_138698_object85921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.404545+0000", 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 253.465659292, 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55058345099999995, 2024-04-06T03:49:11.950 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85922 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404545+0000", 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404545+0000", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404546+0000", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5099999999999999e-07 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404574+0000", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7716999999999999e-05 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404584+0000", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0162e-05 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404592+0000", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3180000000000007e-06 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.406938+0000", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023463440000000002 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.407034+0000", 2024-04-06T03:49:11.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5884999999999994e-05 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.407173+0000", 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013870700000000001 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.409607+0000", 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002433906 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.409760+0000", 2024-04-06T03:49:11.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015334600000000001 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.954986+0000", 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54522571900000005 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.955063+0000", 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6916999999999996e-05 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.955128+0000", 2024-04-06T03:49:11.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5679000000000006e-05 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85926 11.2 11:42250466:::benchmark_data_smithi050_138698_object85925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.421342+0000", 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 253.44886260300001, 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53753072999999996, 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:11.956 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85926 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421342+0000", 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421342+0000", 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421343+0000", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1299999999999998e-07 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421369+0000", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6653999999999999e-05 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421380+0000", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0788000000000001e-05 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421389+0000", 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0329999999999997e-06 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421436+0000", 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7283000000000001e-05 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421535+0000", 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8783000000000006e-05 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:49:11.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.421626+0000", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0870000000000002e-05 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.423810+0000", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021844339999999999 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.424114+0000", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000303145 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.958732+0000", 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53461868300000004 2024-04-06T03:49:11.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.958803+0000", 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1055000000000003e-05 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.958872+0000", 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9089000000000002e-05 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:5496 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94633190265216 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:48:20.254149+0000", 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.616055359999997, 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.082046563, 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5496 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.962 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254149+0000", 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254149+0000", 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254151+0000", 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7689999999999999e-06 2024-04-06T03:49:11.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254160+0000", 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7450000000000007e-06 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254202+0000", 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1440000000000003e-05 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254218+0000", 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6469000000000001e-05 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254306+0000", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7417999999999995e-05 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254314+0000", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0129999999999993e-06 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.267092+0000", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012778326 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:11.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.267105+0000", 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3186999999999999e-05 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.300539+0000", 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0334336630000001 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.301202+0000", 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066282500000000002 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.315145+0000", 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013943514000000001 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.315245+0000", 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9729999999999996e-05 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.336196+0000", 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020950463999999999 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:5595 5.d 5:b4812045:::notify.4:head [watch ping cookie 94633190264064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:49:07.524787+0000", 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.3454177119999997, 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083472500000000003, 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5595 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524787+0000", 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524787+0000", 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524788+0000", 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.204e-06 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524795+0000", 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0960000000000003e-06 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524808+0000", 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.332e-05 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524827+0000", 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8833000000000001e-05 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.525057+0000", 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022954299999999999 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.525296+0000", 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023976299999999999 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.525621+0000", 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032496599999999999 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.971 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:5596 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94633190265216 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:49:07.524923+0000", 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.3452815859999996, 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083983199999999999, 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5596 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524923+0000", 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524923+0000", 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524925+0000", 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7570000000000001e-06 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524935+0000", 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0156999999999999e-05 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524960+0000", 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5439000000000002e-05 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.524980+0000", 2024-04-06T03:49:11.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0146e-05 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.525132+0000", 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000152032 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.525356+0000", 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022327999999999999 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:07.525763+0000", 2024-04-06T03:49:11.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040702099999999998 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(2.1c,RELEASE,e:411,from:7,reservation_nonce:0)", 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:49:08.264450+0000", 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6057546770000002, 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037503699999999999, 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264450+0000", 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264450+0000", 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264451+0000", 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1000000000000001e-06 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264458+0000", 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4000000000000003e-06 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264481+0000", 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2589e-05 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264500+0000", 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8773000000000001e-05 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264736+0000", 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023605300000000001 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264771+0000", 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5669e-05 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:08.264825+0000", 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3452999999999998e-05 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.979 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(3.13,QREQUEST,e:411,from:7,reservation_nonce:24)", 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:49:11.283184+0000", 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.58702047099999999, 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000372975, 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283184+0000", 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283184+0000", 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283187+0000", 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.683e-06 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283203+0000", 2024-04-06T03:49:11.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6390000000000001e-05 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283240+0000", 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7274999999999998e-05 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283260+0000", 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9479000000000001e-05 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:11.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283400+0000", 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140173 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283442+0000", 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2215999999999997e-05 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:11.283557+0000", 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114759 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.983 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:11.984 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:11.984 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:49:11.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: audit 2024-04-06T03:49:10.424215+0000 mgr.y (mgr.24368) 1035 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:11.984 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:10.616348+0000 mgr.y (mgr.24368) 1036 : cluster 0 pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:10.836886+0000 osd.3 (osd.3) 659 : cluster 0 4.0 scrub starts 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:10.839973+0000 osd.3 (osd.3) 660 : cluster 0 4.0 scrub ok 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:11.028137+0000 osd.0 (osd.0) 650 : cluster 0 2.4 scrub starts 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:11.031445+0000 osd.0 (osd.0) 651 : cluster 0 2.4 scrub ok 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:11.067093+0000 osd.4 (osd.4) 629 : cluster 0 2.1a scrub starts 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:11.069503+0000 osd.4 (osd.4) 630 : cluster 0 2.1a scrub ok 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:11.284007+0000 osd.7 (osd.7) 673 : cluster 0 3.13 scrub starts 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[29221]: cluster 2024-04-06T03:49:11.287955+0000 osd.7 (osd.7) 674 : cluster 0 3.13 scrub ok 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: audit 2024-04-06T03:49:10.424215+0000 mgr.y (mgr.24368) 1035 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:10.616348+0000 mgr.y (mgr.24368) 1036 : cluster 0 pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:10.836886+0000 osd.3 (osd.3) 659 : cluster 0 4.0 scrub starts 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:10.839973+0000 osd.3 (osd.3) 660 : cluster 0 4.0 scrub ok 2024-04-06T03:49:11.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:11.028137+0000 osd.0 (osd.0) 650 : cluster 0 2.4 scrub starts 2024-04-06T03:49:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:11.031445+0000 osd.0 (osd.0) 651 : cluster 0 2.4 scrub ok 2024-04-06T03:49:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:11.067093+0000 osd.4 (osd.4) 629 : cluster 0 2.1a scrub starts 2024-04-06T03:49:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:11.069503+0000 osd.4 (osd.4) 630 : cluster 0 2.1a scrub ok 2024-04-06T03:49:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:11.284007+0000 osd.7 (osd.7) 673 : cluster 0 3.13 scrub starts 2024-04-06T03:49:11.986 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:11 smithi050 bash[21467]: cluster 2024-04-06T03:49:11.287955+0000 osd.7 (osd.7) 674 : cluster 0 3.13 scrub ok 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: audit 2024-04-06T03:49:10.424215+0000 mgr.y (mgr.24368) 1035 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:10.616348+0000 mgr.y (mgr.24368) 1036 : cluster 0 pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:10.836886+0000 osd.3 (osd.3) 659 : cluster 0 4.0 scrub starts 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:10.839973+0000 osd.3 (osd.3) 660 : cluster 0 4.0 scrub ok 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:11.028137+0000 osd.0 (osd.0) 650 : cluster 0 2.4 scrub starts 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:11.031445+0000 osd.0 (osd.0) 651 : cluster 0 2.4 scrub ok 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:11.067093+0000 osd.4 (osd.4) 629 : cluster 0 2.1a scrub starts 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:11.069503+0000 osd.4 (osd.4) 630 : cluster 0 2.1a scrub ok 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:11.284007+0000 osd.7 (osd.7) 673 : cluster 0 3.13 scrub starts 2024-04-06T03:49:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:11 smithi153 bash[23852]: cluster 2024-04-06T03:49:11.287955+0000 osd.7 (osd.7) 674 : cluster 0 3.13 scrub ok 2024-04-06T03:49:12.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:12 smithi050 bash[21467]: cluster 2024-04-06T03:49:12.268523+0000 osd.7 (osd.7) 675 : cluster 0 3.16 scrub starts 2024-04-06T03:49:12.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:12 smithi050 bash[21467]: cluster 2024-04-06T03:49:12.272865+0000 osd.7 (osd.7) 676 : cluster 0 3.16 scrub ok 2024-04-06T03:49:12.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:12 smithi050 bash[29221]: cluster 2024-04-06T03:49:12.268523+0000 osd.7 (osd.7) 675 : cluster 0 3.16 scrub starts 2024-04-06T03:49:12.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:12 smithi050 bash[29221]: cluster 2024-04-06T03:49:12.272865+0000 osd.7 (osd.7) 676 : cluster 0 3.16 scrub ok 2024-04-06T03:49:13.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:12 smithi153 bash[23852]: cluster 2024-04-06T03:49:12.268523+0000 osd.7 (osd.7) 675 : cluster 0 3.16 scrub starts 2024-04-06T03:49:13.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:12 smithi153 bash[23852]: cluster 2024-04-06T03:49:12.272865+0000 osd.7 (osd.7) 676 : cluster 0 3.16 scrub ok 2024-04-06T03:49:13.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-06T03:49:13.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:13 smithi050 bash[21467]: cluster 2024-04-06T03:49:12.617116+0000 mgr.y (mgr.24368) 1037 : cluster 0 pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:13.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:13 smithi050 bash[21467]: cluster 2024-04-06T03:49:13.221470+0000 osd.7 (osd.7) 677 : cluster 0 6.3 scrub starts 2024-04-06T03:49:13.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:13 smithi050 bash[21467]: cluster 2024-04-06T03:49:13.381221+0000 osd.7 (osd.7) 678 : cluster 0 6.3 scrub ok 2024-04-06T03:49:13.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:13 smithi050 bash[29221]: cluster 2024-04-06T03:49:12.617116+0000 mgr.y (mgr.24368) 1037 : cluster 0 pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:13.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:13 smithi050 bash[29221]: cluster 2024-04-06T03:49:13.221470+0000 osd.7 (osd.7) 677 : cluster 0 6.3 scrub starts 2024-04-06T03:49:13.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:13 smithi050 bash[29221]: cluster 2024-04-06T03:49:13.381221+0000 osd.7 (osd.7) 678 : cluster 0 6.3 scrub ok 2024-04-06T03:49:14.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:13 smithi153 bash[23852]: cluster 2024-04-06T03:49:12.617116+0000 mgr.y (mgr.24368) 1037 : cluster 0 pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:14.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:13 smithi153 bash[23852]: cluster 2024-04-06T03:49:13.221470+0000 osd.7 (osd.7) 677 : cluster 0 6.3 scrub starts 2024-04-06T03:49:14.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:13 smithi153 bash[23852]: cluster 2024-04-06T03:49:13.381221+0000 osd.7 (osd.7) 678 : cluster 0 6.3 scrub ok 2024-04-06T03:49:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:15 smithi050 bash[21467]: cluster 2024-04-06T03:49:13.930071+0000 osd.0 (osd.0) 652 : cluster 0 5.1e scrub starts 2024-04-06T03:49:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:15 smithi050 bash[21467]: cluster 2024-04-06T03:49:13.932719+0000 osd.0 (osd.0) 653 : cluster 0 5.1e scrub ok 2024-04-06T03:49:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:15 smithi050 bash[21467]: cluster 2024-04-06T03:49:14.618113+0000 mgr.y (mgr.24368) 1038 : cluster 0 pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-04-06T03:49:15.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:15 smithi050 bash[29221]: cluster 2024-04-06T03:49:13.930071+0000 osd.0 (osd.0) 652 : cluster 0 5.1e scrub starts 2024-04-06T03:49:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:15 smithi050 bash[29221]: cluster 2024-04-06T03:49:13.932719+0000 osd.0 (osd.0) 653 : cluster 0 5.1e scrub ok 2024-04-06T03:49:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:15 smithi050 bash[29221]: cluster 2024-04-06T03:49:14.618113+0000 mgr.y (mgr.24368) 1038 : cluster 0 pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-04-06T03:49:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:15 smithi153 bash[23852]: cluster 2024-04-06T03:49:13.930071+0000 osd.0 (osd.0) 652 : cluster 0 5.1e scrub starts 2024-04-06T03:49:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:15 smithi153 bash[23852]: cluster 2024-04-06T03:49:13.932719+0000 osd.0 (osd.0) 653 : cluster 0 5.1e scrub ok 2024-04-06T03:49:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:15 smithi153 bash[23852]: cluster 2024-04-06T03:49:14.618113+0000 mgr.y (mgr.24368) 1038 : cluster 0 pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-04-06T03:49:16.079 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-04-06T03:49:16.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: cluster 2024-04-06T03:49:14.958178+0000 osd.0 (osd.0) 654 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:49:16.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: cluster 2024-04-06T03:49:14.960665+0000 osd.0 (osd.0) 655 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:49:16.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: audit 2024-04-06T03:49:15.137972+0000 mon.c (mon.2) 125 : audit 1 from='client.? 172.21.15.50:0/303771216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:16.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: audit 2024-04-06T03:49:15.138587+0000 mon.a (mon.0) 1849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:16.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: cluster 2024-04-06T03:49:15.139028+0000 osd.7 (osd.7) 679 : cluster 0 2.0 scrub starts 2024-04-06T03:49:16.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: cluster 2024-04-06T03:49:15.141749+0000 osd.7 (osd.7) 680 : cluster 0 2.0 scrub ok 2024-04-06T03:49:16.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: cluster 2024-04-06T03:49:15.153779+0000 osd.4 (osd.4) 631 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:49:16.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[21467]: cluster 2024-04-06T03:49:15.156464+0000 osd.4 (osd.4) 632 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: cluster 2024-04-06T03:49:14.958178+0000 osd.0 (osd.0) 654 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: cluster 2024-04-06T03:49:14.960665+0000 osd.0 (osd.0) 655 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: audit 2024-04-06T03:49:15.137972+0000 mon.c (mon.2) 125 : audit 1 from='client.? 172.21.15.50:0/303771216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: audit 2024-04-06T03:49:15.138587+0000 mon.a (mon.0) 1849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:16.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: cluster 2024-04-06T03:49:15.139028+0000 osd.7 (osd.7) 679 : cluster 0 2.0 scrub starts 2024-04-06T03:49:16.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: cluster 2024-04-06T03:49:15.141749+0000 osd.7 (osd.7) 680 : cluster 0 2.0 scrub ok 2024-04-06T03:49:16.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: cluster 2024-04-06T03:49:15.153779+0000 osd.4 (osd.4) 631 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:49:16.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:16 smithi050 bash[29221]: cluster 2024-04-06T03:49:15.156464+0000 osd.4 (osd.4) 632 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:49:16.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: cluster 2024-04-06T03:49:14.958178+0000 osd.0 (osd.0) 654 : cluster 0 5.5 deep-scrub starts 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: cluster 2024-04-06T03:49:14.960665+0000 osd.0 (osd.0) 655 : cluster 0 5.5 deep-scrub ok 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: audit 2024-04-06T03:49:15.137972+0000 mon.c (mon.2) 125 : audit 1 from='client.? 172.21.15.50:0/303771216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: audit 2024-04-06T03:49:15.138587+0000 mon.a (mon.0) 1849 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: cluster 2024-04-06T03:49:15.139028+0000 osd.7 (osd.7) 679 : cluster 0 2.0 scrub starts 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: cluster 2024-04-06T03:49:15.141749+0000 osd.7 (osd.7) 680 : cluster 0 2.0 scrub ok 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: cluster 2024-04-06T03:49:15.153779+0000 osd.4 (osd.4) 631 : cluster 0 2.18 deep-scrub starts 2024-04-06T03:49:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:16 smithi153 bash[23852]: cluster 2024-04-06T03:49:15.156464+0000 osd.4 (osd.4) 632 : cluster 0 2.18 deep-scrub ok 2024-04-06T03:49:16.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 1 1 2024-04-06T03:49:17.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[21467]: cluster 2024-04-06T03:49:15.997426+0000 osd.0 (osd.0) 656 : cluster 0 6.10 scrub starts 2024-04-06T03:49:17.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[21467]: audit 2024-04-06T03:49:16.068049+0000 mon.a (mon.0) 1850 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:49:17.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[21467]: cluster 2024-04-06T03:49:16.076361+0000 mon.a (mon.0) 1851 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-04-06T03:49:17.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[21467]: cluster 2024-04-06T03:49:16.097742+0000 osd.0 (osd.0) 657 : cluster 0 6.10 scrub ok 2024-04-06T03:49:17.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[21467]: cluster 2024-04-06T03:49:16.618797+0000 mgr.y (mgr.24368) 1039 : cluster 0 pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:17.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[29221]: cluster 2024-04-06T03:49:15.997426+0000 osd.0 (osd.0) 656 : cluster 0 6.10 scrub starts 2024-04-06T03:49:17.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[29221]: audit 2024-04-06T03:49:16.068049+0000 mon.a (mon.0) 1850 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:49:17.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[29221]: cluster 2024-04-06T03:49:16.076361+0000 mon.a (mon.0) 1851 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-04-06T03:49:17.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[29221]: cluster 2024-04-06T03:49:16.097742+0000 osd.0 (osd.0) 657 : cluster 0 6.10 scrub ok 2024-04-06T03:49:17.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:17 smithi050 bash[29221]: cluster 2024-04-06T03:49:16.618797+0000 mgr.y (mgr.24368) 1039 : cluster 0 pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:17.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:17 smithi153 bash[23852]: cluster 2024-04-06T03:49:15.997426+0000 osd.0 (osd.0) 656 : cluster 0 6.10 scrub starts 2024-04-06T03:49:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:17 smithi153 bash[23852]: audit 2024-04-06T03:49:16.068049+0000 mon.a (mon.0) 1850 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:49:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:17 smithi153 bash[23852]: cluster 2024-04-06T03:49:16.076361+0000 mon.a (mon.0) 1851 : cluster 0 osdmap e412: 8 total, 8 up, 8 in 2024-04-06T03:49:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:17 smithi153 bash[23852]: cluster 2024-04-06T03:49:16.097742+0000 osd.0 (osd.0) 657 : cluster 0 6.10 scrub ok 2024-04-06T03:49:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:17 smithi153 bash[23852]: cluster 2024-04-06T03:49:16.618797+0000 mgr.y (mgr.24368) 1039 : cluster 0 pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:18.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:18 smithi050 bash[21467]: cluster 2024-04-06T03:49:16.973553+0000 osd.0 (osd.0) 658 : cluster 0 5.1b scrub starts 2024-04-06T03:49:18.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:18 smithi050 bash[21467]: cluster 2024-04-06T03:49:16.975686+0000 osd.0 (osd.0) 659 : cluster 0 5.1b scrub ok 2024-04-06T03:49:18.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:18 smithi050 bash[29221]: cluster 2024-04-06T03:49:16.973553+0000 osd.0 (osd.0) 658 : cluster 0 5.1b scrub starts 2024-04-06T03:49:18.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:18 smithi050 bash[29221]: cluster 2024-04-06T03:49:16.975686+0000 osd.0 (osd.0) 659 : cluster 0 5.1b scrub ok 2024-04-06T03:49:18.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:18 smithi153 bash[23852]: cluster 2024-04-06T03:49:16.973553+0000 osd.0 (osd.0) 658 : cluster 0 5.1b scrub starts 2024-04-06T03:49:18.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:18 smithi153 bash[23852]: cluster 2024-04-06T03:49:16.975686+0000 osd.0 (osd.0) 659 : cluster 0 5.1b scrub ok 2024-04-06T03:49:19.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[29221]: cluster 2024-04-06T03:49:18.152344+0000 osd.4 (osd.4) 633 : cluster 0 6.e scrub starts 2024-04-06T03:49:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[29221]: cluster 2024-04-06T03:49:18.232399+0000 osd.4 (osd.4) 634 : cluster 0 6.e scrub ok 2024-04-06T03:49:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[29221]: cluster 2024-04-06T03:49:18.619858+0000 mgr.y (mgr.24368) 1040 : cluster 0 pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[21467]: cluster 2024-04-06T03:49:18.152344+0000 osd.4 (osd.4) 633 : cluster 0 6.e scrub starts 2024-04-06T03:49:19.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[21467]: cluster 2024-04-06T03:49:18.232399+0000 osd.4 (osd.4) 634 : cluster 0 6.e scrub ok 2024-04-06T03:49:19.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[21467]: cluster 2024-04-06T03:49:18.619858+0000 mgr.y (mgr.24368) 1040 : cluster 0 pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:19.492 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:49:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:49:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:49:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:19 smithi153 bash[23852]: cluster 2024-04-06T03:49:18.152344+0000 osd.4 (osd.4) 633 : cluster 0 6.e scrub starts 2024-04-06T03:49:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:19 smithi153 bash[23852]: cluster 2024-04-06T03:49:18.232399+0000 osd.4 (osd.4) 634 : cluster 0 6.e scrub ok 2024-04-06T03:49:19.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:19 smithi153 bash[23852]: cluster 2024-04-06T03:49:18.619858+0000 mgr.y (mgr.24368) 1040 : cluster 0 pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:19.611 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:20.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:49:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:49:20.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:20 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.140153+0000 osd.4 (osd.4) 635 : cluster 0 6.1a scrub starts 2024-04-06T03:49:20.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:20 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.222155+0000 osd.4 (osd.4) 636 : cluster 0 6.1a scrub ok 2024-04-06T03:49:20.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:20 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.140153+0000 osd.4 (osd.4) 635 : cluster 0 6.1a scrub starts 2024-04-06T03:49:20.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:20 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.222155+0000 osd.4 (osd.4) 636 : cluster 0 6.1a scrub ok 2024-04-06T03:49:20.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:20 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.140153+0000 osd.4 (osd.4) 635 : cluster 0 6.1a scrub starts 2024-04-06T03:49:20.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:20 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.222155+0000 osd.4 (osd.4) 636 : cluster 0 6.1a scrub ok 2024-04-06T03:49:21.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.177449+0000 osd.7 (osd.7) 681 : cluster 0 3.11 scrub starts 2024-04-06T03:49:21.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.182383+0000 osd.7 (osd.7) 682 : cluster 0 3.11 scrub ok 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: audit 2024-04-06T03:49:20.429587+0000 mgr.y (mgr.24368) 1041 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.621154+0000 mgr.y (mgr.24368) 1042 : cluster 0 pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.841281+0000 osd.3 (osd.3) 661 : cluster 0 3.8 scrub starts 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: cluster 2024-04-06T03:49:20.844293+0000 osd.3 (osd.3) 662 : cluster 0 3.8 scrub ok 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[21467]: audit 2024-04-06T03:49:20.873307+0000 mon.a (mon.0) 1852 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.177449+0000 osd.7 (osd.7) 681 : cluster 0 3.11 scrub starts 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.182383+0000 osd.7 (osd.7) 682 : cluster 0 3.11 scrub ok 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: audit 2024-04-06T03:49:20.429587+0000 mgr.y (mgr.24368) 1041 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.621154+0000 mgr.y (mgr.24368) 1042 : cluster 0 pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.841281+0000 osd.3 (osd.3) 661 : cluster 0 3.8 scrub starts 2024-04-06T03:49:21.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: cluster 2024-04-06T03:49:20.844293+0000 osd.3 (osd.3) 662 : cluster 0 3.8 scrub ok 2024-04-06T03:49:21.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:21 smithi050 bash[29221]: audit 2024-04-06T03:49:20.873307+0000 mon.a (mon.0) 1852 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.177449+0000 osd.7 (osd.7) 681 : cluster 0 3.11 scrub starts 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.182383+0000 osd.7 (osd.7) 682 : cluster 0 3.11 scrub ok 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: audit 2024-04-06T03:49:20.429587+0000 mgr.y (mgr.24368) 1041 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.621154+0000 mgr.y (mgr.24368) 1042 : cluster 0 pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.841281+0000 osd.3 (osd.3) 661 : cluster 0 3.8 scrub starts 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: cluster 2024-04-06T03:49:20.844293+0000 osd.3 (osd.3) 662 : cluster 0 3.8 scrub ok 2024-04-06T03:49:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:21 smithi153 bash[23852]: audit 2024-04-06T03:49:20.873307+0000 mon.a (mon.0) 1852 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:22.071 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:22.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[21467]: cluster 2024-04-06T03:49:21.800860+0000 osd.3 (osd.3) 663 : cluster 0 5.17 scrub starts 2024-04-06T03:49:22.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[21467]: cluster 2024-04-06T03:49:21.803227+0000 osd.3 (osd.3) 664 : cluster 0 5.17 scrub ok 2024-04-06T03:49:22.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[21467]: cluster 2024-04-06T03:49:22.130584+0000 osd.4 (osd.4) 637 : cluster 0 6.2 scrub starts 2024-04-06T03:49:22.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[21467]: cluster 2024-04-06T03:49:22.133488+0000 osd.4 (osd.4) 638 : cluster 0 6.2 scrub ok 2024-04-06T03:49:22.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[29221]: cluster 2024-04-06T03:49:21.800860+0000 osd.3 (osd.3) 663 : cluster 0 5.17 scrub starts 2024-04-06T03:49:22.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[29221]: cluster 2024-04-06T03:49:21.803227+0000 osd.3 (osd.3) 664 : cluster 0 5.17 scrub ok 2024-04-06T03:49:22.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[29221]: cluster 2024-04-06T03:49:22.130584+0000 osd.4 (osd.4) 637 : cluster 0 6.2 scrub starts 2024-04-06T03:49:22.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:22 smithi050 bash[29221]: cluster 2024-04-06T03:49:22.133488+0000 osd.4 (osd.4) 638 : cluster 0 6.2 scrub ok 2024-04-06T03:49:23.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:22 smithi153 bash[23852]: cluster 2024-04-06T03:49:21.800860+0000 osd.3 (osd.3) 663 : cluster 0 5.17 scrub starts 2024-04-06T03:49:23.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:22 smithi153 bash[23852]: cluster 2024-04-06T03:49:21.803227+0000 osd.3 (osd.3) 664 : cluster 0 5.17 scrub ok 2024-04-06T03:49:23.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:22 smithi153 bash[23852]: cluster 2024-04-06T03:49:22.130584+0000 osd.4 (osd.4) 637 : cluster 0 6.2 scrub starts 2024-04-06T03:49:23.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:22 smithi153 bash[23852]: cluster 2024-04-06T03:49:22.133488+0000 osd.4 (osd.4) 638 : cluster 0 6.2 scrub ok 2024-04-06T03:49:23.405 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:49:23.406 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:49:23.406 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-06T03:49:23.406 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:49:23.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:23 smithi050 bash[21467]: cluster 2024-04-06T03:49:22.621819+0000 mgr.y (mgr.24368) 1043 : cluster 0 pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:23.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:23 smithi050 bash[21467]: cluster 2024-04-06T03:49:22.785496+0000 osd.3 (osd.3) 665 : cluster 0 6.1f deep-scrub starts 2024-04-06T03:49:23.927 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:23 smithi050 bash[21467]: cluster 2024-04-06T03:49:23.012822+0000 osd.3 (osd.3) 666 : cluster 0 6.1f deep-scrub ok 2024-04-06T03:49:23.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:23 smithi050 bash[29221]: cluster 2024-04-06T03:49:22.621819+0000 mgr.y (mgr.24368) 1043 : cluster 0 pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:23.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:23 smithi050 bash[29221]: cluster 2024-04-06T03:49:22.785496+0000 osd.3 (osd.3) 665 : cluster 0 6.1f deep-scrub starts 2024-04-06T03:49:23.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:23 smithi050 bash[29221]: cluster 2024-04-06T03:49:23.012822+0000 osd.3 (osd.3) 666 : cluster 0 6.1f deep-scrub ok 2024-04-06T03:49:24.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:23 smithi153 bash[23852]: cluster 2024-04-06T03:49:22.621819+0000 mgr.y (mgr.24368) 1043 : cluster 0 pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:24.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:23 smithi153 bash[23852]: cluster 2024-04-06T03:49:22.785496+0000 osd.3 (osd.3) 665 : cluster 0 6.1f deep-scrub starts 2024-04-06T03:49:24.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:23 smithi153 bash[23852]: cluster 2024-04-06T03:49:23.012822+0000 osd.3 (osd.3) 666 : cluster 0 6.1f deep-scrub ok 2024-04-06T03:49:24.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[21467]: cluster 2024-04-06T03:49:23.271848+0000 osd.7 (osd.7) 683 : cluster 0 5.9 scrub starts 2024-04-06T03:49:24.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[21467]: cluster 2024-04-06T03:49:23.274834+0000 osd.7 (osd.7) 684 : cluster 0 5.9 scrub ok 2024-04-06T03:49:24.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[21467]: cluster 2024-04-06T03:49:23.803535+0000 osd.3 (osd.3) 667 : cluster 0 6.7 scrub starts 2024-04-06T03:49:24.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[21467]: cluster 2024-04-06T03:49:23.968750+0000 osd.3 (osd.3) 668 : cluster 0 6.7 scrub ok 2024-04-06T03:49:24.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[29221]: cluster 2024-04-06T03:49:23.271848+0000 osd.7 (osd.7) 683 : cluster 0 5.9 scrub starts 2024-04-06T03:49:24.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[29221]: cluster 2024-04-06T03:49:23.274834+0000 osd.7 (osd.7) 684 : cluster 0 5.9 scrub ok 2024-04-06T03:49:24.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[29221]: cluster 2024-04-06T03:49:23.803535+0000 osd.3 (osd.3) 667 : cluster 0 6.7 scrub starts 2024-04-06T03:49:24.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:24 smithi050 bash[29221]: cluster 2024-04-06T03:49:23.968750+0000 osd.3 (osd.3) 668 : cluster 0 6.7 scrub ok 2024-04-06T03:49:24.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_blocked_ops 2024-04-06T03:49:25.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:24 smithi153 bash[23852]: cluster 2024-04-06T03:49:23.271848+0000 osd.7 (osd.7) 683 : cluster 0 5.9 scrub starts 2024-04-06T03:49:25.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:24 smithi153 bash[23852]: cluster 2024-04-06T03:49:23.274834+0000 osd.7 (osd.7) 684 : cluster 0 5.9 scrub ok 2024-04-06T03:49:25.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:24 smithi153 bash[23852]: cluster 2024-04-06T03:49:23.803535+0000 osd.3 (osd.3) 667 : cluster 0 6.7 scrub starts 2024-04-06T03:49:25.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:24 smithi153 bash[23852]: cluster 2024-04-06T03:49:23.968750+0000 osd.3 (osd.3) 668 : cluster 0 6.7 scrub ok 2024-04-06T03:49:25.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:25 smithi050 bash[29221]: cluster 2024-04-06T03:49:24.446328+0000 osd.2 (osd.2) 559 : cluster 0 2.16 scrub starts 2024-04-06T03:49:25.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:25 smithi050 bash[29221]: cluster 2024-04-06T03:49:24.449464+0000 osd.2 (osd.2) 560 : cluster 0 2.16 scrub ok 2024-04-06T03:49:25.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:25 smithi050 bash[29221]: cluster 2024-04-06T03:49:24.622875+0000 mgr.y (mgr.24368) 1044 : cluster 0 pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:25.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:25 smithi050 bash[21467]: cluster 2024-04-06T03:49:24.446328+0000 osd.2 (osd.2) 559 : cluster 0 2.16 scrub starts 2024-04-06T03:49:25.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:25 smithi050 bash[21467]: cluster 2024-04-06T03:49:24.449464+0000 osd.2 (osd.2) 560 : cluster 0 2.16 scrub ok 2024-04-06T03:49:25.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:25 smithi050 bash[21467]: cluster 2024-04-06T03:49:24.622875+0000 mgr.y (mgr.24368) 1044 : cluster 0 pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:26.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:25 smithi153 bash[23852]: cluster 2024-04-06T03:49:24.446328+0000 osd.2 (osd.2) 559 : cluster 0 2.16 scrub starts 2024-04-06T03:49:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:25 smithi153 bash[23852]: cluster 2024-04-06T03:49:24.449464+0000 osd.2 (osd.2) 560 : cluster 0 2.16 scrub ok 2024-04-06T03:49:26.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:25 smithi153 bash[23852]: cluster 2024-04-06T03:49:24.622875+0000 mgr.y (mgr.24368) 1044 : cluster 0 pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:26.578 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-06T03:49:26.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:26 smithi050 bash[21467]: audit 2024-04-06T03:49:26.488160+0000 mon.a (mon.0) 1853 : audit 1 from='client.? 172.21.15.50:0/906295874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:26.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:26 smithi050 bash[29221]: audit 2024-04-06T03:49:26.488160+0000 mon.a (mon.0) 1853 : audit 1 from='client.? 172.21.15.50:0/906295874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:27.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:26 smithi153 bash[23852]: audit 2024-04-06T03:49:26.488160+0000 mon.a (mon.0) 1853 : audit 1 from='client.? 172.21.15.50:0/906295874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-06T03:49:27.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 2 1 2024-04-06T03:49:27.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:27 smithi050 bash[21467]: audit 2024-04-06T03:49:26.571769+0000 mon.a (mon.0) 1854 : audit 1 from='client.? 172.21.15.50:0/906295874' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:49:27.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:27 smithi050 bash[21467]: cluster 2024-04-06T03:49:26.578615+0000 mon.a (mon.0) 1855 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-04-06T03:49:27.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:27 smithi050 bash[21467]: cluster 2024-04-06T03:49:26.623396+0000 mgr.y (mgr.24368) 1045 : cluster 0 pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:27.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:27 smithi050 bash[29221]: audit 2024-04-06T03:49:26.571769+0000 mon.a (mon.0) 1854 : audit 1 from='client.? 172.21.15.50:0/906295874' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:49:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:27 smithi050 bash[29221]: cluster 2024-04-06T03:49:26.578615+0000 mon.a (mon.0) 1855 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-04-06T03:49:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:27 smithi050 bash[29221]: cluster 2024-04-06T03:49:26.623396+0000 mgr.y (mgr.24368) 1045 : cluster 0 pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:28.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:27 smithi153 bash[23852]: audit 2024-04-06T03:49:26.571769+0000 mon.a (mon.0) 1854 : audit 1 from='client.? 172.21.15.50:0/906295874' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-06T03:49:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:27 smithi153 bash[23852]: cluster 2024-04-06T03:49:26.578615+0000 mon.a (mon.0) 1855 : cluster 0 osdmap e413: 8 total, 8 up, 8 in 2024-04-06T03:49:28.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:27 smithi153 bash[23852]: cluster 2024-04-06T03:49:26.623396+0000 mgr.y (mgr.24368) 1045 : cluster 0 pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:28.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[29221]: cluster 2024-04-06T03:49:27.602069+0000 mon.a (mon.0) 1856 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-04-06T03:49:28.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[29221]: cluster 2024-04-06T03:49:28.014801+0000 osd.0 (osd.0) 660 : cluster 0 5.3 scrub starts 2024-04-06T03:49:28.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[29221]: cluster 2024-04-06T03:49:28.017291+0000 osd.0 (osd.0) 661 : cluster 0 5.3 scrub ok 2024-04-06T03:49:28.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[29221]: cluster 2024-04-06T03:49:28.046754+0000 osd.4 (osd.4) 639 : cluster 0 4.1 scrub starts 2024-04-06T03:49:28.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[29221]: cluster 2024-04-06T03:49:28.049458+0000 osd.4 (osd.4) 640 : cluster 0 4.1 scrub ok 2024-04-06T03:49:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[21467]: cluster 2024-04-06T03:49:27.602069+0000 mon.a (mon.0) 1856 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-04-06T03:49:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[21467]: cluster 2024-04-06T03:49:28.014801+0000 osd.0 (osd.0) 660 : cluster 0 5.3 scrub starts 2024-04-06T03:49:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[21467]: cluster 2024-04-06T03:49:28.017291+0000 osd.0 (osd.0) 661 : cluster 0 5.3 scrub ok 2024-04-06T03:49:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[21467]: cluster 2024-04-06T03:49:28.046754+0000 osd.4 (osd.4) 639 : cluster 0 4.1 scrub starts 2024-04-06T03:49:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:28 smithi050 bash[21467]: cluster 2024-04-06T03:49:28.049458+0000 osd.4 (osd.4) 640 : cluster 0 4.1 scrub ok 2024-04-06T03:49:29.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:28 smithi153 bash[23852]: cluster 2024-04-06T03:49:27.602069+0000 mon.a (mon.0) 1856 : cluster 0 osdmap e414: 8 total, 8 up, 8 in 2024-04-06T03:49:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:28 smithi153 bash[23852]: cluster 2024-04-06T03:49:28.014801+0000 osd.0 (osd.0) 660 : cluster 0 5.3 scrub starts 2024-04-06T03:49:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:28 smithi153 bash[23852]: cluster 2024-04-06T03:49:28.017291+0000 osd.0 (osd.0) 661 : cluster 0 5.3 scrub ok 2024-04-06T03:49:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:28 smithi153 bash[23852]: cluster 2024-04-06T03:49:28.046754+0000 osd.4 (osd.4) 639 : cluster 0 4.1 scrub starts 2024-04-06T03:49:29.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:28 smithi153 bash[23852]: cluster 2024-04-06T03:49:28.049458+0000 osd.4 (osd.4) 640 : cluster 0 4.1 scrub ok 2024-04-06T03:49:29.604 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:49:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:49:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[29221]: cluster 2024-04-06T03:49:28.624290+0000 mgr.y (mgr.24368) 1046 : cluster 0 pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[29221]: audit 2024-04-06T03:49:28.924163+0000 mon.a (mon.0) 1857 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:49:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[29221]: cluster 2024-04-06T03:49:29.095214+0000 osd.4 (osd.4) 641 : cluster 0 5.1 scrub starts 2024-04-06T03:49:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[29221]: cluster 2024-04-06T03:49:29.097370+0000 osd.4 (osd.4) 642 : cluster 0 5.1 scrub ok 2024-04-06T03:49:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[21467]: cluster 2024-04-06T03:49:28.624290+0000 mgr.y (mgr.24368) 1046 : cluster 0 pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[21467]: audit 2024-04-06T03:49:28.924163+0000 mon.a (mon.0) 1857 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:49:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[21467]: cluster 2024-04-06T03:49:29.095214+0000 osd.4 (osd.4) 641 : cluster 0 5.1 scrub starts 2024-04-06T03:49:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:29 smithi050 bash[21467]: cluster 2024-04-06T03:49:29.097370+0000 osd.4 (osd.4) 642 : cluster 0 5.1 scrub ok 2024-04-06T03:49:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:30.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:29 smithi153 bash[23852]: cluster 2024-04-06T03:49:28.624290+0000 mgr.y (mgr.24368) 1046 : cluster 0 pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:29 smithi153 bash[23852]: audit 2024-04-06T03:49:28.924163+0000 mon.a (mon.0) 1857 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:49:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:29 smithi153 bash[23852]: cluster 2024-04-06T03:49:29.095214+0000 osd.4 (osd.4) 641 : cluster 0 5.1 scrub starts 2024-04-06T03:49:30.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:29 smithi153 bash[23852]: cluster 2024-04-06T03:49:29.097370+0000 osd.4 (osd.4) 642 : cluster 0 5.1 scrub ok 2024-04-06T03:49:30.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:49:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:49:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:30 smithi153 bash[23852]: cluster 2024-04-06T03:49:29.638052+0000 osd.1 (osd.1) 431 : cluster 0 6.d scrub starts 2024-04-06T03:49:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:30 smithi153 bash[23852]: audit 2024-04-06T03:49:29.653877+0000 mon.a (mon.0) 1858 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:49:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:30 smithi153 bash[23852]: audit 2024-04-06T03:49:29.655239+0000 mon.a (mon.0) 1859 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:49:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:30 smithi153 bash[23852]: audit 2024-04-06T03:49:29.664280+0000 mon.a (mon.0) 1860 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:49:30.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:30 smithi153 bash[23852]: cluster 2024-04-06T03:49:29.770579+0000 osd.1 (osd.1) 432 : cluster 0 6.d scrub ok 2024-04-06T03:49:30.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[21467]: cluster 2024-04-06T03:49:29.638052+0000 osd.1 (osd.1) 431 : cluster 0 6.d scrub starts 2024-04-06T03:49:30.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[21467]: audit 2024-04-06T03:49:29.653877+0000 mon.a (mon.0) 1858 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[21467]: audit 2024-04-06T03:49:29.655239+0000 mon.a (mon.0) 1859 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[21467]: audit 2024-04-06T03:49:29.664280+0000 mon.a (mon.0) 1860 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[21467]: cluster 2024-04-06T03:49:29.770579+0000 osd.1 (osd.1) 432 : cluster 0 6.d scrub ok 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[29221]: cluster 2024-04-06T03:49:29.638052+0000 osd.1 (osd.1) 431 : cluster 0 6.d scrub starts 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[29221]: audit 2024-04-06T03:49:29.653877+0000 mon.a (mon.0) 1858 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[29221]: audit 2024-04-06T03:49:29.655239+0000 mon.a (mon.0) 1859 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[29221]: audit 2024-04-06T03:49:29.664280+0000 mon.a (mon.0) 1860 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:49:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:30 smithi050 bash[29221]: cluster 2024-04-06T03:49:29.770579+0000 osd.1 (osd.1) 432 : cluster 0 6.d scrub ok 2024-04-06T03:49:31.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[29221]: audit 2024-04-06T03:49:30.440855+0000 mgr.y (mgr.24368) 1047 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:31.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[29221]: cluster 2024-04-06T03:49:30.625498+0000 mgr.y (mgr.24368) 1048 : cluster 0 pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:31.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[29221]: cluster 2024-04-06T03:49:31.131345+0000 osd.4 (osd.4) 643 : cluster 0 3.14 scrub starts 2024-04-06T03:49:31.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[29221]: cluster 2024-04-06T03:49:31.135776+0000 osd.4 (osd.4) 644 : cluster 0 3.14 scrub ok 2024-04-06T03:49:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[21467]: audit 2024-04-06T03:49:30.440855+0000 mgr.y (mgr.24368) 1047 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[21467]: cluster 2024-04-06T03:49:30.625498+0000 mgr.y (mgr.24368) 1048 : cluster 0 pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[21467]: cluster 2024-04-06T03:49:31.131345+0000 osd.4 (osd.4) 643 : cluster 0 3.14 scrub starts 2024-04-06T03:49:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:31 smithi050 bash[21467]: cluster 2024-04-06T03:49:31.135776+0000 osd.4 (osd.4) 644 : cluster 0 3.14 scrub ok 2024-04-06T03:49:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:31 smithi153 bash[23852]: audit 2024-04-06T03:49:30.440855+0000 mgr.y (mgr.24368) 1047 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:31 smithi153 bash[23852]: cluster 2024-04-06T03:49:30.625498+0000 mgr.y (mgr.24368) 1048 : cluster 0 pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:31 smithi153 bash[23852]: cluster 2024-04-06T03:49:31.131345+0000 osd.4 (osd.4) 643 : cluster 0 3.14 scrub starts 2024-04-06T03:49:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:31 smithi153 bash[23852]: cluster 2024-04-06T03:49:31.135776+0000 osd.4 (osd.4) 644 : cluster 0 3.14 scrub ok 2024-04-06T03:49:32.618 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[21467]: cluster 2024-04-06T03:49:31.647083+0000 osd.3 (osd.3) 669 : cluster 0 3.5 scrub starts 2024-04-06T03:49:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[21467]: cluster 2024-04-06T03:49:31.652130+0000 osd.3 (osd.3) 670 : cluster 0 3.5 scrub ok 2024-04-06T03:49:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[21467]: cluster 2024-04-06T03:49:31.937342+0000 osd.0 (osd.0) 662 : cluster 0 3.12 scrub starts 2024-04-06T03:49:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[21467]: cluster 2024-04-06T03:49:31.941723+0000 osd.0 (osd.0) 663 : cluster 0 3.12 scrub ok 2024-04-06T03:49:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[29221]: cluster 2024-04-06T03:49:31.647083+0000 osd.3 (osd.3) 669 : cluster 0 3.5 scrub starts 2024-04-06T03:49:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[29221]: cluster 2024-04-06T03:49:31.652130+0000 osd.3 (osd.3) 670 : cluster 0 3.5 scrub ok 2024-04-06T03:49:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[29221]: cluster 2024-04-06T03:49:31.937342+0000 osd.0 (osd.0) 662 : cluster 0 3.12 scrub starts 2024-04-06T03:49:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:32 smithi050 bash[29221]: cluster 2024-04-06T03:49:31.941723+0000 osd.0 (osd.0) 663 : cluster 0 3.12 scrub ok 2024-04-06T03:49:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:32 smithi153 bash[23852]: cluster 2024-04-06T03:49:31.647083+0000 osd.3 (osd.3) 669 : cluster 0 3.5 scrub starts 2024-04-06T03:49:33.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:32 smithi153 bash[23852]: cluster 2024-04-06T03:49:31.652130+0000 osd.3 (osd.3) 670 : cluster 0 3.5 scrub ok 2024-04-06T03:49:33.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:32 smithi153 bash[23852]: cluster 2024-04-06T03:49:31.937342+0000 osd.0 (osd.0) 662 : cluster 0 3.12 scrub starts 2024-04-06T03:49:33.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:32 smithi153 bash[23852]: cluster 2024-04-06T03:49:31.941723+0000 osd.0 (osd.0) 663 : cluster 0 3.12 scrub ok 2024-04-06T03:49:33.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[21467]: cluster 2024-04-06T03:49:32.626351+0000 mgr.y (mgr.24368) 1049 : cluster 0 pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:33.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.140946+0000 osd.4 (osd.4) 645 : cluster 0 5.e scrub starts 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.144060+0000 osd.4 (osd.4) 646 : cluster 0 5.e scrub ok 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.490108+0000 osd.2 (osd.2) 561 : cluster 0 6.14 scrub starts 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.493744+0000 osd.2 (osd.2) 562 : cluster 0 6.14 scrub ok 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[29221]: cluster 2024-04-06T03:49:32.626351+0000 mgr.y (mgr.24368) 1049 : cluster 0 pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.140946+0000 osd.4 (osd.4) 645 : cluster 0 5.e scrub starts 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.144060+0000 osd.4 (osd.4) 646 : cluster 0 5.e scrub ok 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.490108+0000 osd.2 (osd.2) 561 : cluster 0 6.14 scrub starts 2024-04-06T03:49:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:33 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.493744+0000 osd.2 (osd.2) 562 : cluster 0 6.14 scrub ok 2024-04-06T03:49:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:33 smithi153 bash[23852]: cluster 2024-04-06T03:49:32.626351+0000 mgr.y (mgr.24368) 1049 : cluster 0 pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:33 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.140946+0000 osd.4 (osd.4) 645 : cluster 0 5.e scrub starts 2024-04-06T03:49:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:33 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.144060+0000 osd.4 (osd.4) 646 : cluster 0 5.e scrub ok 2024-04-06T03:49:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:33 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.490108+0000 osd.2 (osd.2) 561 : cluster 0 6.14 scrub starts 2024-04-06T03:49:34.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:33 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.493744+0000 osd.2 (osd.2) 562 : cluster 0 6.14 scrub ok 2024-04-06T03:49:34.517 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:49:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-06T03:49:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-06T03:49:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-06T03:49:34.517 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:49:35.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.561075+0000 osd.1 (osd.1) 433 : cluster 0 5.1d scrub starts 2024-04-06T03:49:35.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.562795+0000 osd.1 (osd.1) 434 : cluster 0 5.1d scrub ok 2024-04-06T03:49:35.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.621955+0000 osd.3 (osd.3) 671 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:49:35.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[29221]: cluster 2024-04-06T03:49:33.624937+0000 osd.3 (osd.3) 672 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:49:35.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[29221]: cluster 2024-04-06T03:49:34.626976+0000 mgr.y (mgr.24368) 1050 : cluster 0 pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:35.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.561075+0000 osd.1 (osd.1) 433 : cluster 0 5.1d scrub starts 2024-04-06T03:49:35.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.562795+0000 osd.1 (osd.1) 434 : cluster 0 5.1d scrub ok 2024-04-06T03:49:35.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.621955+0000 osd.3 (osd.3) 671 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:49:35.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[21467]: cluster 2024-04-06T03:49:33.624937+0000 osd.3 (osd.3) 672 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:49:35.243 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:35 smithi050 bash[21467]: cluster 2024-04-06T03:49:34.626976+0000 mgr.y (mgr.24368) 1050 : cluster 0 pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:35 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.561075+0000 osd.1 (osd.1) 433 : cluster 0 5.1d scrub starts 2024-04-06T03:49:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:35 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.562795+0000 osd.1 (osd.1) 434 : cluster 0 5.1d scrub ok 2024-04-06T03:49:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:35 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.621955+0000 osd.3 (osd.3) 671 : cluster 0 5.13 deep-scrub starts 2024-04-06T03:49:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:35 smithi153 bash[23852]: cluster 2024-04-06T03:49:33.624937+0000 osd.3 (osd.3) 672 : cluster 0 5.13 deep-scrub ok 2024-04-06T03:49:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:35 smithi153 bash[23852]: cluster 2024-04-06T03:49:34.626976+0000 mgr.y (mgr.24368) 1050 : cluster 0 pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:35.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.3 dump_historic_ops 2024-04-06T03:49:36.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[29221]: cluster 2024-04-06T03:49:34.530108+0000 osd.1 (osd.1) 435 : cluster 0 6.1d scrub starts 2024-04-06T03:49:36.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[29221]: cluster 2024-04-06T03:49:34.654101+0000 osd.1 (osd.1) 436 : cluster 0 6.1d scrub ok 2024-04-06T03:49:36.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[29221]: cluster 2024-04-06T03:49:34.980888+0000 osd.0 (osd.0) 664 : cluster 0 3.6 scrub starts 2024-04-06T03:49:36.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[29221]: cluster 2024-04-06T03:49:34.985693+0000 osd.0 (osd.0) 665 : cluster 0 3.6 scrub ok 2024-04-06T03:49:36.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[29221]: audit 2024-04-06T03:49:35.890062+0000 mon.a (mon.0) 1861 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:36.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[21467]: cluster 2024-04-06T03:49:34.530108+0000 osd.1 (osd.1) 435 : cluster 0 6.1d scrub starts 2024-04-06T03:49:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[21467]: cluster 2024-04-06T03:49:34.654101+0000 osd.1 (osd.1) 436 : cluster 0 6.1d scrub ok 2024-04-06T03:49:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[21467]: cluster 2024-04-06T03:49:34.980888+0000 osd.0 (osd.0) 664 : cluster 0 3.6 scrub starts 2024-04-06T03:49:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[21467]: cluster 2024-04-06T03:49:34.985693+0000 osd.0 (osd.0) 665 : cluster 0 3.6 scrub ok 2024-04-06T03:49:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:36 smithi050 bash[21467]: audit 2024-04-06T03:49:35.890062+0000 mon.a (mon.0) 1861 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:36 smithi153 bash[23852]: cluster 2024-04-06T03:49:34.530108+0000 osd.1 (osd.1) 435 : cluster 0 6.1d scrub starts 2024-04-06T03:49:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:36 smithi153 bash[23852]: cluster 2024-04-06T03:49:34.654101+0000 osd.1 (osd.1) 436 : cluster 0 6.1d scrub ok 2024-04-06T03:49:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:36 smithi153 bash[23852]: cluster 2024-04-06T03:49:34.980888+0000 osd.0 (osd.0) 664 : cluster 0 3.6 scrub starts 2024-04-06T03:49:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:36 smithi153 bash[23852]: cluster 2024-04-06T03:49:34.985693+0000 osd.0 (osd.0) 665 : cluster 0 3.6 scrub ok 2024-04-06T03:49:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:36 smithi153 bash[23852]: audit 2024-04-06T03:49:35.890062+0000 mon.a (mon.0) 1861 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[21467]: cluster 2024-04-06T03:49:35.504370+0000 osd.1 (osd.1) 437 : cluster 0 4.2 scrub starts 2024-04-06T03:49:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[21467]: cluster 2024-04-06T03:49:35.507504+0000 osd.1 (osd.1) 438 : cluster 0 4.2 scrub ok 2024-04-06T03:49:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[21467]: cluster 2024-04-06T03:49:35.964901+0000 osd.0 (osd.0) 666 : cluster 0 5.2 scrub starts 2024-04-06T03:49:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[21467]: cluster 2024-04-06T03:49:35.967222+0000 osd.0 (osd.0) 667 : cluster 0 5.2 scrub ok 2024-04-06T03:49:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[21467]: cluster 2024-04-06T03:49:36.628210+0000 mgr.y (mgr.24368) 1051 : cluster 0 pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[29221]: cluster 2024-04-06T03:49:35.504370+0000 osd.1 (osd.1) 437 : cluster 0 4.2 scrub starts 2024-04-06T03:49:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[29221]: cluster 2024-04-06T03:49:35.507504+0000 osd.1 (osd.1) 438 : cluster 0 4.2 scrub ok 2024-04-06T03:49:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[29221]: cluster 2024-04-06T03:49:35.964901+0000 osd.0 (osd.0) 666 : cluster 0 5.2 scrub starts 2024-04-06T03:49:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[29221]: cluster 2024-04-06T03:49:35.967222+0000 osd.0 (osd.0) 667 : cluster 0 5.2 scrub ok 2024-04-06T03:49:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:37 smithi050 bash[29221]: cluster 2024-04-06T03:49:36.628210+0000 mgr.y (mgr.24368) 1051 : cluster 0 pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:37 smithi153 bash[23852]: cluster 2024-04-06T03:49:35.504370+0000 osd.1 (osd.1) 437 : cluster 0 4.2 scrub starts 2024-04-06T03:49:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:37 smithi153 bash[23852]: cluster 2024-04-06T03:49:35.507504+0000 osd.1 (osd.1) 438 : cluster 0 4.2 scrub ok 2024-04-06T03:49:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:37 smithi153 bash[23852]: cluster 2024-04-06T03:49:35.964901+0000 osd.0 (osd.0) 666 : cluster 0 5.2 scrub starts 2024-04-06T03:49:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:37 smithi153 bash[23852]: cluster 2024-04-06T03:49:35.967222+0000 osd.0 (osd.0) 667 : cluster 0 5.2 scrub ok 2024-04-06T03:49:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:37 smithi153 bash[23852]: cluster 2024-04-06T03:49:36.628210+0000 mgr.y (mgr.24368) 1051 : cluster 0 pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:38.091 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-06T03:49:38.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[21467]: cluster 2024-04-06T03:49:36.550962+0000 osd.1 (osd.1) 439 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:49:38.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[21467]: cluster 2024-04-06T03:49:36.552911+0000 osd.1 (osd.1) 440 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[21467]: cluster 2024-04-06T03:49:36.651128+0000 osd.3 (osd.3) 673 : cluster 0 2.19 deep-scrub starts 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[21467]: cluster 2024-04-06T03:49:36.655597+0000 osd.3 (osd.3) 674 : cluster 0 2.19 deep-scrub ok 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[21467]: audit 2024-04-06T03:49:37.223662+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.50:0/4073716021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[21467]: audit 2024-04-06T03:49:37.224370+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[29221]: cluster 2024-04-06T03:49:36.550962+0000 osd.1 (osd.1) 439 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[29221]: cluster 2024-04-06T03:49:36.552911+0000 osd.1 (osd.1) 440 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[29221]: cluster 2024-04-06T03:49:36.651128+0000 osd.3 (osd.3) 673 : cluster 0 2.19 deep-scrub starts 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[29221]: cluster 2024-04-06T03:49:36.655597+0000 osd.3 (osd.3) 674 : cluster 0 2.19 deep-scrub ok 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[29221]: audit 2024-04-06T03:49:37.223662+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.50:0/4073716021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:38.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:38 smithi050 bash[29221]: audit 2024-04-06T03:49:37.224370+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:38 smithi153 bash[23852]: cluster 2024-04-06T03:49:36.550962+0000 osd.1 (osd.1) 439 : cluster 0 6.4 deep-scrub starts 2024-04-06T03:49:38.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:38 smithi153 bash[23852]: cluster 2024-04-06T03:49:36.552911+0000 osd.1 (osd.1) 440 : cluster 0 6.4 deep-scrub ok 2024-04-06T03:49:38.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:38 smithi153 bash[23852]: cluster 2024-04-06T03:49:36.651128+0000 osd.3 (osd.3) 673 : cluster 0 2.19 deep-scrub starts 2024-04-06T03:49:38.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:38 smithi153 bash[23852]: cluster 2024-04-06T03:49:36.655597+0000 osd.3 (osd.3) 674 : cluster 0 2.19 deep-scrub ok 2024-04-06T03:49:38.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:38 smithi153 bash[23852]: audit 2024-04-06T03:49:37.223662+0000 mon.c (mon.2) 126 : audit 1 from='client.? 172.21.15.50:0/4073716021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:38.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:38 smithi153 bash[23852]: audit 2024-04-06T03:49:37.224370+0000 mon.a (mon.0) 1862 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:38.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 2 1 2024-04-06T03:49:39.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[29221]: audit 2024-04-06T03:49:38.078697+0000 mon.a (mon.0) 1863 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:49:39.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[29221]: cluster 2024-04-06T03:49:38.086392+0000 mon.a (mon.0) 1864 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-04-06T03:49:39.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[29221]: cluster 2024-04-06T03:49:38.288859+0000 osd.7 (osd.7) 685 : cluster 0 3.15 scrub starts 2024-04-06T03:49:39.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[29221]: cluster 2024-04-06T03:49:38.293004+0000 osd.7 (osd.7) 686 : cluster 0 3.15 scrub ok 2024-04-06T03:49:39.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[29221]: cluster 2024-04-06T03:49:38.629209+0000 mgr.y (mgr.24368) 1052 : cluster 0 pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[21467]: audit 2024-04-06T03:49:38.078697+0000 mon.a (mon.0) 1863 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:49:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[21467]: cluster 2024-04-06T03:49:38.086392+0000 mon.a (mon.0) 1864 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-04-06T03:49:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[21467]: cluster 2024-04-06T03:49:38.288859+0000 osd.7 (osd.7) 685 : cluster 0 3.15 scrub starts 2024-04-06T03:49:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[21467]: cluster 2024-04-06T03:49:38.293004+0000 osd.7 (osd.7) 686 : cluster 0 3.15 scrub ok 2024-04-06T03:49:39.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[21467]: cluster 2024-04-06T03:49:38.629209+0000 mgr.y (mgr.24368) 1052 : cluster 0 pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:39.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:49:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:49:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:49:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:39 smithi153 bash[23852]: audit 2024-04-06T03:49:38.078697+0000 mon.a (mon.0) 1863 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:49:39.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:39 smithi153 bash[23852]: cluster 2024-04-06T03:49:38.086392+0000 mon.a (mon.0) 1864 : cluster 0 osdmap e415: 8 total, 8 up, 8 in 2024-04-06T03:49:39.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:39 smithi153 bash[23852]: cluster 2024-04-06T03:49:38.288859+0000 osd.7 (osd.7) 685 : cluster 0 3.15 scrub starts 2024-04-06T03:49:39.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:39 smithi153 bash[23852]: cluster 2024-04-06T03:49:38.293004+0000 osd.7 (osd.7) 686 : cluster 0 3.15 scrub ok 2024-04-06T03:49:39.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:39 smithi153 bash[23852]: cluster 2024-04-06T03:49:38.629209+0000 mgr.y (mgr.24368) 1052 : cluster 0 pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:49:40.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:40 smithi153 bash[23852]: cluster 2024-04-06T03:49:38.855156+0000 osd.0 (osd.0) 668 : cluster 0 3.1f scrub starts 2024-04-06T03:49:40.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:40 smithi153 bash[23852]: cluster 2024-04-06T03:49:38.859586+0000 osd.0 (osd.0) 669 : cluster 0 3.1f scrub ok 2024-04-06T03:49:40.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:40 smithi153 bash[23852]: cluster 2024-04-06T03:49:39.153718+0000 osd.4 (osd.4) 647 : cluster 0 5.1c deep-scrub starts 2024-04-06T03:49:40.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:40 smithi153 bash[23852]: cluster 2024-04-06T03:49:39.155856+0000 osd.4 (osd.4) 648 : cluster 0 5.1c deep-scrub ok 2024-04-06T03:49:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[21467]: cluster 2024-04-06T03:49:38.855156+0000 osd.0 (osd.0) 668 : cluster 0 3.1f scrub starts 2024-04-06T03:49:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[21467]: cluster 2024-04-06T03:49:38.859586+0000 osd.0 (osd.0) 669 : cluster 0 3.1f scrub ok 2024-04-06T03:49:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[21467]: cluster 2024-04-06T03:49:39.153718+0000 osd.4 (osd.4) 647 : cluster 0 5.1c deep-scrub starts 2024-04-06T03:49:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[21467]: cluster 2024-04-06T03:49:39.155856+0000 osd.4 (osd.4) 648 : cluster 0 5.1c deep-scrub ok 2024-04-06T03:49:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[29221]: cluster 2024-04-06T03:49:38.855156+0000 osd.0 (osd.0) 668 : cluster 0 3.1f scrub starts 2024-04-06T03:49:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[29221]: cluster 2024-04-06T03:49:38.859586+0000 osd.0 (osd.0) 669 : cluster 0 3.1f scrub ok 2024-04-06T03:49:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[29221]: cluster 2024-04-06T03:49:39.153718+0000 osd.4 (osd.4) 647 : cluster 0 5.1c deep-scrub starts 2024-04-06T03:49:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:40 smithi050 bash[29221]: cluster 2024-04-06T03:49:39.155856+0000 osd.4 (osd.4) 648 : cluster 0 5.1c deep-scrub ok 2024-04-06T03:49:40.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:49:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:49:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[21467]: cluster 2024-04-06T03:49:39.768667+0000 osd.5 (osd.5) 157 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[21467]: cluster 2024-04-06T03:49:39.768766+0000 osd.5 (osd.5) 158 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[21467]: cluster 2024-04-06T03:49:40.119517+0000 mon.a (mon.0) 1865 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-04-06T03:49:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[21467]: audit 2024-04-06T03:49:40.448571+0000 mgr.y (mgr.24368) 1053 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[21467]: cluster 2024-04-06T03:49:40.630049+0000 mgr.y (mgr.24368) 1054 : cluster 0 pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[29221]: cluster 2024-04-06T03:49:39.768667+0000 osd.5 (osd.5) 157 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[29221]: cluster 2024-04-06T03:49:39.768766+0000 osd.5 (osd.5) 158 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[29221]: cluster 2024-04-06T03:49:40.119517+0000 mon.a (mon.0) 1865 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-04-06T03:49:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[29221]: audit 2024-04-06T03:49:40.448571+0000 mgr.y (mgr.24368) 1053 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:41 smithi050 bash[29221]: cluster 2024-04-06T03:49:40.630049+0000 mgr.y (mgr.24368) 1054 : cluster 0 pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:41 smithi153 bash[23852]: cluster 2024-04-06T03:49:39.768667+0000 osd.5 (osd.5) 157 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:41.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:41 smithi153 bash[23852]: cluster 2024-04-06T03:49:39.768766+0000 osd.5 (osd.5) 158 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:41.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:41 smithi153 bash[23852]: cluster 2024-04-06T03:49:40.119517+0000 mon.a (mon.0) 1865 : cluster 0 osdmap e416: 8 total, 8 up, 8 in 2024-04-06T03:49:41.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:41 smithi153 bash[23852]: audit 2024-04-06T03:49:40.448571+0000 mgr.y (mgr.24368) 1053 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:41.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:41 smithi153 bash[23852]: cluster 2024-04-06T03:49:40.630049+0000 mgr.y (mgr.24368) 1054 : cluster 0 pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:41.712 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:42.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:42 smithi050 bash[21467]: cluster 2024-04-06T03:49:41.176745+0000 osd.4 (osd.4) 649 : cluster 0 3.3 scrub starts 2024-04-06T03:49:42.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:42 smithi050 bash[21467]: cluster 2024-04-06T03:49:41.181933+0000 osd.4 (osd.4) 650 : cluster 0 3.3 scrub ok 2024-04-06T03:49:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:42 smithi050 bash[29221]: cluster 2024-04-06T03:49:41.176745+0000 osd.4 (osd.4) 649 : cluster 0 3.3 scrub starts 2024-04-06T03:49:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:42 smithi050 bash[29221]: cluster 2024-04-06T03:49:41.181933+0000 osd.4 (osd.4) 650 : cluster 0 3.3 scrub ok 2024-04-06T03:49:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:42 smithi153 bash[23852]: cluster 2024-04-06T03:49:41.176745+0000 osd.4 (osd.4) 649 : cluster 0 3.3 scrub starts 2024-04-06T03:49:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:42 smithi153 bash[23852]: cluster 2024-04-06T03:49:41.181933+0000 osd.4 (osd.4) 650 : cluster 0 3.3 scrub ok 2024-04-06T03:49:43.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:43 smithi050 bash[21467]: cluster 2024-04-06T03:49:42.630894+0000 mgr.y (mgr.24368) 1055 : cluster 0 pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:43.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:43 smithi050 bash[29221]: cluster 2024-04-06T03:49:42.630894+0000 mgr.y (mgr.24368) 1055 : cluster 0 pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:43 smithi153 bash[23852]: cluster 2024-04-06T03:49:42.630894+0000 mgr.y (mgr.24368) 1055 : cluster 0 pgmap v1158: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:44.221 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:44.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[21467]: cluster 2024-04-06T03:49:42.433562+0000 osd.1 (osd.1) 441 : cluster 0 2.a scrub starts 2024-04-06T03:49:44.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[21467]: cluster 2024-04-06T03:49:42.435865+0000 osd.1 (osd.1) 442 : cluster 0 2.a scrub ok 2024-04-06T03:49:44.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[21467]: cluster 2024-04-06T03:49:42.949282+0000 osd.0 (osd.0) 670 : cluster 0 6.a scrub starts 2024-04-06T03:49:44.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[21467]: cluster 2024-04-06T03:49:43.017368+0000 osd.0 (osd.0) 671 : cluster 0 6.a scrub ok 2024-04-06T03:49:44.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[21467]: cluster 2024-04-06T03:49:43.203982+0000 osd.4 (osd.4) 651 : cluster 0 3.9 scrub starts 2024-04-06T03:49:44.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[21467]: cluster 2024-04-06T03:49:43.208735+0000 osd.4 (osd.4) 652 : cluster 0 3.9 scrub ok 2024-04-06T03:49:44.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[29221]: cluster 2024-04-06T03:49:42.433562+0000 osd.1 (osd.1) 441 : cluster 0 2.a scrub starts 2024-04-06T03:49:44.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[29221]: cluster 2024-04-06T03:49:42.435865+0000 osd.1 (osd.1) 442 : cluster 0 2.a scrub ok 2024-04-06T03:49:44.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[29221]: cluster 2024-04-06T03:49:42.949282+0000 osd.0 (osd.0) 670 : cluster 0 6.a scrub starts 2024-04-06T03:49:44.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[29221]: cluster 2024-04-06T03:49:43.017368+0000 osd.0 (osd.0) 671 : cluster 0 6.a scrub ok 2024-04-06T03:49:44.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[29221]: cluster 2024-04-06T03:49:43.203982+0000 osd.4 (osd.4) 651 : cluster 0 3.9 scrub starts 2024-04-06T03:49:44.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:44 smithi050 bash[29221]: cluster 2024-04-06T03:49:43.208735+0000 osd.4 (osd.4) 652 : cluster 0 3.9 scrub ok 2024-04-06T03:49:44.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:44 smithi153 bash[23852]: cluster 2024-04-06T03:49:42.433562+0000 osd.1 (osd.1) 441 : cluster 0 2.a scrub starts 2024-04-06T03:49:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:44 smithi153 bash[23852]: cluster 2024-04-06T03:49:42.435865+0000 osd.1 (osd.1) 442 : cluster 0 2.a scrub ok 2024-04-06T03:49:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:44 smithi153 bash[23852]: cluster 2024-04-06T03:49:42.949282+0000 osd.0 (osd.0) 670 : cluster 0 6.a scrub starts 2024-04-06T03:49:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:44 smithi153 bash[23852]: cluster 2024-04-06T03:49:43.017368+0000 osd.0 (osd.0) 671 : cluster 0 6.a scrub ok 2024-04-06T03:49:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:44 smithi153 bash[23852]: cluster 2024-04-06T03:49:43.203982+0000 osd.4 (osd.4) 651 : cluster 0 3.9 scrub starts 2024-04-06T03:49:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:44 smithi153 bash[23852]: cluster 2024-04-06T03:49:43.208735+0000 osd.4 (osd.4) 652 : cluster 0 3.9 scrub ok 2024-04-06T03:49:45.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:43.430801+0000 osd.1 (osd.1) 443 : cluster 0 5.c scrub starts 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:43.433163+0000 osd.1 (osd.1) 444 : cluster 0 5.c scrub ok 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.025627+0000 osd.4 (osd.4) 653 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.025776+0000 osd.4 (osd.4) 654 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.149583+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.204955+0000 osd.4 (osd.4) 655 : cluster 0 6.1e scrub starts 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.281176+0000 osd.4 (osd.4) 656 : cluster 0 6.1e scrub ok 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.534261+0000 osd.2 (osd.2) 563 : cluster 0 5.a scrub starts 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.536126+0000 osd.2 (osd.2) 564 : cluster 0 5.a scrub ok 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[21467]: cluster 2024-04-06T03:49:44.631785+0000 mgr.y (mgr.24368) 1056 : cluster 0 pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:43.430801+0000 osd.1 (osd.1) 443 : cluster 0 5.c scrub starts 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:43.433163+0000 osd.1 (osd.1) 444 : cluster 0 5.c scrub ok 2024-04-06T03:49:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.025627+0000 osd.4 (osd.4) 653 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.025776+0000 osd.4 (osd.4) 654 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.149583+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.204955+0000 osd.4 (osd.4) 655 : cluster 0 6.1e scrub starts 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.281176+0000 osd.4 (osd.4) 656 : cluster 0 6.1e scrub ok 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.534261+0000 osd.2 (osd.2) 563 : cluster 0 5.a scrub starts 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.536126+0000 osd.2 (osd.2) 564 : cluster 0 5.a scrub ok 2024-04-06T03:49:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:45 smithi050 bash[29221]: cluster 2024-04-06T03:49:44.631785+0000 mgr.y (mgr.24368) 1056 : cluster 0 pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:43.430801+0000 osd.1 (osd.1) 443 : cluster 0 5.c scrub starts 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:43.433163+0000 osd.1 (osd.1) 444 : cluster 0 5.c scrub ok 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.025627+0000 osd.4 (osd.4) 653 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.025776+0000 osd.4 (osd.4) 654 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.149583+0000 mon.a (mon.0) 1866 : cluster 0 osdmap e417: 8 total, 8 up, 8 in 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.204955+0000 osd.4 (osd.4) 655 : cluster 0 6.1e scrub starts 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.281176+0000 osd.4 (osd.4) 656 : cluster 0 6.1e scrub ok 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.534261+0000 osd.2 (osd.2) 563 : cluster 0 5.a scrub starts 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.536126+0000 osd.2 (osd.2) 564 : cluster 0 5.a scrub ok 2024-04-06T03:49:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:45 smithi153 bash[23852]: cluster 2024-04-06T03:49:44.631785+0000 mgr.y (mgr.24368) 1056 : cluster 0 pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167155 10.f 10:f88a001f:::benchmark_data_smithi050_121293_object68338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:02.570787+0000", 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 463.33394975099998, 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.02469991, 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.907 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167155 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570787+0000", 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570787+0000", 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570788+0000", 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4959999999999999e-06 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570795+0000", 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3980000000000004e-06 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570811+0000", 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5991999999999999e-05 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570819+0000", 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2989999999999997e-06 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570882+0000", 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3016999999999997e-05 2024-04-06T03:49:45.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.570891+0000", 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0469999999999999e-06 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.585308+0000", 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0144173320000001 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.585410+0000", 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010163800000000001 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.585501+0000", 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1327000000000002e-05 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595166+0000", 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.009664242 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595188+0000", 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2198e-05 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595342+0000", 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015427799999999999 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595434+0000", 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1459999999999995e-05 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595487+0000", 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3186000000000002e-05 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.913 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167187 10.f 10:fea20798:::benchmark_data_smithi050_121293_object68370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:02.610165+0000", 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 463.29457185299998, 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98557857800000004, 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167187 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.610165+0000", 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.610165+0000", 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.610166+0000", 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4729999999999999e-06 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.610172+0000", 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6169999999999998e-06 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.610185+0000", 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2838e-05 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.610192+0000", 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7559999999999999e-06 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.611011+0000", 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081886299999999997 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.611016+0000", 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1109999999999999e-06 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.588885+0000", 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97786903000000003 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.589019+0000", 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013367199999999999 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.589151+0000", 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000131693 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595559+0000", 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006408642 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595578+0000", 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8740999999999999e-05 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595627+0000", 2024-04-06T03:49:45.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9138999999999998e-05 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595692+0000", 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5018000000000007e-05 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595743+0000", 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0985000000000001e-05 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167210 10.f 10:fb564b44:::benchmark_data_smithi050_121293_object68393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:02.631627+0000", 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 463.27310983799998, 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.97049160300000004, 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167210 2024-04-06T03:49:45.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.631627+0000", 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.631627+0000", 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.631628+0000", 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0380000000000001e-06 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.631633+0000", 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9649999999999999e-06 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.631644+0000", 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1096e-05 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.631649+0000", 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5670000000000002e-06 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.632464+0000", 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081479399999999998 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.632473+0000", 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.986e-06 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.592114+0000", 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95964070599999995 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.592204+0000", 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9603000000000003e-05 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.592295+0000", 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1642000000000001e-05 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595585+0000", 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032899909999999999 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595820+0000", 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023515400000000001 2024-04-06T03:49:45.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.601998+0000", 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061779139999999996 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.602061+0000", 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2230999999999994e-05 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.602118+0000", 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7915999999999999e-05 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167221 10.f 10:f614658a:::benchmark_data_smithi050_121293_object68404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:02.644306+0000", 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 463.26043040500002, 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.960894939, 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.928 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167221 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644306+0000", 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644306+0000", 2024-04-06T03:49:45.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644308+0000", 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.452e-06 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644314+0000", 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8540000000000002e-06 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644326+0000", 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2585000000000001e-05 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644334+0000", 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1689999999999992e-06 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644384+0000", 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9218000000000002e-05 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.644390+0000", 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8900000000000001e-06 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.595888+0000", 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95149753400000003 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.596038+0000", 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015044900000000001 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.596204+0000", 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016599099999999999 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.602183+0000", 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059788519999999998 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605029+0000", 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028452719999999998 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605075+0000", 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6971999999999997e-05 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605142+0000", 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6755999999999996e-05 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605201+0000", 2024-04-06T03:49:45.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8944999999999999e-05 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167233 10.f 10:fea63890:::benchmark_data_smithi050_121293_object68416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:02.663529+0000", 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 463.241207938, 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94805911600000004, 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.935 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167233 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663529+0000", 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663529+0000", 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663530+0000", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3519999999999999e-06 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663536+0000", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.48e-06 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663547+0000", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.098e-05 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663554+0000", 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2799999999999998e-06 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663587+0000", 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2984999999999997e-05 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:02.663592+0000", 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3519999999999997e-06 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.599189+0000", 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93559668299999998 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.599314+0000", 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000125091 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,4", 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.599445+0000", 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013064700000000001 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605219+0000", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057742590000000003 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.608032+0000", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028127030000000002 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.611472+0000", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034409480000000001 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.611534+0000", 2024-04-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1853999999999994e-05 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.611588+0000", 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3501999999999997e-05 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167238 10.7 10:e31ba40c:::benchmark_data_smithi050_121293_object68421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:03.605380+0000", 2024-04-06T03:49:45.941 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 462.29935702400002, 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.028351687, 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.942 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167238 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605380+0000", 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605380+0000", 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605381+0000", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.879999999999999e-07 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605387+0000", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1399999999999997e-06 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605399+0000", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.206e-05 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.605407+0000", 2024-04-06T03:49:45.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6100000000000006e-06 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.608163+0000", 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002755267 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.608173+0000", 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0396e-05 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.603783+0000", 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.99560961400000003 2024-04-06T03:49:45.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.604139+0000", 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035599500000000001 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.604589+0000", 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044991999999999997 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.611268+0000", 2024-04-06T03:49:45.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066791560000000003 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.622265+0000", 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010997398 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633549+0000", 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011283837999999999 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633669+0000", 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120352 2024-04-06T03:49:45.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633731+0000", 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1952999999999995e-05 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167256 10.7 10:e6a0ff5f:::benchmark_data_smithi050_121293_object68439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:03.650688+0000", 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 462.25404844299999, 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98340314699999998, 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.948 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167256 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650688+0000", 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650688+0000", 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650690+0000", 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.869e-06 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650698+0000", 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7270000000000007e-06 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650717+0000", 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8799000000000001e-05 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650728+0000", 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.17e-05 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650790+0000", 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1792999999999996e-05 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.650800+0000", 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6979999999999993e-06 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.611350+0000", 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.96055001100000004 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.611606+0000", 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000256032 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:49:45.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.611882+0000", 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027649700000000001 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.622387+0000", 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010504650000000001 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633522+0000", 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011134549000000001 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633877+0000", 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035587200000000002 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633986+0000", 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108353 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.634091+0000", 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000105597 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167267 10.7 10:ec9f1a5e:::benchmark_data_smithi050_121293_object68450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:03.664077+0000", 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 462.24065987199998, 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.98883352400000002, 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167267 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.955 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.664077+0000", 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.664077+0000", 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.664077+0000", 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0699999999999997e-07 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.664080+0000", 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8090000000000001e-06 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.664087+0000", 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4350000000000002e-06 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.664092+0000", 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5079999999999996e-06 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.668319+0000", 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042273160000000001 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.668325+0000", 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0050000000000001e-06 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.616853+0000", 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94852725599999999 2024-04-06T03:49:45.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.617122+0000", 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026975499999999999 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.617334+0000", 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000211564 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.633423+0000", 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016088716999999999 2024-04-06T03:49:45.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.634107+0000", 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068386300000000005 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.652769+0000", 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018662134 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.652849+0000", 2024-04-06T03:49:45.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0099999999999995e-05 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.652910+0000", 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1555000000000003e-05 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167278 10.7 10:edabc334:::benchmark_data_smithi050_121293_object68461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:03.678551+0000", 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 462.22618549200001, 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.974551944, 2024-04-06T03:49:45.961 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167278 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678551+0000", 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678551+0000", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678553+0000", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5060000000000001e-06 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678558+0000", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6540000000000004e-06 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678571+0000", 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3093e-05 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678580+0000", 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.952e-06 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678648+0000", 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7507000000000003e-05 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.678659+0000", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0631999999999999e-05 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.622473+0000", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.94381421899999995 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.622756+0000", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000283751 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.623049+0000", 2024-04-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029232100000000001 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.634117+0000", 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011068305000000001 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.646432+0000", 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012314584 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.652975+0000", 2024-04-06T03:49:45.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065434020000000002 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653048+0000", 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3386999999999994e-05 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653103+0000", 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4630999999999999e-05 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167318 10.7 10:e54525c8:::benchmark_data_smithi050_121293_object68501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:03.732890+0000", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 462.17184632300001, 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.92038722500000003, 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167318 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.732890+0000", 2024-04-06T03:49:45.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.732890+0000", 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.732892+0000", 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.409e-06 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.732898+0000", 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5780000000000004e-06 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.732913+0000", 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5119999999999999e-05 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.732922+0000", 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3920000000000003e-06 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.733010+0000", 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7907999999999999e-05 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.733020+0000", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0237999999999999e-05 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.627556+0000", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.89453553200000002 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.627781+0000", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022507500000000001 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:49:45.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.628520+0000", 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00073917899999999996 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.646332+0000", 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017811984999999999 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.646599+0000", 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026738499999999998 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653161+0000", 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0065617890000000002 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653228+0000", 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7482000000000002e-05 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653278+0000", 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9153000000000002e-05 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15306.0:167324 10.7 10:efacf144:::benchmark_data_smithi050_121293_object68507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:42:03.741562+0000", 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 462.163174975, 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.91190012799999998, 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15306", 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 167324 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741562+0000", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741562+0000", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741563+0000", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2249999999999999e-06 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741568+0000", 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1209999999999998e-06 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741579+0000", 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1199000000000001e-05 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741587+0000", 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3259999999999998e-06 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741643+0000", 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6109000000000001e-05 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:03.741649+0000", 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.635e-06 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.634178+0000", 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.892528879 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.634414+0000", 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000235548 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.634692+0000", 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00027851599999999997 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.649619+0000", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014926671000000001 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.650228+0000", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060937199999999995 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653332+0000", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003103424 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653412+0000", 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0161999999999996e-05 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:42:04.653462+0000", 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9941000000000002e-05 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85895 11.1 11:88aec6b4:::benchmark_data_smithi050_138698_object85894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.381897+0000", 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.522839159, 2024-04-06T03:49:45.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56325134600000004, 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85895 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.381897+0000", 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.381897+0000", 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.381899+0000", 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4130000000000001e-06 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.381958+0000", 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9537000000000003e-05 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.381971+0000", 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2340999999999999e-05 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.381979+0000", 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3780000000000001e-06 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.382234+0000", 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025533199999999999 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.382334+0000", 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9862e-05 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.382444+0000", 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109268 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.388429+0000", 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005985471 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.391340+0000", 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029109460000000002 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.944759+0000", 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.553418937 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.944896+0000", 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137167 2024-04-06T03:49:45.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.945149+0000", 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025269399999999998 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85900 11.9 11:94c931ba:::benchmark_data_smithi050_138698_object85899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.383279+0000", 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.52145732000002, 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.57299175400000002, 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.987 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85900 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383279+0000", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383279+0000", 2024-04-06T03:49:45.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383280+0000", 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0900000000000001e-07 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383324+0000", 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4042000000000003e-05 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383332+0000", 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9410000000000007e-06 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383338+0000", 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9689999999999999e-06 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383951+0000", 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0006135 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.384066+0000", 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115052 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.384199+0000", 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000132507 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.391334+0000", 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007135504 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404573+0000", 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013238405999999999 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956144+0000", 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55157150300000002 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956214+0000", 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9611999999999997e-05 2024-04-06T03:49:45.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956271+0000", 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7108999999999998e-05 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85901 11.1 11:85bbe781:::benchmark_data_smithi050_138698_object85900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.383347+0000", 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.52138984200002, 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.57270235000000003, 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:45.993 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85901 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383347+0000", 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383347+0000", 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383347+0000", 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7100000000000002e-07 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383387+0000", 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9650000000000002e-05 2024-04-06T03:49:45.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383394+0000", 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.905e-06 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.383399+0000", 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5890000000000002e-06 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.388459+0000", 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0050593490000000003 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.388572+0000", 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011371599999999999 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.388707+0000", 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013489 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.391525+0000", 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028172789999999998 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.404609+0000", 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013084446 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.955930+0000", 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55132057599999995 2024-04-06T03:49:45.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.955995+0000", 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4838999999999994e-05 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956049+0000", 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4639999999999999e-05 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:45.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85913 11.9 11:90ac36b1:::benchmark_data_smithi050_138698_object85912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.395089+0000", 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.50964768400002, 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.56137314299999996, 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85913 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.000 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395089+0000", 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395089+0000", 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395090+0000", 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1570000000000001e-06 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395119+0000", 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8683e-05 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395129+0000", 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0185e-05 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395135+0000", 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9059999999999999e-06 2024-04-06T03:49:46.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395178+0000", 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3463e-05 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395274+0000", 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6068000000000002e-05 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.395367+0000", 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2326999999999999e-05 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.397500+0000", 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021336620000000001 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.408100+0000", 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010599101999999999 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956333+0000", 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54823379000000005 2024-04-06T03:49:46.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956399+0000", 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5443000000000003e-05 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.956462+0000", 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3356999999999997e-05 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15498.0:85917 11.9 11:90480a71:::benchmark_data_smithi050_138698_object85916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:49:46.005 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:44:58.400126+0000", 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 287.50461068099997, 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.558943737, 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15498", 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 85917 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400126+0000", 2024-04-06T03:49:46.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400126+0000", 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400127+0000", 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400146+0000", 2024-04-06T03:49:46.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9646000000000001e-05 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400157+0000", 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0312999999999999e-05 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400165+0000", 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.259e-06 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400205+0000", 2024-04-06T03:49:46.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9711e-05 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400294+0000", 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9142999999999997e-05 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,6", 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.400388+0000", 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3854e-05 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.403028+0000", 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026402029999999998 2024-04-06T03:49:46.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.408176+0000", 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005148484 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.958969+0000", 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.55079217700000005 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.959025+0000", 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6549999999999999e-05 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:44:58.959070+0000", 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4579999999999997e-05 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:5497 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94633190260608 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:48:20.254520+0000", 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.650216849000003, 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.099315518, 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:46.011 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5497 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254520+0000", 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254520+0000", 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254521+0000", 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1030000000000001e-06 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254526+0000", 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0900000000000004e-06 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254538+0000", 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1887e-05 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254548+0000", 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.859e-06 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254627+0000", 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9431999999999994e-05 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254636+0000", 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4319999999999999e-06 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.275731+0000", 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021095309999999999 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.275752+0000", 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0922999999999999e-05 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.347662+0000", 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0719098810000001 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.349561+0000", 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001899055 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.352334+0000", 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002773635 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.352429+0000", 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5013000000000007e-05 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.353835+0000", 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001405898 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24278.0:5498 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94633190261760 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:48:20.254574+0000", 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.650163143, 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.097361408, 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:46.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24278", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5498 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254574+0000", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254574+0000", 2024-04-06T03:49:46.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254574+0000", 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8400000000000005e-07 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254577+0000", 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1209999999999998e-06 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254585+0000", 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.644e-06 2024-04-06T03:49:46.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254591+0000", 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6610000000000003e-06 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254679+0000", 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7238999999999996e-05 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.254691+0000", 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2721999999999999e-05 2024-04-06T03:49:46.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.264448+0000", 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0097566330000000007 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.264468+0000", 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9494e-05 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.325695+0000", 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0612278770000001 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for degraded object", 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.326126+0000", 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043013799999999999 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.350519+0000", 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024393649999999999 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.350615+0000", 2024-04-06T03:49:46.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5997999999999998e-05 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.351935+0000", 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001319747 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24418.0:1609 4.0 4:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e395)", 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:48:20.690777+0000", 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 85.213959885999998, 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.65679439299999998, 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:46.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24418", 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.153:0/2744702004", 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1609 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.690777+0000", 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:46.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.690777+0000", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.690780+0000", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.886e-06 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.690805+0000", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4887000000000001e-05 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.690845+0000", 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0501999999999999e-05 2024-04-06T03:49:46.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.690864+0000", 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8856e-05 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.691062+0000", 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000198269 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:20.691101+0000", 2024-04-06T03:49:46.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8526999999999999e-05 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.347280+0000", 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.65617944500000003 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.347425+0000", 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014496800000000001 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:46.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:48:21.347571+0000", 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014605299999999999 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(5.a,QREQUEST,e:417,from:2,reservation_nonce:32)", 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-06T03:49:44.524280+0000", 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.380457096, 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029487929999999999, 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-06T03:49:46.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524280+0000", 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524280+0000", 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524281+0000", 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1179999999999999e-06 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524287+0000", 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4320000000000004e-06 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524302+0000", 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5204e-05 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524313+0000", 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0457e-05 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524393+0000", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9857000000000005e-05 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.524417+0000", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4664e-05 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.527147+0000", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002730049 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.527170+0000", 2024-04-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2348e-05 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-06T03:49:44.527228+0000", 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8663999999999999e-05 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-06T03:49:46.032 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-06T03:49:46.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:46 smithi050 bash[29221]: cluster 2024-04-06T03:49:45.518140+0000 osd.2 (osd.2) 565 : cluster 0 5.6 scrub starts 2024-04-06T03:49:46.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:46 smithi050 bash[29221]: cluster 2024-04-06T03:49:45.520826+0000 osd.2 (osd.2) 566 : cluster 0 5.6 scrub ok 2024-04-06T03:49:46.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:46 smithi050 bash[21467]: cluster 2024-04-06T03:49:45.518140+0000 osd.2 (osd.2) 565 : cluster 0 5.6 scrub starts 2024-04-06T03:49:46.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:46 smithi050 bash[21467]: cluster 2024-04-06T03:49:45.520826+0000 osd.2 (osd.2) 566 : cluster 0 5.6 scrub ok 2024-04-06T03:49:46.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:46 smithi153 bash[23852]: cluster 2024-04-06T03:49:45.518140+0000 osd.2 (osd.2) 565 : cluster 0 5.6 scrub starts 2024-04-06T03:49:46.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:46 smithi153 bash[23852]: cluster 2024-04-06T03:49:45.520826+0000 osd.2 (osd.2) 566 : cluster 0 5.6 scrub ok 2024-04-06T03:49:47.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-06T03:49:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[21467]: cluster 2024-04-06T03:49:46.225458+0000 osd.4 (osd.4) 657 : cluster 0 3.a scrub starts 2024-04-06T03:49:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[21467]: cluster 2024-04-06T03:49:46.231039+0000 osd.4 (osd.4) 658 : cluster 0 3.a scrub ok 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[21467]: cluster 2024-04-06T03:49:46.632963+0000 mgr.y (mgr.24368) 1057 : cluster 0 pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[21467]: cluster 2024-04-06T03:49:46.683406+0000 osd.3 (osd.3) 675 : cluster 0 6.13 scrub starts 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[21467]: cluster 2024-04-06T03:49:46.860693+0000 osd.3 (osd.3) 676 : cluster 0 6.13 scrub ok 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[29221]: cluster 2024-04-06T03:49:46.225458+0000 osd.4 (osd.4) 657 : cluster 0 3.a scrub starts 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[29221]: cluster 2024-04-06T03:49:46.231039+0000 osd.4 (osd.4) 658 : cluster 0 3.a scrub ok 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[29221]: cluster 2024-04-06T03:49:46.632963+0000 mgr.y (mgr.24368) 1057 : cluster 0 pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[29221]: cluster 2024-04-06T03:49:46.683406+0000 osd.3 (osd.3) 675 : cluster 0 6.13 scrub starts 2024-04-06T03:49:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:47 smithi050 bash[29221]: cluster 2024-04-06T03:49:46.860693+0000 osd.3 (osd.3) 676 : cluster 0 6.13 scrub ok 2024-04-06T03:49:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:47 smithi153 bash[23852]: cluster 2024-04-06T03:49:46.225458+0000 osd.4 (osd.4) 657 : cluster 0 3.a scrub starts 2024-04-06T03:49:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:47 smithi153 bash[23852]: cluster 2024-04-06T03:49:46.231039+0000 osd.4 (osd.4) 658 : cluster 0 3.a scrub ok 2024-04-06T03:49:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:47 smithi153 bash[23852]: cluster 2024-04-06T03:49:46.632963+0000 mgr.y (mgr.24368) 1057 : cluster 0 pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:47 smithi153 bash[23852]: cluster 2024-04-06T03:49:46.683406+0000 osd.3 (osd.3) 675 : cluster 0 6.13 scrub starts 2024-04-06T03:49:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:47 smithi153 bash[23852]: cluster 2024-04-06T03:49:46.860693+0000 osd.3 (osd.3) 676 : cluster 0 6.13 scrub ok 2024-04-06T03:49:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[21467]: cluster 2024-04-06T03:49:47.441227+0000 osd.1 (osd.1) 445 : cluster 0 3.4 deep-scrub starts 2024-04-06T03:49:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[21467]: cluster 2024-04-06T03:49:47.447687+0000 osd.1 (osd.1) 446 : cluster 0 3.4 deep-scrub ok 2024-04-06T03:49:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[21467]: cluster 2024-04-06T03:49:47.692857+0000 osd.3 (osd.3) 677 : cluster 0 3.c scrub starts 2024-04-06T03:49:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[21467]: cluster 2024-04-06T03:49:47.697355+0000 osd.3 (osd.3) 678 : cluster 0 3.c scrub ok 2024-04-06T03:49:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[29221]: cluster 2024-04-06T03:49:47.441227+0000 osd.1 (osd.1) 445 : cluster 0 3.4 deep-scrub starts 2024-04-06T03:49:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[29221]: cluster 2024-04-06T03:49:47.447687+0000 osd.1 (osd.1) 446 : cluster 0 3.4 deep-scrub ok 2024-04-06T03:49:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[29221]: cluster 2024-04-06T03:49:47.692857+0000 osd.3 (osd.3) 677 : cluster 0 3.c scrub starts 2024-04-06T03:49:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:48 smithi050 bash[29221]: cluster 2024-04-06T03:49:47.697355+0000 osd.3 (osd.3) 678 : cluster 0 3.c scrub ok 2024-04-06T03:49:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:48 smithi153 bash[23852]: cluster 2024-04-06T03:49:47.441227+0000 osd.1 (osd.1) 445 : cluster 0 3.4 deep-scrub starts 2024-04-06T03:49:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:48 smithi153 bash[23852]: cluster 2024-04-06T03:49:47.447687+0000 osd.1 (osd.1) 446 : cluster 0 3.4 deep-scrub ok 2024-04-06T03:49:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:48 smithi153 bash[23852]: cluster 2024-04-06T03:49:47.692857+0000 osd.3 (osd.3) 677 : cluster 0 3.c scrub starts 2024-04-06T03:49:48.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:48 smithi153 bash[23852]: cluster 2024-04-06T03:49:47.697355+0000 osd.3 (osd.3) 678 : cluster 0 3.c scrub ok 2024-04-06T03:49:49.182 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-06T03:49:49.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:49:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:49:49.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: cluster 2024-04-06T03:49:48.227103+0000 osd.6 (osd.6) 115 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:49.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: cluster 2024-04-06T03:49:48.227220+0000 osd.6 (osd.6) 116 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:49.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: audit 2024-04-06T03:49:48.610328+0000 mon.c (mon.2) 127 : audit 1 from='client.? 172.21.15.50:0/2458374004' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: audit 2024-04-06T03:49:48.611056+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: cluster 2024-04-06T03:49:48.633940+0000 mgr.y (mgr.24368) 1058 : cluster 0 pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: cluster 2024-04-06T03:49:48.705660+0000 osd.3 (osd.3) 679 : cluster 0 6.1b scrub starts 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[29221]: cluster 2024-04-06T03:49:48.913290+0000 osd.3 (osd.3) 680 : cluster 0 6.1b scrub ok 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: cluster 2024-04-06T03:49:48.227103+0000 osd.6 (osd.6) 115 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: cluster 2024-04-06T03:49:48.227220+0000 osd.6 (osd.6) 116 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: audit 2024-04-06T03:49:48.610328+0000 mon.c (mon.2) 127 : audit 1 from='client.? 172.21.15.50:0/2458374004' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: audit 2024-04-06T03:49:48.611056+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: cluster 2024-04-06T03:49:48.633940+0000 mgr.y (mgr.24368) 1058 : cluster 0 pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: cluster 2024-04-06T03:49:48.705660+0000 osd.3 (osd.3) 679 : cluster 0 6.1b scrub starts 2024-04-06T03:49:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:49 smithi050 bash[21467]: cluster 2024-04-06T03:49:48.913290+0000 osd.3 (osd.3) 680 : cluster 0 6.1b scrub ok 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: cluster 2024-04-06T03:49:48.227103+0000 osd.6 (osd.6) 115 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: cluster 2024-04-06T03:49:48.227220+0000 osd.6 (osd.6) 116 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: audit 2024-04-06T03:49:48.610328+0000 mon.c (mon.2) 127 : audit 1 from='client.? 172.21.15.50:0/2458374004' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: audit 2024-04-06T03:49:48.611056+0000 mon.a (mon.0) 1867 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: cluster 2024-04-06T03:49:48.633940+0000 mgr.y (mgr.24368) 1058 : cluster 0 pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: cluster 2024-04-06T03:49:48.705660+0000 osd.3 (osd.3) 679 : cluster 0 6.1b scrub starts 2024-04-06T03:49:49.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:49 smithi153 bash[23852]: cluster 2024-04-06T03:49:48.913290+0000 osd.3 (osd.3) 680 : cluster 0 6.1b scrub ok 2024-04-06T03:49:50.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 3 1 2024-04-06T03:49:50.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[23852]: audit 2024-04-06T03:49:49.166461+0000 mon.a (mon.0) 1868 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:49:50.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[23852]: cluster 2024-04-06T03:49:49.175440+0000 mon.a (mon.0) 1869 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-04-06T03:49:50.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[23852]: cluster 2024-04-06T03:49:49.345556+0000 osd.7 (osd.7) 687 : cluster 0 2.1d scrub starts 2024-04-06T03:49:50.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[23852]: cluster 2024-04-06T03:49:49.350893+0000 osd.7 (osd.7) 688 : cluster 0 2.1d scrub ok 2024-04-06T03:49:50.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[23852]: cluster 2024-04-06T03:49:49.467677+0000 osd.2 (osd.2) 567 : cluster 0 2.17 deep-scrub starts 2024-04-06T03:49:50.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[23852]: cluster 2024-04-06T03:49:49.470394+0000 osd.2 (osd.2) 568 : cluster 0 2.17 deep-scrub ok 2024-04-06T03:49:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[21467]: audit 2024-04-06T03:49:49.166461+0000 mon.a (mon.0) 1868 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:49:50.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[21467]: cluster 2024-04-06T03:49:49.175440+0000 mon.a (mon.0) 1869 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[21467]: cluster 2024-04-06T03:49:49.345556+0000 osd.7 (osd.7) 687 : cluster 0 2.1d scrub starts 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[21467]: cluster 2024-04-06T03:49:49.350893+0000 osd.7 (osd.7) 688 : cluster 0 2.1d scrub ok 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[21467]: cluster 2024-04-06T03:49:49.467677+0000 osd.2 (osd.2) 567 : cluster 0 2.17 deep-scrub starts 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[21467]: cluster 2024-04-06T03:49:49.470394+0000 osd.2 (osd.2) 568 : cluster 0 2.17 deep-scrub ok 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[29221]: audit 2024-04-06T03:49:49.166461+0000 mon.a (mon.0) 1868 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[29221]: cluster 2024-04-06T03:49:49.175440+0000 mon.a (mon.0) 1869 : cluster 0 osdmap e418: 8 total, 8 up, 8 in 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[29221]: cluster 2024-04-06T03:49:49.345556+0000 osd.7 (osd.7) 687 : cluster 0 2.1d scrub starts 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[29221]: cluster 2024-04-06T03:49:49.350893+0000 osd.7 (osd.7) 688 : cluster 0 2.1d scrub ok 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[29221]: cluster 2024-04-06T03:49:49.467677+0000 osd.2 (osd.2) 567 : cluster 0 2.17 deep-scrub starts 2024-04-06T03:49:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:50 smithi050 bash[29221]: cluster 2024-04-06T03:49:49.470394+0000 osd.2 (osd.2) 568 : cluster 0 2.17 deep-scrub ok 2024-04-06T03:49:50.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:49:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:49:51.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[29221]: cluster 2024-04-06T03:49:50.378388+0000 osd.7 (osd.7) 689 : cluster 0 6.15 scrub starts 2024-04-06T03:49:51.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[29221]: audit 2024-04-06T03:49:50.456514+0000 mgr.y (mgr.24368) 1059 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[29221]: cluster 2024-04-06T03:49:50.476337+0000 osd.7 (osd.7) 690 : cluster 0 6.15 scrub ok 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[29221]: cluster 2024-04-06T03:49:50.635270+0000 mgr.y (mgr.24368) 1060 : cluster 0 pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[29221]: audit 2024-04-06T03:49:50.921294+0000 mon.a (mon.0) 1870 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[21467]: cluster 2024-04-06T03:49:50.378388+0000 osd.7 (osd.7) 689 : cluster 0 6.15 scrub starts 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[21467]: audit 2024-04-06T03:49:50.456514+0000 mgr.y (mgr.24368) 1059 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[21467]: cluster 2024-04-06T03:49:50.476337+0000 osd.7 (osd.7) 690 : cluster 0 6.15 scrub ok 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[21467]: cluster 2024-04-06T03:49:50.635270+0000 mgr.y (mgr.24368) 1060 : cluster 0 pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:51 smithi050 bash[21467]: audit 2024-04-06T03:49:50.921294+0000 mon.a (mon.0) 1870 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:51 smithi153 bash[23852]: cluster 2024-04-06T03:49:50.378388+0000 osd.7 (osd.7) 689 : cluster 0 6.15 scrub starts 2024-04-06T03:49:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:51 smithi153 bash[23852]: audit 2024-04-06T03:49:50.456514+0000 mgr.y (mgr.24368) 1059 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:49:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:51 smithi153 bash[23852]: cluster 2024-04-06T03:49:50.476337+0000 osd.7 (osd.7) 690 : cluster 0 6.15 scrub ok 2024-04-06T03:49:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:51 smithi153 bash[23852]: cluster 2024-04-06T03:49:50.635270+0000 mgr.y (mgr.24368) 1060 : cluster 0 pgmap v1164: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:51 smithi153 bash[23852]: audit 2024-04-06T03:49:50.921294+0000 mon.a (mon.0) 1870 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:49:51.985 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:49:52.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:52 smithi153 bash[23852]: cluster 2024-04-06T03:49:51.335269+0000 osd.1 (osd.1) 447 : cluster 0 2.2 deep-scrub starts 2024-04-06T03:49:52.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:52 smithi153 bash[23852]: cluster 2024-04-06T03:49:51.337968+0000 osd.1 (osd.1) 448 : cluster 0 2.2 deep-scrub ok 2024-04-06T03:49:52.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:52 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.036669+0000 osd.3 (osd.3) 681 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:52.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:52 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.036773+0000 osd.3 (osd.3) 682 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:52.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[29221]: cluster 2024-04-06T03:49:51.335269+0000 osd.1 (osd.1) 447 : cluster 0 2.2 deep-scrub starts 2024-04-06T03:49:52.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[29221]: cluster 2024-04-06T03:49:51.337968+0000 osd.1 (osd.1) 448 : cluster 0 2.2 deep-scrub ok 2024-04-06T03:49:52.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.036669+0000 osd.3 (osd.3) 681 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:52.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.036773+0000 osd.3 (osd.3) 682 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:52.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[21467]: cluster 2024-04-06T03:49:51.335269+0000 osd.1 (osd.1) 447 : cluster 0 2.2 deep-scrub starts 2024-04-06T03:49:52.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[21467]: cluster 2024-04-06T03:49:51.337968+0000 osd.1 (osd.1) 448 : cluster 0 2.2 deep-scrub ok 2024-04-06T03:49:52.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.036669+0000 osd.3 (osd.3) 681 : cluster 0 purged_snaps scrub starts 2024-04-06T03:49:52.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:52 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.036773+0000 osd.3 (osd.3) 682 : cluster 0 purged_snaps scrub ok 2024-04-06T03:49:53.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.207192+0000 mon.a (mon.0) 1871 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.342477+0000 osd.7 (osd.7) 691 : cluster 0 3.d deep-scrub starts 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.351706+0000 osd.7 (osd.7) 692 : cluster 0 3.d deep-scrub ok 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.357888+0000 osd.1 (osd.1) 449 : cluster 0 3.19 scrub starts 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.363583+0000 osd.1 (osd.1) 450 : cluster 0 3.19 scrub ok 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[21467]: cluster 2024-04-06T03:49:52.636153+0000 mgr.y (mgr.24368) 1061 : cluster 0 pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.207192+0000 mon.a (mon.0) 1871 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.342477+0000 osd.7 (osd.7) 691 : cluster 0 3.d deep-scrub starts 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.351706+0000 osd.7 (osd.7) 692 : cluster 0 3.d deep-scrub ok 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.357888+0000 osd.1 (osd.1) 449 : cluster 0 3.19 scrub starts 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.363583+0000 osd.1 (osd.1) 450 : cluster 0 3.19 scrub ok 2024-04-06T03:49:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:53 smithi050 bash[29221]: cluster 2024-04-06T03:49:52.636153+0000 mgr.y (mgr.24368) 1061 : cluster 0 pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:53.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:53 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.207192+0000 mon.a (mon.0) 1871 : cluster 0 osdmap e419: 8 total, 8 up, 8 in 2024-04-06T03:49:53.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:53 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.342477+0000 osd.7 (osd.7) 691 : cluster 0 3.d deep-scrub starts 2024-04-06T03:49:53.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:53 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.351706+0000 osd.7 (osd.7) 692 : cluster 0 3.d deep-scrub ok 2024-04-06T03:49:53.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:53 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.357888+0000 osd.1 (osd.1) 449 : cluster 0 3.19 scrub starts 2024-04-06T03:49:53.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:53 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.363583+0000 osd.1 (osd.1) 450 : cluster 0 3.19 scrub ok 2024-04-06T03:49:53.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:53 smithi153 bash[23852]: cluster 2024-04-06T03:49:52.636153+0000 mgr.y (mgr.24368) 1061 : cluster 0 pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:54.806 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:49:55.073 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:49:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:49:55.073 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:49:55.073 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:49:55.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[21467]: cluster 2024-04-06T03:49:53.599486+0000 osd.2 (osd.2) 569 : cluster 0 2.e scrub starts 2024-04-06T03:49:55.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[21467]: cluster 2024-04-06T03:49:53.602049+0000 osd.2 (osd.2) 570 : cluster 0 2.e scrub ok 2024-04-06T03:49:55.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[21467]: cluster 2024-04-06T03:49:54.256025+0000 osd.4 (osd.4) 659 : cluster 0 6.17 scrub starts 2024-04-06T03:49:55.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[21467]: cluster 2024-04-06T03:49:54.463681+0000 osd.4 (osd.4) 660 : cluster 0 6.17 scrub ok 2024-04-06T03:49:55.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[21467]: cluster 2024-04-06T03:49:54.637210+0000 mgr.y (mgr.24368) 1062 : cluster 0 pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:55.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[29221]: cluster 2024-04-06T03:49:53.599486+0000 osd.2 (osd.2) 569 : cluster 0 2.e scrub starts 2024-04-06T03:49:55.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[29221]: cluster 2024-04-06T03:49:53.602049+0000 osd.2 (osd.2) 570 : cluster 0 2.e scrub ok 2024-04-06T03:49:55.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[29221]: cluster 2024-04-06T03:49:54.256025+0000 osd.4 (osd.4) 659 : cluster 0 6.17 scrub starts 2024-04-06T03:49:55.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[29221]: cluster 2024-04-06T03:49:54.463681+0000 osd.4 (osd.4) 660 : cluster 0 6.17 scrub ok 2024-04-06T03:49:55.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:55 smithi050 bash[29221]: cluster 2024-04-06T03:49:54.637210+0000 mgr.y (mgr.24368) 1062 : cluster 0 pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:55.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:55 smithi153 bash[23852]: cluster 2024-04-06T03:49:53.599486+0000 osd.2 (osd.2) 569 : cluster 0 2.e scrub starts 2024-04-06T03:49:55.295 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:55 smithi153 bash[23852]: cluster 2024-04-06T03:49:53.602049+0000 osd.2 (osd.2) 570 : cluster 0 2.e scrub ok 2024-04-06T03:49:55.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:55 smithi153 bash[23852]: cluster 2024-04-06T03:49:54.256025+0000 osd.4 (osd.4) 659 : cluster 0 6.17 scrub starts 2024-04-06T03:49:55.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:55 smithi153 bash[23852]: cluster 2024-04-06T03:49:54.463681+0000 osd.4 (osd.4) 660 : cluster 0 6.17 scrub ok 2024-04-06T03:49:55.296 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:55 smithi153 bash[23852]: cluster 2024-04-06T03:49:54.637210+0000 mgr.y (mgr.24368) 1062 : cluster 0 pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:49:55.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_blocked_ops 2024-04-06T03:49:56.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.183629+0000 osd.0 (osd.0) 672 : cluster 0 6.18 deep-scrub starts 2024-04-06T03:49:56.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.288074+0000 osd.0 (osd.0) 673 : cluster 0 6.18 deep-scrub ok 2024-04-06T03:49:56.416 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.364254+0000 osd.7 (osd.7) 693 : cluster 0 6.5 scrub starts 2024-04-06T03:49:56.416 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.462762+0000 osd.7 (osd.7) 694 : cluster 0 6.5 scrub ok 2024-04-06T03:49:56.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.183629+0000 osd.0 (osd.0) 672 : cluster 0 6.18 deep-scrub starts 2024-04-06T03:49:56.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.288074+0000 osd.0 (osd.0) 673 : cluster 0 6.18 deep-scrub ok 2024-04-06T03:49:56.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.364254+0000 osd.7 (osd.7) 693 : cluster 0 6.5 scrub starts 2024-04-06T03:49:56.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:56 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.462762+0000 osd.7 (osd.7) 694 : cluster 0 6.5 scrub ok 2024-04-06T03:49:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:56 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.183629+0000 osd.0 (osd.0) 672 : cluster 0 6.18 deep-scrub starts 2024-04-06T03:49:56.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:56 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.288074+0000 osd.0 (osd.0) 673 : cluster 0 6.18 deep-scrub ok 2024-04-06T03:49:56.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:56 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.364254+0000 osd.7 (osd.7) 693 : cluster 0 6.5 scrub starts 2024-04-06T03:49:56.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:56 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.462762+0000 osd.7 (osd.7) 694 : cluster 0 6.5 scrub ok 2024-04-06T03:49:57.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.438485+0000 osd.1 (osd.1) 451 : cluster 0 2.d scrub starts 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.440970+0000 osd.1 (osd.1) 452 : cluster 0 2.d scrub ok 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.826614+0000 osd.3 (osd.3) 683 : cluster 0 6.11 deep-scrub starts 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:55.881147+0000 osd.3 (osd.3) 684 : cluster 0 6.11 deep-scrub ok 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:56.257253+0000 osd.4 (osd.4) 661 : cluster 0 3.1c scrub starts 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:56.261516+0000 osd.4 (osd.4) 662 : cluster 0 3.1c scrub ok 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:56.490819+0000 osd.2 (osd.2) 571 : cluster 0 2.1 scrub starts 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:56.493457+0000 osd.2 (osd.2) 572 : cluster 0 2.1 scrub ok 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[21467]: cluster 2024-04-06T03:49:56.638380+0000 mgr.y (mgr.24368) 1063 : cluster 0 pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.438485+0000 osd.1 (osd.1) 451 : cluster 0 2.d scrub starts 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.440970+0000 osd.1 (osd.1) 452 : cluster 0 2.d scrub ok 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.826614+0000 osd.3 (osd.3) 683 : cluster 0 6.11 deep-scrub starts 2024-04-06T03:49:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:55.881147+0000 osd.3 (osd.3) 684 : cluster 0 6.11 deep-scrub ok 2024-04-06T03:49:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:56.257253+0000 osd.4 (osd.4) 661 : cluster 0 3.1c scrub starts 2024-04-06T03:49:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:56.261516+0000 osd.4 (osd.4) 662 : cluster 0 3.1c scrub ok 2024-04-06T03:49:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:56.490819+0000 osd.2 (osd.2) 571 : cluster 0 2.1 scrub starts 2024-04-06T03:49:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:56.493457+0000 osd.2 (osd.2) 572 : cluster 0 2.1 scrub ok 2024-04-06T03:49:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:57 smithi050 bash[29221]: cluster 2024-04-06T03:49:56.638380+0000 mgr.y (mgr.24368) 1063 : cluster 0 pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.438485+0000 osd.1 (osd.1) 451 : cluster 0 2.d scrub starts 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.440970+0000 osd.1 (osd.1) 452 : cluster 0 2.d scrub ok 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.826614+0000 osd.3 (osd.3) 683 : cluster 0 6.11 deep-scrub starts 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:55.881147+0000 osd.3 (osd.3) 684 : cluster 0 6.11 deep-scrub ok 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:56.257253+0000 osd.4 (osd.4) 661 : cluster 0 3.1c scrub starts 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:56.261516+0000 osd.4 (osd.4) 662 : cluster 0 3.1c scrub ok 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:56.490819+0000 osd.2 (osd.2) 571 : cluster 0 2.1 scrub starts 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:56.493457+0000 osd.2 (osd.2) 572 : cluster 0 2.1 scrub ok 2024-04-06T03:49:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:57 smithi153 bash[23852]: cluster 2024-04-06T03:49:56.638380+0000 mgr.y (mgr.24368) 1063 : cluster 0 pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:49:58.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:58 smithi050 bash[21467]: cluster 2024-04-06T03:49:57.820847+0000 osd.3 (osd.3) 685 : cluster 0 5.16 scrub starts 2024-04-06T03:49:58.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:58 smithi050 bash[21467]: cluster 2024-04-06T03:49:57.822835+0000 osd.3 (osd.3) 686 : cluster 0 5.16 scrub ok 2024-04-06T03:49:58.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:58 smithi050 bash[29221]: cluster 2024-04-06T03:49:57.820847+0000 osd.3 (osd.3) 685 : cluster 0 5.16 scrub starts 2024-04-06T03:49:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:58 smithi050 bash[29221]: cluster 2024-04-06T03:49:57.822835+0000 osd.3 (osd.3) 686 : cluster 0 5.16 scrub ok 2024-04-06T03:49:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:58 smithi153 bash[23852]: cluster 2024-04-06T03:49:57.820847+0000 osd.3 (osd.3) 685 : cluster 0 5.16 scrub starts 2024-04-06T03:49:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:58 smithi153 bash[23852]: cluster 2024-04-06T03:49:57.822835+0000 osd.3 (osd.3) 686 : cluster 0 5.16 scrub ok 2024-04-06T03:49:59.110 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-06T03:49:59.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:49:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:49:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:49:59.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:59 smithi050 bash[21467]: audit 2024-04-06T03:49:58.479095+0000 mon.a (mon.0) 1872 : audit 1 from='client.? 172.21.15.50:0/1429450541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:49:59.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:49:59 smithi050 bash[21467]: cluster 2024-04-06T03:49:58.639430+0000 mgr.y (mgr.24368) 1064 : cluster 0 pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:49:59.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:59 smithi050 bash[29221]: audit 2024-04-06T03:49:58.479095+0000 mon.a (mon.0) 1872 : audit 1 from='client.? 172.21.15.50:0/1429450541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:49:59.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:49:59 smithi050 bash[29221]: cluster 2024-04-06T03:49:58.639430+0000 mgr.y (mgr.24368) 1064 : cluster 0 pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:49:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:59 smithi153 bash[23852]: audit 2024-04-06T03:49:58.479095+0000 mon.a (mon.0) 1872 : audit 1 from='client.? 172.21.15.50:0/1429450541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:49:59.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:49:59 smithi153 bash[23852]: cluster 2024-04-06T03:49:58.639430+0000 mgr.y (mgr.24368) 1064 : cluster 0 pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:49:59.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 3 1 2024-04-06T03:50:00.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[21467]: audit 2024-04-06T03:49:59.102328+0000 mon.a (mon.0) 1873 : audit 1 from='client.? 172.21.15.50:0/1429450541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:50:00.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[21467]: cluster 2024-04-06T03:49:59.108891+0000 mon.a (mon.0) 1874 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[21467]: cluster 2024-04-06T03:49:59.843459+0000 osd.3 (osd.3) 687 : cluster 0 3.2 scrub starts 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[21467]: cluster 2024-04-06T03:49:59.847927+0000 osd.3 (osd.3) 688 : cluster 0 3.2 scrub ok 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[21467]: cluster 2024-04-06T03:50:00.000142+0000 mon.a (mon.0) 1875 : cluster 1 overall HEALTH_OK 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[29221]: audit 2024-04-06T03:49:59.102328+0000 mon.a (mon.0) 1873 : audit 1 from='client.? 172.21.15.50:0/1429450541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[29221]: cluster 2024-04-06T03:49:59.108891+0000 mon.a (mon.0) 1874 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[29221]: cluster 2024-04-06T03:49:59.843459+0000 osd.3 (osd.3) 687 : cluster 0 3.2 scrub starts 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[29221]: cluster 2024-04-06T03:49:59.847927+0000 osd.3 (osd.3) 688 : cluster 0 3.2 scrub ok 2024-04-06T03:50:00.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:00 smithi050 bash[29221]: cluster 2024-04-06T03:50:00.000142+0000 mon.a (mon.0) 1875 : cluster 1 overall HEALTH_OK 2024-04-06T03:50:00.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:00 smithi153 bash[23852]: audit 2024-04-06T03:49:59.102328+0000 mon.a (mon.0) 1873 : audit 1 from='client.? 172.21.15.50:0/1429450541' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:50:00.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:00 smithi153 bash[23852]: cluster 2024-04-06T03:49:59.108891+0000 mon.a (mon.0) 1874 : cluster 0 osdmap e420: 8 total, 8 up, 8 in 2024-04-06T03:50:00.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:00 smithi153 bash[23852]: cluster 2024-04-06T03:49:59.843459+0000 osd.3 (osd.3) 687 : cluster 0 3.2 scrub starts 2024-04-06T03:50:00.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:00 smithi153 bash[23852]: cluster 2024-04-06T03:49:59.847927+0000 osd.3 (osd.3) 688 : cluster 0 3.2 scrub ok 2024-04-06T03:50:00.467 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:00 smithi153 bash[23852]: cluster 2024-04-06T03:50:00.000142+0000 mon.a (mon.0) 1875 : cluster 1 overall HEALTH_OK 2024-04-06T03:50:00.592 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:00.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:50:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:50:01.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[21467]: cluster 2024-04-06T03:50:00.396068+0000 osd.7 (osd.7) 695 : cluster 0 6.1c scrub starts 2024-04-06T03:50:01.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[21467]: cluster 2024-04-06T03:50:00.399109+0000 osd.7 (osd.7) 696 : cluster 0 6.1c scrub ok 2024-04-06T03:50:01.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[21467]: audit 2024-04-06T03:50:00.466754+0000 mgr.y (mgr.24368) 1065 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[21467]: cluster 2024-04-06T03:50:00.640697+0000 mgr.y (mgr.24368) 1066 : cluster 0 pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[21467]: cluster 2024-04-06T03:50:00.827870+0000 osd.3 (osd.3) 689 : cluster 0 2.1b scrub starts 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[21467]: cluster 2024-04-06T03:50:00.830495+0000 osd.3 (osd.3) 690 : cluster 0 2.1b scrub ok 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[29221]: cluster 2024-04-06T03:50:00.396068+0000 osd.7 (osd.7) 695 : cluster 0 6.1c scrub starts 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[29221]: cluster 2024-04-06T03:50:00.399109+0000 osd.7 (osd.7) 696 : cluster 0 6.1c scrub ok 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[29221]: audit 2024-04-06T03:50:00.466754+0000 mgr.y (mgr.24368) 1065 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[29221]: cluster 2024-04-06T03:50:00.640697+0000 mgr.y (mgr.24368) 1066 : cluster 0 pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[29221]: cluster 2024-04-06T03:50:00.827870+0000 osd.3 (osd.3) 689 : cluster 0 2.1b scrub starts 2024-04-06T03:50:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:01 smithi050 bash[29221]: cluster 2024-04-06T03:50:00.830495+0000 osd.3 (osd.3) 690 : cluster 0 2.1b scrub ok 2024-04-06T03:50:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:01 smithi153 bash[23852]: cluster 2024-04-06T03:50:00.396068+0000 osd.7 (osd.7) 695 : cluster 0 6.1c scrub starts 2024-04-06T03:50:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:01 smithi153 bash[23852]: cluster 2024-04-06T03:50:00.399109+0000 osd.7 (osd.7) 696 : cluster 0 6.1c scrub ok 2024-04-06T03:50:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:01 smithi153 bash[23852]: audit 2024-04-06T03:50:00.466754+0000 mgr.y (mgr.24368) 1065 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:01 smithi153 bash[23852]: cluster 2024-04-06T03:50:00.640697+0000 mgr.y (mgr.24368) 1066 : cluster 0 pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:01 smithi153 bash[23852]: cluster 2024-04-06T03:50:00.827870+0000 osd.3 (osd.3) 689 : cluster 0 2.1b scrub starts 2024-04-06T03:50:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:01 smithi153 bash[23852]: cluster 2024-04-06T03:50:00.830495+0000 osd.3 (osd.3) 690 : cluster 0 2.1b scrub ok 2024-04-06T03:50:02.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.092607+0000 osd.0 (osd.0) 674 : cluster 0 2.1f scrub starts 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.095149+0000 osd.0 (osd.0) 675 : cluster 0 2.1f scrub ok 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.350312+0000 osd.4 (osd.4) 663 : cluster 0 3.1d scrub starts 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.354457+0000 osd.4 (osd.4) 664 : cluster 0 3.1d scrub ok 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.441870+0000 osd.7 (osd.7) 697 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.484410+0000 osd.7 (osd.7) 698 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.860469+0000 osd.3 (osd.3) 691 : cluster 0 3.1e scrub starts 2024-04-06T03:50:02.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:02 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.864805+0000 osd.3 (osd.3) 692 : cluster 0 3.1e scrub ok 2024-04-06T03:50:02.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.092607+0000 osd.0 (osd.0) 674 : cluster 0 2.1f scrub starts 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.095149+0000 osd.0 (osd.0) 675 : cluster 0 2.1f scrub ok 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.350312+0000 osd.4 (osd.4) 663 : cluster 0 3.1d scrub starts 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.354457+0000 osd.4 (osd.4) 664 : cluster 0 3.1d scrub ok 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.441870+0000 osd.7 (osd.7) 697 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.484410+0000 osd.7 (osd.7) 698 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.860469+0000 osd.3 (osd.3) 691 : cluster 0 3.1e scrub starts 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.864805+0000 osd.3 (osd.3) 692 : cluster 0 3.1e scrub ok 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.092607+0000 osd.0 (osd.0) 674 : cluster 0 2.1f scrub starts 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.095149+0000 osd.0 (osd.0) 675 : cluster 0 2.1f scrub ok 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.350312+0000 osd.4 (osd.4) 663 : cluster 0 3.1d scrub starts 2024-04-06T03:50:02.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.354457+0000 osd.4 (osd.4) 664 : cluster 0 3.1d scrub ok 2024-04-06T03:50:02.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.441870+0000 osd.7 (osd.7) 697 : cluster 0 6.12 deep-scrub starts 2024-04-06T03:50:02.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.484410+0000 osd.7 (osd.7) 698 : cluster 0 6.12 deep-scrub ok 2024-04-06T03:50:02.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.860469+0000 osd.3 (osd.3) 691 : cluster 0 3.1e scrub starts 2024-04-06T03:50:02.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:02 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.864805+0000 osd.3 (osd.3) 692 : cluster 0 3.1e scrub ok 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.424431+0000 osd.1 (osd.1) 453 : cluster 0 6.19 scrub starts 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.547065+0000 osd.1 (osd.1) 454 : cluster 0 6.19 scrub ok 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.554807+0000 osd.1 (osd.1) 455 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:01.554901+0000 osd.1 (osd.1) 456 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:02.131701+0000 mon.a (mon.0) 1876 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:02.641559+0000 mgr.y (mgr.24368) 1067 : cluster 0 pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:02.840948+0000 osd.3 (osd.3) 693 : cluster 0 2.14 scrub starts 2024-04-06T03:50:03.403 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:03 smithi153 bash[23852]: cluster 2024-04-06T03:50:02.843677+0000 osd.3 (osd.3) 694 : cluster 0 2.14 scrub ok 2024-04-06T03:50:03.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.424431+0000 osd.1 (osd.1) 453 : cluster 0 6.19 scrub starts 2024-04-06T03:50:03.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.547065+0000 osd.1 (osd.1) 454 : cluster 0 6.19 scrub ok 2024-04-06T03:50:03.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.554807+0000 osd.1 (osd.1) 455 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:01.554901+0000 osd.1 (osd.1) 456 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:02.131701+0000 mon.a (mon.0) 1876 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:02.641559+0000 mgr.y (mgr.24368) 1067 : cluster 0 pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:02.840948+0000 osd.3 (osd.3) 693 : cluster 0 2.14 scrub starts 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[21467]: cluster 2024-04-06T03:50:02.843677+0000 osd.3 (osd.3) 694 : cluster 0 2.14 scrub ok 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.424431+0000 osd.1 (osd.1) 453 : cluster 0 6.19 scrub starts 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.547065+0000 osd.1 (osd.1) 454 : cluster 0 6.19 scrub ok 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.554807+0000 osd.1 (osd.1) 455 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:01.554901+0000 osd.1 (osd.1) 456 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:02.131701+0000 mon.a (mon.0) 1876 : cluster 0 osdmap e421: 8 total, 8 up, 8 in 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:02.641559+0000 mgr.y (mgr.24368) 1067 : cluster 0 pgmap v1173: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:03.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:02.840948+0000 osd.3 (osd.3) 693 : cluster 0 2.14 scrub starts 2024-04-06T03:50:03.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:03 smithi050 bash[29221]: cluster 2024-04-06T03:50:02.843677+0000 osd.3 (osd.3) 694 : cluster 0 2.14 scrub ok 2024-04-06T03:50:03.577 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:03.577 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:50:03.577 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:50:03.577 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:50:03.578 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:04.319 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.4 dump_historic_ops 2024-04-06T03:50:04.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:04 smithi050 bash[29221]: cluster 2024-04-06T03:50:03.422808+0000 osd.2 (osd.2) 573 : cluster 0 6.f scrub starts 2024-04-06T03:50:04.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:04 smithi050 bash[29221]: cluster 2024-04-06T03:50:03.880205+0000 osd.3 (osd.3) 695 : cluster 0 3.18 scrub starts 2024-04-06T03:50:04.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:04 smithi050 bash[29221]: cluster 2024-04-06T03:50:03.884845+0000 osd.3 (osd.3) 696 : cluster 0 3.18 scrub ok 2024-04-06T03:50:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:04 smithi050 bash[21467]: cluster 2024-04-06T03:50:03.422808+0000 osd.2 (osd.2) 573 : cluster 0 6.f scrub starts 2024-04-06T03:50:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:04 smithi050 bash[21467]: cluster 2024-04-06T03:50:03.880205+0000 osd.3 (osd.3) 695 : cluster 0 3.18 scrub starts 2024-04-06T03:50:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:04 smithi050 bash[21467]: cluster 2024-04-06T03:50:03.884845+0000 osd.3 (osd.3) 696 : cluster 0 3.18 scrub ok 2024-04-06T03:50:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:04 smithi153 bash[23852]: cluster 2024-04-06T03:50:03.422808+0000 osd.2 (osd.2) 573 : cluster 0 6.f scrub starts 2024-04-06T03:50:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:04 smithi153 bash[23852]: cluster 2024-04-06T03:50:03.880205+0000 osd.3 (osd.3) 695 : cluster 0 3.18 scrub starts 2024-04-06T03:50:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:04 smithi153 bash[23852]: cluster 2024-04-06T03:50:03.884845+0000 osd.3 (osd.3) 696 : cluster 0 3.18 scrub ok 2024-04-06T03:50:04.705 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:50:05.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[21467]: cluster 2024-04-06T03:50:03.389425+0000 osd.1 (osd.1) 457 : cluster 0 5.12 scrub starts 2024-04-06T03:50:05.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[21467]: cluster 2024-04-06T03:50:03.391680+0000 osd.1 (osd.1) 458 : cluster 0 5.12 scrub ok 2024-04-06T03:50:05.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[21467]: cluster 2024-04-06T03:50:03.646587+0000 osd.2 (osd.2) 574 : cluster 0 6.f scrub ok 2024-04-06T03:50:05.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[21467]: cluster 2024-04-06T03:50:04.642630+0000 mgr.y (mgr.24368) 1068 : cluster 0 pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:05.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[29221]: cluster 2024-04-06T03:50:03.389425+0000 osd.1 (osd.1) 457 : cluster 0 5.12 scrub starts 2024-04-06T03:50:05.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[29221]: cluster 2024-04-06T03:50:03.391680+0000 osd.1 (osd.1) 458 : cluster 0 5.12 scrub ok 2024-04-06T03:50:05.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[29221]: cluster 2024-04-06T03:50:03.646587+0000 osd.2 (osd.2) 574 : cluster 0 6.f scrub ok 2024-04-06T03:50:05.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:05 smithi050 bash[29221]: cluster 2024-04-06T03:50:04.642630+0000 mgr.y (mgr.24368) 1068 : cluster 0 pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:05 smithi153 bash[23852]: cluster 2024-04-06T03:50:03.389425+0000 osd.1 (osd.1) 457 : cluster 0 5.12 scrub starts 2024-04-06T03:50:05.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:05 smithi153 bash[23852]: cluster 2024-04-06T03:50:03.391680+0000 osd.1 (osd.1) 458 : cluster 0 5.12 scrub ok 2024-04-06T03:50:05.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:05 smithi153 bash[23852]: cluster 2024-04-06T03:50:03.646587+0000 osd.2 (osd.2) 574 : cluster 0 6.f scrub ok 2024-04-06T03:50:05.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:05 smithi153 bash[23852]: cluster 2024-04-06T03:50:04.642630+0000 mgr.y (mgr.24368) 1068 : cluster 0 pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:06.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:06 smithi050 bash[29221]: audit 2024-04-06T03:50:05.952669+0000 mon.a (mon.0) 1877 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:06.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:06 smithi050 bash[21467]: audit 2024-04-06T03:50:05.952669+0000 mon.a (mon.0) 1877 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:06 smithi153 bash[23852]: audit 2024-04-06T03:50:05.952669+0000 mon.a (mon.0) 1877 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:07.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:07 smithi050 bash[21467]: cluster 2024-04-06T03:50:05.466851+0000 osd.1 (osd.1) 459 : cluster 0 2.6 scrub starts 2024-04-06T03:50:07.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:07 smithi050 bash[21467]: cluster 2024-04-06T03:50:05.470444+0000 osd.1 (osd.1) 460 : cluster 0 2.6 scrub ok 2024-04-06T03:50:07.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:07 smithi050 bash[21467]: cluster 2024-04-06T03:50:06.643708+0000 mgr.y (mgr.24368) 1069 : cluster 0 pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:07.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:07 smithi050 bash[29221]: cluster 2024-04-06T03:50:05.466851+0000 osd.1 (osd.1) 459 : cluster 0 2.6 scrub starts 2024-04-06T03:50:07.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:07 smithi050 bash[29221]: cluster 2024-04-06T03:50:05.470444+0000 osd.1 (osd.1) 460 : cluster 0 2.6 scrub ok 2024-04-06T03:50:07.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:07 smithi050 bash[29221]: cluster 2024-04-06T03:50:06.643708+0000 mgr.y (mgr.24368) 1069 : cluster 0 pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:07 smithi153 bash[23852]: cluster 2024-04-06T03:50:05.466851+0000 osd.1 (osd.1) 459 : cluster 0 2.6 scrub starts 2024-04-06T03:50:07.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:07 smithi153 bash[23852]: cluster 2024-04-06T03:50:05.470444+0000 osd.1 (osd.1) 460 : cluster 0 2.6 scrub ok 2024-04-06T03:50:07.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:07 smithi153 bash[23852]: cluster 2024-04-06T03:50:06.643708+0000 mgr.y (mgr.24368) 1069 : cluster 0 pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:08.185 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-06T03:50:08.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[21467]: cluster 2024-04-06T03:50:06.461374+0000 osd.1 (osd.1) 461 : cluster 0 5.7 scrub starts 2024-04-06T03:50:08.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[21467]: cluster 2024-04-06T03:50:06.515707+0000 osd.1 (osd.1) 462 : cluster 0 5.7 scrub ok 2024-04-06T03:50:08.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[21467]: cluster 2024-04-06T03:50:07.475872+0000 osd.2 (osd.2) 575 : cluster 0 5.d scrub starts 2024-04-06T03:50:08.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[21467]: cluster 2024-04-06T03:50:07.479338+0000 osd.2 (osd.2) 576 : cluster 0 5.d scrub ok 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[21467]: audit 2024-04-06T03:50:08.056213+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.50:0/2200378438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[21467]: audit 2024-04-06T03:50:08.056834+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[29221]: cluster 2024-04-06T03:50:06.461374+0000 osd.1 (osd.1) 461 : cluster 0 5.7 scrub starts 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[29221]: cluster 2024-04-06T03:50:06.515707+0000 osd.1 (osd.1) 462 : cluster 0 5.7 scrub ok 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[29221]: cluster 2024-04-06T03:50:07.475872+0000 osd.2 (osd.2) 575 : cluster 0 5.d scrub starts 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[29221]: cluster 2024-04-06T03:50:07.479338+0000 osd.2 (osd.2) 576 : cluster 0 5.d scrub ok 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[29221]: audit 2024-04-06T03:50:08.056213+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.50:0/2200378438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:50:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:08 smithi050 bash[29221]: audit 2024-04-06T03:50:08.056834+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:50:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:08 smithi153 bash[23852]: cluster 2024-04-06T03:50:06.461374+0000 osd.1 (osd.1) 461 : cluster 0 5.7 scrub starts 2024-04-06T03:50:08.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:08 smithi153 bash[23852]: cluster 2024-04-06T03:50:06.515707+0000 osd.1 (osd.1) 462 : cluster 0 5.7 scrub ok 2024-04-06T03:50:08.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:08 smithi153 bash[23852]: cluster 2024-04-06T03:50:07.475872+0000 osd.2 (osd.2) 575 : cluster 0 5.d scrub starts 2024-04-06T03:50:08.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:08 smithi153 bash[23852]: cluster 2024-04-06T03:50:07.479338+0000 osd.2 (osd.2) 576 : cluster 0 5.d scrub ok 2024-04-06T03:50:08.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:08 smithi153 bash[23852]: audit 2024-04-06T03:50:08.056213+0000 mon.c (mon.2) 128 : audit 1 from='client.? 172.21.15.50:0/2200378438' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:50:08.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:08 smithi153 bash[23852]: audit 2024-04-06T03:50:08.056834+0000 mon.a (mon.0) 1878 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-06T03:50:09.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 4 1 2024-04-06T03:50:09.110 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: audit 2024-04-06T03:50:08.171047+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: cluster 2024-04-06T03:50:08.178233+0000 mon.a (mon.0) 1880 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: cluster 2024-04-06T03:50:08.358366+0000 osd.4 (osd.4) 665 : cluster 0 2.f scrub starts 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: cluster 2024-04-06T03:50:08.361477+0000 osd.4 (osd.4) 666 : cluster 0 2.f scrub ok 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: cluster 2024-04-06T03:50:08.644653+0000 mgr.y (mgr.24368) 1070 : cluster 0 pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: cluster 2024-04-06T03:50:08.889971+0000 osd.3 (osd.3) 697 : cluster 0 3.b scrub starts 2024-04-06T03:50:09.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21467]: cluster 2024-04-06T03:50:08.894500+0000 osd.3 (osd.3) 698 : cluster 0 3.b scrub ok 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: audit 2024-04-06T03:50:08.171047+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: cluster 2024-04-06T03:50:08.178233+0000 mon.a (mon.0) 1880 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: cluster 2024-04-06T03:50:08.358366+0000 osd.4 (osd.4) 665 : cluster 0 2.f scrub starts 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: cluster 2024-04-06T03:50:08.361477+0000 osd.4 (osd.4) 666 : cluster 0 2.f scrub ok 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: cluster 2024-04-06T03:50:08.644653+0000 mgr.y (mgr.24368) 1070 : cluster 0 pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: cluster 2024-04-06T03:50:08.889971+0000 osd.3 (osd.3) 697 : cluster 0 3.b scrub starts 2024-04-06T03:50:09.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[29221]: cluster 2024-04-06T03:50:08.894500+0000 osd.3 (osd.3) 698 : cluster 0 3.b scrub ok 2024-04-06T03:50:09.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:50:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:50:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:50:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: audit 2024-04-06T03:50:08.171047+0000 mon.a (mon.0) 1879 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-06T03:50:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: cluster 2024-04-06T03:50:08.178233+0000 mon.a (mon.0) 1880 : cluster 0 osdmap e422: 8 total, 8 up, 8 in 2024-04-06T03:50:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: cluster 2024-04-06T03:50:08.358366+0000 osd.4 (osd.4) 665 : cluster 0 2.f scrub starts 2024-04-06T03:50:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: cluster 2024-04-06T03:50:08.361477+0000 osd.4 (osd.4) 666 : cluster 0 2.f scrub ok 2024-04-06T03:50:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: cluster 2024-04-06T03:50:08.644653+0000 mgr.y (mgr.24368) 1070 : cluster 0 pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: cluster 2024-04-06T03:50:08.889971+0000 osd.3 (osd.3) 697 : cluster 0 3.b scrub starts 2024-04-06T03:50:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:09 smithi153 bash[23852]: cluster 2024-04-06T03:50:08.894500+0000 osd.3 (osd.3) 698 : cluster 0 3.b scrub ok 2024-04-06T03:50:10.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:10 smithi153 bash[23852]: cluster 2024-04-06T03:50:09.372297+0000 osd.4 (osd.4) 667 : cluster 0 3.1a scrub starts 2024-04-06T03:50:10.476 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:10 smithi153 bash[23852]: cluster 2024-04-06T03:50:09.378676+0000 osd.4 (osd.4) 668 : cluster 0 3.1a scrub ok 2024-04-06T03:50:10.476 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:10 smithi153 bash[23852]: cluster 2024-04-06T03:50:09.460532+0000 osd.2 (osd.2) 577 : cluster 0 5.8 scrub starts 2024-04-06T03:50:10.476 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:10 smithi153 bash[23852]: cluster 2024-04-06T03:50:09.462929+0000 osd.2 (osd.2) 578 : cluster 0 5.8 scrub ok 2024-04-06T03:50:10.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[29221]: cluster 2024-04-06T03:50:09.372297+0000 osd.4 (osd.4) 667 : cluster 0 3.1a scrub starts 2024-04-06T03:50:10.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[29221]: cluster 2024-04-06T03:50:09.378676+0000 osd.4 (osd.4) 668 : cluster 0 3.1a scrub ok 2024-04-06T03:50:10.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[29221]: cluster 2024-04-06T03:50:09.460532+0000 osd.2 (osd.2) 577 : cluster 0 5.8 scrub starts 2024-04-06T03:50:10.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[29221]: cluster 2024-04-06T03:50:09.462929+0000 osd.2 (osd.2) 578 : cluster 0 5.8 scrub ok 2024-04-06T03:50:10.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[21467]: cluster 2024-04-06T03:50:09.372297+0000 osd.4 (osd.4) 667 : cluster 0 3.1a scrub starts 2024-04-06T03:50:10.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[21467]: cluster 2024-04-06T03:50:09.378676+0000 osd.4 (osd.4) 668 : cluster 0 3.1a scrub ok 2024-04-06T03:50:10.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[21467]: cluster 2024-04-06T03:50:09.460532+0000 osd.2 (osd.2) 577 : cluster 0 5.8 scrub starts 2024-04-06T03:50:10.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:10 smithi050 bash[21467]: cluster 2024-04-06T03:50:09.462929+0000 osd.2 (osd.2) 578 : cluster 0 5.8 scrub ok 2024-04-06T03:50:10.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:50:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:50:11.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:11 smithi153 bash[23852]: audit 2024-04-06T03:50:10.476295+0000 mgr.y (mgr.24368) 1071 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:11.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:11 smithi153 bash[23852]: cluster 2024-04-06T03:50:10.645980+0000 mgr.y (mgr.24368) 1072 : cluster 0 pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:11.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:11 smithi050 bash[21467]: audit 2024-04-06T03:50:10.476295+0000 mgr.y (mgr.24368) 1071 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:11.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:11 smithi050 bash[21467]: cluster 2024-04-06T03:50:10.645980+0000 mgr.y (mgr.24368) 1072 : cluster 0 pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:11.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:11 smithi050 bash[29221]: audit 2024-04-06T03:50:10.476295+0000 mgr.y (mgr.24368) 1071 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:11.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:11 smithi050 bash[29221]: cluster 2024-04-06T03:50:10.645980+0000 mgr.y (mgr.24368) 1072 : cluster 0 pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:12.023 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166296 10.e 10:70aa3018:::benchmark_data_smithi050_121293_object67479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.552144+0000", 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 492.46892554200002, 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.006335285, 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.024 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166296 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552144+0000", 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552144+0000", 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.025 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552145+0000", 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.161e-06 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552151+0000", 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4280000000000004e-06 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552166+0000", 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.47e-05 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.026 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552174+0000", 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0099999999999995e-06 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552227+0000", 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3377000000000001e-05 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.552235+0000", 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4230000000000005e-06 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.027 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.552020+0000", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.99978509699999996 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.552127+0000", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000106659 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.552202+0000", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4739000000000004e-05 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.028 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554697+0000", 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024956169999999999 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558325+0000", 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036280129999999998 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558379+0000", 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3653999999999998e-05 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.029 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558436+0000", 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6956000000000002e-05 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558480+0000", 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3451000000000001e-05 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.030 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166306 10.e 10:7a0341ac:::benchmark_data_smithi050_121293_object67489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.563338+0000", 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 492.45773163600001, 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0000557219999999, 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166306 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.031 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563338+0000", 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563338+0000", 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563339+0000", 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1009999999999999e-06 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.032 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563345+0000", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6269999999999997e-06 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563355+0000", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.9699999999999994e-06 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563360+0000", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5010000000000004e-06 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.033 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563403+0000", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2561000000000001e-05 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.563409+0000", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7259999999999999e-06 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554736+0000", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.99132699700000004 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554814+0000", 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8863999999999996e-05 2024-04-06T03:50:12.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.554886+0000", 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1253000000000004e-05 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558532+0000", 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036460559999999999 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558595+0000", 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3361000000000005e-05 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.035 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563293+0000", 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0046974119999999998 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563351+0000", 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8588000000000002e-05 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563394+0000", 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2704999999999999e-05 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.036 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166355 10.e 10:7bc3e0da:::benchmark_data_smithi050_121293_object67538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.617833+0000", 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 492.40323687400002, 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94572550600000005, 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166355 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.037 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617833+0000", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617833+0000", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617834+0000", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0780000000000001e-06 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617840+0000", 2024-04-06T03:50:12.038 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8490000000000002e-06 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617852+0000", 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2264e-05 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617861+0000", 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.85e-06 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.039 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617907+0000", 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6387000000000002e-05 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.617917+0000", 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2900000000000008e-06 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.556421+0000", 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93850430200000001 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.556533+0000", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112378 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.556666+0000", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013266600000000001 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558635+0000", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019686730000000002 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.041 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558648+0000", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2917e-05 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563449+0000", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0048014570000000003 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563505+0000", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5788999999999997e-05 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563558+0000", 2024-04-06T03:50:12.042 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3606000000000001e-05 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166405 10.e 10:73562072:::benchmark_data_smithi050_121293_object67588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.665681+0000", 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 492.35538832399999, 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89810087199999999, 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.043 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166405 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665681+0000", 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665681+0000", 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665682+0000", 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5599999999999995e-07 2024-04-06T03:50:12.044 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665685+0000", 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7389999999999999e-06 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665690+0000", 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4310000000000002e-06 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665694+0000", 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0389999999999998e-06 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.045 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665748+0000", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3936e-05 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.665757+0000", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4500000000000004e-06 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558697+0000", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89294079199999998 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.046 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558802+0000", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010429000000000001 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.558925+0000", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012352700000000001 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563572+0000", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0046464970000000003 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563634+0000", 2024-04-06T03:50:12.047 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2106999999999993e-05 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563690+0000", 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6029000000000002e-05 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563741+0000", 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0942000000000001e-05 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563782+0000", 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1536999999999999e-05 2024-04-06T03:50:12.048 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166462 10.e 10:7a1da6ec:::benchmark_data_smithi050_121293_object67645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:59.739365+0000", 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 492.28170499999999, 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.82941186099999997, 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.049 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166462 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739365+0000", 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739365+0000", 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.050 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739366+0000", 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0240000000000001e-06 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739370+0000", 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8099999999999999e-06 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739382+0000", 2024-04-06T03:50:12.051 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1929999999999999e-05 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739390+0000", 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4160000000000004e-06 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739453+0000", 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3379000000000004e-05 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.739462+0000", 2024-04-06T03:50:12.052 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1239999999999994e-06 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.561006+0000", 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.82154333899999998 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.561132+0000", 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000126327 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,5", 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.561263+0000", 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013106200000000001 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.053 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563794+0000", 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002530568 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.563850+0000", 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6280000000000003e-05 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.568671+0000", 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.004821377 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.054 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.568727+0000", 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5729999999999997e-05 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.568777+0000", 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9495e-05 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.055 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166657 10.0 10:0c5fca79:::benchmark_data_smithi050_121293_object67840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 491.44143307899998, 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.97418345500000003, 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166657 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:50:12.056 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579637+0000", 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0999999999999998e-07 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579640+0000", 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1590000000000002e-06 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.057 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579649+0000", 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1049999999999995e-06 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579655+0000", 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5769999999999999e-06 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579785+0000", 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013009499999999999 2024-04-06T03:50:12.058 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.579794+0000", 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8040000000000007e-06 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.546805+0000", 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96701049500000003 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.547033+0000", 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022838900000000001 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.059 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.547269+0000", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00023642800000000001 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550576+0000", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033069089999999998 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550589+0000", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2306999999999999e-05 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.060 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553711+0000", 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031220969999999999 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553768+0000", 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7395000000000002e-05 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553820+0000", 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2085e-05 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.061 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166669 10.0 10:0ee64f06:::benchmark_data_smithi050_121293_object67852:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.596319+0000", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 491.42475053300001, 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96643771099999998, 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166669 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596319+0000", 2024-04-06T03:50:12.062 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596319+0000", 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596320+0000", 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9500000000000002e-07 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596323+0000", 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0089999999999999e-06 2024-04-06T03:50:12.063 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596328+0000", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5400000000000003e-06 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596333+0000", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3229999999999999e-06 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596372+0000", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9202999999999997e-05 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.064 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.596377+0000", 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4909999999999996e-06 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550616+0000", 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.95423870799999999 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550730+0000", 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000113606 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:50:12.065 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.550853+0000", 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000123683 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558454+0000", 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0076005459999999997 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558518+0000", 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3930000000000006e-05 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.066 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562684+0000", 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0041664179999999999 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562726+0000", 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1202999999999998e-05 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562757+0000", 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1455999999999998e-05 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.067 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166688 10.0 10:0f46cca1:::benchmark_data_smithi050_121293_object67871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.613205+0000", 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 491.407864776, 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94968637499999997, 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.068 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166688 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613205+0000", 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613205+0000", 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.069 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613206+0000", 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9999999999999996e-07 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613211+0000", 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8300000000000003e-06 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613219+0000", 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4489999999999999e-06 2024-04-06T03:50:12.070 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613226+0000", 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.63e-06 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613284+0000", 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7806999999999997e-05 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.613292+0000", 2024-04-06T03:50:12.071 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5159999999999994e-06 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.552513+0000", 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93922135100000004 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.552574+0000", 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0516000000000001e-05 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.552638+0000", 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4424999999999994e-05 2024-04-06T03:50:12.072 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558467+0000", 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0058283010000000001 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562800+0000", 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0043330269999999997 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562839+0000", 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8983999999999998e-05 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.073 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562868+0000", 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9204999999999999e-05 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562891+0000", 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3533999999999999e-05 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.074 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166694 10.0 10:0549138b:::benchmark_data_smithi050_121293_object67877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.617633+0000", 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 491.40343642300002, 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.95598773000000004, 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166694 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.075 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617633+0000", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617633+0000", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617634+0000", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4499999999999995e-07 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617639+0000", 2024-04-06T03:50:12.076 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6349999999999997e-06 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617648+0000", 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9390000000000003e-06 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617654+0000", 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7660000000000004e-06 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617699+0000", 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5386000000000002e-05 2024-04-06T03:50:12.077 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.617705+0000", 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1229999999999997e-06 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553859+0000", 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93615364999999995 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553926+0000", 2024-04-06T03:50:12.078 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6758e-05 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.553991+0000", 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5053000000000002e-05 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558533+0000", 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0045419589999999999 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.562954+0000", 2024-04-06T03:50:12.079 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044217409999999999 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573507+0000", 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010552601 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573570+0000", 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2898000000000007e-05 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573621+0000", 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1276000000000002e-05 2024-04-06T03:50:12.080 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:166721 10.0 10:046565d5:::benchmark_data_smithi050_121293_object67904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 491.36981792900002, 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.92257131400000003, 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 166721 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651252+0000", 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7899999999999999e-07 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651255+0000", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6790000000000001e-06 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651261+0000", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0499999999999997e-06 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651267+0000", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5949999999999998e-06 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651320+0000", 2024-04-06T03:50:12.083 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3390000000000002e-05 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:00.651329+0000", 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9809999999999992e-06 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.555249+0000", 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.90391947399999995 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.555365+0000", 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011683700000000001 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 1,7", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.555438+0000", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.224e-05 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.558539+0000", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031015090000000001 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573446+0000", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.014907388000000001 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.085 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573720+0000", 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00027335100000000002 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573781+0000", 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0846e-05 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:01.573823+0000", 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2494999999999999e-05 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167251 10.d 10:b458ba05:::benchmark_data_smithi050_121293_object68434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.632106+0000", 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.38896418299998, 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96111719799999995, 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167251 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.087 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632106+0000", 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632106+0000", 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632110+0000", 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0330000000000002e-06 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.088 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632125+0000", 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5639999999999999e-05 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632166+0000", 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0365999999999999e-05 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632190+0000", 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4083e-05 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.089 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632315+0000", 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000125217 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.632331+0000", 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5639999999999999e-05 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.573490+0000", 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94115940200000003 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.573684+0000", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000193582 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.573881+0000", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00019781599999999999 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.579581+0000", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0056996119999999997 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591195+0000", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.011613627 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593068+0000", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001873535 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593159+0000", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0503999999999999e-05 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593223+0000", 2024-04-06T03:50:12.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4140999999999995e-05 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167253 10.d 10:b8271380:::benchmark_data_smithi050_121293_object68436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.634581+0000", 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.38648910000001, 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.96146916199999999, 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.093 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167253 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634581+0000", 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634581+0000", 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634583+0000", 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0930000000000001e-06 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634596+0000", 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3635000000000001e-05 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634631+0000", 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4479e-05 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634651+0000", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9927000000000001e-05 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634783+0000", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013174000000000001 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.634802+0000", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9519000000000001e-05 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.576782+0000", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.94197949299999995 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.576874+0000", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2299999999999994e-05 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.576949+0000", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5228999999999994e-05 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584102+0000", 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0071525440000000003 2024-04-06T03:50:12.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593342+0000", 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0092403950000000002 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.595806+0000", 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024639929999999998 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.595946+0000", 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014025499999999999 2024-04-06T03:50:12.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.596050+0000", 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010356 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167259 10.d 10:ba4c23a1:::benchmark_data_smithi050_121293_object68442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.660890+0000", 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.36018011499999, 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93821357100000002, 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.099 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167259 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660890+0000", 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660890+0000", 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660891+0000", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5829999999999999e-06 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660900+0000", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8100000000000004e-06 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660918+0000", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8215000000000001e-05 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660931+0000", 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3030999999999999e-05 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.661043+0000", 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111331 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.661065+0000", 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2019999999999999e-05 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.578359+0000", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.91729432499999997 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.578426+0000", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7316000000000004e-05 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.578496+0000", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0028000000000001e-05 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585367+0000", 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006871116 2024-04-06T03:50:12.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.596108+0000", 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.010740201 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.598848+0000", 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0027407600000000001 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.599002+0000", 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000153492 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.599103+0000", 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000101343 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167275 10.d 10:b830d4fc:::benchmark_data_smithi050_121293_object68458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.677093+0000", 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.34397710799999, 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93899825599999998, 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.105 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167275 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677093+0000", 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677093+0000", 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677094+0000", 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4559999999999999e-06 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677102+0000", 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4800000000000004e-06 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677115+0000", 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3818000000000001e-05 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677127+0000", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1504e-05 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677215+0000", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8467999999999994e-05 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.677228+0000", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2238999999999999e-05 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.579719+0000", 2024-04-06T03:50:12.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.90249140900000002 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.579947+0000", 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000227813 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.580172+0000", 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022501800000000001 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585379+0000", 2024-04-06T03:50:12.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0052066550000000001 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.603365+0000", 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017986565999999999 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.615933+0000", 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012568119000000001 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.615997+0000", 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3530999999999999e-05 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616091+0000", 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4179999999999996e-05 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167283 10.d 10:b6ba4186:::benchmark_data_smithi050_121293_object68466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.684070+0000", 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.33699957599998, 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.93226327499999995, 2024-04-06T03:50:12.111 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167283 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684070+0000", 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684070+0000", 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684071+0000", 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.235e-06 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684078+0000", 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8600000000000004e-06 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684095+0000", 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6354e-05 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684105+0000", 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0496000000000001e-05 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684187+0000", 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1683999999999999e-05 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.684196+0000", 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7679999999999997e-06 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.582837+0000", 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89864155400000001 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.582938+0000", 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000100642 2024-04-06T03:50:12.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.583033+0000", 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5432000000000004e-05 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.589783+0000", 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.006750196 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.603462+0000", 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.013678350000000001 2024-04-06T03:50:12.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616167+0000", 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.012705363000000001 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616263+0000", 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5742000000000006e-05 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616333+0000", 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0599000000000006e-05 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.117 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167293 10.d 10:bee55742:::benchmark_data_smithi050_121293_object68476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.696438+0000", 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.32463194299999, 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.92009158000000002, 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167293 2024-04-06T03:50:12.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696438+0000", 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696438+0000", 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696439+0000", 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.226e-06 2024-04-06T03:50:12.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696445+0000", 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8769999999999998e-06 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696458+0000", 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3541e-05 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696472+0000", 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3951000000000001e-05 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697727+0000", 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0012547299999999999 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697740+0000", 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2476999999999999e-05 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584136+0000", 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.88639684500000004 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584233+0000", 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6334000000000003e-05 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.584316+0000", 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3487999999999998e-05 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591066+0000", 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0067497080000000001 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.607549+0000", 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.016483003999999999 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616409+0000", 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0088596300000000003 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616465+0000", 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6335000000000002e-05 2024-04-06T03:50:12.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616529+0000", 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4434e-05 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167294 10.d 10:b609007b:::benchmark_data_smithi050_121293_object68477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.696514+0000", 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.32455561299997, 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.92033769799999998, 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.124 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167294 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696514+0000", 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696514+0000", 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696515+0000", 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9400000000000004e-07 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696520+0000", 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0250000000000002e-06 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696528+0000", 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8609999999999998e-06 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.696535+0000", 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4769999999999997e-06 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697771+0000", 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001235547 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.697779+0000", 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9130000000000003e-06 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585469+0000", 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.88769055399999997 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585697+0000", 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022735 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.585918+0000", 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00022151199999999999 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591072+0000", 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.005153393 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.607634+0000", 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.01656241 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616606+0000", 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0089722410000000006 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616727+0000", 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012099000000000001 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616852+0000", 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012463099999999999 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.130 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167314 10.d 10:be2d3a0d:::benchmark_data_smithi050_121293_object68497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.720937+0000", 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.30013244999998, 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89619347000000005, 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167314 2024-04-06T03:50:12.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720937+0000", 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720937+0000", 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720938+0000", 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.54e-07 2024-04-06T03:50:12.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720943+0000", 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3460000000000001e-06 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720955+0000", 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1399999999999999e-05 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.720966+0000", 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1372000000000001e-05 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.721036+0000", 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9919999999999995e-05 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.721046+0000", 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3940000000000002e-06 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.588520+0000", 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.867474262 2024-04-06T03:50:12.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.588618+0000", 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.8269000000000004e-05 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.588706+0000", 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.8362999999999994e-05 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593009+0000", 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0043024109999999999 2024-04-06T03:50:12.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.607684+0000", 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.014675308 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.616940+0000", 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0092559180000000001 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.617046+0000", 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000105504 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.617131+0000", 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5248999999999996e-05 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167321 10.d 10:b0b421cc:::benchmark_data_smithi050_121293_object68504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.736250+0000", 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.28481931300001, 2024-04-06T03:50:12.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.89355279600000004, 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167321 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736250+0000", 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736250+0000", 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736252+0000", 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5570000000000001e-06 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736258+0000", 2024-04-06T03:50:12.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4400000000000002e-06 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736273+0000", 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4924e-05 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736284+0000", 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0149999999999999e-05 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736353+0000", 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9795000000000005e-05 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.736364+0000", 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0353000000000001e-05 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.589820+0000", 2024-04-06T03:50:12.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.85345636800000002 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.589917+0000", 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7200000000000004e-05 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.590004+0000", 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6927000000000003e-05 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593239+0000", 2024-04-06T03:50:12.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0032349380000000001 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.612346+0000", 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.019107049000000001 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629497+0000", 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017150971000000001 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629665+0000", 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016773600000000001 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629803+0000", 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000138388 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:167328 10.d 10:bc0aacba:::benchmark_data_smithi050_121293_object68511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.777788+0000", 2024-04-06T03:50:12.144 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 488.24328167800002, 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.85257628699999999, 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 167328 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777788+0000", 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777788+0000", 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777791+0000", 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.683e-06 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777807+0000", 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6599000000000002e-05 2024-04-06T03:50:12.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777844+0000", 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6470000000000001e-05 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.777863+0000", 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9606000000000002e-05 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.778040+0000", 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000176217 2024-04-06T03:50:12.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for peered", 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.778067+0000", 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7452000000000001e-05 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591245+0000", 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.81317800600000001 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591320+0000", 2024-04-06T03:50:12.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4659999999999993e-05 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,3", 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.591427+0000", 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010718299999999999 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.593247+0000", 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001819747 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.612617+0000", 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.019370396000000002 2024-04-06T03:50:12.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.629941+0000", 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.017323966 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.630188+0000", 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00024702099999999999 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:04.630364+0000", 2024-04-06T03:50:12.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000176281 2024-04-06T03:50:12.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:12.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:12.151 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:12.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:12 smithi153 bash[23852]: cluster 2024-04-06T03:50:11.393007+0000 osd.4 (osd.4) 669 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:50:12.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:12 smithi153 bash[23852]: cluster 2024-04-06T03:50:11.395385+0000 osd.4 (osd.4) 670 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:50:12.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:12 smithi153 bash[23852]: cluster 2024-04-06T03:50:11.950864+0000 osd.3 (osd.3) 699 : cluster 0 5.0 scrub starts 2024-04-06T03:50:12.376 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:12 smithi153 bash[23852]: cluster 2024-04-06T03:50:11.953875+0000 osd.3 (osd.3) 700 : cluster 0 5.0 scrub ok 2024-04-06T03:50:12.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[21467]: cluster 2024-04-06T03:50:11.393007+0000 osd.4 (osd.4) 669 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:50:12.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[21467]: cluster 2024-04-06T03:50:11.395385+0000 osd.4 (osd.4) 670 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:50:12.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[21467]: cluster 2024-04-06T03:50:11.950864+0000 osd.3 (osd.3) 699 : cluster 0 5.0 scrub starts 2024-04-06T03:50:12.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[21467]: cluster 2024-04-06T03:50:11.953875+0000 osd.3 (osd.3) 700 : cluster 0 5.0 scrub ok 2024-04-06T03:50:12.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[29221]: cluster 2024-04-06T03:50:11.393007+0000 osd.4 (osd.4) 669 : cluster 0 5.18 deep-scrub starts 2024-04-06T03:50:12.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[29221]: cluster 2024-04-06T03:50:11.395385+0000 osd.4 (osd.4) 670 : cluster 0 5.18 deep-scrub ok 2024-04-06T03:50:12.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[29221]: cluster 2024-04-06T03:50:11.950864+0000 osd.3 (osd.3) 699 : cluster 0 5.0 scrub starts 2024-04-06T03:50:12.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:12 smithi050 bash[29221]: cluster 2024-04-06T03:50:11.953875+0000 osd.3 (osd.3) 700 : cluster 0 5.0 scrub ok 2024-04-06T03:50:12.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-06T03:50:13.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.392850+0000 osd.4 (osd.4) 671 : cluster 0 5.11 scrub starts 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.395115+0000 osd.4 (osd.4) 672 : cluster 0 5.11 scrub ok 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.444342+0000 osd.2 (osd.2) 579 : cluster 0 2.c deep-scrub starts 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.446864+0000 osd.2 (osd.2) 580 : cluster 0 2.c deep-scrub ok 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.566217+0000 osd.1 (osd.1) 463 : cluster 0 2.15 scrub starts 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.568522+0000 osd.1 (osd.1) 464 : cluster 0 2.15 scrub ok 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.646865+0000 mgr.y (mgr.24368) 1073 : cluster 0 pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:12.999669+0000 osd.3 (osd.3) 701 : cluster 0 2.1e scrub starts 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.002039+0000 osd.3 (osd.3) 702 : cluster 0 2.1e scrub ok 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.392850+0000 osd.4 (osd.4) 671 : cluster 0 5.11 scrub starts 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.395115+0000 osd.4 (osd.4) 672 : cluster 0 5.11 scrub ok 2024-04-06T03:50:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.444342+0000 osd.2 (osd.2) 579 : cluster 0 2.c deep-scrub starts 2024-04-06T03:50:13.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.446864+0000 osd.2 (osd.2) 580 : cluster 0 2.c deep-scrub ok 2024-04-06T03:50:13.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.566217+0000 osd.1 (osd.1) 463 : cluster 0 2.15 scrub starts 2024-04-06T03:50:13.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.568522+0000 osd.1 (osd.1) 464 : cluster 0 2.15 scrub ok 2024-04-06T03:50:13.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.646865+0000 mgr.y (mgr.24368) 1073 : cluster 0 pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:13.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:12.999669+0000 osd.3 (osd.3) 701 : cluster 0 2.1e scrub starts 2024-04-06T03:50:13.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:13 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.002039+0000 osd.3 (osd.3) 702 : cluster 0 2.1e scrub ok 2024-04-06T03:50:13.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.392850+0000 osd.4 (osd.4) 671 : cluster 0 5.11 scrub starts 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.395115+0000 osd.4 (osd.4) 672 : cluster 0 5.11 scrub ok 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.444342+0000 osd.2 (osd.2) 579 : cluster 0 2.c deep-scrub starts 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.446864+0000 osd.2 (osd.2) 580 : cluster 0 2.c deep-scrub ok 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.566217+0000 osd.1 (osd.1) 463 : cluster 0 2.15 scrub starts 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.568522+0000 osd.1 (osd.1) 464 : cluster 0 2.15 scrub ok 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.646865+0000 mgr.y (mgr.24368) 1073 : cluster 0 pgmap v1179: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:12.999669+0000 osd.3 (osd.3) 701 : cluster 0 2.1e scrub starts 2024-04-06T03:50:13.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:13 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.002039+0000 osd.3 (osd.3) 702 : cluster 0 2.1e scrub ok 2024-04-06T03:50:13.851 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:50:14.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.177165+0000 osd.0 (osd.0) 676 : cluster 0 6.0 scrub starts 2024-04-06T03:50:14.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.259824+0000 osd.0 (osd.0) 677 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.259945+0000 osd.0 (osd.0) 678 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.276687+0000 osd.0 (osd.0) 679 : cluster 0 6.0 scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.392399+0000 osd.4 (osd.4) 673 : cluster 0 2.11 scrub starts 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.394819+0000 osd.4 (osd.4) 674 : cluster 0 2.11 scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.608705+0000 osd.1 (osd.1) 465 : cluster 0 2.9 scrub starts 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[21467]: cluster 2024-04-06T03:50:13.611296+0000 osd.1 (osd.1) 466 : cluster 0 2.9 scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.177165+0000 osd.0 (osd.0) 676 : cluster 0 6.0 scrub starts 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.259824+0000 osd.0 (osd.0) 677 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.259945+0000 osd.0 (osd.0) 678 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.276687+0000 osd.0 (osd.0) 679 : cluster 0 6.0 scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.392399+0000 osd.4 (osd.4) 673 : cluster 0 2.11 scrub starts 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.394819+0000 osd.4 (osd.4) 674 : cluster 0 2.11 scrub ok 2024-04-06T03:50:14.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.608705+0000 osd.1 (osd.1) 465 : cluster 0 2.9 scrub starts 2024-04-06T03:50:14.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:14 smithi050 bash[29221]: cluster 2024-04-06T03:50:13.611296+0000 osd.1 (osd.1) 466 : cluster 0 2.9 scrub ok 2024-04-06T03:50:14.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.177165+0000 osd.0 (osd.0) 676 : cluster 0 6.0 scrub starts 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.259824+0000 osd.0 (osd.0) 677 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.259945+0000 osd.0 (osd.0) 678 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.276687+0000 osd.0 (osd.0) 679 : cluster 0 6.0 scrub ok 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.392399+0000 osd.4 (osd.4) 673 : cluster 0 2.11 scrub starts 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.394819+0000 osd.4 (osd.4) 674 : cluster 0 2.11 scrub ok 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.608705+0000 osd.1 (osd.1) 465 : cluster 0 2.9 scrub starts 2024-04-06T03:50:14.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:14 smithi153 bash[23852]: cluster 2024-04-06T03:50:13.611296+0000 osd.1 (osd.1) 466 : cluster 0 2.9 scrub ok 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[29221]: cluster 2024-04-06T03:50:14.232988+0000 mon.a (mon.0) 1881 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[29221]: cluster 2024-04-06T03:50:14.406595+0000 osd.7 (osd.7) 699 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[29221]: cluster 2024-04-06T03:50:14.406740+0000 osd.7 (osd.7) 700 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[29221]: cluster 2024-04-06T03:50:14.625064+0000 osd.1 (osd.1) 467 : cluster 0 3.0 scrub starts 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[29221]: cluster 2024-04-06T03:50:14.634824+0000 osd.1 (osd.1) 468 : cluster 0 3.0 scrub ok 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[29221]: cluster 2024-04-06T03:50:14.647742+0000 mgr.y (mgr.24368) 1074 : cluster 0 pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[21467]: cluster 2024-04-06T03:50:14.232988+0000 mon.a (mon.0) 1881 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[21467]: cluster 2024-04-06T03:50:14.406595+0000 osd.7 (osd.7) 699 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:15.413 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[21467]: cluster 2024-04-06T03:50:14.406740+0000 osd.7 (osd.7) 700 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:15.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[21467]: cluster 2024-04-06T03:50:14.625064+0000 osd.1 (osd.1) 467 : cluster 0 3.0 scrub starts 2024-04-06T03:50:15.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[21467]: cluster 2024-04-06T03:50:14.634824+0000 osd.1 (osd.1) 468 : cluster 0 3.0 scrub ok 2024-04-06T03:50:15.414 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:15 smithi050 bash[21467]: cluster 2024-04-06T03:50:14.647742+0000 mgr.y (mgr.24368) 1074 : cluster 0 pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:15 smithi153 bash[23852]: cluster 2024-04-06T03:50:14.232988+0000 mon.a (mon.0) 1881 : cluster 0 osdmap e423: 8 total, 8 up, 8 in 2024-04-06T03:50:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:15 smithi153 bash[23852]: cluster 2024-04-06T03:50:14.406595+0000 osd.7 (osd.7) 699 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:15 smithi153 bash[23852]: cluster 2024-04-06T03:50:14.406740+0000 osd.7 (osd.7) 700 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:15 smithi153 bash[23852]: cluster 2024-04-06T03:50:14.625064+0000 osd.1 (osd.1) 467 : cluster 0 3.0 scrub starts 2024-04-06T03:50:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:15 smithi153 bash[23852]: cluster 2024-04-06T03:50:14.634824+0000 osd.1 (osd.1) 468 : cluster 0 3.0 scrub ok 2024-04-06T03:50:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:15 smithi153 bash[23852]: cluster 2024-04-06T03:50:14.647742+0000 mgr.y (mgr.24368) 1074 : cluster 0 pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:16 smithi050 bash[29221]: cluster 2024-04-06T03:50:15.238464+0000 mon.a (mon.0) 1882 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-04-06T03:50:16.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:16 smithi050 bash[21467]: cluster 2024-04-06T03:50:15.238464+0000 mon.a (mon.0) 1882 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-04-06T03:50:16.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:16 smithi153 bash[23852]: cluster 2024-04-06T03:50:15.238464+0000 mon.a (mon.0) 1882 : cluster 0 osdmap e424: 8 total, 8 up, 8 in 2024-04-06T03:50:17.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:17 smithi153 bash[23852]: cluster 2024-04-06T03:50:16.502005+0000 osd.7 (osd.7) 701 : cluster 0 5.1a scrub starts 2024-04-06T03:50:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:17 smithi153 bash[23852]: cluster 2024-04-06T03:50:16.504204+0000 osd.7 (osd.7) 702 : cluster 0 5.1a scrub ok 2024-04-06T03:50:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:17 smithi153 bash[23852]: cluster 2024-04-06T03:50:16.621167+0000 osd.1 (osd.1) 469 : cluster 0 5.15 scrub starts 2024-04-06T03:50:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:17 smithi153 bash[23852]: cluster 2024-04-06T03:50:16.624150+0000 osd.1 (osd.1) 470 : cluster 0 5.15 scrub ok 2024-04-06T03:50:17.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:17 smithi153 bash[23852]: cluster 2024-04-06T03:50:16.648645+0000 mgr.y (mgr.24368) 1075 : cluster 0 pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:17.678 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:17.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[21467]: cluster 2024-04-06T03:50:16.502005+0000 osd.7 (osd.7) 701 : cluster 0 5.1a scrub starts 2024-04-06T03:50:17.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[21467]: cluster 2024-04-06T03:50:16.504204+0000 osd.7 (osd.7) 702 : cluster 0 5.1a scrub ok 2024-04-06T03:50:17.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[21467]: cluster 2024-04-06T03:50:16.621167+0000 osd.1 (osd.1) 469 : cluster 0 5.15 scrub starts 2024-04-06T03:50:17.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[21467]: cluster 2024-04-06T03:50:16.624150+0000 osd.1 (osd.1) 470 : cluster 0 5.15 scrub ok 2024-04-06T03:50:17.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[21467]: cluster 2024-04-06T03:50:16.648645+0000 mgr.y (mgr.24368) 1075 : cluster 0 pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:17.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[29221]: cluster 2024-04-06T03:50:16.502005+0000 osd.7 (osd.7) 701 : cluster 0 5.1a scrub starts 2024-04-06T03:50:17.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[29221]: cluster 2024-04-06T03:50:16.504204+0000 osd.7 (osd.7) 702 : cluster 0 5.1a scrub ok 2024-04-06T03:50:17.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[29221]: cluster 2024-04-06T03:50:16.621167+0000 osd.1 (osd.1) 469 : cluster 0 5.15 scrub starts 2024-04-06T03:50:17.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[29221]: cluster 2024-04-06T03:50:16.624150+0000 osd.1 (osd.1) 470 : cluster 0 5.15 scrub ok 2024-04-06T03:50:17.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:17 smithi050 bash[29221]: cluster 2024-04-06T03:50:16.648645+0000 mgr.y (mgr.24368) 1075 : cluster 0 pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:18.266 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-06T03:50:18.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:18 smithi153 bash[23852]: audit 2024-04-06T03:50:17.331481+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.50:0/2358896262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:18.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:18 smithi153 bash[23852]: audit 2024-04-06T03:50:17.332168+0000 mon.a (mon.0) 1883 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:18.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:18 smithi050 bash[21467]: audit 2024-04-06T03:50:17.331481+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.50:0/2358896262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:18.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:18 smithi050 bash[21467]: audit 2024-04-06T03:50:17.332168+0000 mon.a (mon.0) 1883 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:18.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:18 smithi050 bash[29221]: audit 2024-04-06T03:50:17.331481+0000 mon.c (mon.2) 129 : audit 1 from='client.? 172.21.15.50:0/2358896262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:18.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:18 smithi050 bash[29221]: audit 2024-04-06T03:50:17.332168+0000 mon.a (mon.0) 1883 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:19.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 4 1 2024-04-06T03:50:19.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:19 smithi153 bash[23852]: audit 2024-04-06T03:50:18.254965+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:50:19.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:19 smithi153 bash[23852]: cluster 2024-04-06T03:50:18.261915+0000 mon.a (mon.0) 1885 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-04-06T03:50:19.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:19 smithi153 bash[23852]: cluster 2024-04-06T03:50:18.301896+0000 osd.0 (osd.0) 680 : cluster 0 3.1 scrub starts 2024-04-06T03:50:19.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:19 smithi153 bash[23852]: cluster 2024-04-06T03:50:18.307568+0000 osd.0 (osd.0) 681 : cluster 0 3.1 scrub ok 2024-04-06T03:50:19.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:19 smithi153 bash[23852]: cluster 2024-04-06T03:50:18.649603+0000 mgr.y (mgr.24368) 1076 : cluster 0 pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:50:19.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:50:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:50:19.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[21467]: audit 2024-04-06T03:50:18.254965+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[21467]: cluster 2024-04-06T03:50:18.261915+0000 mon.a (mon.0) 1885 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[21467]: cluster 2024-04-06T03:50:18.301896+0000 osd.0 (osd.0) 680 : cluster 0 3.1 scrub starts 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[21467]: cluster 2024-04-06T03:50:18.307568+0000 osd.0 (osd.0) 681 : cluster 0 3.1 scrub ok 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[21467]: cluster 2024-04-06T03:50:18.649603+0000 mgr.y (mgr.24368) 1076 : cluster 0 pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[29221]: audit 2024-04-06T03:50:18.254965+0000 mon.a (mon.0) 1884 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[29221]: cluster 2024-04-06T03:50:18.261915+0000 mon.a (mon.0) 1885 : cluster 0 osdmap e425: 8 total, 8 up, 8 in 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[29221]: cluster 2024-04-06T03:50:18.301896+0000 osd.0 (osd.0) 680 : cluster 0 3.1 scrub starts 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[29221]: cluster 2024-04-06T03:50:18.307568+0000 osd.0 (osd.0) 681 : cluster 0 3.1 scrub ok 2024-04-06T03:50:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:19 smithi050 bash[29221]: cluster 2024-04-06T03:50:18.649603+0000 mgr.y (mgr.24368) 1076 : cluster 0 pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:50:20.547 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:50:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:50:20.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:20 smithi153 bash[23852]: cluster 2024-04-06T03:50:19.275579+0000 osd.0 (osd.0) 682 : cluster 0 3.17 scrub starts 2024-04-06T03:50:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:20 smithi153 bash[23852]: cluster 2024-04-06T03:50:19.277846+0000 osd.0 (osd.0) 683 : cluster 0 3.17 scrub ok 2024-04-06T03:50:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:20 smithi153 bash[23852]: cluster 2024-04-06T03:50:20.076246+0000 osd.3 (osd.3) 703 : cluster 0 5.14 deep-scrub starts 2024-04-06T03:50:20.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:20 smithi153 bash[23852]: cluster 2024-04-06T03:50:20.079516+0000 osd.3 (osd.3) 704 : cluster 0 5.14 deep-scrub ok 2024-04-06T03:50:20.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[29221]: cluster 2024-04-06T03:50:19.275579+0000 osd.0 (osd.0) 682 : cluster 0 3.17 scrub starts 2024-04-06T03:50:20.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[29221]: cluster 2024-04-06T03:50:19.277846+0000 osd.0 (osd.0) 683 : cluster 0 3.17 scrub ok 2024-04-06T03:50:20.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[29221]: cluster 2024-04-06T03:50:20.076246+0000 osd.3 (osd.3) 703 : cluster 0 5.14 deep-scrub starts 2024-04-06T03:50:20.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[29221]: cluster 2024-04-06T03:50:20.079516+0000 osd.3 (osd.3) 704 : cluster 0 5.14 deep-scrub ok 2024-04-06T03:50:20.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[21467]: cluster 2024-04-06T03:50:19.275579+0000 osd.0 (osd.0) 682 : cluster 0 3.17 scrub starts 2024-04-06T03:50:20.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[21467]: cluster 2024-04-06T03:50:19.277846+0000 osd.0 (osd.0) 683 : cluster 0 3.17 scrub ok 2024-04-06T03:50:20.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[21467]: cluster 2024-04-06T03:50:20.076246+0000 osd.3 (osd.3) 703 : cluster 0 5.14 deep-scrub starts 2024-04-06T03:50:20.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:20 smithi050 bash[21467]: cluster 2024-04-06T03:50:20.079516+0000 osd.3 (osd.3) 704 : cluster 0 5.14 deep-scrub ok 2024-04-06T03:50:20.757 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:20.757 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:50:20.757 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:50:20.757 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: audit 2024-04-06T03:50:20.484172+0000 mgr.y (mgr.24368) 1077 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: cluster 2024-04-06T03:50:20.573378+0000 osd.7 (osd.7) 703 : cluster 0 6.8 deep-scrub starts 2024-04-06T03:50:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: cluster 2024-04-06T03:50:20.575358+0000 osd.7 (osd.7) 704 : cluster 0 6.8 deep-scrub ok 2024-04-06T03:50:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: cluster 2024-04-06T03:50:20.650828+0000 mgr.y (mgr.24368) 1078 : cluster 0 pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-04-06T03:50:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: audit 2024-04-06T03:50:20.968305+0000 mon.a (mon.0) 1886 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: cluster 2024-04-06T03:50:21.059782+0000 osd.3 (osd.3) 705 : cluster 0 3.7 scrub starts 2024-04-06T03:50:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:21 smithi153 bash[23852]: cluster 2024-04-06T03:50:21.067082+0000 osd.3 (osd.3) 706 : cluster 0 3.7 scrub ok 2024-04-06T03:50:21.567 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_blocked_ops 2024-04-06T03:50:21.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: audit 2024-04-06T03:50:20.484172+0000 mgr.y (mgr.24368) 1077 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:21.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: cluster 2024-04-06T03:50:20.573378+0000 osd.7 (osd.7) 703 : cluster 0 6.8 deep-scrub starts 2024-04-06T03:50:21.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: cluster 2024-04-06T03:50:20.575358+0000 osd.7 (osd.7) 704 : cluster 0 6.8 deep-scrub ok 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: cluster 2024-04-06T03:50:20.650828+0000 mgr.y (mgr.24368) 1078 : cluster 0 pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: audit 2024-04-06T03:50:20.968305+0000 mon.a (mon.0) 1886 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: cluster 2024-04-06T03:50:21.059782+0000 osd.3 (osd.3) 705 : cluster 0 3.7 scrub starts 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[21467]: cluster 2024-04-06T03:50:21.067082+0000 osd.3 (osd.3) 706 : cluster 0 3.7 scrub ok 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: audit 2024-04-06T03:50:20.484172+0000 mgr.y (mgr.24368) 1077 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: cluster 2024-04-06T03:50:20.573378+0000 osd.7 (osd.7) 703 : cluster 0 6.8 deep-scrub starts 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: cluster 2024-04-06T03:50:20.575358+0000 osd.7 (osd.7) 704 : cluster 0 6.8 deep-scrub ok 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: cluster 2024-04-06T03:50:20.650828+0000 mgr.y (mgr.24368) 1078 : cluster 0 pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: audit 2024-04-06T03:50:20.968305+0000 mon.a (mon.0) 1886 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: cluster 2024-04-06T03:50:21.059782+0000 osd.3 (osd.3) 705 : cluster 0 3.7 scrub starts 2024-04-06T03:50:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:21 smithi050 bash[29221]: cluster 2024-04-06T03:50:21.067082+0000 osd.3 (osd.3) 706 : cluster 0 3.7 scrub ok 2024-04-06T03:50:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:22 smithi153 bash[23852]: cluster 2024-04-06T03:50:21.198264+0000 osd.0 (osd.0) 684 : cluster 0 3.10 scrub starts 2024-04-06T03:50:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:22 smithi153 bash[23852]: cluster 2024-04-06T03:50:21.201926+0000 osd.0 (osd.0) 685 : cluster 0 3.10 scrub ok 2024-04-06T03:50:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:22 smithi153 bash[23852]: cluster 2024-04-06T03:50:21.595175+0000 osd.1 (osd.1) 471 : cluster 0 6.1 scrub starts 2024-04-06T03:50:22.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:22 smithi153 bash[23852]: cluster 2024-04-06T03:50:21.676143+0000 osd.1 (osd.1) 472 : cluster 0 6.1 scrub ok 2024-04-06T03:50:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[21467]: cluster 2024-04-06T03:50:21.198264+0000 osd.0 (osd.0) 684 : cluster 0 3.10 scrub starts 2024-04-06T03:50:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[21467]: cluster 2024-04-06T03:50:21.201926+0000 osd.0 (osd.0) 685 : cluster 0 3.10 scrub ok 2024-04-06T03:50:22.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[21467]: cluster 2024-04-06T03:50:21.595175+0000 osd.1 (osd.1) 471 : cluster 0 6.1 scrub starts 2024-04-06T03:50:22.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[21467]: cluster 2024-04-06T03:50:21.676143+0000 osd.1 (osd.1) 472 : cluster 0 6.1 scrub ok 2024-04-06T03:50:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[29221]: cluster 2024-04-06T03:50:21.198264+0000 osd.0 (osd.0) 684 : cluster 0 3.10 scrub starts 2024-04-06T03:50:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[29221]: cluster 2024-04-06T03:50:21.201926+0000 osd.0 (osd.0) 685 : cluster 0 3.10 scrub ok 2024-04-06T03:50:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[29221]: cluster 2024-04-06T03:50:21.595175+0000 osd.1 (osd.1) 471 : cluster 0 6.1 scrub starts 2024-04-06T03:50:22.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:22 smithi050 bash[29221]: cluster 2024-04-06T03:50:21.676143+0000 osd.1 (osd.1) 472 : cluster 0 6.1 scrub ok 2024-04-06T03:50:23.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:23 smithi153 bash[23852]: cluster 2024-04-06T03:50:22.451130+0000 osd.4 (osd.4) 675 : cluster 0 5.1f deep-scrub starts 2024-04-06T03:50:23.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:23 smithi153 bash[23852]: cluster 2024-04-06T03:50:22.454497+0000 osd.4 (osd.4) 676 : cluster 0 5.1f deep-scrub ok 2024-04-06T03:50:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:23 smithi153 bash[23852]: cluster 2024-04-06T03:50:22.487806+0000 osd.2 (osd.2) 581 : cluster 0 2.3 scrub starts 2024-04-06T03:50:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:23 smithi153 bash[23852]: cluster 2024-04-06T03:50:22.490371+0000 osd.2 (osd.2) 582 : cluster 0 2.3 scrub ok 2024-04-06T03:50:23.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:23 smithi153 bash[23852]: cluster 2024-04-06T03:50:22.651723+0000 mgr.y (mgr.24368) 1079 : cluster 0 pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:23.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[29221]: cluster 2024-04-06T03:50:22.451130+0000 osd.4 (osd.4) 675 : cluster 0 5.1f deep-scrub starts 2024-04-06T03:50:23.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[29221]: cluster 2024-04-06T03:50:22.454497+0000 osd.4 (osd.4) 676 : cluster 0 5.1f deep-scrub ok 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[29221]: cluster 2024-04-06T03:50:22.487806+0000 osd.2 (osd.2) 581 : cluster 0 2.3 scrub starts 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[29221]: cluster 2024-04-06T03:50:22.490371+0000 osd.2 (osd.2) 582 : cluster 0 2.3 scrub ok 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[29221]: cluster 2024-04-06T03:50:22.651723+0000 mgr.y (mgr.24368) 1079 : cluster 0 pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[21467]: cluster 2024-04-06T03:50:22.451130+0000 osd.4 (osd.4) 675 : cluster 0 5.1f deep-scrub starts 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[21467]: cluster 2024-04-06T03:50:22.454497+0000 osd.4 (osd.4) 676 : cluster 0 5.1f deep-scrub ok 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[21467]: cluster 2024-04-06T03:50:22.487806+0000 osd.2 (osd.2) 581 : cluster 0 2.3 scrub starts 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[21467]: cluster 2024-04-06T03:50:22.490371+0000 osd.2 (osd.2) 582 : cluster 0 2.3 scrub ok 2024-04-06T03:50:23.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:23 smithi050 bash[21467]: cluster 2024-04-06T03:50:22.651723+0000 mgr.y (mgr.24368) 1079 : cluster 0 pgmap v1187: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:50:24.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:24 smithi153 bash[23852]: cluster 2024-04-06T03:50:23.500036+0000 osd.2 (osd.2) 583 : cluster 0 5.b scrub starts 2024-04-06T03:50:24.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:24 smithi153 bash[23852]: cluster 2024-04-06T03:50:23.502723+0000 osd.2 (osd.2) 584 : cluster 0 5.b scrub ok 2024-04-06T03:50:24.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:24 smithi153 bash[23852]: cluster 2024-04-06T03:50:23.526495+0000 osd.7 (osd.7) 705 : cluster 0 5.4 scrub starts 2024-04-06T03:50:24.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:24 smithi153 bash[23852]: cluster 2024-04-06T03:50:23.528876+0000 osd.7 (osd.7) 706 : cluster 0 5.4 scrub ok 2024-04-06T03:50:24.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[21467]: cluster 2024-04-06T03:50:23.500036+0000 osd.2 (osd.2) 583 : cluster 0 5.b scrub starts 2024-04-06T03:50:24.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[21467]: cluster 2024-04-06T03:50:23.502723+0000 osd.2 (osd.2) 584 : cluster 0 5.b scrub ok 2024-04-06T03:50:24.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[21467]: cluster 2024-04-06T03:50:23.526495+0000 osd.7 (osd.7) 705 : cluster 0 5.4 scrub starts 2024-04-06T03:50:24.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[21467]: cluster 2024-04-06T03:50:23.528876+0000 osd.7 (osd.7) 706 : cluster 0 5.4 scrub ok 2024-04-06T03:50:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[29221]: cluster 2024-04-06T03:50:23.500036+0000 osd.2 (osd.2) 583 : cluster 0 5.b scrub starts 2024-04-06T03:50:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[29221]: cluster 2024-04-06T03:50:23.502723+0000 osd.2 (osd.2) 584 : cluster 0 5.b scrub ok 2024-04-06T03:50:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[29221]: cluster 2024-04-06T03:50:23.526495+0000 osd.7 (osd.7) 705 : cluster 0 5.4 scrub starts 2024-04-06T03:50:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:24 smithi050 bash[29221]: cluster 2024-04-06T03:50:23.528876+0000 osd.7 (osd.7) 706 : cluster 0 5.4 scrub ok 2024-04-06T03:50:25.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[21467]: cluster 2024-04-06T03:50:24.134464+0000 osd.3 (osd.3) 707 : cluster 0 6.6 scrub starts 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[21467]: cluster 2024-04-06T03:50:24.177612+0000 osd.3 (osd.3) 708 : cluster 0 6.6 scrub ok 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[21467]: cluster 2024-04-06T03:50:24.231559+0000 osd.0 (osd.0) 686 : cluster 0 3.1b scrub starts 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[21467]: cluster 2024-04-06T03:50:24.235666+0000 osd.0 (osd.0) 687 : cluster 0 3.1b scrub ok 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[21467]: cluster 2024-04-06T03:50:24.652803+0000 mgr.y (mgr.24368) 1080 : cluster 0 pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[29221]: cluster 2024-04-06T03:50:24.134464+0000 osd.3 (osd.3) 707 : cluster 0 6.6 scrub starts 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[29221]: cluster 2024-04-06T03:50:24.177612+0000 osd.3 (osd.3) 708 : cluster 0 6.6 scrub ok 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[29221]: cluster 2024-04-06T03:50:24.231559+0000 osd.0 (osd.0) 686 : cluster 0 3.1b scrub starts 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[29221]: cluster 2024-04-06T03:50:24.235666+0000 osd.0 (osd.0) 687 : cluster 0 3.1b scrub ok 2024-04-06T03:50:25.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:25 smithi050 bash[29221]: cluster 2024-04-06T03:50:24.652803+0000 mgr.y (mgr.24368) 1080 : cluster 0 pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:50:25.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:25 smithi153 bash[23852]: cluster 2024-04-06T03:50:24.134464+0000 osd.3 (osd.3) 707 : cluster 0 6.6 scrub starts 2024-04-06T03:50:25.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:25 smithi153 bash[23852]: cluster 2024-04-06T03:50:24.177612+0000 osd.3 (osd.3) 708 : cluster 0 6.6 scrub ok 2024-04-06T03:50:25.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:25 smithi153 bash[23852]: cluster 2024-04-06T03:50:24.231559+0000 osd.0 (osd.0) 686 : cluster 0 3.1b scrub starts 2024-04-06T03:50:25.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:25 smithi153 bash[23852]: cluster 2024-04-06T03:50:24.235666+0000 osd.0 (osd.0) 687 : cluster 0 3.1b scrub ok 2024-04-06T03:50:25.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:25 smithi153 bash[23852]: cluster 2024-04-06T03:50:24.652803+0000 mgr.y (mgr.24368) 1080 : cluster 0 pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:50:26.360 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:26.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:26 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.478889+0000 osd.2 (osd.2) 585 : cluster 0 2.10 scrub starts 2024-04-06T03:50:26.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:26 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.481308+0000 osd.2 (osd.2) 586 : cluster 0 2.10 scrub ok 2024-04-06T03:50:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:26 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.533451+0000 osd.7 (osd.7) 707 : cluster 0 2.b scrub starts 2024-04-06T03:50:26.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:26 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.535742+0000 osd.7 (osd.7) 708 : cluster 0 2.b scrub ok 2024-04-06T03:50:26.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.478889+0000 osd.2 (osd.2) 585 : cluster 0 2.10 scrub starts 2024-04-06T03:50:26.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.481308+0000 osd.2 (osd.2) 586 : cluster 0 2.10 scrub ok 2024-04-06T03:50:26.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.533451+0000 osd.7 (osd.7) 707 : cluster 0 2.b scrub starts 2024-04-06T03:50:26.900 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.535742+0000 osd.7 (osd.7) 708 : cluster 0 2.b scrub ok 2024-04-06T03:50:26.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.478889+0000 osd.2 (osd.2) 585 : cluster 0 2.10 scrub starts 2024-04-06T03:50:26.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.481308+0000 osd.2 (osd.2) 586 : cluster 0 2.10 scrub ok 2024-04-06T03:50:26.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.533451+0000 osd.7 (osd.7) 707 : cluster 0 2.b scrub starts 2024-04-06T03:50:26.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:26 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.535742+0000 osd.7 (osd.7) 708 : cluster 0 2.b scrub ok 2024-04-06T03:50:27.642 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-06T03:50:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:27 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.546090+0000 osd.4 (osd.4) 677 : cluster 0 2.1a scrub starts 2024-04-06T03:50:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:27 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.548762+0000 osd.4 (osd.4) 678 : cluster 0 2.1a scrub ok 2024-04-06T03:50:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:27 smithi153 bash[23852]: cluster 2024-04-06T03:50:26.653936+0000 mgr.y (mgr.24368) 1081 : cluster 0 pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:27 smithi153 bash[23852]: audit 2024-04-06T03:50:27.295186+0000 mon.c (mon.2) 130 : audit 1 from='client.? 172.21.15.50:0/791817309' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:27.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:27 smithi153 bash[23852]: audit 2024-04-06T03:50:27.295975+0000 mon.a (mon.0) 1887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:27.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.546090+0000 osd.4 (osd.4) 677 : cluster 0 2.1a scrub starts 2024-04-06T03:50:27.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.548762+0000 osd.4 (osd.4) 678 : cluster 0 2.1a scrub ok 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[21467]: cluster 2024-04-06T03:50:26.653936+0000 mgr.y (mgr.24368) 1081 : cluster 0 pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[21467]: audit 2024-04-06T03:50:27.295186+0000 mon.c (mon.2) 130 : audit 1 from='client.? 172.21.15.50:0/791817309' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[21467]: audit 2024-04-06T03:50:27.295975+0000 mon.a (mon.0) 1887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.546090+0000 osd.4 (osd.4) 677 : cluster 0 2.1a scrub starts 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.548762+0000 osd.4 (osd.4) 678 : cluster 0 2.1a scrub ok 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[29221]: cluster 2024-04-06T03:50:26.653936+0000 mgr.y (mgr.24368) 1081 : cluster 0 pgmap v1189: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[29221]: audit 2024-04-06T03:50:27.295186+0000 mon.c (mon.2) 130 : audit 1 from='client.? 172.21.15.50:0/791817309' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:27.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:27 smithi050 bash[29221]: audit 2024-04-06T03:50:27.295975+0000 mon.a (mon.0) 1887 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-06T03:50:28.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 5 1 2024-04-06T03:50:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:27.567967+0000 osd.4 (osd.4) 679 : cluster 0 5.f deep-scrub starts 2024-04-06T03:50:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:27.569059+0000 osd.4 (osd.4) 680 : cluster 0 5.f deep-scrub ok 2024-04-06T03:50:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: audit 2024-04-06T03:50:27.628993+0000 mon.a (mon.0) 1888 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:27.635749+0000 mon.a (mon.0) 1889 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.002748+0000 osd.2 (osd.2) 587 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.002860+0000 osd.2 (osd.2) 588 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.262724+0000 osd.0 (osd.0) 688 : cluster 0 2.13 scrub starts 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.265262+0000 osd.0 (osd.0) 689 : cluster 0 2.13 scrub ok 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.481729+0000 osd.7 (osd.7) 709 : cluster 0 2.5 scrub starts 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.484495+0000 osd.7 (osd.7) 710 : cluster 0 2.5 scrub ok 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:27.567967+0000 osd.4 (osd.4) 679 : cluster 0 5.f deep-scrub starts 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:27.569059+0000 osd.4 (osd.4) 680 : cluster 0 5.f deep-scrub ok 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: audit 2024-04-06T03:50:27.628993+0000 mon.a (mon.0) 1888 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:27.635749+0000 mon.a (mon.0) 1889 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-04-06T03:50:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.002748+0000 osd.2 (osd.2) 587 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:28.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.002860+0000 osd.2 (osd.2) 588 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:28.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.262724+0000 osd.0 (osd.0) 688 : cluster 0 2.13 scrub starts 2024-04-06T03:50:28.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.265262+0000 osd.0 (osd.0) 689 : cluster 0 2.13 scrub ok 2024-04-06T03:50:28.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.481729+0000 osd.7 (osd.7) 709 : cluster 0 2.5 scrub starts 2024-04-06T03:50:28.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:28 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.484495+0000 osd.7 (osd.7) 710 : cluster 0 2.5 scrub ok 2024-04-06T03:50:28.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:27.567967+0000 osd.4 (osd.4) 679 : cluster 0 5.f deep-scrub starts 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:27.569059+0000 osd.4 (osd.4) 680 : cluster 0 5.f deep-scrub ok 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: audit 2024-04-06T03:50:27.628993+0000 mon.a (mon.0) 1888 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:27.635749+0000 mon.a (mon.0) 1889 : cluster 0 osdmap e426: 8 total, 8 up, 8 in 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.002748+0000 osd.2 (osd.2) 587 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.002860+0000 osd.2 (osd.2) 588 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.262724+0000 osd.0 (osd.0) 688 : cluster 0 2.13 scrub starts 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.265262+0000 osd.0 (osd.0) 689 : cluster 0 2.13 scrub ok 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.481729+0000 osd.7 (osd.7) 709 : cluster 0 2.5 scrub starts 2024-04-06T03:50:28.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:28 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.484495+0000 osd.7 (osd.7) 710 : cluster 0 2.5 scrub ok 2024-04-06T03:50:29.460 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:29.460 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:50:29.460 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:50:29.460 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:50:29.460 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:29.659 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:50:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:50:29.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:29 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.542648+0000 osd.4 (osd.4) 681 : cluster 0 6.2 scrub starts 2024-04-06T03:50:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:29 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.545029+0000 osd.4 (osd.4) 682 : cluster 0 6.2 scrub ok 2024-04-06T03:50:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:29 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.640056+0000 osd.1 (osd.1) 473 : cluster 0 5.19 deep-scrub starts 2024-04-06T03:50:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:29 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.642149+0000 osd.1 (osd.1) 474 : cluster 0 5.19 deep-scrub ok 2024-04-06T03:50:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:29 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.655848+0000 mgr.y (mgr.24368) 1082 : cluster 0 pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:29 smithi153 bash[23852]: cluster 2024-04-06T03:50:28.659793+0000 mon.a (mon.0) 1890 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-04-06T03:50:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.542648+0000 osd.4 (osd.4) 681 : cluster 0 6.2 scrub starts 2024-04-06T03:50:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.545029+0000 osd.4 (osd.4) 682 : cluster 0 6.2 scrub ok 2024-04-06T03:50:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.640056+0000 osd.1 (osd.1) 473 : cluster 0 5.19 deep-scrub starts 2024-04-06T03:50:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.642149+0000 osd.1 (osd.1) 474 : cluster 0 5.19 deep-scrub ok 2024-04-06T03:50:29.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.655848+0000 mgr.y (mgr.24368) 1082 : cluster 0 pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[29221]: cluster 2024-04-06T03:50:28.659793+0000 mon.a (mon.0) 1890 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.542648+0000 osd.4 (osd.4) 681 : cluster 0 6.2 scrub starts 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.545029+0000 osd.4 (osd.4) 682 : cluster 0 6.2 scrub ok 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.640056+0000 osd.1 (osd.1) 473 : cluster 0 5.19 deep-scrub starts 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.642149+0000 osd.1 (osd.1) 474 : cluster 0 5.19 deep-scrub ok 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.655848+0000 mgr.y (mgr.24368) 1082 : cluster 0 pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:29.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:29 smithi050 bash[21467]: cluster 2024-04-06T03:50:28.659793+0000 mon.a (mon.0) 1890 : cluster 0 osdmap e427: 8 total, 8 up, 8 in 2024-04-06T03:50:30.307 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.5 dump_historic_ops 2024-04-06T03:50:30.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:50:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:50:30.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:30 smithi153 bash[23852]: audit 2024-04-06T03:50:29.744777+0000 mon.a (mon.0) 1891 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:50:30.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:30 smithi153 bash[23852]: cluster 2024-04-06T03:50:30.411098+0000 osd.7 (osd.7) 711 : cluster 0 1.0 scrub starts 2024-04-06T03:50:30.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:30 smithi153 bash[23852]: cluster 2024-04-06T03:50:30.415109+0000 osd.7 (osd.7) 712 : cluster 0 1.0 scrub ok 2024-04-06T03:50:30.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:30 smithi050 bash[29221]: audit 2024-04-06T03:50:29.744777+0000 mon.a (mon.0) 1891 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:50:30.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:30 smithi050 bash[29221]: cluster 2024-04-06T03:50:30.411098+0000 osd.7 (osd.7) 711 : cluster 0 1.0 scrub starts 2024-04-06T03:50:30.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:30 smithi050 bash[29221]: cluster 2024-04-06T03:50:30.415109+0000 osd.7 (osd.7) 712 : cluster 0 1.0 scrub ok 2024-04-06T03:50:30.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:30 smithi050 bash[21467]: audit 2024-04-06T03:50:29.744777+0000 mon.a (mon.0) 1891 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:50:30.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:30 smithi050 bash[21467]: cluster 2024-04-06T03:50:30.411098+0000 osd.7 (osd.7) 711 : cluster 0 1.0 scrub starts 2024-04-06T03:50:30.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:30 smithi050 bash[21467]: cluster 2024-04-06T03:50:30.415109+0000 osd.7 (osd.7) 712 : cluster 0 1.0 scrub ok 2024-04-06T03:50:31.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:30.200273+0000 osd.3 (osd.3) 709 : cluster 0 6.b scrub starts 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:30.351996+0000 osd.3 (osd.3) 710 : cluster 0 6.b scrub ok 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: audit 2024-04-06T03:50:30.492001+0000 mgr.y (mgr.24368) 1083 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:30.657149+0000 mgr.y (mgr.24368) 1084 : cluster 0 pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:31.200745+0000 osd.0 (osd.0) 690 : cluster 0 6.16 scrub starts 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:31.280744+0000 osd.0 (osd.0) 691 : cluster 0 6.16 scrub ok 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:31.365087+0000 osd.2 (osd.2) 589 : cluster 0 2.16 scrub starts 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[29221]: cluster 2024-04-06T03:50:31.367638+0000 osd.2 (osd.2) 590 : cluster 0 2.16 scrub ok 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:30.200273+0000 osd.3 (osd.3) 709 : cluster 0 6.b scrub starts 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:30.351996+0000 osd.3 (osd.3) 710 : cluster 0 6.b scrub ok 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: audit 2024-04-06T03:50:30.492001+0000 mgr.y (mgr.24368) 1083 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:30.657149+0000 mgr.y (mgr.24368) 1084 : cluster 0 pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:31.200745+0000 osd.0 (osd.0) 690 : cluster 0 6.16 scrub starts 2024-04-06T03:50:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:31.280744+0000 osd.0 (osd.0) 691 : cluster 0 6.16 scrub ok 2024-04-06T03:50:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:31.365087+0000 osd.2 (osd.2) 589 : cluster 0 2.16 scrub starts 2024-04-06T03:50:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:31 smithi050 bash[21467]: cluster 2024-04-06T03:50:31.367638+0000 osd.2 (osd.2) 590 : cluster 0 2.16 scrub ok 2024-04-06T03:50:32.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:30.200273+0000 osd.3 (osd.3) 709 : cluster 0 6.b scrub starts 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:30.351996+0000 osd.3 (osd.3) 710 : cluster 0 6.b scrub ok 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: audit 2024-04-06T03:50:30.492001+0000 mgr.y (mgr.24368) 1083 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:30.657149+0000 mgr.y (mgr.24368) 1084 : cluster 0 pgmap v1193: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:31.200745+0000 osd.0 (osd.0) 690 : cluster 0 6.16 scrub starts 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:31.280744+0000 osd.0 (osd.0) 691 : cluster 0 6.16 scrub ok 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:31.365087+0000 osd.2 (osd.2) 589 : cluster 0 2.16 scrub starts 2024-04-06T03:50:32.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:31 smithi153 bash[23852]: cluster 2024-04-06T03:50:31.367638+0000 osd.2 (osd.2) 590 : cluster 0 2.16 scrub ok 2024-04-06T03:50:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[21467]: cluster 2024-04-06T03:50:31.220290+0000 osd.3 (osd.3) 711 : cluster 0 6.c scrub starts 2024-04-06T03:50:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[21467]: cluster 2024-04-06T03:50:31.223059+0000 osd.3 (osd.3) 712 : cluster 0 6.c scrub ok 2024-04-06T03:50:32.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[21467]: cluster 2024-04-06T03:50:32.442184+0000 osd.7 (osd.7) 713 : cluster 0 2.8 scrub starts 2024-04-06T03:50:32.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[21467]: cluster 2024-04-06T03:50:32.445405+0000 osd.7 (osd.7) 714 : cluster 0 2.8 scrub ok 2024-04-06T03:50:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[29221]: cluster 2024-04-06T03:50:31.220290+0000 osd.3 (osd.3) 711 : cluster 0 6.c scrub starts 2024-04-06T03:50:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[29221]: cluster 2024-04-06T03:50:31.223059+0000 osd.3 (osd.3) 712 : cluster 0 6.c scrub ok 2024-04-06T03:50:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[29221]: cluster 2024-04-06T03:50:32.442184+0000 osd.7 (osd.7) 713 : cluster 0 2.8 scrub starts 2024-04-06T03:50:32.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:32 smithi050 bash[29221]: cluster 2024-04-06T03:50:32.445405+0000 osd.7 (osd.7) 714 : cluster 0 2.8 scrub ok 2024-04-06T03:50:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:32 smithi153 bash[23852]: cluster 2024-04-06T03:50:31.220290+0000 osd.3 (osd.3) 711 : cluster 0 6.c scrub starts 2024-04-06T03:50:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:32 smithi153 bash[23852]: cluster 2024-04-06T03:50:31.223059+0000 osd.3 (osd.3) 712 : cluster 0 6.c scrub ok 2024-04-06T03:50:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:32 smithi153 bash[23852]: cluster 2024-04-06T03:50:32.442184+0000 osd.7 (osd.7) 713 : cluster 0 2.8 scrub starts 2024-04-06T03:50:33.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:32 smithi153 bash[23852]: cluster 2024-04-06T03:50:32.445405+0000 osd.7 (osd.7) 714 : cluster 0 2.8 scrub ok 2024-04-06T03:50:33.267 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:50:33.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:33 smithi050 bash[21467]: cluster 2024-04-06T03:50:32.658019+0000 mgr.y (mgr.24368) 1085 : cluster 0 pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:33.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:33 smithi050 bash[21467]: cluster 2024-04-06T03:50:33.136288+0000 osd.0 (osd.0) 692 : cluster 0 6.9 scrub starts 2024-04-06T03:50:33.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:33 smithi050 bash[21467]: cluster 2024-04-06T03:50:33.296954+0000 osd.0 (osd.0) 693 : cluster 0 6.9 scrub ok 2024-04-06T03:50:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:33 smithi050 bash[29221]: cluster 2024-04-06T03:50:32.658019+0000 mgr.y (mgr.24368) 1085 : cluster 0 pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:33 smithi050 bash[29221]: cluster 2024-04-06T03:50:33.136288+0000 osd.0 (osd.0) 692 : cluster 0 6.9 scrub starts 2024-04-06T03:50:33.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:33 smithi050 bash[29221]: cluster 2024-04-06T03:50:33.296954+0000 osd.0 (osd.0) 693 : cluster 0 6.9 scrub ok 2024-04-06T03:50:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:33 smithi153 bash[23852]: cluster 2024-04-06T03:50:32.658019+0000 mgr.y (mgr.24368) 1085 : cluster 0 pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:33 smithi153 bash[23852]: cluster 2024-04-06T03:50:33.136288+0000 osd.0 (osd.0) 692 : cluster 0 6.9 scrub starts 2024-04-06T03:50:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:33 smithi153 bash[23852]: cluster 2024-04-06T03:50:33.296954+0000 osd.0 (osd.0) 693 : cluster 0 6.9 scrub ok 2024-04-06T03:50:35.082 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.453961+0000 osd.4 (osd.4) 683 : cluster 0 2.18 scrub starts 2024-04-06T03:50:35.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.456606+0000 osd.4 (osd.4) 684 : cluster 0 2.18 scrub ok 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.473892+0000 osd.7 (osd.7) 715 : cluster 0 2.7 scrub starts 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.476242+0000 osd.7 (osd.7) 716 : cluster 0 2.7 scrub ok 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.658652+0000 mgr.y (mgr.24368) 1086 : cluster 0 pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.453961+0000 osd.4 (osd.4) 683 : cluster 0 2.18 scrub starts 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.456606+0000 osd.4 (osd.4) 684 : cluster 0 2.18 scrub ok 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.473892+0000 osd.7 (osd.7) 715 : cluster 0 2.7 scrub starts 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.476242+0000 osd.7 (osd.7) 716 : cluster 0 2.7 scrub ok 2024-04-06T03:50:35.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:35 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.658652+0000 mgr.y (mgr.24368) 1086 : cluster 0 pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:35 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.453961+0000 osd.4 (osd.4) 683 : cluster 0 2.18 scrub starts 2024-04-06T03:50:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:35 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.456606+0000 osd.4 (osd.4) 684 : cluster 0 2.18 scrub ok 2024-04-06T03:50:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:35 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.473892+0000 osd.7 (osd.7) 715 : cluster 0 2.7 scrub starts 2024-04-06T03:50:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:35 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.476242+0000 osd.7 (osd.7) 716 : cluster 0 2.7 scrub ok 2024-04-06T03:50:35.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:35 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.658652+0000 mgr.y (mgr.24368) 1086 : cluster 0 pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:50:36.394 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:36 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.227297+0000 osd.3 (osd.3) 713 : cluster 0 2.12 scrub starts 2024-04-06T03:50:36.394 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:36 smithi153 bash[23852]: cluster 2024-04-06T03:50:34.229759+0000 osd.3 (osd.3) 714 : cluster 0 2.12 scrub ok 2024-04-06T03:50:36.394 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:36 smithi153 bash[23852]: audit 2024-04-06T03:50:36.000666+0000 mon.a (mon.0) 1892 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:36.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:36 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.227297+0000 osd.3 (osd.3) 713 : cluster 0 2.12 scrub starts 2024-04-06T03:50:36.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:36 smithi050 bash[21467]: cluster 2024-04-06T03:50:34.229759+0000 osd.3 (osd.3) 714 : cluster 0 2.12 scrub ok 2024-04-06T03:50:36.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:36 smithi050 bash[21467]: audit 2024-04-06T03:50:36.000666+0000 mon.a (mon.0) 1892 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:36.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:36 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.227297+0000 osd.3 (osd.3) 713 : cluster 0 2.12 scrub starts 2024-04-06T03:50:36.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:36 smithi050 bash[29221]: cluster 2024-04-06T03:50:34.229759+0000 osd.3 (osd.3) 714 : cluster 0 2.12 scrub ok 2024-04-06T03:50:36.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:36 smithi050 bash[29221]: audit 2024-04-06T03:50:36.000666+0000 mon.a (mon.0) 1892 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:37.108 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-04-06T03:50:37.364 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:37 smithi153 bash[23852]: cluster 2024-04-06T03:50:36.487961+0000 osd.7 (osd.7) 717 : cluster 0 3.e deep-scrub starts 2024-04-06T03:50:37.364 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:37 smithi153 bash[23852]: cluster 2024-04-06T03:50:36.490184+0000 osd.7 (osd.7) 718 : cluster 0 3.e deep-scrub ok 2024-04-06T03:50:37.364 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:37 smithi153 bash[23852]: cluster 2024-04-06T03:50:36.659596+0000 mgr.y (mgr.24368) 1087 : cluster 0 pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:50:37.365 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:37 smithi153 bash[23852]: audit 2024-04-06T03:50:36.866938+0000 mon.c (mon.2) 131 : audit 1 from='client.? 172.21.15.50:0/1503359816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:37.365 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:37 smithi153 bash[23852]: audit 2024-04-06T03:50:36.867620+0000 mon.a (mon.0) 1893 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:37.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[29221]: cluster 2024-04-06T03:50:36.487961+0000 osd.7 (osd.7) 717 : cluster 0 3.e deep-scrub starts 2024-04-06T03:50:37.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[29221]: cluster 2024-04-06T03:50:36.490184+0000 osd.7 (osd.7) 718 : cluster 0 3.e deep-scrub ok 2024-04-06T03:50:37.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[29221]: cluster 2024-04-06T03:50:36.659596+0000 mgr.y (mgr.24368) 1087 : cluster 0 pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:50:37.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[29221]: audit 2024-04-06T03:50:36.866938+0000 mon.c (mon.2) 131 : audit 1 from='client.? 172.21.15.50:0/1503359816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[29221]: audit 2024-04-06T03:50:36.867620+0000 mon.a (mon.0) 1893 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[21467]: cluster 2024-04-06T03:50:36.487961+0000 osd.7 (osd.7) 717 : cluster 0 3.e deep-scrub starts 2024-04-06T03:50:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[21467]: cluster 2024-04-06T03:50:36.490184+0000 osd.7 (osd.7) 718 : cluster 0 3.e deep-scrub ok 2024-04-06T03:50:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[21467]: cluster 2024-04-06T03:50:36.659596+0000 mgr.y (mgr.24368) 1087 : cluster 0 pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:50:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[21467]: audit 2024-04-06T03:50:36.866938+0000 mon.c (mon.2) 131 : audit 1 from='client.? 172.21.15.50:0/1503359816' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:37 smithi050 bash[21467]: audit 2024-04-06T03:50:36.867620+0000 mon.a (mon.0) 1893 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:37.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 5 1 2024-04-06T03:50:38.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[21467]: cluster 2024-04-06T03:50:36.272451+0000 osd.3 (osd.3) 715 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:50:38.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[21467]: cluster 2024-04-06T03:50:36.438119+0000 osd.3 (osd.3) 716 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:50:38.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[21467]: audit 2024-04-06T03:50:37.096327+0000 mon.a (mon.0) 1894 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[21467]: cluster 2024-04-06T03:50:37.105658+0000 mon.a (mon.0) 1895 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[21467]: cluster 2024-04-06T03:50:37.429628+0000 osd.4 (osd.4) 685 : cluster 0 6.e deep-scrub starts 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[29221]: cluster 2024-04-06T03:50:36.272451+0000 osd.3 (osd.3) 715 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[29221]: cluster 2024-04-06T03:50:36.438119+0000 osd.3 (osd.3) 716 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[29221]: audit 2024-04-06T03:50:37.096327+0000 mon.a (mon.0) 1894 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[29221]: cluster 2024-04-06T03:50:37.105658+0000 mon.a (mon.0) 1895 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-04-06T03:50:38.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:38 smithi050 bash[29221]: cluster 2024-04-06T03:50:37.429628+0000 osd.4 (osd.4) 685 : cluster 0 6.e deep-scrub starts 2024-04-06T03:50:38.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:38 smithi153 bash[23852]: cluster 2024-04-06T03:50:36.272451+0000 osd.3 (osd.3) 715 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:50:38.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:38 smithi153 bash[23852]: cluster 2024-04-06T03:50:36.438119+0000 osd.3 (osd.3) 716 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:50:38.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:38 smithi153 bash[23852]: audit 2024-04-06T03:50:37.096327+0000 mon.a (mon.0) 1894 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:50:38.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:38 smithi153 bash[23852]: cluster 2024-04-06T03:50:37.105658+0000 mon.a (mon.0) 1895 : cluster 0 osdmap e428: 8 total, 8 up, 8 in 2024-04-06T03:50:38.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:38 smithi153 bash[23852]: cluster 2024-04-06T03:50:37.429628+0000 osd.4 (osd.4) 685 : cluster 0 6.e deep-scrub starts 2024-04-06T03:50:39.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:39 smithi153 bash[23852]: cluster 2024-04-06T03:50:37.509486+0000 osd.4 (osd.4) 686 : cluster 0 6.e deep-scrub ok 2024-04-06T03:50:39.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:39 smithi153 bash[23852]: cluster 2024-04-06T03:50:37.539964+0000 osd.1 (osd.1) 475 : cluster 0 6.d deep-scrub starts 2024-04-06T03:50:39.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:39 smithi153 bash[23852]: cluster 2024-04-06T03:50:37.734835+0000 osd.1 (osd.1) 476 : cluster 0 6.d deep-scrub ok 2024-04-06T03:50:39.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:39 smithi153 bash[23852]: cluster 2024-04-06T03:50:38.359821+0000 osd.3 (osd.3) 717 : cluster 0 3.8 deep-scrub starts 2024-04-06T03:50:39.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:39 smithi153 bash[23852]: cluster 2024-04-06T03:50:38.375213+0000 osd.3 (osd.3) 718 : cluster 0 3.8 deep-scrub ok 2024-04-06T03:50:39.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:39 smithi153 bash[23852]: cluster 2024-04-06T03:50:38.660668+0000 mgr.y (mgr.24368) 1088 : cluster 0 pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:39.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:50:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21467]: cluster 2024-04-06T03:50:37.509486+0000 osd.4 (osd.4) 686 : cluster 0 6.e deep-scrub ok 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21467]: cluster 2024-04-06T03:50:37.539964+0000 osd.1 (osd.1) 475 : cluster 0 6.d deep-scrub starts 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21467]: cluster 2024-04-06T03:50:37.734835+0000 osd.1 (osd.1) 476 : cluster 0 6.d deep-scrub ok 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21467]: cluster 2024-04-06T03:50:38.359821+0000 osd.3 (osd.3) 717 : cluster 0 3.8 deep-scrub starts 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21467]: cluster 2024-04-06T03:50:38.375213+0000 osd.3 (osd.3) 718 : cluster 0 3.8 deep-scrub ok 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[21467]: cluster 2024-04-06T03:50:38.660668+0000 mgr.y (mgr.24368) 1088 : cluster 0 pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[29221]: cluster 2024-04-06T03:50:37.509486+0000 osd.4 (osd.4) 686 : cluster 0 6.e deep-scrub ok 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[29221]: cluster 2024-04-06T03:50:37.539964+0000 osd.1 (osd.1) 475 : cluster 0 6.d deep-scrub starts 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[29221]: cluster 2024-04-06T03:50:37.734835+0000 osd.1 (osd.1) 476 : cluster 0 6.d deep-scrub ok 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[29221]: cluster 2024-04-06T03:50:38.359821+0000 osd.3 (osd.3) 717 : cluster 0 3.8 deep-scrub starts 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[29221]: cluster 2024-04-06T03:50:38.375213+0000 osd.3 (osd.3) 718 : cluster 0 3.8 deep-scrub ok 2024-04-06T03:50:39.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:39 smithi050 bash[29221]: cluster 2024-04-06T03:50:38.660668+0000 mgr.y (mgr.24368) 1088 : cluster 0 pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:39.596 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:39.596 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:121281 10.7 e307/280 10:e86db8b7:::benchmark_data_smithi050_121293_object22464:head v 307'7621, mlcod=307'7619)", 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:01.238145+0000", 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 578.35730590399999, 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.059678084999999999, 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238145+0000", 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238145+0000", 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238146+0000", 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.7300000000000005e-07 2024-04-06T03:50:39.598 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238158+0000", 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1812e-05 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238170+0000", 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2108e-05 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238179+0000", 2024-04-06T03:50:39.599 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9500000000000007e-06 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238254+0000", 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4880000000000001e-05 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.238274+0000", 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9653e-05 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297782+0000", 2024-04-06T03:50:39.600 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.059508013999999998 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297823+0000", 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1795000000000002e-05 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:121285 10.7 e307/280 10:e18fdbd2:::benchmark_data_smithi050_121293_object22468:head v 307'7622, mlcod=307'7619)", 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 578.35119373500004, 2024-04-06T03:50:39.601 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053663272999999997, 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.602 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244258+0000", 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9199999999999999e-07 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244268+0000", 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7389999999999994e-06 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244281+0000", 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3088000000000001e-05 2024-04-06T03:50:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244289+0000", 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9149999999999996e-06 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244345+0000", 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5809e-05 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.244365+0000", 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0466e-05 2024-04-06T03:50:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297828+0000", 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053462618000000003 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297921+0000", 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2745999999999996e-05 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.605 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:121288 10.7 e307/280 10:e6b948e9:::benchmark_data_smithi050_121293_object22471:head v 307'7623, mlcod=307'7619)", 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:01.247618+0000", 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 578.34783295800003, 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.050333576999999997, 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247618+0000", 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.606 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247618+0000", 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247619+0000", 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5499999999999997e-07 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247629+0000", 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0178e-05 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.607 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247643+0000", 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3675999999999999e-05 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247651+0000", 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2090000000000005e-06 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247731+0000", 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9338999999999994e-05 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.608 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.247753+0000", 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2084000000000001e-05 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297923+0000", 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.050170807999999997 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:01.297952+0000", 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8428e-05 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167168 10.a e323/322 10:5d6f7159:::benchmark_data_smithi050_121293_object68351:head v 323'10421, mlcod=0'0)", 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.603535+0000", 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 515.99191623700005, 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.056436527, 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603535+0000", 2024-04-06T03:50:39.610 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603535+0000", 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603537+0000", 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7400000000000001e-06 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603550+0000", 2024-04-06T03:50:39.611 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2877000000000001e-05 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603576+0000", 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6196000000000001e-05 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.603591+0000", 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5123000000000001e-05 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605536+0000", 2024-04-06T03:50:39.612 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001944999 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605585+0000", 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9082999999999997e-05 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.659866+0000", 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.054280796999999999 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.659972+0000", 2024-04-06T03:50:39.613 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010571199999999999 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167177 10.a e323/322 10:53f21d76:::benchmark_data_smithi050_121293_object68360:head v 323'10422, mlcod=0'0)", 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.605736+0000", 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 515.98971491600003, 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.054311015999999997, 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.614 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605736+0000", 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605736+0000", 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605738+0000", 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7439999999999999e-06 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.615 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605753+0000", 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5233e-05 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605784+0000", 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0306000000000001e-05 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.605805+0000", 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1416e-05 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.616 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.609992+0000", 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0041874030000000001 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.610042+0000", 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9784999999999999e-05 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.659984+0000", 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.049941343999999999 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.617 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660047+0000", 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.3785e-05 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167203 10.a e323/322 10:5cd24c27:::benchmark_data_smithi050_121293_object68386:head v 323'10423, mlcod=321'10417)", 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.608662+0000", 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 515.98678880800003, 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.051455097999999998, 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.618 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608662+0000", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608662+0000", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608664+0000", 2024-04-06T03:50:39.619 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5629999999999999e-06 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608676+0000", 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2043e-05 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608702+0000", 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5629000000000001e-05 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.608714+0000", 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2697000000000001e-05 2024-04-06T03:50:39.620 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614826+0000", 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0061117819999999996 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.614881+0000", 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4412e-05 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660053+0000", 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.045172333000000002 2024-04-06T03:50:39.621 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660118+0000", 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4639000000000002e-05 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:167209 10.a e323/322 10:5c8a3898:::benchmark_data_smithi050_121293_object68392:head v 323'10424, mlcod=321'10417)", 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:03.611549+0000", 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 515.98390225399999, 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.048630194000000002, 2024-04-06T03:50:39.622 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611549+0000", 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611549+0000", 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.623 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611552+0000", 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6960000000000001e-06 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611568+0000", 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6537000000000001e-05 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611598+0000", 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0086e-05 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.624 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.611621+0000", 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2909999999999999e-05 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.619572+0000", 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0079503170000000001 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.619624+0000", 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2534999999999998e-05 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.625 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660121+0000", 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.040496826 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:03.660179+0000", 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8286999999999999e-05 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15306.0:180404 10.a e324/322 10:5f8d1ba9:::benchmark_data_smithi050_121293_object81587:head v 324'11261, mlcod=324'11255)", 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.518578+0000", 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 499.07687289900002, 2024-04-06T03:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.049958333000000001, 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518578+0000", 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518578+0000", 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.627 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518579+0000", 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5899999999999995e-07 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.518586+0000", 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0450000000000003e-06 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559643+0000", 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.041056895000000003 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.628 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559653+0000", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5920000000000007e-06 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559742+0000", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9581999999999997e-05 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.559764+0000", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1433e-05 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.629 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.568500+0000", 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0087363819999999991 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.568537+0000", 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6545000000000003e-05 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:139511 11.8 e370/345 11:1f0fb34c:::benchmark_data_smithi050_138698_object33537:head v 370'8982, mlcod=370'8979)", 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:54.035475+0000", 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.55997659399998, 2024-04-06T03:50:39.630 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20158874600000001, 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035475+0000", 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035475+0000", 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.631 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035476+0000", 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3900000000000004e-07 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035482+0000", 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7569999999999998e-06 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035493+0000", 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0742e-05 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.632 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035501+0000", 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5109999999999997e-06 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035581+0000", 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0234999999999993e-05 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.035599+0000", 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8442999999999999e-05 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.633 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.237002+0000", 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.201402951 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.237063+0000", 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1267999999999997e-05 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:139515 11.7 e370/345 11:ecd6dfe9:::benchmark_data_smithi050_138698_object33541:head v 370'8744, mlcod=370'8742)", 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:54.036074+0000", 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.55937706499998, 2024-04-06T03:50:39.634 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.22350033, 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.036074+0000", 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.036074+0000", 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.635 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.036075+0000", 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8800000000000002e-07 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.036081+0000", 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2439999999999998e-06 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.036091+0000", 2024-04-06T03:50:39.636 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0010000000000001e-05 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.036098+0000", 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.985e-06 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232031+0000", 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19593299 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232063+0000", 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1826999999999999e-05 2024-04-06T03:50:39.637 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.259474+0000", 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.027411205000000001 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.259575+0000", 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000100381 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.638 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:139518 11.5 e370/345 11:a4264d6c:::benchmark_data_smithi050_138698_object33544:head v 370'8663, mlcod=370'8659)", 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:54.038467+0000", 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.55698396999998, 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.20210424699999999, 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.038467+0000", 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.639 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.038467+0000", 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.038468+0000", 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.61e-07 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.038475+0000", 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.024e-06 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.640 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.231987+0000", 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.19351194599999999 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232006+0000", 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8597000000000002e-05 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232110+0000", 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010460599999999999 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.641 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232132+0000", 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1486999999999999e-05 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.240453+0000", 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0083210679999999992 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.240572+0000", 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000118758 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.642 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:139520 11.8 e370/345 11:1d511cdb:::benchmark_data_smithi050_138698_object33546:head v 370'8983, mlcod=370'8980)", 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:54.039986+0000", 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 285.55546504799997, 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.197121669, 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.039986+0000", 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.643 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.039986+0000", 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.039987+0000", 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.7000000000000004e-07 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.039994+0000", 2024-04-06T03:50:39.644 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1999999999999997e-06 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.231985+0000", 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.191990677 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232005+0000", 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0000000000000002e-05 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232087+0000", 2024-04-06T03:50:39.645 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2422000000000005e-05 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.232116+0000", 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8280999999999999e-05 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.237070+0000", 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.004954277 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.646 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:54.237108+0000", 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8041999999999999e-05 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:200022 11.7 e377/345 11:e7ee41d3:::benchmark_data_smithi050_138698_object94048:head v 377'12575, mlcod=377'12573)", 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:47:01.162381+0000", 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 218.43307018799999, 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.057205170999999999, 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.647 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162381+0000", 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162381+0000", 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162382+0000", 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3600000000000003e-07 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.648 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162388+0000", 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4289999999999998e-06 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162399+0000", 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.048e-05 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162406+0000", 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8009999999999998e-06 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.649 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162456+0000", 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0346000000000002e-05 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.162474+0000", 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8198000000000001e-05 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.219561+0000", 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.057087369999999998 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.650 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.219586+0000", 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4811e-05 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:200037 11.7 e377/345 11:efa30341:::benchmark_data_smithi050_138698_object94063:head v 377'12576, mlcod=377'12573)", 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:47:01.174935+0000", 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 218.42051610199999, 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.044731387999999997, 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:50:39.651 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.174935+0000", 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.174935+0000", 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.174936+0000", 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9700000000000005e-07 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.652 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.174942+0000", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0469999999999999e-06 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.174955+0000", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3175e-05 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.174974+0000", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8712e-05 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.653 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.175030+0000", 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6184000000000003e-05 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.175049+0000", 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9023999999999999e-05 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.219590+0000", 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.044541010999999998 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.654 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:47:01.219667+0000", 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6338e-05 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(6.a,QREQUEST,e:395,from:2,reservation_nonce:2)", 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:48:15.582398+0000", 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 144.01305350300001, 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044184180000000003, 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:50:39.655 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582398+0000", 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582398+0000", 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582400+0000", 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0899999999999999e-06 2024-04-06T03:50:39.656 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582417+0000", 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7416000000000001e-05 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582446+0000", 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8974000000000001e-05 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.657 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582462+0000", 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5832e-05 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582624+0000", 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00016162699999999999 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.582635+0000", 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0787e-05 2024-04-06T03:50:39.658 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.586646+0000", 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0040111319999999997 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.586694+0000", 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8782000000000002e-05 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:15.586816+0000", 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121778 2024-04-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(5.1d,QREQUEST,e:400,from:1,reservation_nonce:3)", 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:48:24.680955+0000", 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 134.91449652599999, 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0047405290000000003, 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.660 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.680955+0000", 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.680955+0000", 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.680957+0000", 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9240000000000001e-06 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.661 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.680966+0000", 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4800000000000007e-06 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.680997+0000", 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0568999999999998e-05 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.681012+0000", 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5679999999999999e-05 2024-04-06T03:50:39.662 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.681137+0000", 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000124874 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.681148+0000", 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.094e-05 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.685574+0000", 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0044255409999999998 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.685610+0000", 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6572000000000002e-05 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:24.685695+0000", 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.4949000000000003e-05 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(5.6,QREQUEST,e:402,from:2,reservation_nonce:32)", 2024-04-06T03:50:39.664 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:48:30.518197+0000", 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 129.07725445700001, 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0047033969999999998, 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518197+0000", 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518197+0000", 2024-04-06T03:50:39.665 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518199+0000", 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.463e-06 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518216+0000", 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.685e-05 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518248+0000", 2024-04-06T03:50:39.666 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.1785999999999997e-05 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518265+0000", 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7356000000000001e-05 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518424+0000", 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015863699999999999 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.518436+0000", 2024-04-06T03:50:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.198e-05 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.522744+0000", 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0043076080000000001 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.522788+0000", 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4759000000000003e-05 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.668 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:30.522900+0000", 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000111958 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(6.19,QREQUEST,e:405,from:1,reservation_nonce:3)", 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:48:33.682946+0000", 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 125.91250551, 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003080593, 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:50:39.669 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.682946+0000", 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.682946+0000", 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.682948+0000", 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0999999999999998e-06 2024-04-06T03:50:39.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.682978+0000", 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9833e-05 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.683014+0000", 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.6001999999999999e-05 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.683029+0000", 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5546999999999999e-05 2024-04-06T03:50:39.671 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.683163+0000", 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000133981 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.683175+0000", 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1892999999999999e-05 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.685863+0000", 2024-04-06T03:50:39.672 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0026873969999999998 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.685906+0000", 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3742000000000002e-05 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:33.686026+0000", 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000120098 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.673 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(2.17,QREQUEST,e:408,from:2,reservation_nonce:6)", 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:48:40.396192+0000", 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 119.19925956199999, 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0048734210000000002, 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396192+0000", 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.674 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396192+0000", 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396194+0000", 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4360000000000001e-06 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396207+0000", 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2446000000000001e-05 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.675 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396239+0000", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2440000000000001e-05 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396256+0000", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7312999999999998e-05 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396382+0000", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012606800000000001 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:50:39.676 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.396393+0000", 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0665999999999999e-05 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.400947+0000", 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0045535310000000004 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.400988+0000", 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1829000000000002e-05 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.677 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:48:40.401065+0000", 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6692000000000004e-05 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(6.10,RELEASE,e:412,from:0,reservation_nonce:0)", 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:49:16.097363+0000", 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 83.498088616999993, 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0031138730000000001, 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:50:39.678 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097363+0000", 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097363+0000", 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097364+0000", 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1999999999999998e-07 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.679 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097371+0000", 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.96e-06 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097393+0000", 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1180000000000001e-05 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097413+0000", 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0548999999999999e-05 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.680 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097529+0000", 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000115605 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.097540+0000", 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1423000000000001e-05 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.100366+0000", 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0028256079999999999 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.681 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.100408+0000", 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2435999999999998e-05 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:49:16.100477+0000", 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8292000000000003e-05 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:50:39.682 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[21467]: cluster 2024-04-06T03:50:39.175616+0000 osd.0 (osd.0) 694 : cluster 0 6.10 scrub starts 2024-04-06T03:50:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[21467]: cluster 2024-04-06T03:50:39.268872+0000 osd.0 (osd.0) 695 : cluster 0 6.10 scrub ok 2024-04-06T03:50:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[21467]: cluster 2024-04-06T03:50:39.429130+0000 osd.4 (osd.4) 687 : cluster 0 5.1 deep-scrub starts 2024-04-06T03:50:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[21467]: cluster 2024-04-06T03:50:39.430706+0000 osd.4 (osd.4) 688 : cluster 0 5.1 deep-scrub ok 2024-04-06T03:50:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[21467]: cluster 2024-04-06T03:50:39.543541+0000 osd.7 (osd.7) 719 : cluster 0 6.3 scrub starts 2024-04-06T03:50:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[29221]: cluster 2024-04-06T03:50:39.175616+0000 osd.0 (osd.0) 694 : cluster 0 6.10 scrub starts 2024-04-06T03:50:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[29221]: cluster 2024-04-06T03:50:39.268872+0000 osd.0 (osd.0) 695 : cluster 0 6.10 scrub ok 2024-04-06T03:50:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[29221]: cluster 2024-04-06T03:50:39.429130+0000 osd.4 (osd.4) 687 : cluster 0 5.1 deep-scrub starts 2024-04-06T03:50:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[29221]: cluster 2024-04-06T03:50:39.430706+0000 osd.4 (osd.4) 688 : cluster 0 5.1 deep-scrub ok 2024-04-06T03:50:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:40 smithi050 bash[29221]: cluster 2024-04-06T03:50:39.543541+0000 osd.7 (osd.7) 719 : cluster 0 6.3 scrub starts 2024-04-06T03:50:40.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:40 smithi153 bash[23852]: cluster 2024-04-06T03:50:39.175616+0000 osd.0 (osd.0) 694 : cluster 0 6.10 scrub starts 2024-04-06T03:50:40.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:40 smithi153 bash[23852]: cluster 2024-04-06T03:50:39.268872+0000 osd.0 (osd.0) 695 : cluster 0 6.10 scrub ok 2024-04-06T03:50:40.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:40 smithi153 bash[23852]: cluster 2024-04-06T03:50:39.429130+0000 osd.4 (osd.4) 687 : cluster 0 5.1 deep-scrub starts 2024-04-06T03:50:40.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:40 smithi153 bash[23852]: cluster 2024-04-06T03:50:39.430706+0000 osd.4 (osd.4) 688 : cluster 0 5.1 deep-scrub ok 2024-04-06T03:50:40.499 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:40 smithi153 bash[23852]: cluster 2024-04-06T03:50:39.543541+0000 osd.7 (osd.7) 719 : cluster 0 6.3 scrub starts 2024-04-06T03:50:40.738 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-06T03:50:40.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:50:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:50:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:41 smithi050 bash[21467]: cluster 2024-04-06T03:50:39.680328+0000 osd.7 (osd.7) 720 : cluster 0 6.3 scrub ok 2024-04-06T03:50:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:41 smithi050 bash[21467]: audit 2024-04-06T03:50:40.500143+0000 mgr.y (mgr.24368) 1089 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:41 smithi050 bash[21467]: cluster 2024-04-06T03:50:40.661927+0000 mgr.y (mgr.24368) 1090 : cluster 0 pgmap v1199: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:41 smithi050 bash[29221]: cluster 2024-04-06T03:50:39.680328+0000 osd.7 (osd.7) 720 : cluster 0 6.3 scrub ok 2024-04-06T03:50:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:41 smithi050 bash[29221]: audit 2024-04-06T03:50:40.500143+0000 mgr.y (mgr.24368) 1089 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:41 smithi050 bash[29221]: cluster 2024-04-06T03:50:40.661927+0000 mgr.y (mgr.24368) 1090 : cluster 0 pgmap v1199: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:41 smithi153 bash[23852]: cluster 2024-04-06T03:50:39.680328+0000 osd.7 (osd.7) 720 : cluster 0 6.3 scrub ok 2024-04-06T03:50:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:41 smithi153 bash[23852]: audit 2024-04-06T03:50:40.500143+0000 mgr.y (mgr.24368) 1089 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:41 smithi153 bash[23852]: cluster 2024-04-06T03:50:40.661927+0000 mgr.y (mgr.24368) 1090 : cluster 0 pgmap v1199: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:42.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: cluster 2024-04-06T03:50:40.442073+0000 osd.1 (osd.1) 477 : cluster 0 5.1d scrub starts 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: cluster 2024-04-06T03:50:40.444332+0000 osd.1 (osd.1) 478 : cluster 0 5.1d scrub ok 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: cluster 2024-04-06T03:50:41.151527+0000 osd.0 (osd.0) 696 : cluster 0 5.2 scrub starts 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: cluster 2024-04-06T03:50:41.153722+0000 osd.0 (osd.0) 697 : cluster 0 5.2 scrub ok 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: cluster 2024-04-06T03:50:41.365069+0000 osd.3 (osd.3) 719 : cluster 0 5.17 scrub starts 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: cluster 2024-04-06T03:50:41.367555+0000 osd.3 (osd.3) 720 : cluster 0 5.17 scrub ok 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: audit 2024-04-06T03:50:41.978293+0000 mon.a (mon.0) 1896 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: audit 2024-04-06T03:50:41.991055+0000 mon.a (mon.0) 1897 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: audit 2024-04-06T03:50:41.993773+0000 mon.a (mon.0) 1898 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: audit 2024-04-06T03:50:41.995848+0000 mon.a (mon.0) 1899 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[21467]: audit 2024-04-06T03:50:42.006149+0000 mon.a (mon.0) 1900 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: cluster 2024-04-06T03:50:40.442073+0000 osd.1 (osd.1) 477 : cluster 0 5.1d scrub starts 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: cluster 2024-04-06T03:50:40.444332+0000 osd.1 (osd.1) 478 : cluster 0 5.1d scrub ok 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: cluster 2024-04-06T03:50:41.151527+0000 osd.0 (osd.0) 696 : cluster 0 5.2 scrub starts 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: cluster 2024-04-06T03:50:41.153722+0000 osd.0 (osd.0) 697 : cluster 0 5.2 scrub ok 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: cluster 2024-04-06T03:50:41.365069+0000 osd.3 (osd.3) 719 : cluster 0 5.17 scrub starts 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: cluster 2024-04-06T03:50:41.367555+0000 osd.3 (osd.3) 720 : cluster 0 5.17 scrub ok 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: audit 2024-04-06T03:50:41.978293+0000 mon.a (mon.0) 1896 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: audit 2024-04-06T03:50:41.991055+0000 mon.a (mon.0) 1897 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: audit 2024-04-06T03:50:41.993773+0000 mon.a (mon.0) 1898 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: audit 2024-04-06T03:50:41.995848+0000 mon.a (mon.0) 1899 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:50:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:42 smithi050 bash[29221]: audit 2024-04-06T03:50:42.006149+0000 mon.a (mon.0) 1900 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: cluster 2024-04-06T03:50:40.442073+0000 osd.1 (osd.1) 477 : cluster 0 5.1d scrub starts 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: cluster 2024-04-06T03:50:40.444332+0000 osd.1 (osd.1) 478 : cluster 0 5.1d scrub ok 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: cluster 2024-04-06T03:50:41.151527+0000 osd.0 (osd.0) 696 : cluster 0 5.2 scrub starts 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: cluster 2024-04-06T03:50:41.153722+0000 osd.0 (osd.0) 697 : cluster 0 5.2 scrub ok 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: cluster 2024-04-06T03:50:41.365069+0000 osd.3 (osd.3) 719 : cluster 0 5.17 scrub starts 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: cluster 2024-04-06T03:50:41.367555+0000 osd.3 (osd.3) 720 : cluster 0 5.17 scrub ok 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: audit 2024-04-06T03:50:41.978293+0000 mon.a (mon.0) 1896 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: audit 2024-04-06T03:50:41.991055+0000 mon.a (mon.0) 1897 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: audit 2024-04-06T03:50:41.993773+0000 mon.a (mon.0) 1898 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: audit 2024-04-06T03:50:41.995848+0000 mon.a (mon.0) 1899 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:50:42.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:42 smithi153 bash[23852]: audit 2024-04-06T03:50:42.006149+0000 mon.a (mon.0) 1900 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:50:42.720 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:50:43.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:43 smithi050 bash[29221]: cluster 2024-04-06T03:50:42.381010+0000 osd.3 (osd.3) 721 : cluster 0 4.0 deep-scrub starts 2024-04-06T03:50:43.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:43 smithi050 bash[29221]: cluster 2024-04-06T03:50:42.386886+0000 osd.3 (osd.3) 722 : cluster 0 4.0 deep-scrub ok 2024-04-06T03:50:43.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:43 smithi050 bash[29221]: cluster 2024-04-06T03:50:42.662814+0000 mgr.y (mgr.24368) 1091 : cluster 0 pgmap v1200: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:43.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:43 smithi050 bash[21467]: cluster 2024-04-06T03:50:42.381010+0000 osd.3 (osd.3) 721 : cluster 0 4.0 deep-scrub starts 2024-04-06T03:50:43.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:43 smithi050 bash[21467]: cluster 2024-04-06T03:50:42.386886+0000 osd.3 (osd.3) 722 : cluster 0 4.0 deep-scrub ok 2024-04-06T03:50:43.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:43 smithi050 bash[21467]: cluster 2024-04-06T03:50:42.662814+0000 mgr.y (mgr.24368) 1091 : cluster 0 pgmap v1200: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:43.562 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:43 smithi153 bash[23852]: cluster 2024-04-06T03:50:42.381010+0000 osd.3 (osd.3) 721 : cluster 0 4.0 deep-scrub starts 2024-04-06T03:50:43.562 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:43 smithi153 bash[23852]: cluster 2024-04-06T03:50:42.386886+0000 osd.3 (osd.3) 722 : cluster 0 4.0 deep-scrub ok 2024-04-06T03:50:43.562 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:43 smithi153 bash[23852]: cluster 2024-04-06T03:50:42.662814+0000 mgr.y (mgr.24368) 1091 : cluster 0 pgmap v1200: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:44.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[29221]: cluster 2024-04-06T03:50:42.420687+0000 osd.1 (osd.1) 479 : cluster 0 2.4 scrub starts 2024-04-06T03:50:44.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[29221]: cluster 2024-04-06T03:50:42.423606+0000 osd.1 (osd.1) 480 : cluster 0 2.4 scrub ok 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[29221]: cluster 2024-04-06T03:50:43.210906+0000 osd.0 (osd.0) 698 : cluster 0 5.5 scrub starts 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[29221]: cluster 2024-04-06T03:50:43.213299+0000 osd.0 (osd.0) 699 : cluster 0 5.5 scrub ok 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[29221]: cluster 2024-04-06T03:50:43.417167+0000 osd.3 (osd.3) 723 : cluster 0 6.1f scrub starts 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[29221]: cluster 2024-04-06T03:50:43.628769+0000 osd.3 (osd.3) 724 : cluster 0 6.1f scrub ok 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[21467]: cluster 2024-04-06T03:50:42.420687+0000 osd.1 (osd.1) 479 : cluster 0 2.4 scrub starts 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[21467]: cluster 2024-04-06T03:50:42.423606+0000 osd.1 (osd.1) 480 : cluster 0 2.4 scrub ok 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[21467]: cluster 2024-04-06T03:50:43.210906+0000 osd.0 (osd.0) 698 : cluster 0 5.5 scrub starts 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[21467]: cluster 2024-04-06T03:50:43.213299+0000 osd.0 (osd.0) 699 : cluster 0 5.5 scrub ok 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[21467]: cluster 2024-04-06T03:50:43.417167+0000 osd.3 (osd.3) 723 : cluster 0 6.1f scrub starts 2024-04-06T03:50:44.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:44 smithi050 bash[21467]: cluster 2024-04-06T03:50:43.628769+0000 osd.3 (osd.3) 724 : cluster 0 6.1f scrub ok 2024-04-06T03:50:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:44 smithi153 bash[23852]: cluster 2024-04-06T03:50:42.420687+0000 osd.1 (osd.1) 479 : cluster 0 2.4 scrub starts 2024-04-06T03:50:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:44 smithi153 bash[23852]: cluster 2024-04-06T03:50:42.423606+0000 osd.1 (osd.1) 480 : cluster 0 2.4 scrub ok 2024-04-06T03:50:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:44 smithi153 bash[23852]: cluster 2024-04-06T03:50:43.210906+0000 osd.0 (osd.0) 698 : cluster 0 5.5 scrub starts 2024-04-06T03:50:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:44 smithi153 bash[23852]: cluster 2024-04-06T03:50:43.213299+0000 osd.0 (osd.0) 699 : cluster 0 5.5 scrub ok 2024-04-06T03:50:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:44 smithi153 bash[23852]: cluster 2024-04-06T03:50:43.417167+0000 osd.3 (osd.3) 723 : cluster 0 6.1f scrub starts 2024-04-06T03:50:44.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:44 smithi153 bash[23852]: cluster 2024-04-06T03:50:43.628769+0000 osd.3 (osd.3) 724 : cluster 0 6.1f scrub ok 2024-04-06T03:50:45.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[29221]: cluster 2024-04-06T03:50:43.621284+0000 osd.7 (osd.7) 721 : cluster 0 2.0 scrub starts 2024-04-06T03:50:45.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[29221]: cluster 2024-04-06T03:50:43.625193+0000 osd.7 (osd.7) 722 : cluster 0 2.0 scrub ok 2024-04-06T03:50:45.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.176641+0000 osd.0 (osd.0) 700 : cluster 0 5.1b scrub starts 2024-04-06T03:50:45.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.178525+0000 osd.0 (osd.0) 701 : cluster 0 5.1b scrub ok 2024-04-06T03:50:45.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.663840+0000 mgr.y (mgr.24368) 1092 : cluster 0 pgmap v1201: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[21467]: cluster 2024-04-06T03:50:43.621284+0000 osd.7 (osd.7) 721 : cluster 0 2.0 scrub starts 2024-04-06T03:50:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[21467]: cluster 2024-04-06T03:50:43.625193+0000 osd.7 (osd.7) 722 : cluster 0 2.0 scrub ok 2024-04-06T03:50:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.176641+0000 osd.0 (osd.0) 700 : cluster 0 5.1b scrub starts 2024-04-06T03:50:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.178525+0000 osd.0 (osd.0) 701 : cluster 0 5.1b scrub ok 2024-04-06T03:50:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:45 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.663840+0000 mgr.y (mgr.24368) 1092 : cluster 0 pgmap v1201: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:45.516 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:45 smithi153 bash[23852]: cluster 2024-04-06T03:50:43.621284+0000 osd.7 (osd.7) 721 : cluster 0 2.0 scrub starts 2024-04-06T03:50:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:45 smithi153 bash[23852]: cluster 2024-04-06T03:50:43.625193+0000 osd.7 (osd.7) 722 : cluster 0 2.0 scrub ok 2024-04-06T03:50:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:45 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.176641+0000 osd.0 (osd.0) 700 : cluster 0 5.1b scrub starts 2024-04-06T03:50:45.537 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:45 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.178525+0000 osd.0 (osd.0) 701 : cluster 0 5.1b scrub ok 2024-04-06T03:50:45.538 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:45 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.663840+0000 mgr.y (mgr.24368) 1092 : cluster 0 pgmap v1201: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:50:46.166 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-06T03:50:46.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.406686+0000 osd.1 (osd.1) 481 : cluster 0 6.1d scrub starts 2024-04-06T03:50:46.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.571151+0000 osd.1 (osd.1) 482 : cluster 0 6.1d scrub ok 2024-04-06T03:50:46.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.977817+0000 osd.4 (osd.4) 689 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:46.475 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: cluster 2024-04-06T03:50:44.977974+0000 osd.4 (osd.4) 690 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:46.476 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: cluster 2024-04-06T03:50:45.155160+0000 mon.a (mon.0) 1901 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-04-06T03:50:46.476 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: audit 2024-04-06T03:50:46.087187+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.50:0/2687320005' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:46.476 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:46 smithi153 bash[23852]: audit 2024-04-06T03:50:46.087745+0000 mon.a (mon.0) 1902 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:46.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.406686+0000 osd.1 (osd.1) 481 : cluster 0 6.1d scrub starts 2024-04-06T03:50:46.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.571151+0000 osd.1 (osd.1) 482 : cluster 0 6.1d scrub ok 2024-04-06T03:50:46.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.977817+0000 osd.4 (osd.4) 689 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:46.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: cluster 2024-04-06T03:50:44.977974+0000 osd.4 (osd.4) 690 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: cluster 2024-04-06T03:50:45.155160+0000 mon.a (mon.0) 1901 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: audit 2024-04-06T03:50:46.087187+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.50:0/2687320005' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[29221]: audit 2024-04-06T03:50:46.087745+0000 mon.a (mon.0) 1902 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.406686+0000 osd.1 (osd.1) 481 : cluster 0 6.1d scrub starts 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.571151+0000 osd.1 (osd.1) 482 : cluster 0 6.1d scrub ok 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.977817+0000 osd.4 (osd.4) 689 : cluster 0 purged_snaps scrub starts 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: cluster 2024-04-06T03:50:44.977974+0000 osd.4 (osd.4) 690 : cluster 0 purged_snaps scrub ok 2024-04-06T03:50:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: cluster 2024-04-06T03:50:45.155160+0000 mon.a (mon.0) 1901 : cluster 0 osdmap e429: 8 total, 8 up, 8 in 2024-04-06T03:50:46.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: audit 2024-04-06T03:50:46.087187+0000 mon.c (mon.2) 132 : audit 1 from='client.? 172.21.15.50:0/2687320005' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:46.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:46 smithi050 bash[21467]: audit 2024-04-06T03:50:46.087745+0000 mon.a (mon.0) 1902 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-06T03:50:47.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 6 1 2024-04-06T03:50:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: audit 2024-04-06T03:50:46.147396+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: cluster 2024-04-06T03:50:46.153057+0000 mon.a (mon.0) 1904 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: cluster 2024-04-06T03:50:46.237340+0000 osd.0 (osd.0) 702 : cluster 0 5.1e scrub starts 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: cluster 2024-04-06T03:50:46.238963+0000 osd.0 (osd.0) 703 : cluster 0 5.1e scrub ok 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: cluster 2024-04-06T03:50:46.319944+0000 osd.4 (osd.4) 691 : cluster 0 3.14 deep-scrub starts 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: cluster 2024-04-06T03:50:46.333308+0000 osd.4 (osd.4) 692 : cluster 0 3.14 deep-scrub ok 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[21467]: cluster 2024-04-06T03:50:46.664600+0000 mgr.y (mgr.24368) 1093 : cluster 0 pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: audit 2024-04-06T03:50:46.147396+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: cluster 2024-04-06T03:50:46.153057+0000 mon.a (mon.0) 1904 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: cluster 2024-04-06T03:50:46.237340+0000 osd.0 (osd.0) 702 : cluster 0 5.1e scrub starts 2024-04-06T03:50:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: cluster 2024-04-06T03:50:46.238963+0000 osd.0 (osd.0) 703 : cluster 0 5.1e scrub ok 2024-04-06T03:50:47.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: cluster 2024-04-06T03:50:46.319944+0000 osd.4 (osd.4) 691 : cluster 0 3.14 deep-scrub starts 2024-04-06T03:50:47.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: cluster 2024-04-06T03:50:46.333308+0000 osd.4 (osd.4) 692 : cluster 0 3.14 deep-scrub ok 2024-04-06T03:50:47.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:47 smithi050 bash[29221]: cluster 2024-04-06T03:50:46.664600+0000 mgr.y (mgr.24368) 1093 : cluster 0 pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: audit 2024-04-06T03:50:46.147396+0000 mon.a (mon.0) 1903 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-06T03:50:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: cluster 2024-04-06T03:50:46.153057+0000 mon.a (mon.0) 1904 : cluster 0 osdmap e430: 8 total, 8 up, 8 in 2024-04-06T03:50:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: cluster 2024-04-06T03:50:46.237340+0000 osd.0 (osd.0) 702 : cluster 0 5.1e scrub starts 2024-04-06T03:50:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: cluster 2024-04-06T03:50:46.238963+0000 osd.0 (osd.0) 703 : cluster 0 5.1e scrub ok 2024-04-06T03:50:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: cluster 2024-04-06T03:50:46.319944+0000 osd.4 (osd.4) 691 : cluster 0 3.14 deep-scrub starts 2024-04-06T03:50:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: cluster 2024-04-06T03:50:46.333308+0000 osd.4 (osd.4) 692 : cluster 0 3.14 deep-scrub ok 2024-04-06T03:50:47.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:47 smithi153 bash[23852]: cluster 2024-04-06T03:50:46.664600+0000 mgr.y (mgr.24368) 1093 : cluster 0 pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:50:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:48 smithi050 bash[21467]: cluster 2024-04-06T03:50:47.174857+0000 mon.a (mon.0) 1905 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-04-06T03:50:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:48 smithi050 bash[21467]: cluster 2024-04-06T03:50:47.565098+0000 osd.7 (osd.7) 723 : cluster 0 3.f scrub starts 2024-04-06T03:50:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:48 smithi050 bash[21467]: cluster 2024-04-06T03:50:47.569656+0000 osd.7 (osd.7) 724 : cluster 0 3.f scrub ok 2024-04-06T03:50:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:48 smithi050 bash[29221]: cluster 2024-04-06T03:50:47.174857+0000 mon.a (mon.0) 1905 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-04-06T03:50:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:48 smithi050 bash[29221]: cluster 2024-04-06T03:50:47.565098+0000 osd.7 (osd.7) 723 : cluster 0 3.f scrub starts 2024-04-06T03:50:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:48 smithi050 bash[29221]: cluster 2024-04-06T03:50:47.569656+0000 osd.7 (osd.7) 724 : cluster 0 3.f scrub ok 2024-04-06T03:50:48.500 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:48.500 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:50:48.500 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:50:48.500 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:48.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:48 smithi153 bash[23852]: cluster 2024-04-06T03:50:47.174857+0000 mon.a (mon.0) 1905 : cluster 0 osdmap e431: 8 total, 8 up, 8 in 2024-04-06T03:50:48.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:48 smithi153 bash[23852]: cluster 2024-04-06T03:50:47.565098+0000 osd.7 (osd.7) 723 : cluster 0 3.f scrub starts 2024-04-06T03:50:48.511 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:48 smithi153 bash[23852]: cluster 2024-04-06T03:50:47.569656+0000 osd.7 (osd.7) 724 : cluster 0 3.f scrub ok 2024-04-06T03:50:49.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_blocked_ops 2024-04-06T03:50:49.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:50:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:50:49.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[21467]: cluster 2024-04-06T03:50:48.575782+0000 osd.7 (osd.7) 725 : cluster 0 5.9 scrub starts 2024-04-06T03:50:49.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[21467]: cluster 2024-04-06T03:50:48.577391+0000 osd.7 (osd.7) 726 : cluster 0 5.9 scrub ok 2024-04-06T03:50:49.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[21467]: cluster 2024-04-06T03:50:48.665678+0000 mgr.y (mgr.24368) 1094 : cluster 0 pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:50:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[29221]: cluster 2024-04-06T03:50:48.575782+0000 osd.7 (osd.7) 725 : cluster 0 5.9 scrub starts 2024-04-06T03:50:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[29221]: cluster 2024-04-06T03:50:48.577391+0000 osd.7 (osd.7) 726 : cluster 0 5.9 scrub ok 2024-04-06T03:50:49.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:49 smithi050 bash[29221]: cluster 2024-04-06T03:50:48.665678+0000 mgr.y (mgr.24368) 1094 : cluster 0 pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:50:49.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:49 smithi153 bash[23852]: cluster 2024-04-06T03:50:48.575782+0000 osd.7 (osd.7) 725 : cluster 0 5.9 scrub starts 2024-04-06T03:50:49.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:49 smithi153 bash[23852]: cluster 2024-04-06T03:50:48.577391+0000 osd.7 (osd.7) 726 : cluster 0 5.9 scrub ok 2024-04-06T03:50:49.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:49 smithi153 bash[23852]: cluster 2024-04-06T03:50:48.665678+0000 mgr.y (mgr.24368) 1094 : cluster 0 pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-06T03:50:50.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:50:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:50:50.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:50 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.272660+0000 osd.2 (osd.2) 591 : cluster 0 5.a scrub starts 2024-04-06T03:50:50.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:50 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.275752+0000 osd.2 (osd.2) 592 : cluster 0 5.a scrub ok 2024-04-06T03:50:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:50 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.279127+0000 osd.0 (osd.0) 704 : cluster 0 3.1f scrub starts 2024-04-06T03:50:50.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:50 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.284223+0000 osd.0 (osd.0) 705 : cluster 0 3.1f scrub ok 2024-04-06T03:50:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.272660+0000 osd.2 (osd.2) 591 : cluster 0 5.a scrub starts 2024-04-06T03:50:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.275752+0000 osd.2 (osd.2) 592 : cluster 0 5.a scrub ok 2024-04-06T03:50:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.279127+0000 osd.0 (osd.0) 704 : cluster 0 3.1f scrub starts 2024-04-06T03:50:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.284223+0000 osd.0 (osd.0) 705 : cluster 0 3.1f scrub ok 2024-04-06T03:50:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.272660+0000 osd.2 (osd.2) 591 : cluster 0 5.a scrub starts 2024-04-06T03:50:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.275752+0000 osd.2 (osd.2) 592 : cluster 0 5.a scrub ok 2024-04-06T03:50:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.279127+0000 osd.0 (osd.0) 704 : cluster 0 3.1f scrub starts 2024-04-06T03:50:50.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:50 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.284223+0000 osd.0 (osd.0) 705 : cluster 0 3.1f scrub ok 2024-04-06T03:50:51.823 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:50:51.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:49.612526+0000 osd.7 (osd.7) 727 : cluster 0 2.1c scrub starts 2024-04-06T03:50:51.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:49.615066+0000 osd.7 (osd.7) 728 : cluster 0 2.1c scrub ok 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.432113+0000 osd.1 (osd.1) 483 : cluster 0 2.a scrub starts 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.434540+0000 osd.1 (osd.1) 484 : cluster 0 2.a scrub ok 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: audit 2024-04-06T03:50:50.508750+0000 mgr.y (mgr.24368) 1095 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:50.667077+0000 mgr.y (mgr.24368) 1096 : cluster 0 pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: audit 2024-04-06T03:50:51.033408+0000 mon.a (mon.0) 1906 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:51.268661+0000 osd.0 (osd.0) 706 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[21467]: cluster 2024-04-06T03:50:51.272979+0000 osd.0 (osd.0) 707 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:49.612526+0000 osd.7 (osd.7) 727 : cluster 0 2.1c scrub starts 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:49.615066+0000 osd.7 (osd.7) 728 : cluster 0 2.1c scrub ok 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.432113+0000 osd.1 (osd.1) 483 : cluster 0 2.a scrub starts 2024-04-06T03:50:51.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.434540+0000 osd.1 (osd.1) 484 : cluster 0 2.a scrub ok 2024-04-06T03:50:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: audit 2024-04-06T03:50:50.508750+0000 mgr.y (mgr.24368) 1095 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:50.667077+0000 mgr.y (mgr.24368) 1096 : cluster 0 pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:50:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: audit 2024-04-06T03:50:51.033408+0000 mon.a (mon.0) 1906 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:51.268661+0000 osd.0 (osd.0) 706 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:50:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:51 smithi050 bash[29221]: cluster 2024-04-06T03:50:51.272979+0000 osd.0 (osd.0) 707 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:49.612526+0000 osd.7 (osd.7) 727 : cluster 0 2.1c scrub starts 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:49.615066+0000 osd.7 (osd.7) 728 : cluster 0 2.1c scrub ok 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.432113+0000 osd.1 (osd.1) 483 : cluster 0 2.a scrub starts 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.434540+0000 osd.1 (osd.1) 484 : cluster 0 2.a scrub ok 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: audit 2024-04-06T03:50:50.508750+0000 mgr.y (mgr.24368) 1095 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:50.667077+0000 mgr.y (mgr.24368) 1096 : cluster 0 pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: audit 2024-04-06T03:50:51.033408+0000 mon.a (mon.0) 1906 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:51.268661+0000 osd.0 (osd.0) 706 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:50:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:51 smithi153 bash[23852]: cluster 2024-04-06T03:50:51.272979+0000 osd.0 (osd.0) 707 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:50:52.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[21467]: cluster 2024-04-06T03:50:51.422832+0000 osd.1 (osd.1) 485 : cluster 0 3.4 scrub starts 2024-04-06T03:50:52.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[21467]: cluster 2024-04-06T03:50:51.431590+0000 osd.1 (osd.1) 486 : cluster 0 3.4 scrub ok 2024-04-06T03:50:52.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[21467]: cluster 2024-04-06T03:50:51.984427+0000 osd.5 (osd.5) 159 : cluster 0 3.16 scrub starts 2024-04-06T03:50:52.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[21467]: cluster 2024-04-06T03:50:51.988643+0000 osd.5 (osd.5) 160 : cluster 0 3.16 scrub ok 2024-04-06T03:50:52.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[29221]: cluster 2024-04-06T03:50:51.422832+0000 osd.1 (osd.1) 485 : cluster 0 3.4 scrub starts 2024-04-06T03:50:52.925 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[29221]: cluster 2024-04-06T03:50:51.431590+0000 osd.1 (osd.1) 486 : cluster 0 3.4 scrub ok 2024-04-06T03:50:52.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[29221]: cluster 2024-04-06T03:50:51.984427+0000 osd.5 (osd.5) 159 : cluster 0 3.16 scrub starts 2024-04-06T03:50:52.926 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:52 smithi050 bash[29221]: cluster 2024-04-06T03:50:51.988643+0000 osd.5 (osd.5) 160 : cluster 0 3.16 scrub ok 2024-04-06T03:50:53.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:52 smithi153 bash[23852]: cluster 2024-04-06T03:50:51.422832+0000 osd.1 (osd.1) 485 : cluster 0 3.4 scrub starts 2024-04-06T03:50:53.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:52 smithi153 bash[23852]: cluster 2024-04-06T03:50:51.431590+0000 osd.1 (osd.1) 486 : cluster 0 3.4 scrub ok 2024-04-06T03:50:53.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:52 smithi153 bash[23852]: cluster 2024-04-06T03:50:51.984427+0000 osd.5 (osd.5) 159 : cluster 0 3.16 scrub starts 2024-04-06T03:50:53.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:52 smithi153 bash[23852]: cluster 2024-04-06T03:50:51.988643+0000 osd.5 (osd.5) 160 : cluster 0 3.16 scrub ok 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:52.668014+0000 mgr.y (mgr.24368) 1097 : cluster 0 pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 136 B/s rd, 0 op/s 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:52.965815+0000 osd.5 (osd.5) 161 : cluster 0 3.5 scrub starts 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:52.970299+0000 osd.5 (osd.5) 162 : cluster 0 3.5 scrub ok 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.321288+0000 osd.0 (osd.0) 708 : cluster 0 3.6 scrub starts 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.324892+0000 osd.0 (osd.0) 709 : cluster 0 3.6 scrub ok 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.350266+0000 osd.2 (osd.2) 593 : cluster 0 5.6 scrub starts 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.352484+0000 osd.2 (osd.2) 594 : cluster 0 5.6 scrub ok 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:52.668014+0000 mgr.y (mgr.24368) 1097 : cluster 0 pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 136 B/s rd, 0 op/s 2024-04-06T03:50:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:52.965815+0000 osd.5 (osd.5) 161 : cluster 0 3.5 scrub starts 2024-04-06T03:50:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:52.970299+0000 osd.5 (osd.5) 162 : cluster 0 3.5 scrub ok 2024-04-06T03:50:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.321288+0000 osd.0 (osd.0) 708 : cluster 0 3.6 scrub starts 2024-04-06T03:50:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.324892+0000 osd.0 (osd.0) 709 : cluster 0 3.6 scrub ok 2024-04-06T03:50:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.350266+0000 osd.2 (osd.2) 593 : cluster 0 5.6 scrub starts 2024-04-06T03:50:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:53 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.352484+0000 osd.2 (osd.2) 594 : cluster 0 5.6 scrub ok 2024-04-06T03:50:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:52.668014+0000 mgr.y (mgr.24368) 1097 : cluster 0 pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 136 B/s rd, 0 op/s 2024-04-06T03:50:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:52.965815+0000 osd.5 (osd.5) 161 : cluster 0 3.5 scrub starts 2024-04-06T03:50:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:52.970299+0000 osd.5 (osd.5) 162 : cluster 0 3.5 scrub ok 2024-04-06T03:50:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.321288+0000 osd.0 (osd.0) 708 : cluster 0 3.6 scrub starts 2024-04-06T03:50:54.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.324892+0000 osd.0 (osd.0) 709 : cluster 0 3.6 scrub ok 2024-04-06T03:50:54.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.350266+0000 osd.2 (osd.2) 593 : cluster 0 5.6 scrub starts 2024-04-06T03:50:54.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:53 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.352484+0000 osd.2 (osd.2) 594 : cluster 0 5.6 scrub ok 2024-04-06T03:50:54.071 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:50:54.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.562399+0000 osd.7 (osd.7) 729 : cluster 0 5.10 scrub starts 2024-04-06T03:50:54.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.564034+0000 osd.7 (osd.7) 730 : cluster 0 5.10 scrub ok 2024-04-06T03:50:54.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[29221]: cluster 2024-04-06T03:50:53.984670+0000 osd.5 (osd.5) 163 : cluster 0 6.a scrub starts 2024-04-06T03:50:54.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[29221]: cluster 2024-04-06T03:50:54.021336+0000 osd.5 (osd.5) 164 : cluster 0 6.a scrub ok 2024-04-06T03:50:54.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.562399+0000 osd.7 (osd.7) 729 : cluster 0 5.10 scrub starts 2024-04-06T03:50:54.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.564034+0000 osd.7 (osd.7) 730 : cluster 0 5.10 scrub ok 2024-04-06T03:50:54.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[21467]: cluster 2024-04-06T03:50:53.984670+0000 osd.5 (osd.5) 163 : cluster 0 6.a scrub starts 2024-04-06T03:50:54.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:54 smithi050 bash[21467]: cluster 2024-04-06T03:50:54.021336+0000 osd.5 (osd.5) 164 : cluster 0 6.a scrub ok 2024-04-06T03:50:55.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:54 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.562399+0000 osd.7 (osd.7) 729 : cluster 0 5.10 scrub starts 2024-04-06T03:50:55.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:54 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.564034+0000 osd.7 (osd.7) 730 : cluster 0 5.10 scrub ok 2024-04-06T03:50:55.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:54 smithi153 bash[23852]: cluster 2024-04-06T03:50:53.984670+0000 osd.5 (osd.5) 163 : cluster 0 6.a scrub starts 2024-04-06T03:50:55.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:54 smithi153 bash[23852]: cluster 2024-04-06T03:50:54.021336+0000 osd.5 (osd.5) 164 : cluster 0 6.a scrub ok 2024-04-06T03:50:55.631 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-06T03:50:55.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: cluster 2024-04-06T03:50:54.669179+0000 mgr.y (mgr.24368) 1098 : cluster 0 pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 721 B/s rd, 0 op/s 2024-04-06T03:50:55.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.306727+0000 osd.0 (osd.0) 710 : cluster 0 5.3 scrub starts 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.309375+0000 osd.0 (osd.0) 711 : cluster 0 5.3 scrub ok 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: audit 2024-04-06T03:50:55.426177+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.50:0/2651090008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: audit 2024-04-06T03:50:55.426784+0000 mon.a (mon.0) 1907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.560628+0000 osd.7 (osd.7) 731 : cluster 0 3.13 scrub starts 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.565006+0000 osd.7 (osd.7) 732 : cluster 0 3.13 scrub ok 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: cluster 2024-04-06T03:50:54.669179+0000 mgr.y (mgr.24368) 1098 : cluster 0 pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 721 B/s rd, 0 op/s 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.306727+0000 osd.0 (osd.0) 710 : cluster 0 5.3 scrub starts 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.309375+0000 osd.0 (osd.0) 711 : cluster 0 5.3 scrub ok 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: audit 2024-04-06T03:50:55.426177+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.50:0/2651090008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: audit 2024-04-06T03:50:55.426784+0000 mon.a (mon.0) 1907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.560628+0000 osd.7 (osd.7) 731 : cluster 0 3.13 scrub starts 2024-04-06T03:50:55.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:55 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.565006+0000 osd.7 (osd.7) 732 : cluster 0 3.13 scrub ok 2024-04-06T03:50:56.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: cluster 2024-04-06T03:50:54.669179+0000 mgr.y (mgr.24368) 1098 : cluster 0 pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 721 B/s rd, 0 op/s 2024-04-06T03:50:56.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.306727+0000 osd.0 (osd.0) 710 : cluster 0 5.3 scrub starts 2024-04-06T03:50:56.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.309375+0000 osd.0 (osd.0) 711 : cluster 0 5.3 scrub ok 2024-04-06T03:50:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: audit 2024-04-06T03:50:55.426177+0000 mon.c (mon.2) 133 : audit 1 from='client.? 172.21.15.50:0/2651090008' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:50:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: audit 2024-04-06T03:50:55.426784+0000 mon.a (mon.0) 1907 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:50:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.560628+0000 osd.7 (osd.7) 731 : cluster 0 3.13 scrub starts 2024-04-06T03:50:56.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:55 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.565006+0000 osd.7 (osd.7) 732 : cluster 0 3.13 scrub ok 2024-04-06T03:50:56.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 6 1 2024-04-06T03:50:56.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.380188+0000 osd.1 (osd.1) 487 : cluster 0 6.4 scrub starts 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.382315+0000 osd.1 (osd.1) 488 : cluster 0 6.4 scrub ok 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[29221]: audit 2024-04-06T03:50:55.615919+0000 mon.a (mon.0) 1908 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[29221]: cluster 2024-04-06T03:50:55.624314+0000 mon.a (mon.0) 1909 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[29221]: cluster 2024-04-06T03:50:56.459584+0000 osd.4 (osd.4) 693 : cluster 0 5.1c scrub starts 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[29221]: cluster 2024-04-06T03:50:56.462120+0000 osd.4 (osd.4) 694 : cluster 0 5.1c scrub ok 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.380188+0000 osd.1 (osd.1) 487 : cluster 0 6.4 scrub starts 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.382315+0000 osd.1 (osd.1) 488 : cluster 0 6.4 scrub ok 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[21467]: audit 2024-04-06T03:50:55.615919+0000 mon.a (mon.0) 1908 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[21467]: cluster 2024-04-06T03:50:55.624314+0000 mon.a (mon.0) 1909 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[21467]: cluster 2024-04-06T03:50:56.459584+0000 osd.4 (osd.4) 693 : cluster 0 5.1c scrub starts 2024-04-06T03:50:56.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:56 smithi050 bash[21467]: cluster 2024-04-06T03:50:56.462120+0000 osd.4 (osd.4) 694 : cluster 0 5.1c scrub ok 2024-04-06T03:50:57.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:56 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.380188+0000 osd.1 (osd.1) 487 : cluster 0 6.4 scrub starts 2024-04-06T03:50:57.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:56 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.382315+0000 osd.1 (osd.1) 488 : cluster 0 6.4 scrub ok 2024-04-06T03:50:57.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:56 smithi153 bash[23852]: audit 2024-04-06T03:50:55.615919+0000 mon.a (mon.0) 1908 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:50:57.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:56 smithi153 bash[23852]: cluster 2024-04-06T03:50:55.624314+0000 mon.a (mon.0) 1909 : cluster 0 osdmap e432: 8 total, 8 up, 8 in 2024-04-06T03:50:57.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:56 smithi153 bash[23852]: cluster 2024-04-06T03:50:56.459584+0000 osd.4 (osd.4) 693 : cluster 0 5.1c scrub starts 2024-04-06T03:50:57.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:56 smithi153 bash[23852]: cluster 2024-04-06T03:50:56.462120+0000 osd.4 (osd.4) 694 : cluster 0 5.1c scrub ok 2024-04-06T03:50:57.282 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:50:57.282 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:50:57.282 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:50:57.282 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:50:57.282 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:50:57.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:57 smithi050 bash[29221]: cluster 2024-04-06T03:50:56.669842+0000 mgr.y (mgr.24368) 1099 : cluster 0 pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-04-06T03:50:57.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:57 smithi050 bash[29221]: cluster 2024-04-06T03:50:57.434224+0000 osd.4 (osd.4) 695 : cluster 0 5.e scrub starts 2024-04-06T03:50:57.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:57 smithi050 bash[29221]: cluster 2024-04-06T03:50:57.436728+0000 osd.4 (osd.4) 696 : cluster 0 5.e scrub ok 2024-04-06T03:50:57.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:57 smithi050 bash[21467]: cluster 2024-04-06T03:50:56.669842+0000 mgr.y (mgr.24368) 1099 : cluster 0 pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-04-06T03:50:57.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:57 smithi050 bash[21467]: cluster 2024-04-06T03:50:57.434224+0000 osd.4 (osd.4) 695 : cluster 0 5.e scrub starts 2024-04-06T03:50:57.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:57 smithi050 bash[21467]: cluster 2024-04-06T03:50:57.436728+0000 osd.4 (osd.4) 696 : cluster 0 5.e scrub ok 2024-04-06T03:50:58.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:57 smithi153 bash[23852]: cluster 2024-04-06T03:50:56.669842+0000 mgr.y (mgr.24368) 1099 : cluster 0 pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-04-06T03:50:58.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:57 smithi153 bash[23852]: cluster 2024-04-06T03:50:57.434224+0000 osd.4 (osd.4) 695 : cluster 0 5.e scrub starts 2024-04-06T03:50:58.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:57 smithi153 bash[23852]: cluster 2024-04-06T03:50:57.436728+0000 osd.4 (osd.4) 696 : cluster 0 5.e scrub ok 2024-04-06T03:50:58.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.6 dump_historic_ops 2024-04-06T03:50:58.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:58 smithi050 bash[21467]: cluster 2024-04-06T03:50:58.388956+0000 osd.4 (osd.4) 697 : cluster 0 6.1a scrub starts 2024-04-06T03:50:58.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:58 smithi050 bash[21467]: cluster 2024-04-06T03:50:58.468015+0000 osd.4 (osd.4) 698 : cluster 0 6.1a scrub ok 2024-04-06T03:50:58.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:58 smithi050 bash[29221]: cluster 2024-04-06T03:50:58.388956+0000 osd.4 (osd.4) 697 : cluster 0 6.1a scrub starts 2024-04-06T03:50:58.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:58 smithi050 bash[29221]: cluster 2024-04-06T03:50:58.468015+0000 osd.4 (osd.4) 698 : cluster 0 6.1a scrub ok 2024-04-06T03:50:59.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:58 smithi153 bash[23852]: cluster 2024-04-06T03:50:58.388956+0000 osd.4 (osd.4) 697 : cluster 0 6.1a scrub starts 2024-04-06T03:50:59.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:58 smithi153 bash[23852]: cluster 2024-04-06T03:50:58.468015+0000 osd.4 (osd.4) 698 : cluster 0 6.1a scrub ok 2024-04-06T03:50:59.654 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:50:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:50:59.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:58.670859+0000 mgr.y (mgr.24368) 1100 : cluster 0 pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-06T03:50:59.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.363326+0000 osd.4 (osd.4) 699 : cluster 0 3.a scrub starts 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.366389+0000 osd.2 (osd.2) 595 : cluster 0 6.14 scrub starts 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.369484+0000 osd.4 (osd.4) 700 : cluster 0 3.a scrub ok 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.369722+0000 osd.2 (osd.2) 596 : cluster 0 6.14 scrub ok 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.521213+0000 osd.7 (osd.7) 733 : cluster 0 3.11 scrub starts 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.527058+0000 osd.7 (osd.7) 734 : cluster 0 3.11 scrub ok 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:58.670859+0000 mgr.y (mgr.24368) 1100 : cluster 0 pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.363326+0000 osd.4 (osd.4) 699 : cluster 0 3.a scrub starts 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.366389+0000 osd.2 (osd.2) 595 : cluster 0 6.14 scrub starts 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.369484+0000 osd.4 (osd.4) 700 : cluster 0 3.a scrub ok 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.369722+0000 osd.2 (osd.2) 596 : cluster 0 6.14 scrub ok 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.521213+0000 osd.7 (osd.7) 733 : cluster 0 3.11 scrub starts 2024-04-06T03:50:59.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:50:59 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.527058+0000 osd.7 (osd.7) 734 : cluster 0 3.11 scrub ok 2024-04-06T03:51:00.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:58.670859+0000 mgr.y (mgr.24368) 1100 : cluster 0 pgmap v1212: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-06T03:51:00.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.363326+0000 osd.4 (osd.4) 699 : cluster 0 3.a scrub starts 2024-04-06T03:51:00.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.366389+0000 osd.2 (osd.2) 595 : cluster 0 6.14 scrub starts 2024-04-06T03:51:00.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.369484+0000 osd.4 (osd.4) 700 : cluster 0 3.a scrub ok 2024-04-06T03:51:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.369722+0000 osd.2 (osd.2) 596 : cluster 0 6.14 scrub ok 2024-04-06T03:51:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.521213+0000 osd.7 (osd.7) 733 : cluster 0 3.11 scrub starts 2024-04-06T03:51:00.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:50:59 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.527058+0000 osd.7 (osd.7) 734 : cluster 0 3.11 scrub ok 2024-04-06T03:51:00.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:51:00.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.218597+0000 osd.3 (osd.3) 725 : cluster 0 6.13 scrub starts 2024-04-06T03:51:00.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.373362+0000 osd.3 (osd.3) 726 : cluster 0 6.13 scrub ok 2024-04-06T03:51:00.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.436464+0000 osd.1 (osd.1) 489 : cluster 0 5.c deep-scrub starts 2024-04-06T03:51:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:50:59.438931+0000 osd.1 (osd.1) 490 : cluster 0 5.c deep-scrub ok 2024-04-06T03:51:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:00.489879+0000 osd.7 (osd.7) 735 : cluster 0 3.15 deep-scrub starts 2024-04-06T03:51:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:00.500689+0000 osd.7 (osd.7) 736 : cluster 0 3.15 deep-scrub ok 2024-04-06T03:51:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:00.536893+0000 osd.6 (osd.6) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:00.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:00.536990+0000 osd.6 (osd.6) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:00.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.218597+0000 osd.3 (osd.3) 725 : cluster 0 6.13 scrub starts 2024-04-06T03:51:00.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.373362+0000 osd.3 (osd.3) 726 : cluster 0 6.13 scrub ok 2024-04-06T03:51:00.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.436464+0000 osd.1 (osd.1) 489 : cluster 0 5.c deep-scrub starts 2024-04-06T03:51:00.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:50:59.438931+0000 osd.1 (osd.1) 490 : cluster 0 5.c deep-scrub ok 2024-04-06T03:51:00.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:00.489879+0000 osd.7 (osd.7) 735 : cluster 0 3.15 deep-scrub starts 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:00.500689+0000 osd.7 (osd.7) 736 : cluster 0 3.15 deep-scrub ok 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:00.536893+0000 osd.6 (osd.6) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:00.536990+0000 osd.6 (osd.6) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.218597+0000 osd.3 (osd.3) 725 : cluster 0 6.13 scrub starts 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.373362+0000 osd.3 (osd.3) 726 : cluster 0 6.13 scrub ok 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.436464+0000 osd.1 (osd.1) 489 : cluster 0 5.c deep-scrub starts 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:50:59.438931+0000 osd.1 (osd.1) 490 : cluster 0 5.c deep-scrub ok 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:00.489879+0000 osd.7 (osd.7) 735 : cluster 0 3.15 deep-scrub starts 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:00.500689+0000 osd.7 (osd.7) 736 : cluster 0 3.15 deep-scrub ok 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:00.536893+0000 osd.6 (osd.6) 117 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:00.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:00.536990+0000 osd.6 (osd.6) 118 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:01.331 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:01.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[21467]: audit 2024-04-06T03:51:00.520083+0000 mgr.y (mgr.24368) 1101 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:00.672663+0000 mgr.y (mgr.24368) 1102 : cluster 0 pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:00.673473+0000 mon.a (mon.0) 1910 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:01.070892+0000 osd.5 (osd.5) 165 : cluster 0 3.c scrub starts 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:01.074432+0000 osd.5 (osd.5) 166 : cluster 0 3.c scrub ok 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[29221]: audit 2024-04-06T03:51:00.520083+0000 mgr.y (mgr.24368) 1101 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:00.672663+0000 mgr.y (mgr.24368) 1102 : cluster 0 pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:00.673473+0000 mon.a (mon.0) 1910 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:01.070892+0000 osd.5 (osd.5) 165 : cluster 0 3.c scrub starts 2024-04-06T03:51:01.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:01.074432+0000 osd.5 (osd.5) 166 : cluster 0 3.c scrub ok 2024-04-06T03:51:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:01 smithi153 bash[23852]: audit 2024-04-06T03:51:00.520083+0000 mgr.y (mgr.24368) 1101 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:00.672663+0000 mgr.y (mgr.24368) 1102 : cluster 0 pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-06T03:51:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:00.673473+0000 mon.a (mon.0) 1910 : cluster 0 osdmap e433: 8 total, 8 up, 8 in 2024-04-06T03:51:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:01.070892+0000 osd.5 (osd.5) 165 : cluster 0 3.c scrub starts 2024-04-06T03:51:02.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:01.074432+0000 osd.5 (osd.5) 166 : cluster 0 3.c scrub ok 2024-04-06T03:51:02.842 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:51:02.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.366467+0000 osd.4 (osd.4) 701 : cluster 0 4.1 scrub starts 2024-04-06T03:51:02.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.368259+0000 osd.4 (osd.4) 702 : cluster 0 4.1 scrub ok 2024-04-06T03:51:02.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.454561+0000 osd.7 (osd.7) 737 : cluster 0 6.1c scrub starts 2024-04-06T03:51:02.973 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.457929+0000 osd.7 (osd.7) 738 : cluster 0 6.1c scrub ok 2024-04-06T03:51:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.366467+0000 osd.4 (osd.4) 701 : cluster 0 4.1 scrub starts 2024-04-06T03:51:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.368259+0000 osd.4 (osd.4) 702 : cluster 0 4.1 scrub ok 2024-04-06T03:51:02.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.454561+0000 osd.7 (osd.7) 737 : cluster 0 6.1c scrub starts 2024-04-06T03:51:02.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:02 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.457929+0000 osd.7 (osd.7) 738 : cluster 0 6.1c scrub ok 2024-04-06T03:51:03.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:02 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.366467+0000 osd.4 (osd.4) 701 : cluster 0 4.1 scrub starts 2024-04-06T03:51:03.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:02 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.368259+0000 osd.4 (osd.4) 702 : cluster 0 4.1 scrub ok 2024-04-06T03:51:03.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:02 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.454561+0000 osd.7 (osd.7) 737 : cluster 0 6.1c scrub starts 2024-04-06T03:51:03.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:02 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.457929+0000 osd.7 (osd.7) 738 : cluster 0 6.1c scrub ok 2024-04-06T03:51:03.951 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.393934+0000 osd.1 (osd.1) 491 : cluster 0 4.2 scrub starts 2024-04-06T03:51:03.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.397248+0000 osd.1 (osd.1) 492 : cluster 0 4.2 scrub ok 2024-04-06T03:51:03.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:02.673306+0000 mgr.y (mgr.24368) 1103 : cluster 0 pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-06T03:51:03.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.062913+0000 osd.5 (osd.5) 167 : cluster 0 5.16 scrub starts 2024-04-06T03:51:03.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.065621+0000 osd.5 (osd.5) 168 : cluster 0 5.16 scrub ok 2024-04-06T03:51:03.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.356321+0000 osd.4 (osd.4) 703 : cluster 0 3.9 scrub starts 2024-04-06T03:51:03.952 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:03 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.360831+0000 osd.4 (osd.4) 704 : cluster 0 3.9 scrub ok 2024-04-06T03:51:03.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.393934+0000 osd.1 (osd.1) 491 : cluster 0 4.2 scrub starts 2024-04-06T03:51:03.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.397248+0000 osd.1 (osd.1) 492 : cluster 0 4.2 scrub ok 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:02.673306+0000 mgr.y (mgr.24368) 1103 : cluster 0 pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.062913+0000 osd.5 (osd.5) 167 : cluster 0 5.16 scrub starts 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.065621+0000 osd.5 (osd.5) 168 : cluster 0 5.16 scrub ok 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.356321+0000 osd.4 (osd.4) 703 : cluster 0 3.9 scrub starts 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.360831+0000 osd.4 (osd.4) 704 : cluster 0 3.9 scrub ok 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.393934+0000 osd.1 (osd.1) 491 : cluster 0 4.2 scrub starts 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.397248+0000 osd.1 (osd.1) 492 : cluster 0 4.2 scrub ok 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:02.673306+0000 mgr.y (mgr.24368) 1103 : cluster 0 pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.062913+0000 osd.5 (osd.5) 167 : cluster 0 5.16 scrub starts 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.065621+0000 osd.5 (osd.5) 168 : cluster 0 5.16 scrub ok 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.356321+0000 osd.4 (osd.4) 703 : cluster 0 3.9 scrub starts 2024-04-06T03:51:03.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:03 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.360831+0000 osd.4 (osd.4) 704 : cluster 0 3.9 scrub ok 2024-04-06T03:51:04.957 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:04 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.146999+0000 osd.3 (osd.3) 727 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:04.957 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:04 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.147092+0000 osd.3 (osd.3) 728 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:04.957 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:04 smithi153 bash[23852]: cluster 2024-04-06T03:51:03.692659+0000 mon.a (mon.0) 1911 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-04-06T03:51:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:04 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.146999+0000 osd.3 (osd.3) 727 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:04 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.147092+0000 osd.3 (osd.3) 728 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:04.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:04 smithi050 bash[21467]: cluster 2024-04-06T03:51:03.692659+0000 mon.a (mon.0) 1911 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-04-06T03:51:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:04 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.146999+0000 osd.3 (osd.3) 727 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:04 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.147092+0000 osd.3 (osd.3) 728 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:04.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:04 smithi050 bash[29221]: cluster 2024-04-06T03:51:03.692659+0000 mon.a (mon.0) 1911 : cluster 0 osdmap e434: 8 total, 8 up, 8 in 2024-04-06T03:51:05.714 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-06T03:51:05.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: cluster 2024-04-06T03:51:04.313177+0000 osd.3 (osd.3) 729 : cluster 0 5.13 scrub starts 2024-04-06T03:51:05.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: cluster 2024-04-06T03:51:04.315583+0000 osd.3 (osd.3) 730 : cluster 0 5.13 scrub ok 2024-04-06T03:51:05.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: cluster 2024-04-06T03:51:04.673902+0000 mgr.y (mgr.24368) 1104 : cluster 0 pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 61 op/s 2024-04-06T03:51:05.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: cluster 2024-04-06T03:51:04.866420+0000 osd.5 (osd.5) 169 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: cluster 2024-04-06T03:51:04.866540+0000 osd.5 (osd.5) 170 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: audit 2024-04-06T03:51:05.063534+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.50:0/608215401' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[21467]: audit 2024-04-06T03:51:05.064299+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: cluster 2024-04-06T03:51:04.313177+0000 osd.3 (osd.3) 729 : cluster 0 5.13 scrub starts 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: cluster 2024-04-06T03:51:04.315583+0000 osd.3 (osd.3) 730 : cluster 0 5.13 scrub ok 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: cluster 2024-04-06T03:51:04.673902+0000 mgr.y (mgr.24368) 1104 : cluster 0 pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 61 op/s 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: cluster 2024-04-06T03:51:04.866420+0000 osd.5 (osd.5) 169 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: cluster 2024-04-06T03:51:04.866540+0000 osd.5 (osd.5) 170 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: audit 2024-04-06T03:51:05.063534+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.50:0/608215401' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:51:05.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:05 smithi050 bash[29221]: audit 2024-04-06T03:51:05.064299+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: cluster 2024-04-06T03:51:04.313177+0000 osd.3 (osd.3) 729 : cluster 0 5.13 scrub starts 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: cluster 2024-04-06T03:51:04.315583+0000 osd.3 (osd.3) 730 : cluster 0 5.13 scrub ok 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: cluster 2024-04-06T03:51:04.673902+0000 mgr.y (mgr.24368) 1104 : cluster 0 pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 61 op/s 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: cluster 2024-04-06T03:51:04.866420+0000 osd.5 (osd.5) 169 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: cluster 2024-04-06T03:51:04.866540+0000 osd.5 (osd.5) 170 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: audit 2024-04-06T03:51:05.063534+0000 mon.c (mon.2) 134 : audit 1 from='client.? 172.21.15.50:0/608215401' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:51:06.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:05 smithi153 bash[23852]: audit 2024-04-06T03:51:05.064299+0000 mon.a (mon.0) 1912 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-06T03:51:06.091 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:84174 11.1 e359/345 11:8ca00a49:::benchmark_data_smithi050_138698_object84173:head v 359'5290, mlcod=359'5288)", 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.664313+0000", 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 369.42651740000002, 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.049978880000000003, 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664313+0000", 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664313+0000", 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664314+0000", 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4499999999999995e-07 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664383+0000", 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9108000000000003e-05 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664392+0000", 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0790000000000007e-06 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664398+0000", 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2620000000000004e-06 2024-04-06T03:51:06.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664452+0000", 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3195000000000001e-05 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.664470+0000", 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8094e-05 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.713978+0000", 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.049508792000000003 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.714292+0000", 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000313405 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:84182 11.d e359/345 11:bdae2efb:::benchmark_data_smithi050_138698_object84181:head v 359'5385, mlcod=359'5383)", 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.665640+0000", 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 369.42519067199999, 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053706983, 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.096 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665640+0000", 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665640+0000", 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665640+0000", 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9400000000000005e-07 2024-04-06T03:51:06.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665700+0000", 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9725000000000002e-05 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665710+0000", 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.713e-06 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665716+0000", 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1800000000000001e-06 2024-04-06T03:51:06.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665762+0000", 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6102e-05 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.665781+0000", 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9315000000000001e-05 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.719303+0000", 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.053521256000000003 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.719347+0000", 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3998000000000001e-05 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:84183 11.8 e359/345 11:1b051640:::benchmark_data_smithi050_138698_object84182:head v 359'5424, mlcod=359'5422)", 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.666714+0000", 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 369.42411631900001, 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.050290296999999998, 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.100 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.666714+0000", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.666714+0000", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.666715+0000", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7999999999999995e-07 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.666791+0000", 2024-04-06T03:51:06.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6080000000000003e-05 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.713930+0000", 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.047139604000000002 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.713953+0000", 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2178999999999999e-05 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.714282+0000", 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00032975700000000002 2024-04-06T03:51:06.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.714301+0000", 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8408999999999999e-05 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.716961+0000", 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0026600460000000001 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.717004+0000", 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3541999999999997e-05 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90065 11.9 e359/345 11:9e91a1d6:::benchmark_data_smithi050_138698_object90064:head v 359'5749, mlcod=359'5746)", 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.684724+0000", 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.40610625900001, 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.35591052899999998, 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.104 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.684724+0000", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.684724+0000", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.684724+0000", 2024-04-06T03:51:06.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8700000000000002e-07 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.684772+0000", 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7157000000000003e-05 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.684780+0000", 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.2560000000000002e-06 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.684788+0000", 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9470000000000003e-06 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.687300+0000", 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025122690000000001 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.687320+0000", 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9519999999999999e-05 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695131+0000", 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0078114500000000002 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040635+0000", 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34550364300000003 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90075 11.d e359/345 11:be745510:::benchmark_data_smithi050_138698_object90074:head v 359'5760, mlcod=359'5758)", 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.691060+0000", 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.39977066199998, 2024-04-06T03:51:06.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34958530900000001, 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.691060+0000", 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.691060+0000", 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.691061+0000", 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6899999999999996e-07 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.691100+0000", 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9610999999999997e-05 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.691114+0000", 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4303e-05 2024-04-06T03:51:06.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.691122+0000", 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6459999999999993e-06 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692615+0000", 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001492497 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692629+0000", 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4586999999999999e-05 2024-04-06T03:51:06.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040607+0000", 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.347977488 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040645+0000", 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8408000000000002e-05 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.112 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90078 11.d e359/345 11:b77bf66b:::benchmark_data_smithi050_138698_object90077:head v 359'5761, mlcod=359'5758)", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.692183+0000", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.39864711500002, 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.348493728, 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692183+0000", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692183+0000", 2024-04-06T03:51:06.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692184+0000", 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.0800000000000004e-07 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692228+0000", 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4594e-05 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692237+0000", 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.0199999999999994e-06 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.692242+0000", 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9899999999999997e-06 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694909+0000", 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002667099 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694928+0000", 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.9341e-05 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040651+0000", 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34572277800000001 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040677+0000", 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6298000000000001e-05 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90070 11.1 e359/345 11:896d1544:::benchmark_data_smithi050_138698_object90069:head v 359'5672, mlcod=359'5670)", 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.694888+0000", 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.39594268399998, 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34869023199999999, 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694888+0000", 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694888+0000", 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694888+0000", 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6400000000000001e-07 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694954+0000", 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5251000000000003e-05 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694979+0000", 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5313999999999999e-05 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.694988+0000", 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5429999999999994e-06 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695061+0000", 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3054999999999998e-05 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695082+0000", 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0862000000000001e-05 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.043522+0000", 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34844002899999998 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.043578+0000", 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6413999999999999e-05 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90076 11.a e359/345 11:5fca8c9e:::benchmark_data_smithi050_138698_object90075:head v 359'5574, mlcod=359'5572)", 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.695006+0000", 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.395824232, 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34853129700000002, 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695006+0000", 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695006+0000", 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695007+0000", 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2800000000000002e-07 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695037+0000", 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9909000000000001e-05 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695044+0000", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.8890000000000002e-06 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.695050+0000", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1150000000000002e-06 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040609+0000", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34555900099999998 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040633+0000", 2024-04-06T03:51:06.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.4816000000000001e-05 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.043479+0000", 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0028452870000000002 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.043537+0000", 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8851999999999998e-05 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90080 11.3 e359/345 11:c39070ca:::benchmark_data_smithi050_138698_object90079:head v 359'5626, mlcod=359'5624)", 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.696226+0000", 2024-04-06T03:51:06.124 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.39460458899998, 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.350701449, 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.696226+0000", 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.696226+0000", 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.696227+0000", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.6700000000000002e-07 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.696247+0000", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0237e-05 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040580+0000", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34433329400000001 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040587+0000", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2060000000000004e-06 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040655+0000", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8201000000000007e-05 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040678+0000", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2869000000000001e-05 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.046886+0000", 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0062087039999999998 2024-04-06T03:51:06.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.046927+0000", 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0970999999999998e-05 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:90086 11.9 e359/345 11:9ced346e:::benchmark_data_smithi050_138698_object90085:head v 359'5750, mlcod=359'5747)", 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:45:03.699475+0000", 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 362.39135551599998, 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.348485712, 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.699475+0000", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.699475+0000", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.699476+0000", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.9100000000000003e-07 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:03.699542+0000", 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6678000000000001e-05 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040561+0000", 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.34101833799999998 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040814+0000", 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025357100000000002 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040865+0000", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1001999999999997e-05 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.040885+0000", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.929e-05 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.047931+0000", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0070467359999999996 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:45:04.047961+0000", 2024-04-06T03:51:06.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9306e-05 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:183157 11.9 e376/345 11:97129759:::benchmark_data_smithi050_138698_object77183:head v 376'11658, mlcod=376'11653)", 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:43.714987+0000", 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 262.37584383199999, 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.072561343, 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.714987+0000", 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.714987+0000", 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.714987+0000", 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1500000000000003e-07 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.714994+0000", 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1870000000000002e-06 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.715004+0000", 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0256e-05 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.715011+0000", 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9009999999999997e-06 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.715053+0000", 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2252000000000001e-05 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.715071+0000", 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7839999999999999e-05 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.718886+0000", 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0038150319999999999 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787548+0000", 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.068662059999999997 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:183154 11.8 e376/345 11:17e17993:::benchmark_data_smithi050_138698_object77180:head v 376'11726, mlcod=376'11722)", 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:43.716643+0000", 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 262.37418757, 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.070919880000000005, 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.136 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716643+0000", 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716643+0000", 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716644+0000", 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5499999999999996e-07 2024-04-06T03:51:06.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716652+0000", 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1839999999999999e-06 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716662+0000", 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0096e-05 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716669+0000", 2024-04-06T03:51:06.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.0990000000000001e-06 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716736+0000", 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6883000000000003e-05 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.716754+0000", 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7737e-05 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787468+0000", 2024-04-06T03:51:06.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.070714677000000004 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787563+0000", 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4248999999999998e-05 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:183156 11.3 e376/345 11:ce8cb4ff:::benchmark_data_smithi050_138698_object77182:head v 376'11483, mlcod=376'11481)", 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:43.717404+0000", 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 262.37342688000001, 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.070079052000000003, 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.140 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717404+0000", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717404+0000", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717404+0000", 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.61e-07 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717412+0000", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2219999999999996e-06 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717422+0000", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0611000000000001e-05 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717429+0000", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.2660000000000004e-06 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717505+0000", 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5449000000000002e-05 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.717523+0000", 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.836e-05 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787426+0000", 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.069903229999999997 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787483+0000", 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6153000000000003e-05 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:183158 11.3 e376/345 11:cff95ce7:::benchmark_data_smithi050_138698_object77184:head v 376'11484, mlcod=376'11481)", 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:43.719096+0000", 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 262.371734796, 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.075440502000000006, 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.144 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719096+0000", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719096+0000", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719096+0000", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.23e-07 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719102+0000", 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7640000000000002e-06 2024-04-06T03:51:06.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787411+0000", 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.068308905000000003 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787427+0000", 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6266999999999999e-05 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787529+0000", 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000101389 2024-04-06T03:51:06.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787571+0000", 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.2431000000000001e-05 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.794373+0000", 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0068014499999999997 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.794536+0000", 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000163573 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:183159 11.8 e376/345 11:19257d24:::benchmark_data_smithi050_138698_object77185:head v 376'11727, mlcod=376'11722)", 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:43.719228+0000", 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 262.37160218899999, 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.076751113999999995, 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719228+0000", 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719228+0000", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719229+0000", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6899999999999997e-07 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.719236+0000", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6790000000000005e-06 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787422+0000", 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.068185348000000007 2024-04-06T03:51:06.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787440+0000", 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8213999999999999e-05 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787642+0000", 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000201654 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.787690+0000", 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7982000000000002e-05 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.795802+0000", 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0081123270000000008 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:43.795979+0000", 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00017734099999999999 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:192987 11.d e377/345 11:b13141bd:::benchmark_data_smithi050_138698_object87013:head v 377'12307, mlcod=377'12303)", 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:53.149533+0000", 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 252.941296967, 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.22604713300000001, 2024-04-06T03:51:06.151 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.149533+0000", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.149533+0000", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.149534+0000", 2024-04-06T03:51:06.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1600000000000001e-07 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.149539+0000", 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0919999999999997e-06 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.149548+0000", 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5299999999999996e-06 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.149553+0000", 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.9699999999999998e-06 2024-04-06T03:51:06.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151364+0000", 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0018115239999999999 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151383+0000", 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8802000000000001e-05 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.375529+0000", 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.224145698 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.375581+0000", 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1901000000000003e-05 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:192991 11.9 e377/345 11:927e3ed3:::benchmark_data_smithi050_138698_object87017:head v 377'12306, mlcod=377'12304)", 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:53.151115+0000", 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 252.93971584100001, 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21338586100000001, 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151115+0000", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151115+0000", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151115+0000", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7700000000000004e-07 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151124+0000", 2024-04-06T03:51:06.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0580000000000004e-06 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151131+0000", 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.7179999999999999e-06 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151136+0000", 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2800000000000003e-06 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151183+0000", 2024-04-06T03:51:06.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6770000000000001e-05 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151197+0000", 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4453e-05 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.364454+0000", 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21325618499999999 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.364500+0000", 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6820000000000002e-05 2024-04-06T03:51:06.158 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:192992 11.1 e377/345 11:8eab93be:::benchmark_data_smithi050_138698_object87018:head v 377'12128, mlcod=377'12126)", 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:53.151231+0000", 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 252.93959893600001, 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21330141499999999, 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151231+0000", 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151231+0000", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151232+0000", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0999999999999999e-07 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151237+0000", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.6959999999999996e-06 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151245+0000", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3799999999999994e-06 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151251+0000", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6189999999999999e-06 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151332+0000", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1328000000000005e-05 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.151350+0000", 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.7683999999999999e-05 2024-04-06T03:51:06.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.364505+0000", 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21315517 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.364533+0000", 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8028e-05 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.162 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:193001 11.d e377/345 11:b539e2c9:::benchmark_data_smithi050_138698_object87027:head v 377'12308, mlcod=377'12305)", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:53.156036+0000", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 252.93479413899999, 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.21957344200000001, 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.156036+0000", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.156036+0000", 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.156037+0000", 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.3799999999999996e-07 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.156043+0000", 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1290000000000001e-06 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.365097+0000", 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.209053556 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.365113+0000", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5854000000000001e-05 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.370445+0000", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0053326049999999998 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.370714+0000", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026891300000000002 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.375586+0000", 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0048720539999999998 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:53.375610+0000", 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3592999999999999e-05 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_repop(client.15498.0:193690 11.8 e377/345 11:120ab871:::benchmark_data_smithi050_138698_object87716:head v 377'12418, mlcod=377'12416)", 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:54.264249+0000", 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 251.82658148300001, 2024-04-06T03:51:06.166 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.18319665299999999, 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264249+0000", 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264249+0000", 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.167 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264250+0000", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3370000000000001e-06 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264265+0000", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4250000000000001e-05 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264292+0000", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7912e-05 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264297+0000", 2024-04-06T03:51:06.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.8239999999999999e-06 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264337+0000", 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9734000000000003e-05 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.264356+0000", 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8995e-05 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.266834+0000", 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0024775940000000001 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:54.447446+0000", 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.18061200699999999 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:51:06.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd reweight 7 1 2024-04-06T03:51:06.865 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-06T03:51:06.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:05.438737+0000 osd.1 (osd.1) 493 : cluster 0 3.19 scrub starts 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:05.444970+0000 osd.1 (osd.1) 494 : cluster 0 3.19 scrub ok 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: audit 2024-04-06T03:51:05.702092+0000 mon.a (mon.0) 1913 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:05.710266+0000 mon.a (mon.0) 1914 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: audit 2024-04-06T03:51:06.065956+0000 mon.a (mon.0) 1915 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.095864+0000 osd.5 (osd.5) 171 : cluster 0 6.19 scrub starts 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.196654+0000 osd.5 (osd.5) 172 : cluster 0 6.19 scrub ok 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.476708+0000 osd.7 (osd.7) 739 : cluster 0 3.d scrub starts 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.482004+0000 osd.7 (osd.7) 740 : cluster 0 3.d scrub ok 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.712131+0000 mon.a (mon.0) 1916 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-04-06T03:51:06.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:05.438737+0000 osd.1 (osd.1) 493 : cluster 0 3.19 scrub starts 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:05.444970+0000 osd.1 (osd.1) 494 : cluster 0 3.19 scrub ok 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: audit 2024-04-06T03:51:05.702092+0000 mon.a (mon.0) 1913 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:05.710266+0000 mon.a (mon.0) 1914 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: audit 2024-04-06T03:51:06.065956+0000 mon.a (mon.0) 1915 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.095864+0000 osd.5 (osd.5) 171 : cluster 0 6.19 scrub starts 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.196654+0000 osd.5 (osd.5) 172 : cluster 0 6.19 scrub ok 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.476708+0000 osd.7 (osd.7) 739 : cluster 0 3.d scrub starts 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.482004+0000 osd.7 (osd.7) 740 : cluster 0 3.d scrub ok 2024-04-06T03:51:06.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:06 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.712131+0000 mon.a (mon.0) 1916 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:05.438737+0000 osd.1 (osd.1) 493 : cluster 0 3.19 scrub starts 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:05.444970+0000 osd.1 (osd.1) 494 : cluster 0 3.19 scrub ok 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: audit 2024-04-06T03:51:05.702092+0000 mon.a (mon.0) 1913 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:05.710266+0000 mon.a (mon.0) 1914 : cluster 0 osdmap e435: 8 total, 8 up, 8 in 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: audit 2024-04-06T03:51:06.065956+0000 mon.a (mon.0) 1915 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.095864+0000 osd.5 (osd.5) 171 : cluster 0 6.19 scrub starts 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.196654+0000 osd.5 (osd.5) 172 : cluster 0 6.19 scrub ok 2024-04-06T03:51:07.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.476708+0000 osd.7 (osd.7) 739 : cluster 0 3.d scrub starts 2024-04-06T03:51:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.482004+0000 osd.7 (osd.7) 740 : cluster 0 3.d scrub ok 2024-04-06T03:51:07.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:06 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.712131+0000 mon.a (mon.0) 1916 : cluster 0 osdmap e436: 8 total, 8 up, 8 in 2024-04-06T03:51:07.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:07 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.316754+0000 osd.3 (osd.3) 731 : cluster 0 2.19 deep-scrub starts 2024-04-06T03:51:07.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:07 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.320127+0000 osd.3 (osd.3) 732 : cluster 0 2.19 deep-scrub ok 2024-04-06T03:51:07.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:07 smithi050 bash[29221]: cluster 2024-04-06T03:51:06.674511+0000 mgr.y (mgr.24368) 1105 : cluster 0 pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-04-06T03:51:07.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:07 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.316754+0000 osd.3 (osd.3) 731 : cluster 0 2.19 deep-scrub starts 2024-04-06T03:51:07.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:07 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.320127+0000 osd.3 (osd.3) 732 : cluster 0 2.19 deep-scrub ok 2024-04-06T03:51:07.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:07 smithi050 bash[21467]: cluster 2024-04-06T03:51:06.674511+0000 mgr.y (mgr.24368) 1105 : cluster 0 pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-04-06T03:51:08.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:07 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.316754+0000 osd.3 (osd.3) 731 : cluster 0 2.19 deep-scrub starts 2024-04-06T03:51:08.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:07 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.320127+0000 osd.3 (osd.3) 732 : cluster 0 2.19 deep-scrub ok 2024-04-06T03:51:08.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:07 smithi153 bash[23852]: cluster 2024-04-06T03:51:06.674511+0000 mgr.y (mgr.24368) 1105 : cluster 0 pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-04-06T03:51:08.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[21467]: cluster 2024-04-06T03:51:07.365785+0000 osd.3 (osd.3) 733 : cluster 0 6.11 scrub starts 2024-04-06T03:51:08.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[21467]: cluster 2024-04-06T03:51:07.421510+0000 osd.3 (osd.3) 734 : cluster 0 6.11 scrub ok 2024-04-06T03:51:08.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[21467]: cluster 2024-04-06T03:51:08.197659+0000 osd.0 (osd.0) 712 : cluster 0 6.18 scrub starts 2024-04-06T03:51:08.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[21467]: cluster 2024-04-06T03:51:08.300675+0000 osd.0 (osd.0) 713 : cluster 0 6.18 scrub ok 2024-04-06T03:51:08.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[21467]: cluster 2024-04-06T03:51:08.409156+0000 osd.7 (osd.7) 741 : cluster 0 6.12 scrub starts 2024-04-06T03:51:09.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:08 smithi153 bash[23852]: cluster 2024-04-06T03:51:07.365785+0000 osd.3 (osd.3) 733 : cluster 0 6.11 scrub starts 2024-04-06T03:51:09.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:08 smithi153 bash[23852]: cluster 2024-04-06T03:51:07.421510+0000 osd.3 (osd.3) 734 : cluster 0 6.11 scrub ok 2024-04-06T03:51:09.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:08 smithi153 bash[23852]: cluster 2024-04-06T03:51:08.197659+0000 osd.0 (osd.0) 712 : cluster 0 6.18 scrub starts 2024-04-06T03:51:09.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:08 smithi153 bash[23852]: cluster 2024-04-06T03:51:08.300675+0000 osd.0 (osd.0) 713 : cluster 0 6.18 scrub ok 2024-04-06T03:51:09.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:08 smithi153 bash[23852]: cluster 2024-04-06T03:51:08.409156+0000 osd.7 (osd.7) 741 : cluster 0 6.12 scrub starts 2024-04-06T03:51:09.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:08 smithi153 bash[23852]: cluster 2024-04-06T03:51:08.451552+0000 osd.7 (osd.7) 742 : cluster 0 6.12 scrub ok 2024-04-06T03:51:09.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[21467]: cluster 2024-04-06T03:51:08.451552+0000 osd.7 (osd.7) 742 : cluster 0 6.12 scrub ok 2024-04-06T03:51:09.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[29221]: cluster 2024-04-06T03:51:07.365785+0000 osd.3 (osd.3) 733 : cluster 0 6.11 scrub starts 2024-04-06T03:51:09.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[29221]: cluster 2024-04-06T03:51:07.421510+0000 osd.3 (osd.3) 734 : cluster 0 6.11 scrub ok 2024-04-06T03:51:09.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[29221]: cluster 2024-04-06T03:51:08.197659+0000 osd.0 (osd.0) 712 : cluster 0 6.18 scrub starts 2024-04-06T03:51:09.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[29221]: cluster 2024-04-06T03:51:08.300675+0000 osd.0 (osd.0) 713 : cluster 0 6.18 scrub ok 2024-04-06T03:51:09.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[29221]: cluster 2024-04-06T03:51:08.409156+0000 osd.7 (osd.7) 741 : cluster 0 6.12 scrub starts 2024-04-06T03:51:09.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:08 smithi050 bash[29221]: cluster 2024-04-06T03:51:08.451552+0000 osd.7 (osd.7) 742 : cluster 0 6.12 scrub ok 2024-04-06T03:51:09.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:51:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:51:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:51:10.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:09 smithi153 bash[23852]: cluster 2024-04-06T03:51:08.675610+0000 mgr.y (mgr.24368) 1106 : cluster 0 pgmap v1221: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-04-06T03:51:10.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:09 smithi153 bash[23852]: cluster 2024-04-06T03:51:09.496171+0000 osd.4 (osd.4) 705 : cluster 0 6.17 scrub starts 2024-04-06T03:51:10.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:09 smithi050 bash[21467]: cluster 2024-04-06T03:51:08.675610+0000 mgr.y (mgr.24368) 1106 : cluster 0 pgmap v1221: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-04-06T03:51:10.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:09 smithi050 bash[21467]: cluster 2024-04-06T03:51:09.496171+0000 osd.4 (osd.4) 705 : cluster 0 6.17 scrub starts 2024-04-06T03:51:10.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:09 smithi050 bash[29221]: cluster 2024-04-06T03:51:08.675610+0000 mgr.y (mgr.24368) 1106 : cluster 0 pgmap v1221: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 94 op/s 2024-04-06T03:51:10.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:09 smithi050 bash[29221]: cluster 2024-04-06T03:51:09.496171+0000 osd.4 (osd.4) 705 : cluster 0 6.17 scrub starts 2024-04-06T03:51:10.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:51:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:51:10.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:10 smithi153 bash[23852]: cluster 2024-04-06T03:51:09.705186+0000 osd.4 (osd.4) 706 : cluster 0 6.17 scrub ok 2024-04-06T03:51:10.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:10 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.460991+0000 osd.4 (osd.4) 707 : cluster 0 3.3 scrub starts 2024-04-06T03:51:10.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:10 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.466094+0000 osd.4 (osd.4) 708 : cluster 0 3.3 scrub ok 2024-04-06T03:51:11.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:10 smithi050 bash[21467]: cluster 2024-04-06T03:51:09.705186+0000 osd.4 (osd.4) 706 : cluster 0 6.17 scrub ok 2024-04-06T03:51:11.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:10 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.460991+0000 osd.4 (osd.4) 707 : cluster 0 3.3 scrub starts 2024-04-06T03:51:11.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:10 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.466094+0000 osd.4 (osd.4) 708 : cluster 0 3.3 scrub ok 2024-04-06T03:51:11.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:10 smithi050 bash[29221]: cluster 2024-04-06T03:51:09.705186+0000 osd.4 (osd.4) 706 : cluster 0 6.17 scrub ok 2024-04-06T03:51:11.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:10 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.460991+0000 osd.4 (osd.4) 707 : cluster 0 3.3 scrub starts 2024-04-06T03:51:11.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:10 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.466094+0000 osd.4 (osd.4) 708 : cluster 0 3.3 scrub ok 2024-04-06T03:51:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:11.651 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:51:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.384256+0000 osd.3 (osd.3) 735 : cluster 0 3.2 scrub starts 2024-04-06T03:51:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.389422+0000 osd.3 (osd.3) 736 : cluster 0 3.2 scrub ok 2024-04-06T03:51:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.430205+0000 osd.1 (osd.1) 495 : cluster 0 2.d scrub starts 2024-04-06T03:51:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.432714+0000 osd.1 (osd.1) 496 : cluster 0 2.d scrub ok 2024-04-06T03:51:12.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: audit 2024-04-06T03:51:10.525999+0000 mgr.y (mgr.24368) 1107 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:12.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.618464+0000 osd.6 (osd.6) 119 : cluster 0 2.17 scrub starts 2024-04-06T03:51:12.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.620258+0000 osd.6 (osd.6) 120 : cluster 0 2.17 scrub ok 2024-04-06T03:51:12.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:10.676942+0000 mgr.y (mgr.24368) 1108 : cluster 0 pgmap v1222: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-04-06T03:51:12.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:11.061867+0000 osd.5 (osd.5) 173 : cluster 0 5.7 scrub starts 2024-04-06T03:51:12.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:11.115772+0000 osd.5 (osd.5) 174 : cluster 0 5.7 scrub ok 2024-04-06T03:51:12.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:11.420883+0000 osd.4 (osd.4) 709 : cluster 0 3.1a scrub starts 2024-04-06T03:51:12.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:11 smithi153 bash[23852]: cluster 2024-04-06T03:51:11.425333+0000 osd.4 (osd.4) 710 : cluster 0 3.1a scrub ok 2024-04-06T03:51:12.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.384256+0000 osd.3 (osd.3) 735 : cluster 0 3.2 scrub starts 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.389422+0000 osd.3 (osd.3) 736 : cluster 0 3.2 scrub ok 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.430205+0000 osd.1 (osd.1) 495 : cluster 0 2.d scrub starts 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.432714+0000 osd.1 (osd.1) 496 : cluster 0 2.d scrub ok 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: audit 2024-04-06T03:51:10.525999+0000 mgr.y (mgr.24368) 1107 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.618464+0000 osd.6 (osd.6) 119 : cluster 0 2.17 scrub starts 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.620258+0000 osd.6 (osd.6) 120 : cluster 0 2.17 scrub ok 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:10.676942+0000 mgr.y (mgr.24368) 1108 : cluster 0 pgmap v1222: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:11.061867+0000 osd.5 (osd.5) 173 : cluster 0 5.7 scrub starts 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:11.115772+0000 osd.5 (osd.5) 174 : cluster 0 5.7 scrub ok 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:11.420883+0000 osd.4 (osd.4) 709 : cluster 0 3.1a scrub starts 2024-04-06T03:51:12.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[21467]: cluster 2024-04-06T03:51:11.425333+0000 osd.4 (osd.4) 710 : cluster 0 3.1a scrub ok 2024-04-06T03:51:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.384256+0000 osd.3 (osd.3) 735 : cluster 0 3.2 scrub starts 2024-04-06T03:51:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.389422+0000 osd.3 (osd.3) 736 : cluster 0 3.2 scrub ok 2024-04-06T03:51:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.430205+0000 osd.1 (osd.1) 495 : cluster 0 2.d scrub starts 2024-04-06T03:51:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.432714+0000 osd.1 (osd.1) 496 : cluster 0 2.d scrub ok 2024-04-06T03:51:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: audit 2024-04-06T03:51:10.525999+0000 mgr.y (mgr.24368) 1107 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:12.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.618464+0000 osd.6 (osd.6) 119 : cluster 0 2.17 scrub starts 2024-04-06T03:51:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.620258+0000 osd.6 (osd.6) 120 : cluster 0 2.17 scrub ok 2024-04-06T03:51:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:10.676942+0000 mgr.y (mgr.24368) 1108 : cluster 0 pgmap v1222: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s 2024-04-06T03:51:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:11.061867+0000 osd.5 (osd.5) 173 : cluster 0 5.7 scrub starts 2024-04-06T03:51:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:11.115772+0000 osd.5 (osd.5) 174 : cluster 0 5.7 scrub ok 2024-04-06T03:51:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:11.420883+0000 osd.4 (osd.4) 709 : cluster 0 3.1a scrub starts 2024-04-06T03:51:12.243 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:11 smithi050 bash[29221]: cluster 2024-04-06T03:51:11.425333+0000 osd.4 (osd.4) 710 : cluster 0 3.1a scrub ok 2024-04-06T03:51:12.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:12 smithi050 bash[21467]: cluster 2024-04-06T03:51:12.464513+0000 osd.4 (osd.4) 711 : cluster 0 6.1e scrub starts 2024-04-06T03:51:12.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:12 smithi050 bash[29221]: cluster 2024-04-06T03:51:12.464513+0000 osd.4 (osd.4) 711 : cluster 0 6.1e scrub starts 2024-04-06T03:51:13.026 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:12 smithi153 bash[23852]: cluster 2024-04-06T03:51:12.464513+0000 osd.4 (osd.4) 711 : cluster 0 6.1e scrub starts 2024-04-06T03:51:13.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[29221]: cluster 2024-04-06T03:51:12.544351+0000 osd.4 (osd.4) 712 : cluster 0 6.1e scrub ok 2024-04-06T03:51:13.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[29221]: cluster 2024-04-06T03:51:12.677738+0000 mgr.y (mgr.24368) 1109 : cluster 0 pgmap v1223: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-06T03:51:13.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[29221]: cluster 2024-04-06T03:51:13.493298+0000 osd.7 (osd.7) 743 : cluster 0 2.1d scrub starts 2024-04-06T03:51:13.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[29221]: cluster 2024-04-06T03:51:13.496398+0000 osd.7 (osd.7) 744 : cluster 0 2.1d scrub ok 2024-04-06T03:51:13.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[21467]: cluster 2024-04-06T03:51:12.544351+0000 osd.4 (osd.4) 712 : cluster 0 6.1e scrub ok 2024-04-06T03:51:13.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[21467]: cluster 2024-04-06T03:51:12.677738+0000 mgr.y (mgr.24368) 1109 : cluster 0 pgmap v1223: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-06T03:51:13.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[21467]: cluster 2024-04-06T03:51:13.493298+0000 osd.7 (osd.7) 743 : cluster 0 2.1d scrub starts 2024-04-06T03:51:13.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:13 smithi050 bash[21467]: cluster 2024-04-06T03:51:13.496398+0000 osd.7 (osd.7) 744 : cluster 0 2.1d scrub ok 2024-04-06T03:51:14.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:13 smithi153 bash[23852]: cluster 2024-04-06T03:51:12.544351+0000 osd.4 (osd.4) 712 : cluster 0 6.1e scrub ok 2024-04-06T03:51:14.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:13 smithi153 bash[23852]: cluster 2024-04-06T03:51:12.677738+0000 mgr.y (mgr.24368) 1109 : cluster 0 pgmap v1223: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-06T03:51:14.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:13 smithi153 bash[23852]: cluster 2024-04-06T03:51:13.493298+0000 osd.7 (osd.7) 743 : cluster 0 2.1d scrub starts 2024-04-06T03:51:14.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:13 smithi153 bash[23852]: cluster 2024-04-06T03:51:13.496398+0000 osd.7 (osd.7) 744 : cluster 0 2.1d scrub ok 2024-04-06T03:51:14.941 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:51:14.942 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:51:14.942 INFO:teuthology.orchestra.run.smithi153.stdout: "num_ops": 0 2024-04-06T03:51:14.942 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:51:15.124 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-06T03:51:15.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.082977+0000 osd.5 (osd.5) 175 : cluster 0 3.1c scrub starts 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.085718+0000 osd.5 (osd.5) 176 : cluster 0 3.1c scrub ok 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.467146+0000 osd.2 (osd.2) 597 : cluster 0 2.1 scrub starts 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.468541+0000 osd.2 (osd.2) 598 : cluster 0 2.1 scrub ok 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.678634+0000 mgr.y (mgr.24368) 1110 : cluster 0 pgmap v1224: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: audit 2024-04-06T03:51:14.815489+0000 mon.c (mon.2) 135 : audit 1 from='client.? 172.21.15.50:0/2289678946' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[29221]: audit 2024-04-06T03:51:14.816063+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.082977+0000 osd.5 (osd.5) 175 : cluster 0 3.1c scrub starts 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.085718+0000 osd.5 (osd.5) 176 : cluster 0 3.1c scrub ok 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.467146+0000 osd.2 (osd.2) 597 : cluster 0 2.1 scrub starts 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.468541+0000 osd.2 (osd.2) 598 : cluster 0 2.1 scrub ok 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.678634+0000 mgr.y (mgr.24368) 1110 : cluster 0 pgmap v1224: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: audit 2024-04-06T03:51:14.815489+0000 mon.c (mon.2) 135 : audit 1 from='client.? 172.21.15.50:0/2289678946' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:15 smithi050 bash[21467]: audit 2024-04-06T03:51:14.816063+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.082977+0000 osd.5 (osd.5) 175 : cluster 0 3.1c scrub starts 2024-04-06T03:51:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.085718+0000 osd.5 (osd.5) 176 : cluster 0 3.1c scrub ok 2024-04-06T03:51:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.467146+0000 osd.2 (osd.2) 597 : cluster 0 2.1 scrub starts 2024-04-06T03:51:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.468541+0000 osd.2 (osd.2) 598 : cluster 0 2.1 scrub ok 2024-04-06T03:51:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.678634+0000 mgr.y (mgr.24368) 1110 : cluster 0 pgmap v1224: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-04-06T03:51:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: audit 2024-04-06T03:51:14.815489+0000 mon.c (mon.2) 135 : audit 1 from='client.? 172.21.15.50:0/2289678946' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:15 smithi153 bash[23852]: audit 2024-04-06T03:51:14.816063+0000 mon.a (mon.0) 1917 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:15.696 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_blocked_ops 2024-04-06T03:51:15.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd primary-affinity 7 1 2024-04-06T03:51:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.302024+0000 osd.1 (osd.1) 497 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:14.302131+0000 osd.1 (osd.1) 498 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: audit 2024-04-06T03:51:15.110710+0000 mon.a (mon.0) 1918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:15.118496+0000 mon.a (mon.0) 1919 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:15.443202+0000 osd.3 (osd.3) 737 : cluster 0 5.0 scrub starts 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:15.447410+0000 osd.3 (osd.3) 738 : cluster 0 5.0 scrub ok 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:15.509914+0000 osd.2 (osd.2) 599 : cluster 0 5.d scrub starts 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[21467]: cluster 2024-04-06T03:51:15.513257+0000 osd.2 (osd.2) 600 : cluster 0 5.d scrub ok 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.302024+0000 osd.1 (osd.1) 497 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:14.302131+0000 osd.1 (osd.1) 498 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: audit 2024-04-06T03:51:15.110710+0000 mon.a (mon.0) 1918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:15.118496+0000 mon.a (mon.0) 1919 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:15.443202+0000 osd.3 (osd.3) 737 : cluster 0 5.0 scrub starts 2024-04-06T03:51:16.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:15.447410+0000 osd.3 (osd.3) 738 : cluster 0 5.0 scrub ok 2024-04-06T03:51:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:15.509914+0000 osd.2 (osd.2) 599 : cluster 0 5.d scrub starts 2024-04-06T03:51:16.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:16 smithi050 bash[29221]: cluster 2024-04-06T03:51:15.513257+0000 osd.2 (osd.2) 600 : cluster 0 5.d scrub ok 2024-04-06T03:51:16.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.302024+0000 osd.1 (osd.1) 497 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:14.302131+0000 osd.1 (osd.1) 498 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: audit 2024-04-06T03:51:15.110710+0000 mon.a (mon.0) 1918 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:15.118496+0000 mon.a (mon.0) 1919 : cluster 0 osdmap e437: 8 total, 8 up, 8 in 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:15.443202+0000 osd.3 (osd.3) 737 : cluster 0 5.0 scrub starts 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:15.447410+0000 osd.3 (osd.3) 738 : cluster 0 5.0 scrub ok 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:15.509914+0000 osd.2 (osd.2) 599 : cluster 0 5.d scrub starts 2024-04-06T03:51:16.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:16 smithi153 bash[23852]: cluster 2024-04-06T03:51:15.513257+0000 osd.2 (osd.2) 600 : cluster 0 5.d scrub ok 2024-04-06T03:51:17.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:17 smithi050 bash[21467]: cluster 2024-04-06T03:51:16.494573+0000 osd.7 (osd.7) 745 : cluster 0 6.15 deep-scrub starts 2024-04-06T03:51:17.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:17 smithi050 bash[21467]: cluster 2024-04-06T03:51:16.588363+0000 osd.7 (osd.7) 746 : cluster 0 6.15 deep-scrub ok 2024-04-06T03:51:17.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:17 smithi050 bash[21467]: cluster 2024-04-06T03:51:16.679418+0000 mgr.y (mgr.24368) 1111 : cluster 0 pgmap v1226: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 821 B/s rd, 0 op/s 2024-04-06T03:51:17.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:17 smithi050 bash[29221]: cluster 2024-04-06T03:51:16.494573+0000 osd.7 (osd.7) 745 : cluster 0 6.15 deep-scrub starts 2024-04-06T03:51:17.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:17 smithi050 bash[29221]: cluster 2024-04-06T03:51:16.588363+0000 osd.7 (osd.7) 746 : cluster 0 6.15 deep-scrub ok 2024-04-06T03:51:17.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:17 smithi050 bash[29221]: cluster 2024-04-06T03:51:16.679418+0000 mgr.y (mgr.24368) 1111 : cluster 0 pgmap v1226: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 821 B/s rd, 0 op/s 2024-04-06T03:51:17.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:17 smithi153 bash[23852]: cluster 2024-04-06T03:51:16.494573+0000 osd.7 (osd.7) 745 : cluster 0 6.15 deep-scrub starts 2024-04-06T03:51:17.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:17 smithi153 bash[23852]: cluster 2024-04-06T03:51:16.588363+0000 osd.7 (osd.7) 746 : cluster 0 6.15 deep-scrub ok 2024-04-06T03:51:17.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:17 smithi153 bash[23852]: cluster 2024-04-06T03:51:16.679418+0000 mgr.y (mgr.24368) 1111 : cluster 0 pgmap v1226: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 821 B/s rd, 0 op/s 2024-04-06T03:51:18.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:18 smithi050 bash[21467]: cluster 2024-04-06T03:51:17.440781+0000 osd.1 (osd.1) 499 : cluster 0 5.12 scrub starts 2024-04-06T03:51:18.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:18 smithi050 bash[21467]: cluster 2024-04-06T03:51:17.443316+0000 osd.1 (osd.1) 500 : cluster 0 5.12 scrub ok 2024-04-06T03:51:18.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:18 smithi050 bash[21467]: cluster 2024-04-06T03:51:18.211319+0000 mon.a (mon.0) 1920 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-04-06T03:51:18.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:18 smithi050 bash[29221]: cluster 2024-04-06T03:51:17.440781+0000 osd.1 (osd.1) 499 : cluster 0 5.12 scrub starts 2024-04-06T03:51:18.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:18 smithi050 bash[29221]: cluster 2024-04-06T03:51:17.443316+0000 osd.1 (osd.1) 500 : cluster 0 5.12 scrub ok 2024-04-06T03:51:18.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:18 smithi050 bash[29221]: cluster 2024-04-06T03:51:18.211319+0000 mon.a (mon.0) 1920 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-04-06T03:51:18.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:18 smithi153 bash[23852]: cluster 2024-04-06T03:51:17.440781+0000 osd.1 (osd.1) 499 : cluster 0 5.12 scrub starts 2024-04-06T03:51:18.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:18 smithi153 bash[23852]: cluster 2024-04-06T03:51:17.443316+0000 osd.1 (osd.1) 500 : cluster 0 5.12 scrub ok 2024-04-06T03:51:18.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:18 smithi153 bash[23852]: cluster 2024-04-06T03:51:18.211319+0000 mon.a (mon.0) 1920 : cluster 0 osdmap e438: 8 total, 8 up, 8 in 2024-04-06T03:51:19.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[21467]: cluster 2024-04-06T03:51:18.150021+0000 osd.0 (osd.0) 714 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:19.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[21467]: cluster 2024-04-06T03:51:18.150125+0000 osd.0 (osd.0) 715 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[21467]: cluster 2024-04-06T03:51:18.470175+0000 osd.2 (osd.2) 601 : cluster 0 6.f scrub starts 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[21467]: cluster 2024-04-06T03:51:18.680339+0000 mgr.y (mgr.24368) 1112 : cluster 0 pgmap v1228: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[29221]: cluster 2024-04-06T03:51:18.150021+0000 osd.0 (osd.0) 714 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[29221]: cluster 2024-04-06T03:51:18.150125+0000 osd.0 (osd.0) 715 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[29221]: cluster 2024-04-06T03:51:18.470175+0000 osd.2 (osd.2) 601 : cluster 0 6.f scrub starts 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[29221]: cluster 2024-04-06T03:51:18.680339+0000 mgr.y (mgr.24368) 1112 : cluster 0 pgmap v1228: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:19.491 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:51:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:51:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:51:19.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:19 smithi153 bash[23852]: cluster 2024-04-06T03:51:18.150021+0000 osd.0 (osd.0) 714 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:19.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:19 smithi153 bash[23852]: cluster 2024-04-06T03:51:18.150125+0000 osd.0 (osd.0) 715 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:19.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:19 smithi153 bash[23852]: cluster 2024-04-06T03:51:18.470175+0000 osd.2 (osd.2) 601 : cluster 0 6.f scrub starts 2024-04-06T03:51:19.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:19 smithi153 bash[23852]: cluster 2024-04-06T03:51:18.680339+0000 mgr.y (mgr.24368) 1112 : cluster 0 pgmap v1228: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:20.484 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:51:20.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:20 smithi153 bash[23852]: cluster 2024-04-06T03:51:18.682678+0000 osd.2 (osd.2) 602 : cluster 0 6.f scrub ok 2024-04-06T03:51:20.736 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:20.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:20 smithi050 bash[21467]: cluster 2024-04-06T03:51:18.682678+0000 osd.2 (osd.2) 602 : cluster 0 6.f scrub ok 2024-04-06T03:51:20.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:20 smithi050 bash[29221]: cluster 2024-04-06T03:51:18.682678+0000 osd.2 (osd.2) 602 : cluster 0 6.f scrub ok 2024-04-06T03:51:20.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:51:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:51:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: cluster 2024-04-06T03:51:20.449734+0000 osd.7 (osd.7) 747 : cluster 0 6.5 scrub starts 2024-04-06T03:51:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: cluster 2024-04-06T03:51:20.465004+0000 osd.1 (osd.1) 501 : cluster 0 2.15 scrub starts 2024-04-06T03:51:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: cluster 2024-04-06T03:51:20.468058+0000 osd.1 (osd.1) 502 : cluster 0 2.15 scrub ok 2024-04-06T03:51:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: cluster 2024-04-06T03:51:20.533020+0000 osd.7 (osd.7) 748 : cluster 0 6.5 scrub ok 2024-04-06T03:51:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: audit 2024-04-06T03:51:20.535985+0000 mgr.y (mgr.24368) 1113 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:21.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: cluster 2024-04-06T03:51:20.681586+0000 mgr.y (mgr.24368) 1114 : cluster 0 pgmap v1229: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:21.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:21 smithi153 bash[23852]: audit 2024-04-06T03:51:21.097434+0000 mon.a (mon.0) 1921 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:21.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: cluster 2024-04-06T03:51:20.449734+0000 osd.7 (osd.7) 747 : cluster 0 6.5 scrub starts 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: cluster 2024-04-06T03:51:20.465004+0000 osd.1 (osd.1) 501 : cluster 0 2.15 scrub starts 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: cluster 2024-04-06T03:51:20.468058+0000 osd.1 (osd.1) 502 : cluster 0 2.15 scrub ok 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: cluster 2024-04-06T03:51:20.533020+0000 osd.7 (osd.7) 748 : cluster 0 6.5 scrub ok 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: audit 2024-04-06T03:51:20.535985+0000 mgr.y (mgr.24368) 1113 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: cluster 2024-04-06T03:51:20.681586+0000 mgr.y (mgr.24368) 1114 : cluster 0 pgmap v1229: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[21467]: audit 2024-04-06T03:51:21.097434+0000 mon.a (mon.0) 1921 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: cluster 2024-04-06T03:51:20.449734+0000 osd.7 (osd.7) 747 : cluster 0 6.5 scrub starts 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: cluster 2024-04-06T03:51:20.465004+0000 osd.1 (osd.1) 501 : cluster 0 2.15 scrub starts 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: cluster 2024-04-06T03:51:20.468058+0000 osd.1 (osd.1) 502 : cluster 0 2.15 scrub ok 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: cluster 2024-04-06T03:51:20.533020+0000 osd.7 (osd.7) 748 : cluster 0 6.5 scrub ok 2024-04-06T03:51:21.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: audit 2024-04-06T03:51:20.535985+0000 mgr.y (mgr.24368) 1113 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:21.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: cluster 2024-04-06T03:51:20.681586+0000 mgr.y (mgr.24368) 1114 : cluster 0 pgmap v1229: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:21.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:21 smithi050 bash[29221]: audit 2024-04-06T03:51:21.097434+0000 mon.a (mon.0) 1921 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:22.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:22 smithi050 bash[21467]: cluster 2024-04-06T03:51:21.480711+0000 osd.7 (osd.7) 749 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:22.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:22 smithi050 bash[21467]: cluster 2024-04-06T03:51:21.480840+0000 osd.7 (osd.7) 750 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:22.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:22 smithi050 bash[29221]: cluster 2024-04-06T03:51:21.480711+0000 osd.7 (osd.7) 749 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:22.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:22 smithi050 bash[29221]: cluster 2024-04-06T03:51:21.480840+0000 osd.7 (osd.7) 750 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:22 smithi153 bash[23852]: cluster 2024-04-06T03:51:21.480711+0000 osd.7 (osd.7) 749 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:22.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:22 smithi153 bash[23852]: cluster 2024-04-06T03:51:21.480840+0000 osd.7 (osd.7) 750 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:23.489 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:51:23.489 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [], 2024-04-06T03:51:23.489 INFO:teuthology.orchestra.run.smithi153.stdout: "complaint_time": 30, 2024-04-06T03:51:23.489 INFO:teuthology.orchestra.run.smithi153.stdout: "num_blocked_ops": 0 2024-04-06T03:51:23.489 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:51:23.529 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:23 smithi153 bash[23852]: cluster 2024-04-06T03:51:22.268995+0000 mon.a (mon.0) 1922 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-04-06T03:51:23.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:23 smithi153 bash[23852]: cluster 2024-04-06T03:51:22.523286+0000 osd.2 (osd.2) 603 : cluster 0 2.e scrub starts 2024-04-06T03:51:23.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:23 smithi153 bash[23852]: cluster 2024-04-06T03:51:22.525786+0000 osd.2 (osd.2) 604 : cluster 0 2.e scrub ok 2024-04-06T03:51:23.530 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:23 smithi153 bash[23852]: cluster 2024-04-06T03:51:22.682431+0000 mgr.y (mgr.24368) 1115 : cluster 0 pgmap v1231: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-06T03:51:23.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[21467]: cluster 2024-04-06T03:51:22.268995+0000 mon.a (mon.0) 1922 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-04-06T03:51:23.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[21467]: cluster 2024-04-06T03:51:22.523286+0000 osd.2 (osd.2) 603 : cluster 0 2.e scrub starts 2024-04-06T03:51:23.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[21467]: cluster 2024-04-06T03:51:22.525786+0000 osd.2 (osd.2) 604 : cluster 0 2.e scrub ok 2024-04-06T03:51:23.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[21467]: cluster 2024-04-06T03:51:22.682431+0000 mgr.y (mgr.24368) 1115 : cluster 0 pgmap v1231: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-06T03:51:23.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[29221]: cluster 2024-04-06T03:51:22.268995+0000 mon.a (mon.0) 1922 : cluster 0 osdmap e439: 8 total, 8 up, 8 in 2024-04-06T03:51:23.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[29221]: cluster 2024-04-06T03:51:22.523286+0000 osd.2 (osd.2) 603 : cluster 0 2.e scrub starts 2024-04-06T03:51:23.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[29221]: cluster 2024-04-06T03:51:22.525786+0000 osd.2 (osd.2) 604 : cluster 0 2.e scrub ok 2024-04-06T03:51:23.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:23 smithi050 bash[29221]: cluster 2024-04-06T03:51:22.682431+0000 mgr.y (mgr.24368) 1115 : cluster 0 pgmap v1231: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-06T03:51:24.228 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph daemon osd.7 dump_historic_ops 2024-04-06T03:51:24.286 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-06T03:51:24.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:24 smithi153 bash[23852]: audit 2024-04-06T03:51:24.105590+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.50:0/2283395858' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:24.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:24 smithi153 bash[23852]: audit 2024-04-06T03:51:24.106393+0000 mon.a (mon.0) 1923 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:24.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:24 smithi050 bash[21467]: audit 2024-04-06T03:51:24.105590+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.50:0/2283395858' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:24.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:24 smithi050 bash[21467]: audit 2024-04-06T03:51:24.106393+0000 mon.a (mon.0) 1923 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:24 smithi050 bash[29221]: audit 2024-04-06T03:51:24.105590+0000 mon.c (mon.2) 136 : audit 1 from='client.? 172.21.15.50:0/2283395858' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:24.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:24 smithi050 bash[29221]: audit 2024-04-06T03:51:24.106393+0000 mon.a (mon.0) 1923 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-06T03:51:25.138 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-06T03:51:25.139 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-06T03:51:25.139 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-06T03:51:25.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:25 smithi050 bash[21467]: audit 2024-04-06T03:51:24.273173+0000 mon.a (mon.0) 1924 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:51:25.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:25 smithi050 bash[21467]: cluster 2024-04-06T03:51:24.280113+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-04-06T03:51:25.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:25 smithi050 bash[21467]: cluster 2024-04-06T03:51:24.683418+0000 mgr.y (mgr.24368) 1116 : cluster 0 pgmap v1233: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:51:25.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:25 smithi050 bash[29221]: audit 2024-04-06T03:51:24.273173+0000 mon.a (mon.0) 1924 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:51:25.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:25 smithi050 bash[29221]: cluster 2024-04-06T03:51:24.280113+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-04-06T03:51:25.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:25 smithi050 bash[29221]: cluster 2024-04-06T03:51:24.683418+0000 mgr.y (mgr.24368) 1116 : cluster 0 pgmap v1233: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:51:25.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:25 smithi153 bash[23852]: audit 2024-04-06T03:51:24.273173+0000 mon.a (mon.0) 1924 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-06T03:51:25.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:25 smithi153 bash[23852]: cluster 2024-04-06T03:51:24.280113+0000 mon.a (mon.0) 1925 : cluster 0 osdmap e440: 8 total, 8 up, 8 in 2024-04-06T03:51:25.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:25 smithi153 bash[23852]: cluster 2024-04-06T03:51:24.683418+0000 mgr.y (mgr.24368) 1116 : cluster 0 pgmap v1233: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-06T03:51:26.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:26 smithi153 bash[23852]: cluster 2024-04-06T03:51:25.159733+0000 osd.0 (osd.0) 716 : cluster 0 3.17 deep-scrub starts 2024-04-06T03:51:26.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:26 smithi153 bash[23852]: cluster 2024-04-06T03:51:25.163860+0000 osd.0 (osd.0) 717 : cluster 0 3.17 deep-scrub ok 2024-04-06T03:51:26.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:26 smithi050 bash[21467]: cluster 2024-04-06T03:51:25.159733+0000 osd.0 (osd.0) 716 : cluster 0 3.17 deep-scrub starts 2024-04-06T03:51:26.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:26 smithi050 bash[21467]: cluster 2024-04-06T03:51:25.163860+0000 osd.0 (osd.0) 717 : cluster 0 3.17 deep-scrub ok 2024-04-06T03:51:26.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:26 smithi050 bash[29221]: cluster 2024-04-06T03:51:25.159733+0000 osd.0 (osd.0) 716 : cluster 0 3.17 deep-scrub starts 2024-04-06T03:51:26.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:26 smithi050 bash[29221]: cluster 2024-04-06T03:51:25.163860+0000 osd.0 (osd.0) 717 : cluster 0 3.17 deep-scrub ok 2024-04-06T03:51:27.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.018358+0000 osd.5 (osd.5) 177 : cluster 0 2.2 scrub starts 2024-04-06T03:51:27.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.020713+0000 osd.5 (osd.5) 178 : cluster 0 2.2 scrub ok 2024-04-06T03:51:27.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.494823+0000 osd.4 (osd.4) 713 : cluster 0 5.18 scrub starts 2024-04-06T03:51:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.496995+0000 osd.4 (osd.4) 714 : cluster 0 5.18 scrub ok 2024-04-06T03:51:27.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.684502+0000 mgr.y (mgr.24368) 1117 : cluster 0 pgmap v1234: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:51:27.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.018358+0000 osd.5 (osd.5) 177 : cluster 0 2.2 scrub starts 2024-04-06T03:51:27.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.020713+0000 osd.5 (osd.5) 178 : cluster 0 2.2 scrub ok 2024-04-06T03:51:27.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.494823+0000 osd.4 (osd.4) 713 : cluster 0 5.18 scrub starts 2024-04-06T03:51:27.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.496995+0000 osd.4 (osd.4) 714 : cluster 0 5.18 scrub ok 2024-04-06T03:51:27.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:27 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.684502+0000 mgr.y (mgr.24368) 1117 : cluster 0 pgmap v1234: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:51:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:27 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.018358+0000 osd.5 (osd.5) 177 : cluster 0 2.2 scrub starts 2024-04-06T03:51:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:27 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.020713+0000 osd.5 (osd.5) 178 : cluster 0 2.2 scrub ok 2024-04-06T03:51:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:27 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.494823+0000 osd.4 (osd.4) 713 : cluster 0 5.18 scrub starts 2024-04-06T03:51:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:27 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.496995+0000 osd.4 (osd.4) 714 : cluster 0 5.18 scrub ok 2024-04-06T03:51:27.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:27 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.684502+0000 mgr.y (mgr.24368) 1117 : cluster 0 pgmap v1234: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:51:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.711098+0000 osd.6 (osd.6) 121 : cluster 0 3.10 scrub starts 2024-04-06T03:51:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[21467]: cluster 2024-04-06T03:51:26.716043+0000 osd.6 (osd.6) 122 : cluster 0 3.10 scrub ok 2024-04-06T03:51:28.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[21467]: cluster 2024-04-06T03:51:27.577483+0000 osd.2 (osd.2) 605 : cluster 0 2.c scrub starts 2024-04-06T03:51:28.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[21467]: cluster 2024-04-06T03:51:27.579872+0000 osd.2 (osd.2) 606 : cluster 0 2.c scrub ok 2024-04-06T03:51:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.711098+0000 osd.6 (osd.6) 121 : cluster 0 3.10 scrub starts 2024-04-06T03:51:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[29221]: cluster 2024-04-06T03:51:26.716043+0000 osd.6 (osd.6) 122 : cluster 0 3.10 scrub ok 2024-04-06T03:51:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[29221]: cluster 2024-04-06T03:51:27.577483+0000 osd.2 (osd.2) 605 : cluster 0 2.c scrub starts 2024-04-06T03:51:28.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:28 smithi050 bash[29221]: cluster 2024-04-06T03:51:27.579872+0000 osd.2 (osd.2) 606 : cluster 0 2.c scrub ok 2024-04-06T03:51:28.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:28 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.711098+0000 osd.6 (osd.6) 121 : cluster 0 3.10 scrub starts 2024-04-06T03:51:28.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:28 smithi153 bash[23852]: cluster 2024-04-06T03:51:26.716043+0000 osd.6 (osd.6) 122 : cluster 0 3.10 scrub ok 2024-04-06T03:51:28.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:28 smithi153 bash[23852]: cluster 2024-04-06T03:51:27.577483+0000 osd.2 (osd.2) 605 : cluster 0 2.c scrub starts 2024-04-06T03:51:28.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:28 smithi153 bash[23852]: cluster 2024-04-06T03:51:27.579872+0000 osd.2 (osd.2) 606 : cluster 0 2.c scrub ok 2024-04-06T03:51:29.026 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.b/config 2024-04-06T03:51:29.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:51:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:27.715334+0000 osd.6 (osd.6) 123 : cluster 0 5.11 scrub starts 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:27.717835+0000 osd.6 (osd.6) 124 : cluster 0 5.11 scrub ok 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.324135+0000 osd.3 (osd.3) 739 : cluster 0 3.18 scrub starts 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.328850+0000 osd.3 (osd.3) 740 : cluster 0 3.18 scrub ok 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.464705+0000 osd.1 (osd.1) 503 : cluster 0 2.6 scrub starts 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.468893+0000 osd.1 (osd.1) 504 : cluster 0 2.6 scrub ok 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.483589+0000 osd.4 (osd.4) 715 : cluster 0 2.f scrub starts 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.487661+0000 osd.4 (osd.4) 716 : cluster 0 2.f scrub ok 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[29221]: cluster 2024-04-06T03:51:28.685483+0000 mgr.y (mgr.24368) 1118 : cluster 0 pgmap v1235: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:29.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:27.715334+0000 osd.6 (osd.6) 123 : cluster 0 5.11 scrub starts 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:27.717835+0000 osd.6 (osd.6) 124 : cluster 0 5.11 scrub ok 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.324135+0000 osd.3 (osd.3) 739 : cluster 0 3.18 scrub starts 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.328850+0000 osd.3 (osd.3) 740 : cluster 0 3.18 scrub ok 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.464705+0000 osd.1 (osd.1) 503 : cluster 0 2.6 scrub starts 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.468893+0000 osd.1 (osd.1) 504 : cluster 0 2.6 scrub ok 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.483589+0000 osd.4 (osd.4) 715 : cluster 0 2.f scrub starts 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.487661+0000 osd.4 (osd.4) 716 : cluster 0 2.f scrub ok 2024-04-06T03:51:29.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:29 smithi050 bash[21467]: cluster 2024-04-06T03:51:28.685483+0000 mgr.y (mgr.24368) 1118 : cluster 0 pgmap v1235: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:29.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:27.715334+0000 osd.6 (osd.6) 123 : cluster 0 5.11 scrub starts 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:27.717835+0000 osd.6 (osd.6) 124 : cluster 0 5.11 scrub ok 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.324135+0000 osd.3 (osd.3) 739 : cluster 0 3.18 scrub starts 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.328850+0000 osd.3 (osd.3) 740 : cluster 0 3.18 scrub ok 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.464705+0000 osd.1 (osd.1) 503 : cluster 0 2.6 scrub starts 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.468893+0000 osd.1 (osd.1) 504 : cluster 0 2.6 scrub ok 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.483589+0000 osd.4 (osd.4) 715 : cluster 0 2.f scrub starts 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.487661+0000 osd.4 (osd.4) 716 : cluster 0 2.f scrub ok 2024-04-06T03:51:29.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:29 smithi153 bash[23852]: cluster 2024-04-06T03:51:28.685483+0000 mgr.y (mgr.24368) 1118 : cluster 0 pgmap v1235: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:30.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:30 smithi153 bash[23852]: cluster 2024-04-06T03:51:30.075405+0000 osd.5 (osd.5) 179 : cluster 0 3.1d scrub starts 2024-04-06T03:51:30.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:30 smithi153 bash[23852]: cluster 2024-04-06T03:51:30.080183+0000 osd.5 (osd.5) 180 : cluster 0 3.1d scrub ok 2024-04-06T03:51:30.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:51:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:51:30.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:30 smithi050 bash[21467]: cluster 2024-04-06T03:51:30.075405+0000 osd.5 (osd.5) 179 : cluster 0 3.1d scrub starts 2024-04-06T03:51:30.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:30 smithi050 bash[21467]: cluster 2024-04-06T03:51:30.080183+0000 osd.5 (osd.5) 180 : cluster 0 3.1d scrub ok 2024-04-06T03:51:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:30 smithi050 bash[29221]: cluster 2024-04-06T03:51:30.075405+0000 osd.5 (osd.5) 179 : cluster 0 3.1d scrub starts 2024-04-06T03:51:30.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:30 smithi050 bash[29221]: cluster 2024-04-06T03:51:30.080183+0000 osd.5 (osd.5) 180 : cluster 0 3.1d scrub ok 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: audit 2024-04-06T03:51:30.544024+0000 mgr.y (mgr.24368) 1119 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:30.686812+0000 mgr.y (mgr.24368) 1120 : cluster 0 pgmap v1236: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:30.731422+0000 osd.6 (osd.6) 125 : cluster 0 2.14 deep-scrub starts 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:30.734180+0000 osd.6 (osd.6) 126 : cluster 0 2.14 deep-scrub ok 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:31.108195+0000 osd.0 (osd.0) 718 : cluster 0 6.0 deep-scrub starts 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:31.197983+0000 osd.0 (osd.0) 719 : cluster 0 6.0 deep-scrub ok 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:31.496384+0000 osd.7 (osd.7) 751 : cluster 0 2.b scrub starts 2024-04-06T03:51:31.931 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:31 smithi153 bash[23852]: cluster 2024-04-06T03:51:31.499115+0000 osd.7 (osd.7) 752 : cluster 0 2.b scrub ok 2024-04-06T03:51:31.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: audit 2024-04-06T03:51:30.544024+0000 mgr.y (mgr.24368) 1119 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:30.686812+0000 mgr.y (mgr.24368) 1120 : cluster 0 pgmap v1236: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:30.731422+0000 osd.6 (osd.6) 125 : cluster 0 2.14 deep-scrub starts 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:30.734180+0000 osd.6 (osd.6) 126 : cluster 0 2.14 deep-scrub ok 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:31.108195+0000 osd.0 (osd.0) 718 : cluster 0 6.0 deep-scrub starts 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:31.197983+0000 osd.0 (osd.0) 719 : cluster 0 6.0 deep-scrub ok 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:31.496384+0000 osd.7 (osd.7) 751 : cluster 0 2.b scrub starts 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[29221]: cluster 2024-04-06T03:51:31.499115+0000 osd.7 (osd.7) 752 : cluster 0 2.b scrub ok 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: audit 2024-04-06T03:51:30.544024+0000 mgr.y (mgr.24368) 1119 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:30.686812+0000 mgr.y (mgr.24368) 1120 : cluster 0 pgmap v1236: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:30.731422+0000 osd.6 (osd.6) 125 : cluster 0 2.14 deep-scrub starts 2024-04-06T03:51:31.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:30.734180+0000 osd.6 (osd.6) 126 : cluster 0 2.14 deep-scrub ok 2024-04-06T03:51:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:31.108195+0000 osd.0 (osd.0) 718 : cluster 0 6.0 deep-scrub starts 2024-04-06T03:51:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:31.197983+0000 osd.0 (osd.0) 719 : cluster 0 6.0 deep-scrub ok 2024-04-06T03:51:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:31.496384+0000 osd.7 (osd.7) 751 : cluster 0 2.b scrub starts 2024-04-06T03:51:31.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:31 smithi050 bash[21467]: cluster 2024-04-06T03:51:31.499115+0000 osd.7 (osd.7) 752 : cluster 0 2.b scrub ok 2024-04-06T03:51:32.088 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-06T03:51:32.088 INFO:teuthology.orchestra.run.smithi153.stdout: "size": 20, 2024-04-06T03:51:32.088 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 600, 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "ops": [ 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:165871 10.c 10:3b4360f5:::benchmark_data_smithi050_121293_object67054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:58.596624+0000", 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 573.48822833899999, 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48529114499999998, 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:51:32.089 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 165871 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596624+0000", 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596624+0000", 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.090 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596625+0000", 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.4279999999999999e-06 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596630+0000", 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7419999999999997e-06 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596641+0000", 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1039999999999999e-05 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.091 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596649+0000", 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3289999999999994e-06 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596724+0000", 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5227000000000003e-05 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596841+0000", 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000116302 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.092 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.596950+0000", 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010900600000000001 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.599506+0000", 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025567010000000002 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.607899+0000", 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0083922719999999992 2024-04-06T03:51:32.093 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081812+0000", 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47391306599999999 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081877+0000", 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5177000000000003e-05 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.094 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081915+0000", 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7855000000000003e-05 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:165873 10.c 10:353b0683:::benchmark_data_smithi050_121293_object67056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:58.601575+0000", 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 573.48327730400001, 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48048355300000001, 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.095 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 165873 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601575+0000", 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601575+0000", 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.096 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601576+0000", 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3069999999999999e-06 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601582+0000", 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.5199999999999997e-06 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601595+0000", 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3142999999999999e-05 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.097 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601601+0000", 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6769999999999998e-06 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601658+0000", 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.6910999999999998e-05 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601797+0000", 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000138935 2024-04-06T03:51:32.098 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.601953+0000", 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000156224 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.604454+0000", 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025002940000000001 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.607939+0000", 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003485546 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.099 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081957+0000", 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47401758300000002 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.081997+0000", 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.0373000000000001e-05 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082058+0000", 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.1039999999999998e-05 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.100 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:165880 10.c 10:3a9cc1ff:::benchmark_data_smithi050_121293_object67063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:41:58.611324+0000", 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 573.47352749200002, 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47127398500000001, 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 165880 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.101 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611324+0000", 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.102 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611324+0000", 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611326+0000", 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.449e-06 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611331+0000", 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.169e-06 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.103 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611345+0000", 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3529e-05 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611354+0000", 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9439999999999994e-06 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611413+0000", 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.9459000000000001e-05 2024-04-06T03:51:32.104 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611528+0000", 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000115103 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.611643+0000", 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011493999999999999 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.613786+0000", 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0021425789999999999 2024-04-06T03:51:32.105 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:58.615791+0000", 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0020058070000000001 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082545+0000", 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.46675355499999999 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082575+0000", 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.9646999999999999e-05 2024-04-06T03:51:32.106 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:41:59.082598+0000", 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.3804000000000001e-05 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:180541 10.c 10:32bb1016:::benchmark_data_smithi050_121293_object81724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.651564+0000", 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 551.43328835299997, 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55442063399999997, 2024-04-06T03:51:32.107 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 180541 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651564+0000", 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.108 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651564+0000", 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651565+0000", 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.61e-06 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651571+0000", 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.604e-06 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.109 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651583+0000", 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2351000000000001e-05 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651590+0000", 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1060000000000002e-06 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651659+0000", 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.8736000000000001e-05 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.110 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651751+0000", 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2001000000000002e-05 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.651827+0000", 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6103e-05 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654354+0000", 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0025271080000000001 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.111 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654607+0000", 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025244599999999997 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.205724+0000", 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55111694300000003 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.205836+0000", 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000112645 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.205984+0000", 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014798099999999999 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15306.0:180542 10.c 10:3a06efca:::benchmark_data_smithi050_121293_object81725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:42:20.654984+0000", 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 551.42986824299999, 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55130753499999996, 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15306", 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3682578803", 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 180542 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654984+0000", 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654984+0000", 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654985+0000", 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.47e-07 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654988+0000", 2024-04-06T03:51:32.115 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.8909999999999997e-06 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.654997+0000", 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.9989999999999997e-06 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655003+0000", 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4140000000000002e-06 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.116 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655044+0000", 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.1588999999999998e-05 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655138+0000", 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3553000000000004e-05 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.655213+0000", 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4663e-05 2024-04-06T03:51:32.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.656889+0000", 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016761370000000001 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:20.657245+0000", 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00035658899999999998 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.206122+0000", 2024-04-06T03:51:32.118 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54887692600000004 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.206215+0000", 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.2174000000000002e-05 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:42:21.206291+0000", 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.6753000000000002e-05 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84079 11.f 11:f336f4e4:::benchmark_data_smithi050_138698_object84078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.082199+0000", 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 396.00265334199997, 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48952396300000001, 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84079 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.120 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082199+0000", 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082199+0000", 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082200+0000", 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0279999999999999e-06 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.121 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082285+0000", 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5451000000000006e-05 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082297+0000", 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.184e-05 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.082326+0000", 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.8679000000000001e-05 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.122 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.084009+0000", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0016828800000000001 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.084144+0000", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013520200000000001 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.084268+0000", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000124234 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.087490+0000", 2024-04-06T03:51:32.123 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003222291 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.087956+0000", 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00046621299999999999 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.571387+0000", 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.483430154 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.124 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.571586+0000", 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000199583 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.571723+0000", 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000136408 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84093 11.f 11:f22a7ccd:::benchmark_data_smithi050_138698_object84092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.094231+0000", 2024-04-06T03:51:32.125 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 395.99062051200002, 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48593634800000002, 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84093 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.126 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094231+0000", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094231+0000", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094232+0000", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.5600000000000004e-07 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094367+0000", 2024-04-06T03:51:32.127 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00013476100000000001 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094379+0000", 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1525999999999999e-05 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094386+0000", 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5730000000000002e-06 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094440+0000", 2024-04-06T03:51:32.128 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4128000000000001e-05 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094559+0000", 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011815399999999999 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.094650+0000", 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1713999999999996e-05 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.096736+0000", 2024-04-06T03:51:32.129 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002085408 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.098839+0000", 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.002103705 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580100+0000", 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48126070799999998 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580135+0000", 2024-04-06T03:51:32.130 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.5182999999999998e-05 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580168+0000", 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.2532e-05 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84100 11.f 11:f2bff20b:::benchmark_data_smithi050_138698_object84099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.102298+0000", 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 395.98255358199998, 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.48398215500000002, 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.131 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84100 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.132 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102298+0000", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102298+0000", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102300+0000", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.108e-06 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102343+0000", 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3961000000000001e-05 2024-04-06T03:51:32.133 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102356+0000", 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2659e-05 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102363+0000", 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.097e-06 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102421+0000", 2024-04-06T03:51:32.134 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7815000000000001e-05 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102542+0000", 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000121045 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.102648+0000", 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000105849 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.104642+0000", 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0019937079999999999 2024-04-06T03:51:32.135 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.580240+0000", 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47559864499999999 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586194+0000", 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.005953814 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586237+0000", 2024-04-06T03:51:32.136 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.244e-05 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586281+0000", 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4014000000000002e-05 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:84103 11.f 11:f0c0dc0b:::benchmark_data_smithi050_138698_object84102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:56.106886+0000", 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 395.97796574699998, 2024-04-06T03:51:32.137 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47961802100000001, 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 84103 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106886+0000", 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.138 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106886+0000", 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106894+0000", 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5650000000000004e-06 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.139 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106944+0000", 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0262000000000001e-05 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106955+0000", 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1015999999999999e-05 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.106982+0000", 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.7263e-05 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.140 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.107027+0000", 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5124999999999999e-05 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.107144+0000", 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00011633600000000001 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.107253+0000", 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000108849 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.141 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.109003+0000", 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0017502349999999999 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.581085+0000", 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.47208220400000001 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586450+0000", 2024-04-06T03:51:32.142 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0053647670000000003 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586484+0000", 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.4044000000000001e-05 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:56.586504+0000", 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0355000000000001e-05 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.143 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:85899 11.f 11:f7a53cd3:::benchmark_data_smithi050_138698_object85898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:58.381838+0000", 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 393.70301437900002, 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55626176100000002, 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.144 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 85899 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381838+0000", 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381838+0000", 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.145 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381839+0000", 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.463e-06 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381890+0000", 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.1119999999999998e-05 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381901+0000", 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1185999999999999e-05 2024-04-06T03:51:32.146 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381909+0000", 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.123e-06 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.381968+0000", 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8616000000000003e-05 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.147 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.382079+0000", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000110776 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.382180+0000", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000101068 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.385840+0000", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0036598479999999998 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.393293+0000", 2024-04-06T03:51:32.148 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00745366 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.937979+0000", 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54468579900000003 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.938045+0000", 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5513999999999996e-05 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.149 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.938099+0000", 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.4588e-05 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:85907 11.f 11:f6c6d927:::benchmark_data_smithi050_138698_object85906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:58.387295+0000", 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 393.697556566, 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.56115197800000005, 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.150 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 85907 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387295+0000", 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387295+0000", 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.151 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387296+0000", 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7100000000000002e-07 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387354+0000", 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.8009e-05 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387360+0000", 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7570000000000001e-06 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.152 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387364+0000", 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.4789999999999998e-06 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387422+0000", 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.7383999999999998e-05 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.153 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387528+0000", 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00010686100000000001 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.387629+0000", 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000100826 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.396378+0000", 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0087485410000000003 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.154 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397753+0000", 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0013752440000000001 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948329+0000", 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55057592899999996 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948395+0000", 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5655999999999996e-05 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.155 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948447+0000", 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.2720999999999999e-05 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:85914 11.f 11:f40a4adc:::benchmark_data_smithi050_138698_object85913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:44:58.397788+0000", 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 393.687064233, 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.55124815999999999, 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.156 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 85914 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397788+0000", 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397788+0000", 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.157 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397789+0000", 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.09e-07 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397810+0000", 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0930000000000001e-05 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397816+0000", 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2600000000000002e-06 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.158 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.397820+0000", 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3309999999999997e-06 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.399239+0000", 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001419078 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.399391+0000", 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00015199899999999999 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.159 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.399795+0000", 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00040332299999999997 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.403164+0000", 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033698999999999999 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.403171+0000", 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6660000000000002e-06 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.160 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.948970+0000", 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.54579887100000002 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.949010+0000", 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.9861999999999999e-05 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:44:58.949036+0000", 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6030999999999999e-05 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.161 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:149170 11.4 11:20e216ff:::benchmark_data_smithi050_138698_object43196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:05.458743+0000", 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 326.626109144, 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43579931799999999, 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 149170 2024-04-06T03:51:32.162 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458743+0000", 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458743+0000", 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.163 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458744+0000", 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.4900000000000005e-07 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458747+0000", 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.613e-06 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458756+0000", 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.6170000000000007e-06 2024-04-06T03:51:32.164 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458762+0000", 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4790000000000001e-06 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458815+0000", 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3087999999999997e-05 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458899+0000", 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.3294999999999994e-05 2024-04-06T03:51:32.165 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,6", 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.458989+0000", 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0365999999999995e-05 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.166 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.462025+0000", 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.003035921 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.462702+0000", 2024-04-06T03:51:32.167 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00067719500000000001 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.894448+0000", 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.431745555 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.168 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.894499+0000", 2024-04-06T03:51:32.169 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0955000000000003e-05 2024-04-06T03:51:32.169 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.169 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.169 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.169 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:05.894542+0000", 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3485000000000001e-05 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.170 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:198205 11.f 11:fb85945d:::benchmark_data_smithi050_138698_object92231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:58.969061+0000", 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 273.11579120499999, 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.42277171299999999, 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.171 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 198205 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969061+0000", 2024-04-06T03:51:32.172 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969061+0000", 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.173 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.174 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969062+0000", 2024-04-06T03:51:32.174 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.251e-06 2024-04-06T03:51:32.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.174 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969068+0000", 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.032e-06 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969081+0000", 2024-04-06T03:51:32.175 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.2449000000000001e-05 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969091+0000", 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0597e-05 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969157+0000", 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.6285000000000007e-05 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969278+0000", 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012081799999999999 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,2", 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.969385+0000", 2024-04-06T03:51:32.177 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000107169 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.971280+0000", 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.001894932 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.972181+0000", 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00090072800000000005 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.178 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:59.391620+0000", 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.419438739 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:59.391739+0000", 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000119073 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:59.391832+0000", 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.3640000000000005e-05 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.179 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.15498.0:198212 11.4 11:235c9794:::benchmark_data_smithi050_138698_object92238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:46:58.977443+0000", 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 273.10740869199998, 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43498639, 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.15498", 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/3150423505", 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 198212 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977443+0000", 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977443+0000", 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977445+0000", 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3230000000000001e-06 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.181 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977450+0000", 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.0989999999999998e-06 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977460+0000", 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0611000000000001e-05 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977467+0000", 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.2369999999999997e-06 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.182 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977514+0000", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.7462000000000001e-05 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977604+0000", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.0248999999999996e-05 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 2,6", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.977680+0000", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.5576999999999998e-05 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.183 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.979941+0000", 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0022609969999999998 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:58.980032+0000", 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.1601000000000006e-05 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:59.412280+0000", 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.43224730700000003 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.184 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:59.412365+0000", 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5657999999999999e-05 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:46:59.412430+0000", 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.4269000000000004e-05 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "rep_scrubmap(6.3 e326 from shard 6)", 2024-04-06T03:51:32.185 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:50:39.629220+0000", 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 52.455631492999998, 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.029947373999999999, 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.629220+0000", 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.186 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.629220+0000", 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.629221+0000", 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.5000000000000002e-07 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.629225+0000", 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.3309999999999997e-06 2024-04-06T03:51:32.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.629235+0000", 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.4909999999999994e-06 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.629246+0000", 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.1056999999999999e-05 2024-04-06T03:51:32.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.659145+0000", 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.029898661999999999 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.659155+0000", 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0138e-05 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:50:39.659168+0000", 2024-04-06T03:51:32.189 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.3045e-05 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.24278.0:5883 3.f 3:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:51:00.799708+0000", 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 31.285143850000001, 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0053417680000000002, 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-06T03:51:32.190 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.24278", 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 5883 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799708+0000", 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.191 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799708+0000", 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799711+0000", 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.067e-06 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799729+0000", 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.8249000000000001e-05 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.192 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799767+0000", 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.7610000000000001e-05 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799783+0000", 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.6288999999999999e-05 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799893+0000", 2024-04-06T03:51:32.193 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000109479 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.799901+0000", 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.1019999999999997e-06 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.194 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.803231+0000", 2024-04-06T03:51:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0033297589999999998 2024-04-06T03:51:32.197 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.803351+0000", 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00012058 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "waiting for subops from 0,4", 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.803501+0000", 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00014966 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op_commit", 2024-04-06T03:51:32.198 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.804125+0000", 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00062437199999999999 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.804515+0000", 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00038988100000000002 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "sub_op_commit_rec", 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.804927+0000", 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00041170300000000003 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "commit_sent", 2024-04-06T03:51:32.199 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.804996+0000", 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 6.9494999999999998e-05 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:00.805050+0000", 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 5.3522e-05 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.200 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "MOSDScrubReserve(5.d,QREQUEST,e:437,from:2,reservation_nonce:37)", 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:51:15.502934+0000", 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 16.581918154, 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0059033940000000002, 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.502934+0000", 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.201 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.502934+0000", 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.502936+0000", 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.787e-06 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.502945+0000", 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 9.7960000000000007e-06 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.202 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.502991+0000", 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 4.5283e-05 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.503006+0000", 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.5186e-05 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.503159+0000", 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000153425 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.203 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "op must wait for map", 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.503186+0000", 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.6342999999999998e-05 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.508734+0000", 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.0055487959999999999 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.204 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.508765+0000", 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 3.0876000000000001e-05 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:15.508837+0000", 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 7.1902000000000004e-05 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.205 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.24278.0:6312 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94633190253696 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:51:22.628992+0000", 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 9.4558597300000002, 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00096102500000000001, 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.24278", 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 6312 2024-04-06T03:51:32.206 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.628992+0000", 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.628992+0000", 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.628994+0000", 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.703e-06 2024-04-06T03:51:32.207 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.629004+0000", 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 1.0380000000000001e-05 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.629025+0000", 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.1118999999999999e-05 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.629048+0000", 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.2234000000000001e-05 2024-04-06T03:51:32.208 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.629361+0000", 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00031323100000000002 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.629617+0000", 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00025626300000000001 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:22.629953+0000", 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00033609500000000001 2024-04-06T03:51:32.209 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "description": "osd_op(client.24278.0:6322 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94633190253696 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "initiated_at": "2024-04-06T03:51:27.632436+0000", 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "age": 4.4524159489999997, 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00099119099999999999, 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "continuous": false, 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "type_data": { 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "flag_point": "started", 2024-04-06T03:51:32.210 INFO:teuthology.orchestra.run.smithi153.stdout: "client_info": { 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "client": "client.24278", 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "client_addr": "172.21.15.50:0/2847340914", 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "tid": 6322 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "events": [ 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "initiated", 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632436+0000", 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "header_read", 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632436+0000", 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0 2024-04-06T03:51:32.211 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "throttled", 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632438+0000", 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0559999999999999e-06 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "all_read", 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632447+0000", 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 8.5329999999999994e-06 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "dispatched", 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632468+0000", 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.0930000000000001e-05 2024-04-06T03:51:32.212 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "queued_for_pg", 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632494+0000", 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 2.5956e-05 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "reached_pg", 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.632762+0000", 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.00026890600000000002 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "started", 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.633054+0000", 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000292062 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-04-06T03:51:32.213 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: "event": "done", 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: "time": "2024-04-06T03:51:27.633427+0000", 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: "duration": 0.000372748 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-06T03:51:32.214 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-06T03:51:32.863 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-06T03:51:32.863 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-06T03:51:32.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:51:34.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:33 smithi153 bash[23852]: cluster 2024-04-06T03:51:32.687577+0000 mgr.y (mgr.24368) 1121 : cluster 0 pgmap v1237: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:34.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:33 smithi050 bash[21467]: cluster 2024-04-06T03:51:32.687577+0000 mgr.y (mgr.24368) 1121 : cluster 0 pgmap v1237: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:34.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:33 smithi050 bash[29221]: cluster 2024-04-06T03:51:32.687577+0000 mgr.y (mgr.24368) 1121 : cluster 0 pgmap v1237: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:35.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:35 smithi050 bash[21467]: cluster 2024-04-06T03:51:34.505731+0000 osd.7 (osd.7) 753 : cluster 0 2.8 deep-scrub starts 2024-04-06T03:51:35.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:35 smithi050 bash[21467]: cluster 2024-04-06T03:51:34.508283+0000 osd.7 (osd.7) 754 : cluster 0 2.8 deep-scrub ok 2024-04-06T03:51:35.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:35 smithi050 bash[21467]: cluster 2024-04-06T03:51:34.688963+0000 mgr.y (mgr.24368) 1122 : cluster 0 pgmap v1238: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-06T03:51:35.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:35 smithi050 bash[29221]: cluster 2024-04-06T03:51:34.505731+0000 osd.7 (osd.7) 753 : cluster 0 2.8 deep-scrub starts 2024-04-06T03:51:35.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:35 smithi050 bash[29221]: cluster 2024-04-06T03:51:34.508283+0000 osd.7 (osd.7) 754 : cluster 0 2.8 deep-scrub ok 2024-04-06T03:51:35.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:35 smithi050 bash[29221]: cluster 2024-04-06T03:51:34.688963+0000 mgr.y (mgr.24368) 1122 : cluster 0 pgmap v1238: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-06T03:51:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:35 smithi153 bash[23852]: cluster 2024-04-06T03:51:34.505731+0000 osd.7 (osd.7) 753 : cluster 0 2.8 deep-scrub starts 2024-04-06T03:51:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:35 smithi153 bash[23852]: cluster 2024-04-06T03:51:34.508283+0000 osd.7 (osd.7) 754 : cluster 0 2.8 deep-scrub ok 2024-04-06T03:51:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:35 smithi153 bash[23852]: cluster 2024-04-06T03:51:34.688963+0000 mgr.y (mgr.24368) 1122 : cluster 0 pgmap v1238: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-06T03:51:36.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[21467]: cluster 2024-04-06T03:51:34.593975+0000 osd.2 (osd.2) 607 : cluster 0 5.8 scrub starts 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[21467]: cluster 2024-04-06T03:51:34.596313+0000 osd.2 (osd.2) 608 : cluster 0 5.8 scrub ok 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[21467]: cluster 2024-04-06T03:51:35.070413+0000 osd.0 (osd.0) 720 : cluster 0 2.1f scrub starts 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[21467]: cluster 2024-04-06T03:51:35.073025+0000 osd.0 (osd.0) 721 : cluster 0 2.1f scrub ok 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[21467]: cluster 2024-04-06T03:51:35.331640+0000 osd.3 (osd.3) 741 : cluster 0 6.1b scrub starts 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[21467]: cluster 2024-04-06T03:51:35.538523+0000 osd.3 (osd.3) 742 : cluster 0 6.1b scrub ok 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[29221]: cluster 2024-04-06T03:51:34.593975+0000 osd.2 (osd.2) 607 : cluster 0 5.8 scrub starts 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[29221]: cluster 2024-04-06T03:51:34.596313+0000 osd.2 (osd.2) 608 : cluster 0 5.8 scrub ok 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[29221]: cluster 2024-04-06T03:51:35.070413+0000 osd.0 (osd.0) 720 : cluster 0 2.1f scrub starts 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[29221]: cluster 2024-04-06T03:51:35.073025+0000 osd.0 (osd.0) 721 : cluster 0 2.1f scrub ok 2024-04-06T03:51:36.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[29221]: cluster 2024-04-06T03:51:35.331640+0000 osd.3 (osd.3) 741 : cluster 0 6.1b scrub starts 2024-04-06T03:51:36.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:36 smithi050 bash[29221]: cluster 2024-04-06T03:51:35.538523+0000 osd.3 (osd.3) 742 : cluster 0 6.1b scrub ok 2024-04-06T03:51:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:36 smithi153 bash[23852]: cluster 2024-04-06T03:51:34.593975+0000 osd.2 (osd.2) 607 : cluster 0 5.8 scrub starts 2024-04-06T03:51:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:36 smithi153 bash[23852]: cluster 2024-04-06T03:51:34.596313+0000 osd.2 (osd.2) 608 : cluster 0 5.8 scrub ok 2024-04-06T03:51:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:36 smithi153 bash[23852]: cluster 2024-04-06T03:51:35.070413+0000 osd.0 (osd.0) 720 : cluster 0 2.1f scrub starts 2024-04-06T03:51:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:36 smithi153 bash[23852]: cluster 2024-04-06T03:51:35.073025+0000 osd.0 (osd.0) 721 : cluster 0 2.1f scrub ok 2024-04-06T03:51:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:36 smithi153 bash[23852]: cluster 2024-04-06T03:51:35.331640+0000 osd.3 (osd.3) 741 : cluster 0 6.1b scrub starts 2024-04-06T03:51:36.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:36 smithi153 bash[23852]: cluster 2024-04-06T03:51:35.538523+0000 osd.3 (osd.3) 742 : cluster 0 6.1b scrub ok 2024-04-06T03:51:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[21467]: audit 2024-04-06T03:51:36.130354+0000 mon.a (mon.0) 1926 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:37.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[21467]: cluster 2024-04-06T03:51:36.447578+0000 osd.7 (osd.7) 755 : cluster 0 6.8 scrub starts 2024-04-06T03:51:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[21467]: cluster 2024-04-06T03:51:36.450143+0000 osd.7 (osd.7) 756 : cluster 0 6.8 scrub ok 2024-04-06T03:51:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[21467]: cluster 2024-04-06T03:51:36.689936+0000 mgr.y (mgr.24368) 1123 : cluster 0 pgmap v1239: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:51:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[29221]: audit 2024-04-06T03:51:36.130354+0000 mon.a (mon.0) 1926 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[29221]: cluster 2024-04-06T03:51:36.447578+0000 osd.7 (osd.7) 755 : cluster 0 6.8 scrub starts 2024-04-06T03:51:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[29221]: cluster 2024-04-06T03:51:36.450143+0000 osd.7 (osd.7) 756 : cluster 0 6.8 scrub ok 2024-04-06T03:51:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:37 smithi050 bash[29221]: cluster 2024-04-06T03:51:36.689936+0000 mgr.y (mgr.24368) 1123 : cluster 0 pgmap v1239: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:51:37.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:37 smithi153 bash[23852]: audit 2024-04-06T03:51:36.130354+0000 mon.a (mon.0) 1926 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:37.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:37 smithi153 bash[23852]: cluster 2024-04-06T03:51:36.447578+0000 osd.7 (osd.7) 755 : cluster 0 6.8 scrub starts 2024-04-06T03:51:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:37 smithi153 bash[23852]: cluster 2024-04-06T03:51:36.450143+0000 osd.7 (osd.7) 756 : cluster 0 6.8 scrub ok 2024-04-06T03:51:37.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:37 smithi153 bash[23852]: cluster 2024-04-06T03:51:36.689936+0000 mgr.y (mgr.24368) 1123 : cluster 0 pgmap v1239: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:51:37.667 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:38.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[29221]: cluster 2024-04-06T03:51:36.408986+0000 osd.1 (osd.1) 505 : cluster 0 2.9 deep-scrub starts 2024-04-06T03:51:38.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[29221]: cluster 2024-04-06T03:51:36.411635+0000 osd.1 (osd.1) 506 : cluster 0 2.9 deep-scrub ok 2024-04-06T03:51:38.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[29221]: cluster 2024-04-06T03:51:37.313855+0000 osd.3 (osd.3) 743 : cluster 0 2.1e scrub starts 2024-04-06T03:51:38.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[29221]: cluster 2024-04-06T03:51:37.316222+0000 osd.3 (osd.3) 744 : cluster 0 2.1e scrub ok 2024-04-06T03:51:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[21467]: cluster 2024-04-06T03:51:36.408986+0000 osd.1 (osd.1) 505 : cluster 0 2.9 deep-scrub starts 2024-04-06T03:51:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[21467]: cluster 2024-04-06T03:51:36.411635+0000 osd.1 (osd.1) 506 : cluster 0 2.9 deep-scrub ok 2024-04-06T03:51:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[21467]: cluster 2024-04-06T03:51:37.313855+0000 osd.3 (osd.3) 743 : cluster 0 2.1e scrub starts 2024-04-06T03:51:38.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:38 smithi050 bash[21467]: cluster 2024-04-06T03:51:37.316222+0000 osd.3 (osd.3) 744 : cluster 0 2.1e scrub ok 2024-04-06T03:51:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:38 smithi153 bash[23852]: cluster 2024-04-06T03:51:36.408986+0000 osd.1 (osd.1) 505 : cluster 0 2.9 deep-scrub starts 2024-04-06T03:51:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:38 smithi153 bash[23852]: cluster 2024-04-06T03:51:36.411635+0000 osd.1 (osd.1) 506 : cluster 0 2.9 deep-scrub ok 2024-04-06T03:51:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:38 smithi153 bash[23852]: cluster 2024-04-06T03:51:37.313855+0000 osd.3 (osd.3) 743 : cluster 0 2.1e scrub starts 2024-04-06T03:51:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:38 smithi153 bash[23852]: cluster 2024-04-06T03:51:37.316222+0000 osd.3 (osd.3) 744 : cluster 0 2.1e scrub ok 2024-04-06T03:51:39.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21467]: cluster 2024-04-06T03:51:37.406853+0000 osd.1 (osd.1) 507 : cluster 0 3.0 scrub starts 2024-04-06T03:51:39.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21467]: cluster 2024-04-06T03:51:37.412994+0000 osd.1 (osd.1) 508 : cluster 0 3.0 scrub ok 2024-04-06T03:51:39.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21467]: cluster 2024-04-06T03:51:37.760276+0000 osd.2 (osd.2) 609 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:39.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21467]: cluster 2024-04-06T03:51:37.760436+0000 osd.2 (osd.2) 610 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:39.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21467]: cluster 2024-04-06T03:51:38.133984+0000 mon.a (mon.0) 1927 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-04-06T03:51:39.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21467]: cluster 2024-04-06T03:51:38.690537+0000 mgr.y (mgr.24368) 1124 : cluster 0 pgmap v1241: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:39.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[29221]: cluster 2024-04-06T03:51:37.406853+0000 osd.1 (osd.1) 507 : cluster 0 3.0 scrub starts 2024-04-06T03:51:39.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[29221]: cluster 2024-04-06T03:51:37.412994+0000 osd.1 (osd.1) 508 : cluster 0 3.0 scrub ok 2024-04-06T03:51:39.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[29221]: cluster 2024-04-06T03:51:37.760276+0000 osd.2 (osd.2) 609 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:39.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[29221]: cluster 2024-04-06T03:51:37.760436+0000 osd.2 (osd.2) 610 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:39.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[29221]: cluster 2024-04-06T03:51:38.133984+0000 mon.a (mon.0) 1927 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-04-06T03:51:39.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[29221]: cluster 2024-04-06T03:51:38.690537+0000 mgr.y (mgr.24368) 1124 : cluster 0 pgmap v1241: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:39 smithi153 bash[23852]: cluster 2024-04-06T03:51:37.406853+0000 osd.1 (osd.1) 507 : cluster 0 3.0 scrub starts 2024-04-06T03:51:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:39 smithi153 bash[23852]: cluster 2024-04-06T03:51:37.412994+0000 osd.1 (osd.1) 508 : cluster 0 3.0 scrub ok 2024-04-06T03:51:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:39 smithi153 bash[23852]: cluster 2024-04-06T03:51:37.760276+0000 osd.2 (osd.2) 609 : cluster 0 purged_snaps scrub starts 2024-04-06T03:51:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:39 smithi153 bash[23852]: cluster 2024-04-06T03:51:37.760436+0000 osd.2 (osd.2) 610 : cluster 0 purged_snaps scrub ok 2024-04-06T03:51:39.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:39 smithi153 bash[23852]: cluster 2024-04-06T03:51:38.133984+0000 mon.a (mon.0) 1927 : cluster 0 osdmap e441: 8 total, 8 up, 8 in 2024-04-06T03:51:39.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:39 smithi153 bash[23852]: cluster 2024-04-06T03:51:38.690537+0000 mgr.y (mgr.24368) 1124 : cluster 0 pgmap v1241: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:39.673 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:51:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:51:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:51:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[21467]: cluster 2024-04-06T03:51:39.093966+0000 osd.5 (osd.5) 181 : cluster 0 2.3 scrub starts 2024-04-06T03:51:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[21467]: cluster 2024-04-06T03:51:39.097861+0000 osd.5 (osd.5) 182 : cluster 0 2.3 scrub ok 2024-04-06T03:51:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[21467]: cluster 2024-04-06T03:51:39.345381+0000 osd.3 (osd.3) 745 : cluster 0 6.6 scrub starts 2024-04-06T03:51:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[21467]: cluster 2024-04-06T03:51:39.387737+0000 osd.3 (osd.3) 746 : cluster 0 6.6 scrub ok 2024-04-06T03:51:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[29221]: cluster 2024-04-06T03:51:39.093966+0000 osd.5 (osd.5) 181 : cluster 0 2.3 scrub starts 2024-04-06T03:51:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[29221]: cluster 2024-04-06T03:51:39.097861+0000 osd.5 (osd.5) 182 : cluster 0 2.3 scrub ok 2024-04-06T03:51:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[29221]: cluster 2024-04-06T03:51:39.345381+0000 osd.3 (osd.3) 745 : cluster 0 6.6 scrub starts 2024-04-06T03:51:40.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:40 smithi050 bash[29221]: cluster 2024-04-06T03:51:39.387737+0000 osd.3 (osd.3) 746 : cluster 0 6.6 scrub ok 2024-04-06T03:51:40.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:40 smithi153 bash[23852]: cluster 2024-04-06T03:51:39.093966+0000 osd.5 (osd.5) 181 : cluster 0 2.3 scrub starts 2024-04-06T03:51:40.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:40 smithi153 bash[23852]: cluster 2024-04-06T03:51:39.097861+0000 osd.5 (osd.5) 182 : cluster 0 2.3 scrub ok 2024-04-06T03:51:40.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:40 smithi153 bash[23852]: cluster 2024-04-06T03:51:39.345381+0000 osd.3 (osd.3) 745 : cluster 0 6.6 scrub starts 2024-04-06T03:51:40.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:40 smithi153 bash[23852]: cluster 2024-04-06T03:51:39.387737+0000 osd.3 (osd.3) 746 : cluster 0 6.6 scrub ok 2024-04-06T03:51:41.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:51:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:51:41.180 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:51:41.181 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":441,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:10:26.969128+0000","modified":"2024-04-06T03:51:38.117473+0000","last_up_change":"2024-04-06T03:20:09.072730+0000","last_in_change":"2024-04-06T03:48:31.370747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:15:48.096900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-06T03:20:50.011930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-06T03:20:51.687715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-06T03:20:53.301921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-06T03:20:53.810316+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-06T03:20:56.321502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e607f527-4592-4acd-a47f-76575615ae1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6803","nonce":3531754646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6805","nonce":3531754646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6809","nonce":3531754646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6807","nonce":3531754646}]},"public_addr":"172.21.15.50:6803/3531754646","cluster_addr":"172.21.15.50:6805/3531754646","heartbeat_back_addr":"172.21.15.50:6809/3531754646","heartbeat_front_addr":"172.21.15.50:6807/3531754646","state":["exists","up"]},{"osd":1,"uuid":"2ae8f784-d36a-4633-8630-2aed04e4ff10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":413,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6811","nonce":4128410069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6813","nonce":4128410069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6817","nonce":4128410069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6815","nonce":4128410069}]},"public_addr":"172.21.15.50:6811/4128410069","cluster_addr":"172.21.15.50:6813/4128410069","heartbeat_back_addr":"172.21.15.50:6817/4128410069","heartbeat_front_addr":"172.21.15.50:6815/4128410069","state":["exists","up"]},{"osd":2,"uuid":"43f31d09-5c16-4aea-8822-e0d877cc931b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6819","nonce":1307247904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6821","nonce":1307247904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6825","nonce":1307247904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6823","nonce":1307247904}]},"public_addr":"172.21.15.50:6819/1307247904","cluster_addr":"172.21.15.50:6821/1307247904","heartbeat_back_addr":"172.21.15.50:6825/1307247904","heartbeat_front_addr":"172.21.15.50:6823/1307247904","state":["exists","up"]},{"osd":3,"uuid":"4c913325-442e-4ee5-94f1-8b5bca0c3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6827","nonce":1397888508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6829","nonce":1397888508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6833","nonce":1397888508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6831","nonce":1397888508}]},"public_addr":"172.21.15.50:6827/1397888508","cluster_addr":"172.21.15.50:6829/1397888508","heartbeat_back_addr":"172.21.15.50:6833/1397888508","heartbeat_front_addr":"172.21.15.50:6831/1397888508","state":["exists","up"]},{"osd":4,"uuid":"50c02bfc-c2f2-4aae-b1d5-c38a86726b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6801","nonce":3377683473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6803","nonce":3377683473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6807","nonce":3377683473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6805","nonce":3377683473}]},"public_addr":"172.21.15.153:6801/3377683473","cluster_addr":"172.21.15.153:6803/3377683473","heartbeat_back_addr":"172.21.15.153:6807/3377683473","heartbeat_front_addr":"172.21.15.153:6805/3377683473","state":["exists","up"]},{"osd":5,"uuid":"c3e8194b-2c20-4ffb-859d-00c33902b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":430,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6809","nonce":357179117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6811","nonce":357179117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6815","nonce":357179117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6813","nonce":357179117}]},"public_addr":"172.21.15.153:6809/357179117","cluster_addr":"172.21.15.153:6811/357179117","heartbeat_back_addr":"172.21.15.153:6815/357179117","heartbeat_front_addr":"172.21.15.153:6813/357179117","state":["exists","up"]},{"osd":6,"uuid":"98027d53-4324-44d1-89ce-8da85be58efb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":435,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6817","nonce":1552538208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6819","nonce":1552538208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6823","nonce":1552538208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6821","nonce":1552538208}]},"public_addr":"172.21.15.153:6817/1552538208","cluster_addr":"172.21.15.153:6819/1552538208","heartbeat_back_addr":"172.21.15.153:6823/1552538208","heartbeat_front_addr":"172.21.15.153:6821/1552538208","state":["exists","up"]},{"osd":7,"uuid":"6b47d95a-1228-4fb1-a315-5371d9b42bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":406,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6825","nonce":2479193924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6827","nonce":2479193924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6831","nonce":2479193924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6829","nonce":2479193924}]},"public_addr":"172.21.15.153:6825/2479193924","cluster_addr":"172.21.15.153:6827/2479193924","heartbeat_back_addr":"172.21.15.153:6831/2479193924","heartbeat_front_addr":"172.21.15.153:6829/2479193924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:18.150130+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:14.302136+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:37.760444+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:03.147097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:50:44.977985+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:04.866545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:00.536995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:21.480847+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.12","mappings":[{"from":7,"to":2}]},{"pgid":"3.4","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/283275195":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2738019433":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1268273764":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/567103210":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/2891059051":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2170219983":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/4082841962":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6800/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/570623774":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6801/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1379780741":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2880368026":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2044358308":"2024-04-07T03:21:47.915758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:51:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:41 smithi050 bash[21467]: audit 2024-04-06T03:51:40.552024+0000 mgr.y (mgr.24368) 1125 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:41 smithi050 bash[21467]: cluster 2024-04-06T03:51:40.691283+0000 mgr.y (mgr.24368) 1126 : cluster 0 pgmap v1242: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:41 smithi050 bash[29221]: audit 2024-04-06T03:51:40.552024+0000 mgr.y (mgr.24368) 1125 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:41 smithi050 bash[29221]: cluster 2024-04-06T03:51:40.691283+0000 mgr.y (mgr.24368) 1126 : cluster 0 pgmap v1242: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:41 smithi153 bash[23852]: audit 2024-04-06T03:51:40.552024+0000 mgr.y (mgr.24368) 1125 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:41 smithi153 bash[23852]: cluster 2024-04-06T03:51:40.691283+0000 mgr.y (mgr.24368) 1126 : cluster 0 pgmap v1242: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:41.946 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-06T03:51:41.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-06T03:51:42.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[21467]: cluster 2024-04-06T03:51:41.063133+0000 osd.5 (osd.5) 183 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:51:42.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[21467]: audit 2024-04-06T03:51:41.180273+0000 mon.a (mon.0) 1928 : audit 0 from='client.? 172.21.15.50:0/233996101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:51:42.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[21467]: cluster 2024-04-06T03:51:41.225566+0000 osd.5 (osd.5) 184 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:51:42.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[21467]: cluster 2024-04-06T03:51:41.332152+0000 osd.3 (osd.3) 747 : cluster 0 2.1b scrub starts 2024-04-06T03:51:42.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[21467]: cluster 2024-04-06T03:51:41.334625+0000 osd.3 (osd.3) 748 : cluster 0 2.1b scrub ok 2024-04-06T03:51:42.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[21467]: audit 2024-04-06T03:51:42.091299+0000 mon.a (mon.0) 1929 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:51:42.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[29221]: cluster 2024-04-06T03:51:41.063133+0000 osd.5 (osd.5) 183 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:51:42.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[29221]: audit 2024-04-06T03:51:41.180273+0000 mon.a (mon.0) 1928 : audit 0 from='client.? 172.21.15.50:0/233996101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:51:42.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[29221]: cluster 2024-04-06T03:51:41.225566+0000 osd.5 (osd.5) 184 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:51:42.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[29221]: cluster 2024-04-06T03:51:41.332152+0000 osd.3 (osd.3) 747 : cluster 0 2.1b scrub starts 2024-04-06T03:51:42.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[29221]: cluster 2024-04-06T03:51:41.334625+0000 osd.3 (osd.3) 748 : cluster 0 2.1b scrub ok 2024-04-06T03:51:42.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:42 smithi050 bash[29221]: audit 2024-04-06T03:51:42.091299+0000 mon.a (mon.0) 1929 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:51:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:42 smithi153 bash[23852]: cluster 2024-04-06T03:51:41.063133+0000 osd.5 (osd.5) 183 : cluster 0 6.7 deep-scrub starts 2024-04-06T03:51:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:42 smithi153 bash[23852]: audit 2024-04-06T03:51:41.180273+0000 mon.a (mon.0) 1928 : audit 0 from='client.? 172.21.15.50:0/233996101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:51:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:42 smithi153 bash[23852]: cluster 2024-04-06T03:51:41.225566+0000 osd.5 (osd.5) 184 : cluster 0 6.7 deep-scrub ok 2024-04-06T03:51:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:42 smithi153 bash[23852]: cluster 2024-04-06T03:51:41.332152+0000 osd.3 (osd.3) 747 : cluster 0 2.1b scrub starts 2024-04-06T03:51:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:42 smithi153 bash[23852]: cluster 2024-04-06T03:51:41.334625+0000 osd.3 (osd.3) 748 : cluster 0 2.1b scrub ok 2024-04-06T03:51:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:42 smithi153 bash[23852]: audit 2024-04-06T03:51:42.091299+0000 mon.a (mon.0) 1929 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: cluster 2024-04-06T03:51:41.755450+0000 osd.6 (osd.6) 127 : cluster 0 2.11 scrub starts 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: cluster 2024-04-06T03:51:41.758236+0000 osd.6 (osd.6) 128 : cluster 0 2.11 scrub ok 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: cluster 2024-04-06T03:51:42.544072+0000 osd.7 (osd.7) 757 : cluster 0 5.4 scrub starts 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: cluster 2024-04-06T03:51:42.545397+0000 osd.7 (osd.7) 758 : cluster 0 5.4 scrub ok 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: cluster 2024-04-06T03:51:42.692141+0000 mgr.y (mgr.24368) 1127 : cluster 0 pgmap v1243: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: audit 2024-04-06T03:51:42.773144+0000 mon.a (mon.0) 1930 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: audit 2024-04-06T03:51:42.774919+0000 mon.a (mon.0) 1931 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[21467]: audit 2024-04-06T03:51:42.787137+0000 mon.a (mon.0) 1932 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: cluster 2024-04-06T03:51:41.755450+0000 osd.6 (osd.6) 127 : cluster 0 2.11 scrub starts 2024-04-06T03:51:43.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: cluster 2024-04-06T03:51:41.758236+0000 osd.6 (osd.6) 128 : cluster 0 2.11 scrub ok 2024-04-06T03:51:43.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: cluster 2024-04-06T03:51:42.544072+0000 osd.7 (osd.7) 757 : cluster 0 5.4 scrub starts 2024-04-06T03:51:43.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: cluster 2024-04-06T03:51:42.545397+0000 osd.7 (osd.7) 758 : cluster 0 5.4 scrub ok 2024-04-06T03:51:43.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: cluster 2024-04-06T03:51:42.692141+0000 mgr.y (mgr.24368) 1127 : cluster 0 pgmap v1243: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:43.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: audit 2024-04-06T03:51:42.773144+0000 mon.a (mon.0) 1930 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:51:43.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: audit 2024-04-06T03:51:42.774919+0000 mon.a (mon.0) 1931 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:51:43.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:43 smithi050 bash[29221]: audit 2024-04-06T03:51:42.787137+0000 mon.a (mon.0) 1932 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:51:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: cluster 2024-04-06T03:51:41.755450+0000 osd.6 (osd.6) 127 : cluster 0 2.11 scrub starts 2024-04-06T03:51:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: cluster 2024-04-06T03:51:41.758236+0000 osd.6 (osd.6) 128 : cluster 0 2.11 scrub ok 2024-04-06T03:51:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: cluster 2024-04-06T03:51:42.544072+0000 osd.7 (osd.7) 757 : cluster 0 5.4 scrub starts 2024-04-06T03:51:43.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: cluster 2024-04-06T03:51:42.545397+0000 osd.7 (osd.7) 758 : cluster 0 5.4 scrub ok 2024-04-06T03:51:43.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: cluster 2024-04-06T03:51:42.692141+0000 mgr.y (mgr.24368) 1127 : cluster 0 pgmap v1243: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:43.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: audit 2024-04-06T03:51:42.773144+0000 mon.a (mon.0) 1930 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:51:43.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: audit 2024-04-06T03:51:42.774919+0000 mon.a (mon.0) 1931 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:51:43.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:43 smithi153 bash[23852]: audit 2024-04-06T03:51:42.787137+0000 mon.a (mon.0) 1932 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:51:45.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:43.330927+0000 osd.3 (osd.3) 749 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:43.335931+0000 osd.3 (osd.3) 750 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:43.426883+0000 osd.1 (osd.1) 509 : cluster 0 5.19 scrub starts 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:43.429700+0000 osd.1 (osd.1) 510 : cluster 0 5.19 scrub ok 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:44.001683+0000 osd.5 (osd.5) 185 : cluster 0 5.15 deep-scrub starts 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:44.005132+0000 osd.5 (osd.5) 186 : cluster 0 5.15 deep-scrub ok 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[21467]: cluster 2024-04-06T03:51:44.693437+0000 mgr.y (mgr.24368) 1128 : cluster 0 pgmap v1244: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:43.330927+0000 osd.3 (osd.3) 749 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:43.335931+0000 osd.3 (osd.3) 750 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:43.426883+0000 osd.1 (osd.1) 509 : cluster 0 5.19 scrub starts 2024-04-06T03:51:45.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:43.429700+0000 osd.1 (osd.1) 510 : cluster 0 5.19 scrub ok 2024-04-06T03:51:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:44.001683+0000 osd.5 (osd.5) 185 : cluster 0 5.15 deep-scrub starts 2024-04-06T03:51:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:44.005132+0000 osd.5 (osd.5) 186 : cluster 0 5.15 deep-scrub ok 2024-04-06T03:51:45.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:45 smithi050 bash[29221]: cluster 2024-04-06T03:51:44.693437+0000 mgr.y (mgr.24368) 1128 : cluster 0 pgmap v1244: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:45.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:43.330927+0000 osd.3 (osd.3) 749 : cluster 0 3.1e deep-scrub starts 2024-04-06T03:51:45.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:43.335931+0000 osd.3 (osd.3) 750 : cluster 0 3.1e deep-scrub ok 2024-04-06T03:51:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:43.426883+0000 osd.1 (osd.1) 509 : cluster 0 5.19 scrub starts 2024-04-06T03:51:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:43.429700+0000 osd.1 (osd.1) 510 : cluster 0 5.19 scrub ok 2024-04-06T03:51:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:44.001683+0000 osd.5 (osd.5) 185 : cluster 0 5.15 deep-scrub starts 2024-04-06T03:51:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:44.005132+0000 osd.5 (osd.5) 186 : cluster 0 5.15 deep-scrub ok 2024-04-06T03:51:45.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:45 smithi153 bash[23852]: cluster 2024-04-06T03:51:44.693437+0000 mgr.y (mgr.24368) 1128 : cluster 0 pgmap v1244: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:46.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[21467]: cluster 2024-04-06T03:51:45.024455+0000 osd.5 (osd.5) 187 : cluster 0 2.16 scrub starts 2024-04-06T03:51:46.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[21467]: cluster 2024-04-06T03:51:45.026829+0000 osd.5 (osd.5) 188 : cluster 0 2.16 scrub ok 2024-04-06T03:51:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[21467]: cluster 2024-04-06T03:51:45.357833+0000 osd.3 (osd.3) 751 : cluster 0 3.b scrub starts 2024-04-06T03:51:46.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[21467]: cluster 2024-04-06T03:51:45.362113+0000 osd.3 (osd.3) 752 : cluster 0 3.b scrub ok 2024-04-06T03:51:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[29221]: cluster 2024-04-06T03:51:45.024455+0000 osd.5 (osd.5) 187 : cluster 0 2.16 scrub starts 2024-04-06T03:51:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[29221]: cluster 2024-04-06T03:51:45.026829+0000 osd.5 (osd.5) 188 : cluster 0 2.16 scrub ok 2024-04-06T03:51:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[29221]: cluster 2024-04-06T03:51:45.357833+0000 osd.3 (osd.3) 751 : cluster 0 3.b scrub starts 2024-04-06T03:51:46.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:46 smithi050 bash[29221]: cluster 2024-04-06T03:51:45.362113+0000 osd.3 (osd.3) 752 : cluster 0 3.b scrub ok 2024-04-06T03:51:46.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:46 smithi153 bash[23852]: cluster 2024-04-06T03:51:45.024455+0000 osd.5 (osd.5) 187 : cluster 0 2.16 scrub starts 2024-04-06T03:51:46.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:46 smithi153 bash[23852]: cluster 2024-04-06T03:51:45.026829+0000 osd.5 (osd.5) 188 : cluster 0 2.16 scrub ok 2024-04-06T03:51:46.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:46 smithi153 bash[23852]: cluster 2024-04-06T03:51:45.357833+0000 osd.3 (osd.3) 751 : cluster 0 3.b scrub starts 2024-04-06T03:51:46.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:46 smithi153 bash[23852]: cluster 2024-04-06T03:51:45.362113+0000 osd.3 (osd.3) 752 : cluster 0 3.b scrub ok 2024-04-06T03:51:46.747 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:47 smithi050 bash[21467]: cluster 2024-04-06T03:51:46.157113+0000 osd.0 (osd.0) 722 : cluster 0 3.1b scrub starts 2024-04-06T03:51:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:47 smithi050 bash[21467]: cluster 2024-04-06T03:51:46.161090+0000 osd.0 (osd.0) 723 : cluster 0 3.1b scrub ok 2024-04-06T03:51:47.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:47 smithi050 bash[21467]: cluster 2024-04-06T03:51:46.694304+0000 mgr.y (mgr.24368) 1129 : cluster 0 pgmap v1245: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:47 smithi050 bash[29221]: cluster 2024-04-06T03:51:46.157113+0000 osd.0 (osd.0) 722 : cluster 0 3.1b scrub starts 2024-04-06T03:51:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:47 smithi050 bash[29221]: cluster 2024-04-06T03:51:46.161090+0000 osd.0 (osd.0) 723 : cluster 0 3.1b scrub ok 2024-04-06T03:51:47.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:47 smithi050 bash[29221]: cluster 2024-04-06T03:51:46.694304+0000 mgr.y (mgr.24368) 1129 : cluster 0 pgmap v1245: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:47 smithi153 bash[23852]: cluster 2024-04-06T03:51:46.157113+0000 osd.0 (osd.0) 722 : cluster 0 3.1b scrub starts 2024-04-06T03:51:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:47 smithi153 bash[23852]: cluster 2024-04-06T03:51:46.161090+0000 osd.0 (osd.0) 723 : cluster 0 3.1b scrub ok 2024-04-06T03:51:47.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:47 smithi153 bash[23852]: cluster 2024-04-06T03:51:46.694304+0000 mgr.y (mgr.24368) 1129 : cluster 0 pgmap v1245: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:51:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[21467]: cluster 2024-04-06T03:51:46.442083+0000 osd.1 (osd.1) 511 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:51:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[21467]: cluster 2024-04-06T03:51:46.526926+0000 osd.1 (osd.1) 512 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:51:48.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[21467]: cluster 2024-04-06T03:51:47.115168+0000 osd.0 (osd.0) 724 : cluster 0 3.1 deep-scrub starts 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[21467]: cluster 2024-04-06T03:51:47.133572+0000 osd.0 (osd.0) 725 : cluster 0 3.1 deep-scrub ok 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[21467]: cluster 2024-04-06T03:51:47.295830+0000 osd.4 (osd.4) 717 : cluster 0 5.1 scrub starts 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[21467]: cluster 2024-04-06T03:51:47.298148+0000 osd.4 (osd.4) 718 : cluster 0 5.1 scrub ok 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[29221]: cluster 2024-04-06T03:51:46.442083+0000 osd.1 (osd.1) 511 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[29221]: cluster 2024-04-06T03:51:46.526926+0000 osd.1 (osd.1) 512 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[29221]: cluster 2024-04-06T03:51:47.115168+0000 osd.0 (osd.0) 724 : cluster 0 3.1 deep-scrub starts 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[29221]: cluster 2024-04-06T03:51:47.133572+0000 osd.0 (osd.0) 725 : cluster 0 3.1 deep-scrub ok 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[29221]: cluster 2024-04-06T03:51:47.295830+0000 osd.4 (osd.4) 717 : cluster 0 5.1 scrub starts 2024-04-06T03:51:48.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:48 smithi050 bash[29221]: cluster 2024-04-06T03:51:47.298148+0000 osd.4 (osd.4) 718 : cluster 0 5.1 scrub ok 2024-04-06T03:51:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:48 smithi153 bash[23852]: cluster 2024-04-06T03:51:46.442083+0000 osd.1 (osd.1) 511 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:51:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:48 smithi153 bash[23852]: cluster 2024-04-06T03:51:46.526926+0000 osd.1 (osd.1) 512 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:51:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:48 smithi153 bash[23852]: cluster 2024-04-06T03:51:47.115168+0000 osd.0 (osd.0) 724 : cluster 0 3.1 deep-scrub starts 2024-04-06T03:51:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:48 smithi153 bash[23852]: cluster 2024-04-06T03:51:47.133572+0000 osd.0 (osd.0) 725 : cluster 0 3.1 deep-scrub ok 2024-04-06T03:51:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:48 smithi153 bash[23852]: cluster 2024-04-06T03:51:47.295830+0000 osd.4 (osd.4) 717 : cluster 0 5.1 scrub starts 2024-04-06T03:51:48.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:48 smithi153 bash[23852]: cluster 2024-04-06T03:51:47.298148+0000 osd.4 (osd.4) 718 : cluster 0 5.1 scrub ok 2024-04-06T03:51:49.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:49 smithi050 bash[21467]: cluster 2024-04-06T03:51:48.695181+0000 mgr.y (mgr.24368) 1130 : cluster 0 pgmap v1246: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:51:49.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:49 smithi050 bash[29221]: cluster 2024-04-06T03:51:48.695181+0000 mgr.y (mgr.24368) 1130 : cluster 0 pgmap v1246: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:51:49.491 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:51:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:51:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:51:49.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:49 smithi153 bash[23852]: cluster 2024-04-06T03:51:48.695181+0000 mgr.y (mgr.24368) 1130 : cluster 0 pgmap v1246: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:51:50.144 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:51:50.144 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":441,"fsid":"0313e520-f3c3-11ee-b649-cb9ed24678a4","created":"2024-04-06T03:10:26.969128+0000","modified":"2024-04-06T03:51:38.117473+0000","last_up_change":"2024-04-06T03:20:09.072730+0000","last_in_change":"2024-04-06T03:48:31.370747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-06T03:15:48.096900+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-06T03:20:50.011930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-06T03:20:51.687715+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-06T03:20:53.301921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":88,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-06T03:20:53.810316+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-06T03:20:56.321502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e607f527-4592-4acd-a47f-76575615ae1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6803","nonce":3531754646}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6805","nonce":3531754646}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6809","nonce":3531754646}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3531754646},{"type":"v1","addr":"172.21.15.50:6807","nonce":3531754646}]},"public_addr":"172.21.15.50:6803/3531754646","cluster_addr":"172.21.15.50:6805/3531754646","heartbeat_back_addr":"172.21.15.50:6809/3531754646","heartbeat_front_addr":"172.21.15.50:6807/3531754646","state":["exists","up"]},{"osd":1,"uuid":"2ae8f784-d36a-4633-8630-2aed04e4ff10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":413,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6811","nonce":4128410069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6813","nonce":4128410069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6817","nonce":4128410069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":4128410069},{"type":"v1","addr":"172.21.15.50:6815","nonce":4128410069}]},"public_addr":"172.21.15.50:6811/4128410069","cluster_addr":"172.21.15.50:6813/4128410069","heartbeat_back_addr":"172.21.15.50:6817/4128410069","heartbeat_front_addr":"172.21.15.50:6815/4128410069","state":["exists","up"]},{"osd":2,"uuid":"43f31d09-5c16-4aea-8822-e0d877cc931b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6819","nonce":1307247904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6821","nonce":1307247904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6825","nonce":1307247904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1307247904},{"type":"v1","addr":"172.21.15.50:6823","nonce":1307247904}]},"public_addr":"172.21.15.50:6819/1307247904","cluster_addr":"172.21.15.50:6821/1307247904","heartbeat_back_addr":"172.21.15.50:6825/1307247904","heartbeat_front_addr":"172.21.15.50:6823/1307247904","state":["exists","up"]},{"osd":3,"uuid":"4c913325-442e-4ee5-94f1-8b5bca0c3240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6827","nonce":1397888508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6829","nonce":1397888508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6833","nonce":1397888508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":1397888508},{"type":"v1","addr":"172.21.15.50:6831","nonce":1397888508}]},"public_addr":"172.21.15.50:6827/1397888508","cluster_addr":"172.21.15.50:6829/1397888508","heartbeat_back_addr":"172.21.15.50:6833/1397888508","heartbeat_front_addr":"172.21.15.50:6831/1397888508","state":["exists","up"]},{"osd":4,"uuid":"50c02bfc-c2f2-4aae-b1d5-c38a86726b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":403,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6801","nonce":3377683473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6803","nonce":3377683473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6807","nonce":3377683473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":3377683473},{"type":"v1","addr":"172.21.15.153:6805","nonce":3377683473}]},"public_addr":"172.21.15.153:6801/3377683473","cluster_addr":"172.21.15.153:6803/3377683473","heartbeat_back_addr":"172.21.15.153:6807/3377683473","heartbeat_front_addr":"172.21.15.153:6805/3377683473","state":["exists","up"]},{"osd":5,"uuid":"c3e8194b-2c20-4ffb-859d-00c33902b241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":430,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6809","nonce":357179117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6811","nonce":357179117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6815","nonce":357179117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":357179117},{"type":"v1","addr":"172.21.15.153:6813","nonce":357179117}]},"public_addr":"172.21.15.153:6809/357179117","cluster_addr":"172.21.15.153:6811/357179117","heartbeat_back_addr":"172.21.15.153:6815/357179117","heartbeat_front_addr":"172.21.15.153:6813/357179117","state":["exists","up"]},{"osd":6,"uuid":"98027d53-4324-44d1-89ce-8da85be58efb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":435,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6817","nonce":1552538208}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6819","nonce":1552538208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6823","nonce":1552538208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":1552538208},{"type":"v1","addr":"172.21.15.153:6821","nonce":1552538208}]},"public_addr":"172.21.15.153:6817/1552538208","cluster_addr":"172.21.15.153:6819/1552538208","heartbeat_back_addr":"172.21.15.153:6823/1552538208","heartbeat_front_addr":"172.21.15.153:6821/1552538208","state":["exists","up"]},{"osd":7,"uuid":"6b47d95a-1228-4fb1-a315-5371d9b42bbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":406,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6825","nonce":2479193924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6827","nonce":2479193924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6831","nonce":2479193924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2479193924},{"type":"v1","addr":"172.21.15.153:6829","nonce":2479193924}]},"public_addr":"172.21.15.153:6825/2479193924","cluster_addr":"172.21.15.153:6827/2479193924","heartbeat_back_addr":"172.21.15.153:6831/2479193924","heartbeat_front_addr":"172.21.15.153:6829/2479193924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:18.150130+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:14.302136+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:37.760444+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:03.147097+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:50:44.977985+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:04.866545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:00.536995+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-06T03:51:21.480847+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.12","mappings":[{"from":7,"to":2}]},{"pgid":"3.4","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/283275195":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2738019433":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1268273764":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/567103210":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/2891059051":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2170219983":"2024-04-07T03:11:33.442338+0000","172.21.15.50:6800/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/4082841962":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6800/1263219005":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/570623774":"2024-04-07T03:21:47.915758+0000","172.21.15.50:6801/889382629":"2024-04-07T03:10:58.614019+0000","172.21.15.50:0/1379780741":"2024-04-07T03:21:47.915758+0000","172.21.15.50:0/2880368026":"2024-04-07T03:10:58.614019+0000","172.21.15.50:6801/132418263":"2024-04-07T03:11:33.442338+0000","172.21.15.50:0/2044358308":"2024-04-07T03:21:47.915758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-06T03:51:50.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[29221]: cluster 2024-04-06T03:51:48.821784+0000 osd.6 (osd.6) 129 : cluster 0 2.7 scrub starts 2024-04-06T03:51:50.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[29221]: cluster 2024-04-06T03:51:48.824453+0000 osd.6 (osd.6) 130 : cluster 0 2.7 scrub ok 2024-04-06T03:51:50.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[29221]: cluster 2024-04-06T03:51:49.164439+0000 osd.0 (osd.0) 726 : cluster 0 6.10 scrub starts 2024-04-06T03:51:50.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[29221]: cluster 2024-04-06T03:51:49.257162+0000 osd.0 (osd.0) 727 : cluster 0 6.10 scrub ok 2024-04-06T03:51:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[21467]: cluster 2024-04-06T03:51:48.821784+0000 osd.6 (osd.6) 129 : cluster 0 2.7 scrub starts 2024-04-06T03:51:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[21467]: cluster 2024-04-06T03:51:48.824453+0000 osd.6 (osd.6) 130 : cluster 0 2.7 scrub ok 2024-04-06T03:51:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[21467]: cluster 2024-04-06T03:51:49.164439+0000 osd.0 (osd.0) 726 : cluster 0 6.10 scrub starts 2024-04-06T03:51:50.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:50 smithi050 bash[21467]: cluster 2024-04-06T03:51:49.257162+0000 osd.0 (osd.0) 727 : cluster 0 6.10 scrub ok 2024-04-06T03:51:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:50 smithi153 bash[23852]: cluster 2024-04-06T03:51:48.821784+0000 osd.6 (osd.6) 129 : cluster 0 2.7 scrub starts 2024-04-06T03:51:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:50 smithi153 bash[23852]: cluster 2024-04-06T03:51:48.824453+0000 osd.6 (osd.6) 130 : cluster 0 2.7 scrub ok 2024-04-06T03:51:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:50 smithi153 bash[23852]: cluster 2024-04-06T03:51:49.164439+0000 osd.0 (osd.0) 726 : cluster 0 6.10 scrub starts 2024-04-06T03:51:50.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:50 smithi153 bash[23852]: cluster 2024-04-06T03:51:49.257162+0000 osd.0 (osd.0) 727 : cluster 0 6.10 scrub ok 2024-04-06T03:51:51.012 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-06T03:51:51.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-06T03:51:51.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-06T03:51:51.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-06T03:51:51.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-06T03:51:51.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-06T03:51:51.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-06T03:51:51.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-06T03:51:51.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:51:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:51:51.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:49.835043+0000 osd.6 (osd.6) 131 : cluster 0 5.1f scrub starts 2024-04-06T03:51:51.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:49.838062+0000 osd.6 (osd.6) 132 : cluster 0 5.1f scrub ok 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: audit 2024-04-06T03:51:50.143673+0000 mon.c (mon.2) 137 : audit 0 from='client.? 172.21.15.50:0/3295821816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:50.434757+0000 osd.3 (osd.3) 753 : cluster 0 6.c scrub starts 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:50.437439+0000 osd.3 (osd.3) 754 : cluster 0 6.c scrub ok 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:50.464545+0000 osd.7 (osd.7) 759 : cluster 0 3.e scrub starts 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:50.468961+0000 osd.7 (osd.7) 760 : cluster 0 3.e scrub ok 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: audit 2024-04-06T03:51:50.559943+0000 mgr.y (mgr.24368) 1131 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[21467]: cluster 2024-04-06T03:51:50.696767+0000 mgr.y (mgr.24368) 1132 : cluster 0 pgmap v1247: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:49.835043+0000 osd.6 (osd.6) 131 : cluster 0 5.1f scrub starts 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:49.838062+0000 osd.6 (osd.6) 132 : cluster 0 5.1f scrub ok 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: audit 2024-04-06T03:51:50.143673+0000 mon.c (mon.2) 137 : audit 0 from='client.? 172.21.15.50:0/3295821816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:51:51.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:50.434757+0000 osd.3 (osd.3) 753 : cluster 0 6.c scrub starts 2024-04-06T03:51:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:50.437439+0000 osd.3 (osd.3) 754 : cluster 0 6.c scrub ok 2024-04-06T03:51:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:50.464545+0000 osd.7 (osd.7) 759 : cluster 0 3.e scrub starts 2024-04-06T03:51:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:50.468961+0000 osd.7 (osd.7) 760 : cluster 0 3.e scrub ok 2024-04-06T03:51:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: audit 2024-04-06T03:51:50.559943+0000 mgr.y (mgr.24368) 1131 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:51.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:51 smithi050 bash[29221]: cluster 2024-04-06T03:51:50.696767+0000 mgr.y (mgr.24368) 1132 : cluster 0 pgmap v1247: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:49.835043+0000 osd.6 (osd.6) 131 : cluster 0 5.1f scrub starts 2024-04-06T03:51:51.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:49.838062+0000 osd.6 (osd.6) 132 : cluster 0 5.1f scrub ok 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: audit 2024-04-06T03:51:50.143673+0000 mon.c (mon.2) 137 : audit 0 from='client.? 172.21.15.50:0/3295821816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:50.434757+0000 osd.3 (osd.3) 753 : cluster 0 6.c scrub starts 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:50.437439+0000 osd.3 (osd.3) 754 : cluster 0 6.c scrub ok 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:50.464545+0000 osd.7 (osd.7) 759 : cluster 0 3.e scrub starts 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:50.468961+0000 osd.7 (osd.7) 760 : cluster 0 3.e scrub ok 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: audit 2024-04-06T03:51:50.559943+0000 mgr.y (mgr.24368) 1131 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:51:51.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:51 smithi153 bash[23852]: cluster 2024-04-06T03:51:50.696767+0000 mgr.y (mgr.24368) 1132 : cluster 0 pgmap v1247: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:52.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:52 smithi050 bash[21467]: audit 2024-04-06T03:51:51.164147+0000 mon.a (mon.0) 1933 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:52.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:52 smithi050 bash[21467]: cluster 2024-04-06T03:51:51.417866+0000 osd.3 (osd.3) 755 : cluster 0 5.14 scrub starts 2024-04-06T03:51:52.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:52 smithi050 bash[21467]: cluster 2024-04-06T03:51:51.419745+0000 osd.3 (osd.3) 756 : cluster 0 5.14 scrub ok 2024-04-06T03:51:52.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:52 smithi050 bash[29221]: audit 2024-04-06T03:51:51.164147+0000 mon.a (mon.0) 1933 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:52.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:52 smithi050 bash[29221]: cluster 2024-04-06T03:51:51.417866+0000 osd.3 (osd.3) 755 : cluster 0 5.14 scrub starts 2024-04-06T03:51:52.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:52 smithi050 bash[29221]: cluster 2024-04-06T03:51:51.419745+0000 osd.3 (osd.3) 756 : cluster 0 5.14 scrub ok 2024-04-06T03:51:52.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:52 smithi153 bash[23852]: audit 2024-04-06T03:51:51.164147+0000 mon.a (mon.0) 1933 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:51:52.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:52 smithi153 bash[23852]: cluster 2024-04-06T03:51:51.417866+0000 osd.3 (osd.3) 755 : cluster 0 5.14 scrub starts 2024-04-06T03:51:52.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:52 smithi153 bash[23852]: cluster 2024-04-06T03:51:51.419745+0000 osd.3 (osd.3) 756 : cluster 0 5.14 scrub ok 2024-04-06T03:51:53.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:53 smithi050 bash[21467]: cluster 2024-04-06T03:51:52.288434+0000 osd.4 (osd.4) 719 : cluster 0 6.e scrub starts 2024-04-06T03:51:53.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:53 smithi050 bash[21467]: cluster 2024-04-06T03:51:52.371592+0000 osd.4 (osd.4) 720 : cluster 0 6.e scrub ok 2024-04-06T03:51:53.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:53 smithi050 bash[21467]: cluster 2024-04-06T03:51:52.697504+0000 mgr.y (mgr.24368) 1133 : cluster 0 pgmap v1248: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:53 smithi050 bash[29221]: cluster 2024-04-06T03:51:52.288434+0000 osd.4 (osd.4) 719 : cluster 0 6.e scrub starts 2024-04-06T03:51:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:53 smithi050 bash[29221]: cluster 2024-04-06T03:51:52.371592+0000 osd.4 (osd.4) 720 : cluster 0 6.e scrub ok 2024-04-06T03:51:53.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:53 smithi050 bash[29221]: cluster 2024-04-06T03:51:52.697504+0000 mgr.y (mgr.24368) 1133 : cluster 0 pgmap v1248: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:53.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:53 smithi153 bash[23852]: cluster 2024-04-06T03:51:52.288434+0000 osd.4 (osd.4) 719 : cluster 0 6.e scrub starts 2024-04-06T03:51:53.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:53 smithi153 bash[23852]: cluster 2024-04-06T03:51:52.371592+0000 osd.4 (osd.4) 720 : cluster 0 6.e scrub ok 2024-04-06T03:51:53.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:53 smithi153 bash[23852]: cluster 2024-04-06T03:51:52.697504+0000 mgr.y (mgr.24368) 1133 : cluster 0 pgmap v1248: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:54.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[21467]: cluster 2024-04-06T03:51:52.900055+0000 osd.5 (osd.5) 189 : cluster 0 2.18 scrub starts 2024-04-06T03:51:54.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[21467]: cluster 2024-04-06T03:51:52.902528+0000 osd.5 (osd.5) 190 : cluster 0 2.18 scrub ok 2024-04-06T03:51:54.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[21467]: cluster 2024-04-06T03:51:53.592195+0000 osd.7 (osd.7) 761 : cluster 0 5.1a scrub starts 2024-04-06T03:51:54.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[21467]: cluster 2024-04-06T03:51:53.594686+0000 osd.7 (osd.7) 762 : cluster 0 5.1a scrub ok 2024-04-06T03:51:54.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[29221]: cluster 2024-04-06T03:51:52.900055+0000 osd.5 (osd.5) 189 : cluster 0 2.18 scrub starts 2024-04-06T03:51:54.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[29221]: cluster 2024-04-06T03:51:52.902528+0000 osd.5 (osd.5) 190 : cluster 0 2.18 scrub ok 2024-04-06T03:51:54.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[29221]: cluster 2024-04-06T03:51:53.592195+0000 osd.7 (osd.7) 761 : cluster 0 5.1a scrub starts 2024-04-06T03:51:54.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:54 smithi050 bash[29221]: cluster 2024-04-06T03:51:53.594686+0000 osd.7 (osd.7) 762 : cluster 0 5.1a scrub ok 2024-04-06T03:51:54.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:54 smithi153 bash[23852]: cluster 2024-04-06T03:51:52.900055+0000 osd.5 (osd.5) 189 : cluster 0 2.18 scrub starts 2024-04-06T03:51:54.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:54 smithi153 bash[23852]: cluster 2024-04-06T03:51:52.902528+0000 osd.5 (osd.5) 190 : cluster 0 2.18 scrub ok 2024-04-06T03:51:54.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:54 smithi153 bash[23852]: cluster 2024-04-06T03:51:53.592195+0000 osd.7 (osd.7) 761 : cluster 0 5.1a scrub starts 2024-04-06T03:51:54.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:54 smithi153 bash[23852]: cluster 2024-04-06T03:51:53.594686+0000 osd.7 (osd.7) 762 : cluster 0 5.1a scrub ok 2024-04-06T03:51:55.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:55 smithi050 bash[21467]: cluster 2024-04-06T03:51:54.699203+0000 mgr.y (mgr.24368) 1134 : cluster 0 pgmap v1249: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:51:55.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:55 smithi050 bash[29221]: cluster 2024-04-06T03:51:54.699203+0000 mgr.y (mgr.24368) 1134 : cluster 0 pgmap v1249: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:51:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:55 smithi153 bash[23852]: cluster 2024-04-06T03:51:54.699203+0000 mgr.y (mgr.24368) 1134 : cluster 0 pgmap v1249: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-06T03:51:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.950 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.951 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.951 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.951 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.952 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.952 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:55.954 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:51:56.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[21467]: cluster 2024-04-06T03:51:54.616030+0000 osd.2 (osd.2) 611 : cluster 0 5.b scrub starts 2024-04-06T03:51:56.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[21467]: cluster 2024-04-06T03:51:54.618570+0000 osd.2 (osd.2) 612 : cluster 0 5.b scrub ok 2024-04-06T03:51:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[21467]: cluster 2024-04-06T03:51:55.398447+0000 osd.3 (osd.3) 757 : cluster 0 5.17 scrub starts 2024-04-06T03:51:56.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[21467]: cluster 2024-04-06T03:51:55.400931+0000 osd.3 (osd.3) 758 : cluster 0 5.17 scrub ok 2024-04-06T03:51:56.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[29221]: cluster 2024-04-06T03:51:54.616030+0000 osd.2 (osd.2) 611 : cluster 0 5.b scrub starts 2024-04-06T03:51:56.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[29221]: cluster 2024-04-06T03:51:54.618570+0000 osd.2 (osd.2) 612 : cluster 0 5.b scrub ok 2024-04-06T03:51:56.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[29221]: cluster 2024-04-06T03:51:55.398447+0000 osd.3 (osd.3) 757 : cluster 0 5.17 scrub starts 2024-04-06T03:51:56.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:56 smithi050 bash[29221]: cluster 2024-04-06T03:51:55.400931+0000 osd.3 (osd.3) 758 : cluster 0 5.17 scrub ok 2024-04-06T03:51:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:56 smithi153 bash[23852]: cluster 2024-04-06T03:51:54.616030+0000 osd.2 (osd.2) 611 : cluster 0 5.b scrub starts 2024-04-06T03:51:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:56 smithi153 bash[23852]: cluster 2024-04-06T03:51:54.618570+0000 osd.2 (osd.2) 612 : cluster 0 5.b scrub ok 2024-04-06T03:51:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:56 smithi153 bash[23852]: cluster 2024-04-06T03:51:55.398447+0000 osd.3 (osd.3) 757 : cluster 0 5.17 scrub starts 2024-04-06T03:51:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:56 smithi153 bash[23852]: cluster 2024-04-06T03:51:55.400931+0000 osd.3 (osd.3) 758 : cluster 0 5.17 scrub ok 2024-04-06T03:51:57.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.447403+0000 osd.3 (osd.3) 759 : cluster 0 3.7 scrub starts 2024-04-06T03:51:57.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.451932+0000 osd.3 (osd.3) 760 : cluster 0 3.7 scrub ok 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.555251+0000 osd.2 (osd.2) 613 : cluster 0 2.10 scrub starts 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.557962+0000 osd.2 (osd.2) 614 : cluster 0 2.10 scrub ok 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.699941+0000 mgr.y (mgr.24368) 1135 : cluster 0 pgmap v1250: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.447403+0000 osd.3 (osd.3) 759 : cluster 0 3.7 scrub starts 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.451932+0000 osd.3 (osd.3) 760 : cluster 0 3.7 scrub ok 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.555251+0000 osd.2 (osd.2) 613 : cluster 0 2.10 scrub starts 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.557962+0000 osd.2 (osd.2) 614 : cluster 0 2.10 scrub ok 2024-04-06T03:51:57.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:57 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.699941+0000 mgr.y (mgr.24368) 1135 : cluster 0 pgmap v1250: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:57 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.447403+0000 osd.3 (osd.3) 759 : cluster 0 3.7 scrub starts 2024-04-06T03:51:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:57 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.451932+0000 osd.3 (osd.3) 760 : cluster 0 3.7 scrub ok 2024-04-06T03:51:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:57 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.555251+0000 osd.2 (osd.2) 613 : cluster 0 2.10 scrub starts 2024-04-06T03:51:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:57 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.557962+0000 osd.2 (osd.2) 614 : cluster 0 2.10 scrub ok 2024-04-06T03:51:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:57 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.699941+0000 mgr.y (mgr.24368) 1135 : cluster 0 pgmap v1250: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:58.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.820443+0000 osd.6 (osd.6) 133 : cluster 0 2.1a scrub starts 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[21467]: cluster 2024-04-06T03:51:56.823060+0000 osd.6 (osd.6) 134 : cluster 0 2.1a scrub ok 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[21467]: cluster 2024-04-06T03:51:57.326088+0000 osd.4 (osd.4) 721 : cluster 0 6.2 scrub starts 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[21467]: cluster 2024-04-06T03:51:57.327972+0000 osd.4 (osd.4) 722 : cluster 0 6.2 scrub ok 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[21467]: cluster 2024-04-06T03:51:57.435668+0000 osd.3 (osd.3) 761 : cluster 0 3.8 scrub starts 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[21467]: cluster 2024-04-06T03:51:57.441530+0000 osd.3 (osd.3) 762 : cluster 0 3.8 scrub ok 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.820443+0000 osd.6 (osd.6) 133 : cluster 0 2.1a scrub starts 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[29221]: cluster 2024-04-06T03:51:56.823060+0000 osd.6 (osd.6) 134 : cluster 0 2.1a scrub ok 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[29221]: cluster 2024-04-06T03:51:57.326088+0000 osd.4 (osd.4) 721 : cluster 0 6.2 scrub starts 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[29221]: cluster 2024-04-06T03:51:57.327972+0000 osd.4 (osd.4) 722 : cluster 0 6.2 scrub ok 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[29221]: cluster 2024-04-06T03:51:57.435668+0000 osd.3 (osd.3) 761 : cluster 0 3.8 scrub starts 2024-04-06T03:51:58.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:58 smithi050 bash[29221]: cluster 2024-04-06T03:51:57.441530+0000 osd.3 (osd.3) 762 : cluster 0 3.8 scrub ok 2024-04-06T03:51:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:58 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.820443+0000 osd.6 (osd.6) 133 : cluster 0 2.1a scrub starts 2024-04-06T03:51:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:58 smithi153 bash[23852]: cluster 2024-04-06T03:51:56.823060+0000 osd.6 (osd.6) 134 : cluster 0 2.1a scrub ok 2024-04-06T03:51:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:58 smithi153 bash[23852]: cluster 2024-04-06T03:51:57.326088+0000 osd.4 (osd.4) 721 : cluster 0 6.2 scrub starts 2024-04-06T03:51:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:58 smithi153 bash[23852]: cluster 2024-04-06T03:51:57.327972+0000 osd.4 (osd.4) 722 : cluster 0 6.2 scrub ok 2024-04-06T03:51:58.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:58 smithi153 bash[23852]: cluster 2024-04-06T03:51:57.435668+0000 osd.3 (osd.3) 761 : cluster 0 3.8 scrub starts 2024-04-06T03:51:58.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:58 smithi153 bash[23852]: cluster 2024-04-06T03:51:57.441530+0000 osd.3 (osd.3) 762 : cluster 0 3.8 scrub ok 2024-04-06T03:51:59.380 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:51:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:51:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[21467]: cluster 2024-04-06T03:51:58.259753+0000 osd.0 (osd.0) 728 : cluster 0 3.1f scrub starts 2024-04-06T03:51:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[21467]: cluster 2024-04-06T03:51:58.262483+0000 osd.0 (osd.0) 729 : cluster 0 3.1f scrub ok 2024-04-06T03:51:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[21467]: cluster 2024-04-06T03:51:58.700879+0000 mgr.y (mgr.24368) 1136 : cluster 0 pgmap v1251: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:59.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[29221]: cluster 2024-04-06T03:51:58.259753+0000 osd.0 (osd.0) 728 : cluster 0 3.1f scrub starts 2024-04-06T03:51:59.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[29221]: cluster 2024-04-06T03:51:58.262483+0000 osd.0 (osd.0) 729 : cluster 0 3.1f scrub ok 2024-04-06T03:51:59.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:51:59 smithi050 bash[29221]: cluster 2024-04-06T03:51:58.700879+0000 mgr.y (mgr.24368) 1136 : cluster 0 pgmap v1251: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:51:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:59 smithi153 bash[23852]: cluster 2024-04-06T03:51:58.259753+0000 osd.0 (osd.0) 728 : cluster 0 3.1f scrub starts 2024-04-06T03:51:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:59 smithi153 bash[23852]: cluster 2024-04-06T03:51:58.262483+0000 osd.0 (osd.0) 729 : cluster 0 3.1f scrub ok 2024-04-06T03:51:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:51:59 smithi153 bash[23852]: cluster 2024-04-06T03:51:58.700879+0000 mgr.y (mgr.24368) 1136 : cluster 0 pgmap v1251: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:52:00.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.466213+0000 osd.3 (osd.3) 763 : cluster 0 4.0 scrub starts 2024-04-06T03:52:00.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.468808+0000 osd.3 (osd.3) 764 : cluster 0 4.0 scrub ok 2024-04-06T03:52:00.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.567333+0000 osd.1 (osd.1) 513 : cluster 0 2.a deep-scrub starts 2024-04-06T03:52:00.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.569682+0000 osd.1 (osd.1) 514 : cluster 0 2.a deep-scrub ok 2024-04-06T03:52:00.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.466213+0000 osd.3 (osd.3) 763 : cluster 0 4.0 scrub starts 2024-04-06T03:52:00.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.468808+0000 osd.3 (osd.3) 764 : cluster 0 4.0 scrub ok 2024-04-06T03:52:00.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.567333+0000 osd.1 (osd.1) 513 : cluster 0 2.a deep-scrub starts 2024-04-06T03:52:00.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:00 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.569682+0000 osd.1 (osd.1) 514 : cluster 0 2.a deep-scrub ok 2024-04-06T03:52:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.466213+0000 osd.3 (osd.3) 763 : cluster 0 4.0 scrub starts 2024-04-06T03:52:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.468808+0000 osd.3 (osd.3) 764 : cluster 0 4.0 scrub ok 2024-04-06T03:52:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.567333+0000 osd.1 (osd.1) 513 : cluster 0 2.a deep-scrub starts 2024-04-06T03:52:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:00 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.569682+0000 osd.1 (osd.1) 514 : cluster 0 2.a deep-scrub ok 2024-04-06T03:52:01.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:52:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:52:01.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.651493+0000 osd.7 (osd.7) 763 : cluster 0 3.f scrub starts 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.656446+0000 osd.7 (osd.7) 764 : cluster 0 3.f scrub ok 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.847273+0000 osd.5 (osd.5) 191 : cluster 0 5.f scrub starts 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:51:59.849728+0000 osd.5 (osd.5) 192 : cluster 0 5.f scrub ok 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:52:00.215528+0000 osd.0 (osd.0) 730 : cluster 0 2.13 scrub starts 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:52:00.217471+0000 osd.0 (osd.0) 731 : cluster 0 2.13 scrub ok 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:52:00.503718+0000 osd.3 (osd.3) 765 : cluster 0 6.b scrub starts 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: audit 2024-04-06T03:52:00.567936+0000 mgr.y (mgr.24368) 1137 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:52:00.702427+0000 mgr.y (mgr.24368) 1138 : cluster 0 pgmap v1252: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[29221]: cluster 2024-04-06T03:52:00.708372+0000 osd.3 (osd.3) 766 : cluster 0 6.b scrub ok 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.651493+0000 osd.7 (osd.7) 763 : cluster 0 3.f scrub starts 2024-04-06T03:52:01.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.656446+0000 osd.7 (osd.7) 764 : cluster 0 3.f scrub ok 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.847273+0000 osd.5 (osd.5) 191 : cluster 0 5.f scrub starts 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:51:59.849728+0000 osd.5 (osd.5) 192 : cluster 0 5.f scrub ok 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:52:00.215528+0000 osd.0 (osd.0) 730 : cluster 0 2.13 scrub starts 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:52:00.217471+0000 osd.0 (osd.0) 731 : cluster 0 2.13 scrub ok 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:52:00.503718+0000 osd.3 (osd.3) 765 : cluster 0 6.b scrub starts 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: audit 2024-04-06T03:52:00.567936+0000 mgr.y (mgr.24368) 1137 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:52:00.702427+0000 mgr.y (mgr.24368) 1138 : cluster 0 pgmap v1252: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:52:01.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:01 smithi050 bash[21467]: cluster 2024-04-06T03:52:00.708372+0000 osd.3 (osd.3) 766 : cluster 0 6.b scrub ok 2024-04-06T03:52:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.651493+0000 osd.7 (osd.7) 763 : cluster 0 3.f scrub starts 2024-04-06T03:52:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.656446+0000 osd.7 (osd.7) 764 : cluster 0 3.f scrub ok 2024-04-06T03:52:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.847273+0000 osd.5 (osd.5) 191 : cluster 0 5.f scrub starts 2024-04-06T03:52:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:51:59.849728+0000 osd.5 (osd.5) 192 : cluster 0 5.f scrub ok 2024-04-06T03:52:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:52:00.215528+0000 osd.0 (osd.0) 730 : cluster 0 2.13 scrub starts 2024-04-06T03:52:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:52:00.217471+0000 osd.0 (osd.0) 731 : cluster 0 2.13 scrub ok 2024-04-06T03:52:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:52:00.503718+0000 osd.3 (osd.3) 765 : cluster 0 6.b scrub starts 2024-04-06T03:52:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: audit 2024-04-06T03:52:00.567936+0000 mgr.y (mgr.24368) 1137 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:52:00.702427+0000 mgr.y (mgr.24368) 1138 : cluster 0 pgmap v1252: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-06T03:52:01.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:01 smithi153 bash[23852]: cluster 2024-04-06T03:52:00.708372+0000 osd.3 (osd.3) 766 : cluster 0 6.b scrub ok 2024-04-06T03:52:02.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[29221]: cluster 2024-04-06T03:52:01.199994+0000 osd.0 (osd.0) 732 : cluster 0 6.16 scrub starts 2024-04-06T03:52:02.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[29221]: cluster 2024-04-06T03:52:01.295933+0000 osd.0 (osd.0) 733 : cluster 0 6.16 scrub ok 2024-04-06T03:52:02.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[29221]: cluster 2024-04-06T03:52:01.386346+0000 osd.4 (osd.4) 723 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:02.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[29221]: cluster 2024-04-06T03:52:01.386445+0000 osd.4 (osd.4) 724 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[21467]: cluster 2024-04-06T03:52:01.199994+0000 osd.0 (osd.0) 732 : cluster 0 6.16 scrub starts 2024-04-06T03:52:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[21467]: cluster 2024-04-06T03:52:01.295933+0000 osd.0 (osd.0) 733 : cluster 0 6.16 scrub ok 2024-04-06T03:52:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[21467]: cluster 2024-04-06T03:52:01.386346+0000 osd.4 (osd.4) 723 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:02.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:02 smithi050 bash[21467]: cluster 2024-04-06T03:52:01.386445+0000 osd.4 (osd.4) 724 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:02 smithi153 bash[23852]: cluster 2024-04-06T03:52:01.199994+0000 osd.0 (osd.0) 732 : cluster 0 6.16 scrub starts 2024-04-06T03:52:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:02 smithi153 bash[23852]: cluster 2024-04-06T03:52:01.295933+0000 osd.0 (osd.0) 733 : cluster 0 6.16 scrub ok 2024-04-06T03:52:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:02 smithi153 bash[23852]: cluster 2024-04-06T03:52:01.386346+0000 osd.4 (osd.4) 723 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:02.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:02 smithi153 bash[23852]: cluster 2024-04-06T03:52:01.386445+0000 osd.4 (osd.4) 724 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:03.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:01.863541+0000 osd.5 (osd.5) 193 : cluster 0 6.d scrub starts 2024-04-06T03:52:03.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.035681+0000 osd.5 (osd.5) 194 : cluster 0 6.d scrub ok 2024-04-06T03:52:03.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.151505+0000 osd.0 (osd.0) 734 : cluster 0 6.9 deep-scrub starts 2024-04-06T03:52:03.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.225082+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-04-06T03:52:03.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.315416+0000 osd.0 (osd.0) 735 : cluster 0 6.9 deep-scrub ok 2024-04-06T03:52:03.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.561057+0000 osd.1 (osd.1) 515 : cluster 0 2.4 scrub starts 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.563102+0000 osd.1 (osd.1) 516 : cluster 0 2.4 scrub ok 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.703099+0000 mgr.y (mgr.24368) 1139 : cluster 0 pgmap v1254: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:01.863541+0000 osd.5 (osd.5) 193 : cluster 0 6.d scrub starts 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.035681+0000 osd.5 (osd.5) 194 : cluster 0 6.d scrub ok 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.151505+0000 osd.0 (osd.0) 734 : cluster 0 6.9 deep-scrub starts 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.225082+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.315416+0000 osd.0 (osd.0) 735 : cluster 0 6.9 deep-scrub ok 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.561057+0000 osd.1 (osd.1) 515 : cluster 0 2.4 scrub starts 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.563102+0000 osd.1 (osd.1) 516 : cluster 0 2.4 scrub ok 2024-04-06T03:52:03.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:03 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.703099+0000 mgr.y (mgr.24368) 1139 : cluster 0 pgmap v1254: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:01.863541+0000 osd.5 (osd.5) 193 : cluster 0 6.d scrub starts 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.035681+0000 osd.5 (osd.5) 194 : cluster 0 6.d scrub ok 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.151505+0000 osd.0 (osd.0) 734 : cluster 0 6.9 deep-scrub starts 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.225082+0000 mon.a (mon.0) 1934 : cluster 0 osdmap e442: 8 total, 8 up, 8 in 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.315416+0000 osd.0 (osd.0) 735 : cluster 0 6.9 deep-scrub ok 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.561057+0000 osd.1 (osd.1) 515 : cluster 0 2.4 scrub starts 2024-04-06T03:52:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.563102+0000 osd.1 (osd.1) 516 : cluster 0 2.4 scrub ok 2024-04-06T03:52:03.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:03 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.703099+0000 mgr.y (mgr.24368) 1139 : cluster 0 pgmap v1254: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:04.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.838832+0000 osd.5 (osd.5) 195 : cluster 0 3.16 scrub starts 2024-04-06T03:52:04.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[29221]: cluster 2024-04-06T03:52:02.843307+0000 osd.5 (osd.5) 196 : cluster 0 3.16 scrub ok 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[29221]: cluster 2024-04-06T03:52:03.144155+0000 osd.0 (osd.0) 736 : cluster 0 5.5 scrub starts 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[29221]: cluster 2024-04-06T03:52:03.146079+0000 osd.0 (osd.0) 737 : cluster 0 5.5 scrub ok 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[29221]: cluster 2024-04-06T03:52:03.507524+0000 osd.4 (osd.4) 725 : cluster 0 3.14 scrub starts 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[29221]: cluster 2024-04-06T03:52:03.512992+0000 osd.4 (osd.4) 726 : cluster 0 3.14 scrub ok 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.838832+0000 osd.5 (osd.5) 195 : cluster 0 3.16 scrub starts 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[21467]: cluster 2024-04-06T03:52:02.843307+0000 osd.5 (osd.5) 196 : cluster 0 3.16 scrub ok 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[21467]: cluster 2024-04-06T03:52:03.144155+0000 osd.0 (osd.0) 736 : cluster 0 5.5 scrub starts 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[21467]: cluster 2024-04-06T03:52:03.146079+0000 osd.0 (osd.0) 737 : cluster 0 5.5 scrub ok 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[21467]: cluster 2024-04-06T03:52:03.507524+0000 osd.4 (osd.4) 725 : cluster 0 3.14 scrub starts 2024-04-06T03:52:04.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:04 smithi050 bash[21467]: cluster 2024-04-06T03:52:03.512992+0000 osd.4 (osd.4) 726 : cluster 0 3.14 scrub ok 2024-04-06T03:52:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:04 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.838832+0000 osd.5 (osd.5) 195 : cluster 0 3.16 scrub starts 2024-04-06T03:52:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:04 smithi153 bash[23852]: cluster 2024-04-06T03:52:02.843307+0000 osd.5 (osd.5) 196 : cluster 0 3.16 scrub ok 2024-04-06T03:52:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:04 smithi153 bash[23852]: cluster 2024-04-06T03:52:03.144155+0000 osd.0 (osd.0) 736 : cluster 0 5.5 scrub starts 2024-04-06T03:52:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:04 smithi153 bash[23852]: cluster 2024-04-06T03:52:03.146079+0000 osd.0 (osd.0) 737 : cluster 0 5.5 scrub ok 2024-04-06T03:52:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:04 smithi153 bash[23852]: cluster 2024-04-06T03:52:03.507524+0000 osd.4 (osd.4) 725 : cluster 0 3.14 scrub starts 2024-04-06T03:52:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:04 smithi153 bash[23852]: cluster 2024-04-06T03:52:03.512992+0000 osd.4 (osd.4) 726 : cluster 0 3.14 scrub ok 2024-04-06T03:52:05.423 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[29221]: cluster 2024-04-06T03:52:03.824150+0000 osd.6 (osd.6) 135 : cluster 0 5.2 scrub starts 2024-04-06T03:52:05.423 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[29221]: cluster 2024-04-06T03:52:03.826775+0000 osd.6 (osd.6) 136 : cluster 0 5.2 scrub ok 2024-04-06T03:52:05.424 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[29221]: cluster 2024-04-06T03:52:04.559779+0000 osd.2 (osd.2) 615 : cluster 0 5.6 deep-scrub starts 2024-04-06T03:52:05.424 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[29221]: cluster 2024-04-06T03:52:04.562391+0000 osd.2 (osd.2) 616 : cluster 0 5.6 deep-scrub ok 2024-04-06T03:52:05.424 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[29221]: cluster 2024-04-06T03:52:04.704670+0000 mgr.y (mgr.24368) 1140 : cluster 0 pgmap v1255: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:05.424 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[21467]: cluster 2024-04-06T03:52:03.824150+0000 osd.6 (osd.6) 135 : cluster 0 5.2 scrub starts 2024-04-06T03:52:05.424 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[21467]: cluster 2024-04-06T03:52:03.826775+0000 osd.6 (osd.6) 136 : cluster 0 5.2 scrub ok 2024-04-06T03:52:05.425 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[21467]: cluster 2024-04-06T03:52:04.559779+0000 osd.2 (osd.2) 615 : cluster 0 5.6 deep-scrub starts 2024-04-06T03:52:05.425 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[21467]: cluster 2024-04-06T03:52:04.562391+0000 osd.2 (osd.2) 616 : cluster 0 5.6 deep-scrub ok 2024-04-06T03:52:05.425 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:05 smithi050 bash[21467]: cluster 2024-04-06T03:52:04.704670+0000 mgr.y (mgr.24368) 1140 : cluster 0 pgmap v1255: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:05 smithi153 bash[23852]: cluster 2024-04-06T03:52:03.824150+0000 osd.6 (osd.6) 135 : cluster 0 5.2 scrub starts 2024-04-06T03:52:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:05 smithi153 bash[23852]: cluster 2024-04-06T03:52:03.826775+0000 osd.6 (osd.6) 136 : cluster 0 5.2 scrub ok 2024-04-06T03:52:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:05 smithi153 bash[23852]: cluster 2024-04-06T03:52:04.559779+0000 osd.2 (osd.2) 615 : cluster 0 5.6 deep-scrub starts 2024-04-06T03:52:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:05 smithi153 bash[23852]: cluster 2024-04-06T03:52:04.562391+0000 osd.2 (osd.2) 616 : cluster 0 5.6 deep-scrub ok 2024-04-06T03:52:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:05 smithi153 bash[23852]: cluster 2024-04-06T03:52:04.704670+0000 mgr.y (mgr.24368) 1140 : cluster 0 pgmap v1255: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:06.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.071417+0000 osd.5 (osd.5) 197 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:06.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.071511+0000 osd.5 (osd.5) 198 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:06.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.245699+0000 mon.a (mon.0) 1935 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-04-06T03:52:06.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.607002+0000 osd.7 (osd.7) 765 : cluster 0 2.0 scrub starts 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.612526+0000 osd.7 (osd.7) 766 : cluster 0 2.0 scrub ok 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[29221]: audit 2024-04-06T03:52:06.193965+0000 mon.a (mon.0) 1936 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.071417+0000 osd.5 (osd.5) 197 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.071511+0000 osd.5 (osd.5) 198 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.245699+0000 mon.a (mon.0) 1935 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.607002+0000 osd.7 (osd.7) 765 : cluster 0 2.0 scrub starts 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.612526+0000 osd.7 (osd.7) 766 : cluster 0 2.0 scrub ok 2024-04-06T03:52:06.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:06 smithi050 bash[21467]: audit 2024-04-06T03:52:06.193965+0000 mon.a (mon.0) 1936 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:06 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.071417+0000 osd.5 (osd.5) 197 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:06 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.071511+0000 osd.5 (osd.5) 198 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:06 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.245699+0000 mon.a (mon.0) 1935 : cluster 0 osdmap e443: 8 total, 8 up, 8 in 2024-04-06T03:52:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:06 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.607002+0000 osd.7 (osd.7) 765 : cluster 0 2.0 scrub starts 2024-04-06T03:52:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:06 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.612526+0000 osd.7 (osd.7) 766 : cluster 0 2.0 scrub ok 2024-04-06T03:52:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:06 smithi153 bash[23852]: audit 2024-04-06T03:52:06.193965+0000 mon.a (mon.0) 1936 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:07 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.921222+0000 osd.5 (osd.5) 199 : cluster 0 5.1b scrub starts 2024-04-06T03:52:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:07 smithi153 bash[23852]: cluster 2024-04-06T03:52:05.923695+0000 osd.5 (osd.5) 200 : cluster 0 5.1b scrub ok 2024-04-06T03:52:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:07 smithi153 bash[23852]: cluster 2024-04-06T03:52:06.531563+0000 osd.1 (osd.1) 517 : cluster 0 5.1d scrub starts 2024-04-06T03:52:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:07 smithi153 bash[23852]: cluster 2024-04-06T03:52:06.534146+0000 osd.1 (osd.1) 518 : cluster 0 5.1d scrub ok 2024-04-06T03:52:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:07 smithi153 bash[23852]: cluster 2024-04-06T03:52:06.705388+0000 mgr.y (mgr.24368) 1141 : cluster 0 pgmap v1257: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:07.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.921222+0000 osd.5 (osd.5) 199 : cluster 0 5.1b scrub starts 2024-04-06T03:52:07.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[21467]: cluster 2024-04-06T03:52:05.923695+0000 osd.5 (osd.5) 200 : cluster 0 5.1b scrub ok 2024-04-06T03:52:07.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[21467]: cluster 2024-04-06T03:52:06.531563+0000 osd.1 (osd.1) 517 : cluster 0 5.1d scrub starts 2024-04-06T03:52:07.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[21467]: cluster 2024-04-06T03:52:06.534146+0000 osd.1 (osd.1) 518 : cluster 0 5.1d scrub ok 2024-04-06T03:52:07.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[21467]: cluster 2024-04-06T03:52:06.705388+0000 mgr.y (mgr.24368) 1141 : cluster 0 pgmap v1257: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:07.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.921222+0000 osd.5 (osd.5) 199 : cluster 0 5.1b scrub starts 2024-04-06T03:52:07.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[29221]: cluster 2024-04-06T03:52:05.923695+0000 osd.5 (osd.5) 200 : cluster 0 5.1b scrub ok 2024-04-06T03:52:07.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[29221]: cluster 2024-04-06T03:52:06.531563+0000 osd.1 (osd.1) 517 : cluster 0 5.1d scrub starts 2024-04-06T03:52:07.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[29221]: cluster 2024-04-06T03:52:06.534146+0000 osd.1 (osd.1) 518 : cluster 0 5.1d scrub ok 2024-04-06T03:52:07.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:07 smithi050 bash[29221]: cluster 2024-04-06T03:52:06.705388+0000 mgr.y (mgr.24368) 1141 : cluster 0 pgmap v1257: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:08.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:08 smithi050 bash[21467]: cluster 2024-04-06T03:52:07.144094+0000 osd.0 (osd.0) 738 : cluster 0 3.6 deep-scrub starts 2024-04-06T03:52:08.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:08 smithi050 bash[21467]: cluster 2024-04-06T03:52:07.147598+0000 osd.0 (osd.0) 739 : cluster 0 3.6 deep-scrub ok 2024-04-06T03:52:08.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:08 smithi050 bash[29221]: cluster 2024-04-06T03:52:07.144094+0000 osd.0 (osd.0) 738 : cluster 0 3.6 deep-scrub starts 2024-04-06T03:52:08.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:08 smithi050 bash[29221]: cluster 2024-04-06T03:52:07.147598+0000 osd.0 (osd.0) 739 : cluster 0 3.6 deep-scrub ok 2024-04-06T03:52:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:08 smithi153 bash[23852]: cluster 2024-04-06T03:52:07.144094+0000 osd.0 (osd.0) 738 : cluster 0 3.6 deep-scrub starts 2024-04-06T03:52:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:08 smithi153 bash[23852]: cluster 2024-04-06T03:52:07.147598+0000 osd.0 (osd.0) 739 : cluster 0 3.6 deep-scrub ok 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:07.811421+0000 osd.6 (osd.6) 137 : cluster 0 3.a scrub starts 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:07.817255+0000 osd.6 (osd.6) 138 : cluster 0 3.a scrub ok 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:08.483455+0000 osd.4 (osd.4) 727 : cluster 0 5.e deep-scrub starts 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:08.486732+0000 osd.4 (osd.4) 728 : cluster 0 5.e deep-scrub ok 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:08.504671+0000 osd.3 (osd.3) 767 : cluster 0 6.1f scrub starts 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:08.706148+0000 mgr.y (mgr.24368) 1142 : cluster 0 pgmap v1258: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21467]: cluster 2024-04-06T03:52:08.753544+0000 osd.3 (osd.3) 768 : cluster 0 6.1f scrub ok 2024-04-06T03:52:09.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:07.811421+0000 osd.6 (osd.6) 137 : cluster 0 3.a scrub starts 2024-04-06T03:52:09.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:07.817255+0000 osd.6 (osd.6) 138 : cluster 0 3.a scrub ok 2024-04-06T03:52:09.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:08.483455+0000 osd.4 (osd.4) 727 : cluster 0 5.e deep-scrub starts 2024-04-06T03:52:09.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:08.486732+0000 osd.4 (osd.4) 728 : cluster 0 5.e deep-scrub ok 2024-04-06T03:52:09.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:08.504671+0000 osd.3 (osd.3) 767 : cluster 0 6.1f scrub starts 2024-04-06T03:52:09.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:08.706148+0000 mgr.y (mgr.24368) 1142 : cluster 0 pgmap v1258: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:09.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[29221]: cluster 2024-04-06T03:52:08.753544+0000 osd.3 (osd.3) 768 : cluster 0 6.1f scrub ok 2024-04-06T03:52:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:07.811421+0000 osd.6 (osd.6) 137 : cluster 0 3.a scrub starts 2024-04-06T03:52:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:07.817255+0000 osd.6 (osd.6) 138 : cluster 0 3.a scrub ok 2024-04-06T03:52:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:08.483455+0000 osd.4 (osd.4) 727 : cluster 0 5.e deep-scrub starts 2024-04-06T03:52:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:08.486732+0000 osd.4 (osd.4) 728 : cluster 0 5.e deep-scrub ok 2024-04-06T03:52:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:08.504671+0000 osd.3 (osd.3) 767 : cluster 0 6.1f scrub starts 2024-04-06T03:52:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:08.706148+0000 mgr.y (mgr.24368) 1142 : cluster 0 pgmap v1258: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:09.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:09 smithi153 bash[23852]: cluster 2024-04-06T03:52:08.753544+0000 osd.3 (osd.3) 768 : cluster 0 6.1f scrub ok 2024-04-06T03:52:09.672 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:52:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:52:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:52:10.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:10 smithi153 bash[23852]: cluster 2024-04-06T03:52:09.570254+0000 osd.2 (osd.2) 617 : cluster 0 6.14 scrub starts 2024-04-06T03:52:10.608 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:10 smithi153 bash[23852]: cluster 2024-04-06T03:52:09.572885+0000 osd.2 (osd.2) 618 : cluster 0 6.14 scrub ok 2024-04-06T03:52:10.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:10 smithi050 bash[29221]: cluster 2024-04-06T03:52:09.570254+0000 osd.2 (osd.2) 617 : cluster 0 6.14 scrub starts 2024-04-06T03:52:10.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:10 smithi050 bash[29221]: cluster 2024-04-06T03:52:09.572885+0000 osd.2 (osd.2) 618 : cluster 0 6.14 scrub ok 2024-04-06T03:52:10.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:10 smithi050 bash[21467]: cluster 2024-04-06T03:52:09.570254+0000 osd.2 (osd.2) 617 : cluster 0 6.14 scrub starts 2024-04-06T03:52:10.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:10 smithi050 bash[21467]: cluster 2024-04-06T03:52:09.572885+0000 osd.2 (osd.2) 618 : cluster 0 6.14 scrub ok 2024-04-06T03:52:11.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:52:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:52:11.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:11 smithi050 bash[21467]: audit 2024-04-06T03:52:10.579284+0000 mgr.y (mgr.24368) 1143 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:11.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:11 smithi050 bash[21467]: cluster 2024-04-06T03:52:10.707805+0000 mgr.y (mgr.24368) 1144 : cluster 0 pgmap v1259: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:11.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:11 smithi050 bash[29221]: audit 2024-04-06T03:52:10.579284+0000 mgr.y (mgr.24368) 1143 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:11.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:11 smithi050 bash[29221]: cluster 2024-04-06T03:52:10.707805+0000 mgr.y (mgr.24368) 1144 : cluster 0 pgmap v1259: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:11.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:11 smithi153 bash[23852]: audit 2024-04-06T03:52:10.579284+0000 mgr.y (mgr.24368) 1143 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:11.550 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:11 smithi153 bash[23852]: cluster 2024-04-06T03:52:10.707805+0000 mgr.y (mgr.24368) 1144 : cluster 0 pgmap v1259: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:12.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:12 smithi153 bash[23852]: cluster 2024-04-06T03:52:10.619562+0000 osd.2 (osd.2) 619 : cluster 0 5.a deep-scrub starts 2024-04-06T03:52:12.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:12 smithi153 bash[23852]: cluster 2024-04-06T03:52:10.622086+0000 osd.2 (osd.2) 620 : cluster 0 5.a deep-scrub ok 2024-04-06T03:52:12.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:12 smithi153 bash[23852]: cluster 2024-04-06T03:52:10.676689+0000 osd.7 (osd.7) 767 : cluster 0 2.5 scrub starts 2024-04-06T03:52:12.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:12 smithi153 bash[23852]: cluster 2024-04-06T03:52:10.678925+0000 osd.7 (osd.7) 768 : cluster 0 2.5 scrub ok 2024-04-06T03:52:12.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:12 smithi153 bash[23852]: cluster 2024-04-06T03:52:11.505097+0000 osd.3 (osd.3) 769 : cluster 0 2.19 scrub starts 2024-04-06T03:52:12.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:12 smithi153 bash[23852]: cluster 2024-04-06T03:52:11.507252+0000 osd.3 (osd.3) 770 : cluster 0 2.19 scrub ok 2024-04-06T03:52:12.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[29221]: cluster 2024-04-06T03:52:10.619562+0000 osd.2 (osd.2) 619 : cluster 0 5.a deep-scrub starts 2024-04-06T03:52:12.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[29221]: cluster 2024-04-06T03:52:10.622086+0000 osd.2 (osd.2) 620 : cluster 0 5.a deep-scrub ok 2024-04-06T03:52:12.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[29221]: cluster 2024-04-06T03:52:10.676689+0000 osd.7 (osd.7) 767 : cluster 0 2.5 scrub starts 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[29221]: cluster 2024-04-06T03:52:10.678925+0000 osd.7 (osd.7) 768 : cluster 0 2.5 scrub ok 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[29221]: cluster 2024-04-06T03:52:11.505097+0000 osd.3 (osd.3) 769 : cluster 0 2.19 scrub starts 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[29221]: cluster 2024-04-06T03:52:11.507252+0000 osd.3 (osd.3) 770 : cluster 0 2.19 scrub ok 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[21467]: cluster 2024-04-06T03:52:10.619562+0000 osd.2 (osd.2) 619 : cluster 0 5.a deep-scrub starts 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[21467]: cluster 2024-04-06T03:52:10.622086+0000 osd.2 (osd.2) 620 : cluster 0 5.a deep-scrub ok 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[21467]: cluster 2024-04-06T03:52:10.676689+0000 osd.7 (osd.7) 767 : cluster 0 2.5 scrub starts 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[21467]: cluster 2024-04-06T03:52:10.678925+0000 osd.7 (osd.7) 768 : cluster 0 2.5 scrub ok 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[21467]: cluster 2024-04-06T03:52:11.505097+0000 osd.3 (osd.3) 769 : cluster 0 2.19 scrub starts 2024-04-06T03:52:12.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:12 smithi050 bash[21467]: cluster 2024-04-06T03:52:11.507252+0000 osd.3 (osd.3) 770 : cluster 0 2.19 scrub ok 2024-04-06T03:52:13.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[21467]: cluster 2024-04-06T03:52:11.951775+0000 osd.5 (osd.5) 201 : cluster 0 2.12 scrub starts 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[21467]: cluster 2024-04-06T03:52:11.953841+0000 osd.5 (osd.5) 202 : cluster 0 2.12 scrub ok 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[21467]: cluster 2024-04-06T03:52:12.198767+0000 osd.0 (osd.0) 740 : cluster 0 5.1e scrub starts 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[21467]: cluster 2024-04-06T03:52:12.201318+0000 osd.0 (osd.0) 741 : cluster 0 5.1e scrub ok 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[21467]: cluster 2024-04-06T03:52:12.708984+0000 mgr.y (mgr.24368) 1145 : cluster 0 pgmap v1260: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[29221]: cluster 2024-04-06T03:52:11.951775+0000 osd.5 (osd.5) 201 : cluster 0 2.12 scrub starts 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[29221]: cluster 2024-04-06T03:52:11.953841+0000 osd.5 (osd.5) 202 : cluster 0 2.12 scrub ok 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[29221]: cluster 2024-04-06T03:52:12.198767+0000 osd.0 (osd.0) 740 : cluster 0 5.1e scrub starts 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[29221]: cluster 2024-04-06T03:52:12.201318+0000 osd.0 (osd.0) 741 : cluster 0 5.1e scrub ok 2024-04-06T03:52:13.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:13 smithi050 bash[29221]: cluster 2024-04-06T03:52:12.708984+0000 mgr.y (mgr.24368) 1145 : cluster 0 pgmap v1260: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:13.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:13 smithi153 bash[23852]: cluster 2024-04-06T03:52:11.951775+0000 osd.5 (osd.5) 201 : cluster 0 2.12 scrub starts 2024-04-06T03:52:13.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:13 smithi153 bash[23852]: cluster 2024-04-06T03:52:11.953841+0000 osd.5 (osd.5) 202 : cluster 0 2.12 scrub ok 2024-04-06T03:52:13.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:13 smithi153 bash[23852]: cluster 2024-04-06T03:52:12.198767+0000 osd.0 (osd.0) 740 : cluster 0 5.1e scrub starts 2024-04-06T03:52:13.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:13 smithi153 bash[23852]: cluster 2024-04-06T03:52:12.201318+0000 osd.0 (osd.0) 741 : cluster 0 5.1e scrub ok 2024-04-06T03:52:13.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:13 smithi153 bash[23852]: cluster 2024-04-06T03:52:12.708984+0000 mgr.y (mgr.24368) 1145 : cluster 0 pgmap v1260: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:14.470 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:14 smithi050 bash[21467]: cluster 2024-04-06T03:52:12.910453+0000 osd.5 (osd.5) 203 : cluster 0 3.5 scrub starts 2024-04-06T03:52:14.470 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:14 smithi050 bash[21467]: cluster 2024-04-06T03:52:12.914726+0000 osd.5 (osd.5) 204 : cluster 0 3.5 scrub ok 2024-04-06T03:52:14.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:14 smithi050 bash[29221]: cluster 2024-04-06T03:52:12.910453+0000 osd.5 (osd.5) 203 : cluster 0 3.5 scrub starts 2024-04-06T03:52:14.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:14 smithi050 bash[29221]: cluster 2024-04-06T03:52:12.914726+0000 osd.5 (osd.5) 204 : cluster 0 3.5 scrub ok 2024-04-06T03:52:14.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:14 smithi153 bash[23852]: cluster 2024-04-06T03:52:12.910453+0000 osd.5 (osd.5) 203 : cluster 0 3.5 scrub starts 2024-04-06T03:52:14.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:14 smithi153 bash[23852]: cluster 2024-04-06T03:52:12.914726+0000 osd.5 (osd.5) 204 : cluster 0 3.5 scrub ok 2024-04-06T03:52:15.234 INFO:teuthology.orchestra.run.smithi050.stdout:34359738828 2024-04-06T03:52:15.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-06T03:52:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[21467]: cluster 2024-04-06T03:52:14.127356+0000 osd.0 (osd.0) 742 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:52:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[21467]: cluster 2024-04-06T03:52:14.130983+0000 osd.0 (osd.0) 743 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:52:15.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[21467]: cluster 2024-04-06T03:52:14.432131+0000 osd.1 (osd.1) 519 : cluster 0 6.1d scrub starts 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[21467]: cluster 2024-04-06T03:52:14.571346+0000 osd.1 (osd.1) 520 : cluster 0 6.1d scrub ok 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[21467]: cluster 2024-04-06T03:52:14.709746+0000 mgr.y (mgr.24368) 1146 : cluster 0 pgmap v1261: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[29221]: cluster 2024-04-06T03:52:14.127356+0000 osd.0 (osd.0) 742 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[29221]: cluster 2024-04-06T03:52:14.130983+0000 osd.0 (osd.0) 743 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[29221]: cluster 2024-04-06T03:52:14.432131+0000 osd.1 (osd.1) 519 : cluster 0 6.1d scrub starts 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[29221]: cluster 2024-04-06T03:52:14.571346+0000 osd.1 (osd.1) 520 : cluster 0 6.1d scrub ok 2024-04-06T03:52:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:15 smithi050 bash[29221]: cluster 2024-04-06T03:52:14.709746+0000 mgr.y (mgr.24368) 1146 : cluster 0 pgmap v1261: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:15 smithi153 bash[23852]: cluster 2024-04-06T03:52:14.127356+0000 osd.0 (osd.0) 742 : cluster 0 3.12 deep-scrub starts 2024-04-06T03:52:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:15 smithi153 bash[23852]: cluster 2024-04-06T03:52:14.130983+0000 osd.0 (osd.0) 743 : cluster 0 3.12 deep-scrub ok 2024-04-06T03:52:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:15 smithi153 bash[23852]: cluster 2024-04-06T03:52:14.432131+0000 osd.1 (osd.1) 519 : cluster 0 6.1d scrub starts 2024-04-06T03:52:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:15 smithi153 bash[23852]: cluster 2024-04-06T03:52:14.571346+0000 osd.1 (osd.1) 520 : cluster 0 6.1d scrub ok 2024-04-06T03:52:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:15 smithi153 bash[23852]: cluster 2024-04-06T03:52:14.709746+0000 mgr.y (mgr.24368) 1146 : cluster 0 pgmap v1261: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:15.747 INFO:teuthology.orchestra.run.smithi050.stdout:55834575298 2024-04-06T03:52:15.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-06T03:52:17.184 INFO:teuthology.orchestra.run.smithi050.stdout:150323855779 2024-04-06T03:52:17.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-06T03:52:17.282 INFO:teuthology.orchestra.run.smithi050.stdout:244813136269 2024-04-06T03:52:17.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-06T03:52:17.337 INFO:teuthology.orchestra.run.smithi050.stdout:120259084717 2024-04-06T03:52:17.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-06T03:52:18.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:17 smithi153 bash[23852]: cluster 2024-04-06T03:52:16.710422+0000 mgr.y (mgr.24368) 1147 : cluster 0 pgmap v1262: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-04-06T03:52:18.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:17 smithi153 bash[23852]: cluster 2024-04-06T03:52:17.574904+0000 osd.7 (osd.7) 769 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:52:18.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:17 smithi153 bash[23852]: cluster 2024-04-06T03:52:17.581979+0000 osd.7 (osd.7) 770 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:52:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:17 smithi050 bash[29221]: cluster 2024-04-06T03:52:16.710422+0000 mgr.y (mgr.24368) 1147 : cluster 0 pgmap v1262: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-04-06T03:52:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:17 smithi050 bash[29221]: cluster 2024-04-06T03:52:17.574904+0000 osd.7 (osd.7) 769 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:52:18.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:17 smithi050 bash[29221]: cluster 2024-04-06T03:52:17.581979+0000 osd.7 (osd.7) 770 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:52:18.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:17 smithi050 bash[21467]: cluster 2024-04-06T03:52:16.710422+0000 mgr.y (mgr.24368) 1147 : cluster 0 pgmap v1262: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-04-06T03:52:18.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:17 smithi050 bash[21467]: cluster 2024-04-06T03:52:17.574904+0000 osd.7 (osd.7) 769 : cluster 0 1.0 deep-scrub starts 2024-04-06T03:52:18.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:17 smithi050 bash[21467]: cluster 2024-04-06T03:52:17.581979+0000 osd.7 (osd.7) 770 : cluster 0 1.0 deep-scrub ok 2024-04-06T03:52:18.686 INFO:teuthology.orchestra.run.smithi050.stdout:81604379064 2024-04-06T03:52:18.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-06T03:52:18.706 INFO:teuthology.orchestra.run.smithi050.stdout:296352743812 2024-04-06T03:52:18.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-06T03:52:18.715 INFO:teuthology.orchestra.run.smithi050.stdout:197568496026 2024-04-06T03:52:18.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-06T03:52:18.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:18 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.448003+0000 osd.4 (osd.4) 729 : cluster 0 6.1a scrub starts 2024-04-06T03:52:18.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:18 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.448003+0000 osd.4 (osd.4) 729 : cluster 0 6.1a scrub starts 2024-04-06T03:52:19.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:18 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.448003+0000 osd.4 (osd.4) 729 : cluster 0 6.1a scrub starts 2024-04-06T03:52:19.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:52:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:52:20.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.471009+0000 osd.1 (osd.1) 521 : cluster 0 6.4 scrub starts 2024-04-06T03:52:20.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.473378+0000 osd.1 (osd.1) 522 : cluster 0 6.4 scrub ok 2024-04-06T03:52:20.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.524747+0000 osd.4 (osd.4) 730 : cluster 0 6.1a scrub ok 2024-04-06T03:52:20.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.710889+0000 mgr.y (mgr.24368) 1148 : cluster 0 pgmap v1263: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:20.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.875853+0000 osd.5 (osd.5) 205 : cluster 0 6.a scrub starts 2024-04-06T03:52:20.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:18.908311+0000 osd.5 (osd.5) 206 : cluster 0 6.a scrub ok 2024-04-06T03:52:20.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:19.406994+0000 osd.4 (osd.4) 731 : cluster 0 3.1a scrub starts 2024-04-06T03:52:20.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:19.411308+0000 osd.4 (osd.4) 732 : cluster 0 3.1a scrub ok 2024-04-06T03:52:20.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:19 smithi153 bash[23852]: cluster 2024-04-06T03:52:19.578265+0000 osd.7 (osd.7) 771 : cluster 0 6.3 scrub starts 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.471009+0000 osd.1 (osd.1) 521 : cluster 0 6.4 scrub starts 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.473378+0000 osd.1 (osd.1) 522 : cluster 0 6.4 scrub ok 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.524747+0000 osd.4 (osd.4) 730 : cluster 0 6.1a scrub ok 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.710889+0000 mgr.y (mgr.24368) 1148 : cluster 0 pgmap v1263: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.875853+0000 osd.5 (osd.5) 205 : cluster 0 6.a scrub starts 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:18.908311+0000 osd.5 (osd.5) 206 : cluster 0 6.a scrub ok 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:19.406994+0000 osd.4 (osd.4) 731 : cluster 0 3.1a scrub starts 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:19.411308+0000 osd.4 (osd.4) 732 : cluster 0 3.1a scrub ok 2024-04-06T03:52:20.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[21467]: cluster 2024-04-06T03:52:19.578265+0000 osd.7 (osd.7) 771 : cluster 0 6.3 scrub starts 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.471009+0000 osd.1 (osd.1) 521 : cluster 0 6.4 scrub starts 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.473378+0000 osd.1 (osd.1) 522 : cluster 0 6.4 scrub ok 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.524747+0000 osd.4 (osd.4) 730 : cluster 0 6.1a scrub ok 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.710889+0000 mgr.y (mgr.24368) 1148 : cluster 0 pgmap v1263: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.875853+0000 osd.5 (osd.5) 205 : cluster 0 6.a scrub starts 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:18.908311+0000 osd.5 (osd.5) 206 : cluster 0 6.a scrub ok 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:19.406994+0000 osd.4 (osd.4) 731 : cluster 0 3.1a scrub starts 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:19.411308+0000 osd.4 (osd.4) 732 : cluster 0 3.1a scrub ok 2024-04-06T03:52:20.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:19 smithi050 bash[29221]: cluster 2024-04-06T03:52:19.578265+0000 osd.7 (osd.7) 771 : cluster 0 6.3 scrub starts 2024-04-06T03:52:21.042 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:21.044 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:21.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:52:20 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:52:21.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:20 smithi153 bash[23852]: cluster 2024-04-06T03:52:19.739021+0000 osd.7 (osd.7) 772 : cluster 0 6.3 scrub ok 2024-04-06T03:52:21.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:20 smithi153 bash[23852]: cluster 2024-04-06T03:52:19.908329+0000 osd.5 (osd.5) 207 : cluster 0 5.16 scrub starts 2024-04-06T03:52:21.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:20 smithi153 bash[23852]: cluster 2024-04-06T03:52:19.911022+0000 osd.5 (osd.5) 208 : cluster 0 5.16 scrub ok 2024-04-06T03:52:21.049 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:21.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:20 smithi050 bash[29221]: cluster 2024-04-06T03:52:19.739021+0000 osd.7 (osd.7) 772 : cluster 0 6.3 scrub ok 2024-04-06T03:52:21.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:20 smithi050 bash[29221]: cluster 2024-04-06T03:52:19.908329+0000 osd.5 (osd.5) 207 : cluster 0 5.16 scrub starts 2024-04-06T03:52:21.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:20 smithi050 bash[29221]: cluster 2024-04-06T03:52:19.911022+0000 osd.5 (osd.5) 208 : cluster 0 5.16 scrub ok 2024-04-06T03:52:21.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:20 smithi050 bash[21467]: cluster 2024-04-06T03:52:19.739021+0000 osd.7 (osd.7) 772 : cluster 0 6.3 scrub ok 2024-04-06T03:52:21.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:20 smithi050 bash[21467]: cluster 2024-04-06T03:52:19.908329+0000 osd.5 (osd.5) 207 : cluster 0 5.16 scrub starts 2024-04-06T03:52:21.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:20 smithi050 bash[21467]: cluster 2024-04-06T03:52:19.911022+0000 osd.5 (osd.5) 208 : cluster 0 5.16 scrub ok 2024-04-06T03:52:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: cluster 2024-04-06T03:52:20.434383+0000 osd.3 (osd.3) 771 : cluster 0 5.13 scrub starts 2024-04-06T03:52:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: cluster 2024-04-06T03:52:20.436927+0000 osd.3 (osd.3) 772 : cluster 0 5.13 scrub ok 2024-04-06T03:52:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: audit 2024-04-06T03:52:20.582581+0000 mgr.y (mgr.24368) 1149 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: cluster 2024-04-06T03:52:20.712462+0000 mgr.y (mgr.24368) 1150 : cluster 0 pgmap v1264: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: cluster 2024-04-06T03:52:20.891732+0000 osd.5 (osd.5) 209 : cluster 0 6.19 scrub starts 2024-04-06T03:52:22.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: cluster 2024-04-06T03:52:21.009403+0000 osd.5 (osd.5) 210 : cluster 0 6.19 scrub ok 2024-04-06T03:52:22.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:21 smithi153 bash[23852]: audit 2024-04-06T03:52:21.228675+0000 mon.a (mon.0) 1937 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:22.062 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:22.062 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:22.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: cluster 2024-04-06T03:52:20.434383+0000 osd.3 (osd.3) 771 : cluster 0 5.13 scrub starts 2024-04-06T03:52:22.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: cluster 2024-04-06T03:52:20.436927+0000 osd.3 (osd.3) 772 : cluster 0 5.13 scrub ok 2024-04-06T03:52:22.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: audit 2024-04-06T03:52:20.582581+0000 mgr.y (mgr.24368) 1149 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:22.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: cluster 2024-04-06T03:52:20.712462+0000 mgr.y (mgr.24368) 1150 : cluster 0 pgmap v1264: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:22.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: cluster 2024-04-06T03:52:20.891732+0000 osd.5 (osd.5) 209 : cluster 0 6.19 scrub starts 2024-04-06T03:52:22.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: cluster 2024-04-06T03:52:21.009403+0000 osd.5 (osd.5) 210 : cluster 0 6.19 scrub ok 2024-04-06T03:52:22.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[21467]: audit 2024-04-06T03:52:21.228675+0000 mon.a (mon.0) 1937 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:22.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: cluster 2024-04-06T03:52:20.434383+0000 osd.3 (osd.3) 771 : cluster 0 5.13 scrub starts 2024-04-06T03:52:22.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: cluster 2024-04-06T03:52:20.436927+0000 osd.3 (osd.3) 772 : cluster 0 5.13 scrub ok 2024-04-06T03:52:22.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: audit 2024-04-06T03:52:20.582581+0000 mgr.y (mgr.24368) 1149 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:22.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: cluster 2024-04-06T03:52:20.712462+0000 mgr.y (mgr.24368) 1150 : cluster 0 pgmap v1264: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:22.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: cluster 2024-04-06T03:52:20.891732+0000 osd.5 (osd.5) 209 : cluster 0 6.19 scrub starts 2024-04-06T03:52:22.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: cluster 2024-04-06T03:52:21.009403+0000 osd.5 (osd.5) 210 : cluster 0 6.19 scrub ok 2024-04-06T03:52:22.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:21 smithi050 bash[29221]: audit 2024-04-06T03:52:21.228675+0000 mon.a (mon.0) 1937 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:23.082 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:23.082 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:23.083 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:24.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:22.390995+0000 osd.3 (osd.3) 773 : cluster 0 6.13 scrub starts 2024-04-06T03:52:24.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:22.571574+0000 osd.3 (osd.3) 774 : cluster 0 6.13 scrub ok 2024-04-06T03:52:24.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:22.713267+0000 mgr.y (mgr.24368) 1151 : cluster 0 pgmap v1265: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-06T03:52:24.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.409914+0000 osd.4 (osd.4) 733 : cluster 0 3.3 scrub starts 2024-04-06T03:52:24.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.414249+0000 osd.4 (osd.4) 734 : cluster 0 3.3 scrub ok 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.506197+0000 osd.7 (osd.7) 773 : cluster 0 5.10 scrub starts 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.508919+0000 osd.7 (osd.7) 774 : cluster 0 5.10 scrub ok 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.528100+0000 osd.6 (osd.6) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.528190+0000 osd.6 (osd.6) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:22.390995+0000 osd.3 (osd.3) 773 : cluster 0 6.13 scrub starts 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:22.571574+0000 osd.3 (osd.3) 774 : cluster 0 6.13 scrub ok 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:22.713267+0000 mgr.y (mgr.24368) 1151 : cluster 0 pgmap v1265: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.409914+0000 osd.4 (osd.4) 733 : cluster 0 3.3 scrub starts 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.414249+0000 osd.4 (osd.4) 734 : cluster 0 3.3 scrub ok 2024-04-06T03:52:24.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.506197+0000 osd.7 (osd.7) 773 : cluster 0 5.10 scrub starts 2024-04-06T03:52:24.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.508919+0000 osd.7 (osd.7) 774 : cluster 0 5.10 scrub ok 2024-04-06T03:52:24.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.528100+0000 osd.6 (osd.6) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:24.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:23 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.528190+0000 osd.6 (osd.6) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:24.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:22.390995+0000 osd.3 (osd.3) 773 : cluster 0 6.13 scrub starts 2024-04-06T03:52:24.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:22.571574+0000 osd.3 (osd.3) 774 : cluster 0 6.13 scrub ok 2024-04-06T03:52:24.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:22.713267+0000 mgr.y (mgr.24368) 1151 : cluster 0 pgmap v1265: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-06T03:52:24.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.409914+0000 osd.4 (osd.4) 733 : cluster 0 3.3 scrub starts 2024-04-06T03:52:24.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.414249+0000 osd.4 (osd.4) 734 : cluster 0 3.3 scrub ok 2024-04-06T03:52:24.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.506197+0000 osd.7 (osd.7) 773 : cluster 0 5.10 scrub starts 2024-04-06T03:52:24.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.508919+0000 osd.7 (osd.7) 774 : cluster 0 5.10 scrub ok 2024-04-06T03:52:24.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.528100+0000 osd.6 (osd.6) 139 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:24.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:23 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.528190+0000 osd.6 (osd.6) 140 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:25.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.439223+0000 osd.1 (osd.1) 523 : cluster 0 2.d scrub starts 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.441681+0000 osd.1 (osd.1) 524 : cluster 0 2.d scrub ok 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.733556+0000 osd.2 (osd.2) 621 : cluster 0 5.d scrub starts 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.736914+0000 osd.2 (osd.2) 622 : cluster 0 5.d scrub ok 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:23.798336+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:24.002462+0000 osd.0 (osd.0) 744 : cluster 0 5.3 scrub starts 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[21467]: cluster 2024-04-06T03:52:24.006221+0000 osd.0 (osd.0) 745 : cluster 0 5.3 scrub ok 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.439223+0000 osd.1 (osd.1) 523 : cluster 0 2.d scrub starts 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.441681+0000 osd.1 (osd.1) 524 : cluster 0 2.d scrub ok 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.733556+0000 osd.2 (osd.2) 621 : cluster 0 5.d scrub starts 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.736914+0000 osd.2 (osd.2) 622 : cluster 0 5.d scrub ok 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:23.798336+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-04-06T03:52:25.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:24.002462+0000 osd.0 (osd.0) 744 : cluster 0 5.3 scrub starts 2024-04-06T03:52:25.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:24 smithi050 bash[29221]: cluster 2024-04-06T03:52:24.006221+0000 osd.0 (osd.0) 745 : cluster 0 5.3 scrub ok 2024-04-06T03:52:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.439223+0000 osd.1 (osd.1) 523 : cluster 0 2.d scrub starts 2024-04-06T03:52:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.441681+0000 osd.1 (osd.1) 524 : cluster 0 2.d scrub ok 2024-04-06T03:52:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.733556+0000 osd.2 (osd.2) 621 : cluster 0 5.d scrub starts 2024-04-06T03:52:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.736914+0000 osd.2 (osd.2) 622 : cluster 0 5.d scrub ok 2024-04-06T03:52:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:23.798336+0000 mon.a (mon.0) 1938 : cluster 0 osdmap e444: 8 total, 8 up, 8 in 2024-04-06T03:52:25.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:24.002462+0000 osd.0 (osd.0) 744 : cluster 0 5.3 scrub starts 2024-04-06T03:52:25.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:24 smithi153 bash[23852]: cluster 2024-04-06T03:52:24.006221+0000 osd.0 (osd.0) 745 : cluster 0 5.3 scrub ok 2024-04-06T03:52:26.140 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:25 smithi050 bash[21467]: cluster 2024-04-06T03:52:24.714890+0000 mgr.y (mgr.24368) 1152 : cluster 0 pgmap v1267: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:52:26.141 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:25 smithi050 bash[29221]: cluster 2024-04-06T03:52:24.714890+0000 mgr.y (mgr.24368) 1152 : cluster 0 pgmap v1267: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:52:26.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:25 smithi153 bash[23852]: cluster 2024-04-06T03:52:24.714890+0000 mgr.y (mgr.24368) 1152 : cluster 0 pgmap v1267: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-06T03:52:27.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[21467]: cluster 2024-04-06T03:52:25.280460+0000 osd.1 (osd.1) 525 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:27.080 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[21467]: cluster 2024-04-06T03:52:25.280577+0000 osd.1 (osd.1) 526 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[21467]: cluster 2024-04-06T03:52:25.599585+0000 osd.3 (osd.3) 775 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[21467]: cluster 2024-04-06T03:52:25.599709+0000 osd.3 (osd.3) 776 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[21467]: cluster 2024-04-06T03:52:26.118469+0000 mon.a (mon.0) 1939 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[29221]: cluster 2024-04-06T03:52:25.280460+0000 osd.1 (osd.1) 525 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[29221]: cluster 2024-04-06T03:52:25.280577+0000 osd.1 (osd.1) 526 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[29221]: cluster 2024-04-06T03:52:25.599585+0000 osd.3 (osd.3) 775 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[29221]: cluster 2024-04-06T03:52:25.599709+0000 osd.3 (osd.3) 776 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:27.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:26 smithi050 bash[29221]: cluster 2024-04-06T03:52:26.118469+0000 mon.a (mon.0) 1939 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-04-06T03:52:27.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:26 smithi153 bash[23852]: cluster 2024-04-06T03:52:25.280460+0000 osd.1 (osd.1) 525 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:27.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:26 smithi153 bash[23852]: cluster 2024-04-06T03:52:25.280577+0000 osd.1 (osd.1) 526 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:27.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:26 smithi153 bash[23852]: cluster 2024-04-06T03:52:25.599585+0000 osd.3 (osd.3) 775 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:27.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:26 smithi153 bash[23852]: cluster 2024-04-06T03:52:25.599709+0000 osd.3 (osd.3) 776 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:27.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:26 smithi153 bash[23852]: cluster 2024-04-06T03:52:26.118469+0000 mon.a (mon.0) 1939 : cluster 0 osdmap e445: 8 total, 8 up, 8 in 2024-04-06T03:52:28.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[21467]: cluster 2024-04-06T03:52:26.389948+0000 osd.3 (osd.3) 777 : cluster 0 3.2 scrub starts 2024-04-06T03:52:28.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[21467]: cluster 2024-04-06T03:52:26.394810+0000 osd.3 (osd.3) 778 : cluster 0 3.2 scrub ok 2024-04-06T03:52:28.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[21467]: cluster 2024-04-06T03:52:26.638685+0000 osd.7 (osd.7) 775 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:28.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[21467]: cluster 2024-04-06T03:52:26.638784+0000 osd.7 (osd.7) 776 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:28.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[21467]: cluster 2024-04-06T03:52:26.715716+0000 mgr.y (mgr.24368) 1153 : cluster 0 pgmap v1269: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[21467]: cluster 2024-04-06T03:52:27.125970+0000 mon.a (mon.0) 1940 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[29221]: cluster 2024-04-06T03:52:26.389948+0000 osd.3 (osd.3) 777 : cluster 0 3.2 scrub starts 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[29221]: cluster 2024-04-06T03:52:26.394810+0000 osd.3 (osd.3) 778 : cluster 0 3.2 scrub ok 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[29221]: cluster 2024-04-06T03:52:26.638685+0000 osd.7 (osd.7) 775 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[29221]: cluster 2024-04-06T03:52:26.638784+0000 osd.7 (osd.7) 776 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[29221]: cluster 2024-04-06T03:52:26.715716+0000 mgr.y (mgr.24368) 1153 : cluster 0 pgmap v1269: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:52:28.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:27 smithi050 bash[29221]: cluster 2024-04-06T03:52:27.125970+0000 mon.a (mon.0) 1940 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-04-06T03:52:28.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:27 smithi153 bash[23852]: cluster 2024-04-06T03:52:26.389948+0000 osd.3 (osd.3) 777 : cluster 0 3.2 scrub starts 2024-04-06T03:52:28.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:27 smithi153 bash[23852]: cluster 2024-04-06T03:52:26.394810+0000 osd.3 (osd.3) 778 : cluster 0 3.2 scrub ok 2024-04-06T03:52:28.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:27 smithi153 bash[23852]: cluster 2024-04-06T03:52:26.638685+0000 osd.7 (osd.7) 775 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:28.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:27 smithi153 bash[23852]: cluster 2024-04-06T03:52:26.638784+0000 osd.7 (osd.7) 776 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:28.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:27 smithi153 bash[23852]: cluster 2024-04-06T03:52:26.715716+0000 mgr.y (mgr.24368) 1153 : cluster 0 pgmap v1269: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-06T03:52:28.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:27 smithi153 bash[23852]: cluster 2024-04-06T03:52:27.125970+0000 mon.a (mon.0) 1940 : cluster 0 osdmap e446: 8 total, 8 up, 8 in 2024-04-06T03:52:29.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:28 smithi050 bash[21467]: cluster 2024-04-06T03:52:27.381761+0000 osd.1 (osd.1) 527 : cluster 0 3.4 scrub starts 2024-04-06T03:52:29.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:28 smithi050 bash[21467]: cluster 2024-04-06T03:52:27.385676+0000 osd.1 (osd.1) 528 : cluster 0 3.4 scrub ok 2024-04-06T03:52:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:28 smithi050 bash[29221]: cluster 2024-04-06T03:52:27.381761+0000 osd.1 (osd.1) 527 : cluster 0 3.4 scrub starts 2024-04-06T03:52:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:28 smithi050 bash[29221]: cluster 2024-04-06T03:52:27.385676+0000 osd.1 (osd.1) 528 : cluster 0 3.4 scrub ok 2024-04-06T03:52:29.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:28 smithi153 bash[23852]: cluster 2024-04-06T03:52:27.381761+0000 osd.1 (osd.1) 527 : cluster 0 3.4 scrub starts 2024-04-06T03:52:29.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:28 smithi153 bash[23852]: cluster 2024-04-06T03:52:27.385676+0000 osd.1 (osd.1) 528 : cluster 0 3.4 scrub ok 2024-04-06T03:52:29.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:52:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:52:30.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.404658+0000 osd.1 (osd.1) 529 : cluster 0 4.2 deep-scrub starts 2024-04-06T03:52:30.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.407138+0000 osd.1 (osd.1) 530 : cluster 0 4.2 deep-scrub ok 2024-04-06T03:52:30.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.634176+0000 osd.7 (osd.7) 777 : cluster 0 3.15 scrub starts 2024-04-06T03:52:30.155 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.638184+0000 osd.7 (osd.7) 778 : cluster 0 3.15 scrub ok 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.716481+0000 mgr.y (mgr.24368) 1154 : cluster 0 pgmap v1271: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.928480+0000 osd.5 (osd.5) 211 : cluster 0 3.c scrub starts 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:28.932824+0000 osd.5 (osd.5) 212 : cluster 0 3.c scrub ok 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:29.265400+0000 osd.4 (osd.4) 735 : cluster 0 5.1c scrub starts 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[29221]: cluster 2024-04-06T03:52:29.267895+0000 osd.4 (osd.4) 736 : cluster 0 5.1c scrub ok 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.404658+0000 osd.1 (osd.1) 529 : cluster 0 4.2 deep-scrub starts 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.407138+0000 osd.1 (osd.1) 530 : cluster 0 4.2 deep-scrub ok 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.634176+0000 osd.7 (osd.7) 777 : cluster 0 3.15 scrub starts 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.638184+0000 osd.7 (osd.7) 778 : cluster 0 3.15 scrub ok 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.716481+0000 mgr.y (mgr.24368) 1154 : cluster 0 pgmap v1271: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:30.156 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.928480+0000 osd.5 (osd.5) 211 : cluster 0 3.c scrub starts 2024-04-06T03:52:30.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:28.932824+0000 osd.5 (osd.5) 212 : cluster 0 3.c scrub ok 2024-04-06T03:52:30.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:29.265400+0000 osd.4 (osd.4) 735 : cluster 0 5.1c scrub starts 2024-04-06T03:52:30.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:29 smithi050 bash[21467]: cluster 2024-04-06T03:52:29.267895+0000 osd.4 (osd.4) 736 : cluster 0 5.1c scrub ok 2024-04-06T03:52:30.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.404658+0000 osd.1 (osd.1) 529 : cluster 0 4.2 deep-scrub starts 2024-04-06T03:52:30.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.407138+0000 osd.1 (osd.1) 530 : cluster 0 4.2 deep-scrub ok 2024-04-06T03:52:30.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.634176+0000 osd.7 (osd.7) 777 : cluster 0 3.15 scrub starts 2024-04-06T03:52:30.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.638184+0000 osd.7 (osd.7) 778 : cluster 0 3.15 scrub ok 2024-04-06T03:52:30.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.716481+0000 mgr.y (mgr.24368) 1154 : cluster 0 pgmap v1271: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:30.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.928480+0000 osd.5 (osd.5) 211 : cluster 0 3.c scrub starts 2024-04-06T03:52:30.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:28.932824+0000 osd.5 (osd.5) 212 : cluster 0 3.c scrub ok 2024-04-06T03:52:30.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:29.265400+0000 osd.4 (osd.4) 735 : cluster 0 5.1c scrub starts 2024-04-06T03:52:30.298 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:29 smithi153 bash[23852]: cluster 2024-04-06T03:52:29.267895+0000 osd.4 (osd.4) 736 : cluster 0 5.1c scrub ok 2024-04-06T03:52:31.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:52:30 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:52:31.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:30 smithi153 bash[23852]: cluster 2024-04-06T03:52:29.667265+0000 osd.7 (osd.7) 779 : cluster 0 6.1c scrub starts 2024-04-06T03:52:31.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:30 smithi153 bash[23852]: cluster 2024-04-06T03:52:29.670526+0000 osd.7 (osd.7) 780 : cluster 0 6.1c scrub ok 2024-04-06T03:52:31.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:30 smithi153 bash[23852]: cluster 2024-04-06T03:52:30.219912+0000 osd.4 (osd.4) 737 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:52:31.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:30 smithi153 bash[23852]: cluster 2024-04-06T03:52:30.295443+0000 osd.4 (osd.4) 738 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:52:31.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[21467]: cluster 2024-04-06T03:52:29.667265+0000 osd.7 (osd.7) 779 : cluster 0 6.1c scrub starts 2024-04-06T03:52:31.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[21467]: cluster 2024-04-06T03:52:29.670526+0000 osd.7 (osd.7) 780 : cluster 0 6.1c scrub ok 2024-04-06T03:52:31.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[21467]: cluster 2024-04-06T03:52:30.219912+0000 osd.4 (osd.4) 737 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:52:31.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[21467]: cluster 2024-04-06T03:52:30.295443+0000 osd.4 (osd.4) 738 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:52:31.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[29221]: cluster 2024-04-06T03:52:29.667265+0000 osd.7 (osd.7) 779 : cluster 0 6.1c scrub starts 2024-04-06T03:52:31.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[29221]: cluster 2024-04-06T03:52:29.670526+0000 osd.7 (osd.7) 780 : cluster 0 6.1c scrub ok 2024-04-06T03:52:31.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[29221]: cluster 2024-04-06T03:52:30.219912+0000 osd.4 (osd.4) 737 : cluster 0 6.1e deep-scrub starts 2024-04-06T03:52:31.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:30 smithi050 bash[29221]: cluster 2024-04-06T03:52:30.295443+0000 osd.4 (osd.4) 738 : cluster 0 6.1e deep-scrub ok 2024-04-06T03:52:31.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[21467]: audit 2024-04-06T03:52:30.588298+0000 mgr.y (mgr.24368) 1155 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:31.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[21467]: cluster 2024-04-06T03:52:30.648073+0000 osd.7 (osd.7) 781 : cluster 0 2.1c scrub starts 2024-04-06T03:52:31.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[21467]: cluster 2024-04-06T03:52:30.650495+0000 osd.7 (osd.7) 782 : cluster 0 2.1c scrub ok 2024-04-06T03:52:31.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[21467]: cluster 2024-04-06T03:52:30.718014+0000 mgr.y (mgr.24368) 1156 : cluster 0 pgmap v1272: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:52:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[29221]: audit 2024-04-06T03:52:30.588298+0000 mgr.y (mgr.24368) 1155 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[29221]: cluster 2024-04-06T03:52:30.648073+0000 osd.7 (osd.7) 781 : cluster 0 2.1c scrub starts 2024-04-06T03:52:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[29221]: cluster 2024-04-06T03:52:30.650495+0000 osd.7 (osd.7) 782 : cluster 0 2.1c scrub ok 2024-04-06T03:52:31.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:31 smithi050 bash[29221]: cluster 2024-04-06T03:52:30.718014+0000 mgr.y (mgr.24368) 1156 : cluster 0 pgmap v1272: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:52:32.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:31 smithi153 bash[23852]: audit 2024-04-06T03:52:30.588298+0000 mgr.y (mgr.24368) 1155 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:32.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:31 smithi153 bash[23852]: cluster 2024-04-06T03:52:30.648073+0000 osd.7 (osd.7) 781 : cluster 0 2.1c scrub starts 2024-04-06T03:52:32.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:31 smithi153 bash[23852]: cluster 2024-04-06T03:52:30.650495+0000 osd.7 (osd.7) 782 : cluster 0 2.1c scrub ok 2024-04-06T03:52:32.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:31 smithi153 bash[23852]: cluster 2024-04-06T03:52:30.718014+0000 mgr.y (mgr.24368) 1156 : cluster 0 pgmap v1272: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-06T03:52:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:32 smithi050 bash[21467]: cluster 2024-04-06T03:52:31.977747+0000 osd.5 (osd.5) 213 : cluster 0 3.1c scrub starts 2024-04-06T03:52:32.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:32 smithi050 bash[21467]: cluster 2024-04-06T03:52:31.982696+0000 osd.5 (osd.5) 214 : cluster 0 3.1c scrub ok 2024-04-06T03:52:32.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:32 smithi050 bash[29221]: cluster 2024-04-06T03:52:31.977747+0000 osd.5 (osd.5) 213 : cluster 0 3.1c scrub starts 2024-04-06T03:52:32.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:32 smithi050 bash[29221]: cluster 2024-04-06T03:52:31.982696+0000 osd.5 (osd.5) 214 : cluster 0 3.1c scrub ok 2024-04-06T03:52:33.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:32 smithi153 bash[23852]: cluster 2024-04-06T03:52:31.977747+0000 osd.5 (osd.5) 213 : cluster 0 3.1c scrub starts 2024-04-06T03:52:33.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:32 smithi153 bash[23852]: cluster 2024-04-06T03:52:31.982696+0000 osd.5 (osd.5) 214 : cluster 0 3.1c scrub ok 2024-04-06T03:52:33.965 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:33 smithi050 bash[21467]: cluster 2024-04-06T03:52:32.718621+0000 mgr.y (mgr.24368) 1157 : cluster 0 pgmap v1273: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:33.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:33 smithi050 bash[21467]: cluster 2024-04-06T03:52:33.102470+0000 osd.4 (osd.4) 739 : cluster 0 6.17 deep-scrub starts 2024-04-06T03:52:33.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:33 smithi050 bash[21467]: cluster 2024-04-06T03:52:33.312332+0000 osd.4 (osd.4) 740 : cluster 0 6.17 deep-scrub ok 2024-04-06T03:52:34.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:33 smithi050 bash[29221]: cluster 2024-04-06T03:52:32.718621+0000 mgr.y (mgr.24368) 1157 : cluster 0 pgmap v1273: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:34.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:33 smithi050 bash[29221]: cluster 2024-04-06T03:52:33.102470+0000 osd.4 (osd.4) 739 : cluster 0 6.17 deep-scrub starts 2024-04-06T03:52:34.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:33 smithi050 bash[29221]: cluster 2024-04-06T03:52:33.312332+0000 osd.4 (osd.4) 740 : cluster 0 6.17 deep-scrub ok 2024-04-06T03:52:34.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:33 smithi153 bash[23852]: cluster 2024-04-06T03:52:32.718621+0000 mgr.y (mgr.24368) 1157 : cluster 0 pgmap v1273: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:34.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:33 smithi153 bash[23852]: cluster 2024-04-06T03:52:33.102470+0000 osd.4 (osd.4) 739 : cluster 0 6.17 deep-scrub starts 2024-04-06T03:52:34.297 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:33 smithi153 bash[23852]: cluster 2024-04-06T03:52:33.312332+0000 osd.4 (osd.4) 740 : cluster 0 6.17 deep-scrub ok 2024-04-06T03:52:35.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[29221]: cluster 2024-04-06T03:52:33.851910+0000 osd.6 (osd.6) 141 : cluster 0 2.17 scrub starts 2024-04-06T03:52:35.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[29221]: cluster 2024-04-06T03:52:33.854422+0000 osd.6 (osd.6) 142 : cluster 0 2.17 scrub ok 2024-04-06T03:52:35.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.014069+0000 osd.0 (osd.0) 746 : cluster 0 6.18 scrub starts 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.107591+0000 osd.0 (osd.0) 747 : cluster 0 6.18 scrub ok 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.719762+0000 mgr.y (mgr.24368) 1158 : cluster 0 pgmap v1274: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[21467]: cluster 2024-04-06T03:52:33.851910+0000 osd.6 (osd.6) 141 : cluster 0 2.17 scrub starts 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[21467]: cluster 2024-04-06T03:52:33.854422+0000 osd.6 (osd.6) 142 : cluster 0 2.17 scrub ok 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.014069+0000 osd.0 (osd.0) 746 : cluster 0 6.18 scrub starts 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.107591+0000 osd.0 (osd.0) 747 : cluster 0 6.18 scrub ok 2024-04-06T03:52:35.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:35 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.719762+0000 mgr.y (mgr.24368) 1158 : cluster 0 pgmap v1274: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:35 smithi153 bash[23852]: cluster 2024-04-06T03:52:33.851910+0000 osd.6 (osd.6) 141 : cluster 0 2.17 scrub starts 2024-04-06T03:52:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:35 smithi153 bash[23852]: cluster 2024-04-06T03:52:33.854422+0000 osd.6 (osd.6) 142 : cluster 0 2.17 scrub ok 2024-04-06T03:52:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:35 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.014069+0000 osd.0 (osd.0) 746 : cluster 0 6.18 scrub starts 2024-04-06T03:52:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:35 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.107591+0000 osd.0 (osd.0) 747 : cluster 0 6.18 scrub ok 2024-04-06T03:52:35.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:35 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.719762+0000 mgr.y (mgr.24368) 1158 : cluster 0 pgmap v1274: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:36.399 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.440558+0000 osd.1 (osd.1) 531 : cluster 0 5.c scrub starts 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.442961+0000 osd.1 (osd.1) 532 : cluster 0 5.c scrub ok 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.656320+0000 osd.7 (osd.7) 783 : cluster 0 6.12 scrub starts 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[29221]: cluster 2024-04-06T03:52:34.698743+0000 osd.7 (osd.7) 784 : cluster 0 6.12 scrub ok 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.440558+0000 osd.1 (osd.1) 531 : cluster 0 5.c scrub starts 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.442961+0000 osd.1 (osd.1) 532 : cluster 0 5.c scrub ok 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.656320+0000 osd.7 (osd.7) 783 : cluster 0 6.12 scrub starts 2024-04-06T03:52:36.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:36 smithi050 bash[21467]: cluster 2024-04-06T03:52:34.698743+0000 osd.7 (osd.7) 784 : cluster 0 6.12 scrub ok 2024-04-06T03:52:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:36 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.440558+0000 osd.1 (osd.1) 531 : cluster 0 5.c scrub starts 2024-04-06T03:52:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:36 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.442961+0000 osd.1 (osd.1) 532 : cluster 0 5.c scrub ok 2024-04-06T03:52:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:36 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.656320+0000 osd.7 (osd.7) 783 : cluster 0 6.12 scrub starts 2024-04-06T03:52:36.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:36 smithi153 bash[23852]: cluster 2024-04-06T03:52:34.698743+0000 osd.7 (osd.7) 784 : cluster 0 6.12 scrub ok 2024-04-06T03:52:36.807 INFO:teuthology.orchestra.run.smithi050.stdout:34359738833 2024-04-06T03:52:37.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:37 smithi050 bash[29221]: audit 2024-04-06T03:52:36.260784+0000 mon.a (mon.0) 1941 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:37 smithi050 bash[29221]: cluster 2024-04-06T03:52:36.720402+0000 mgr.y (mgr.24368) 1159 : cluster 0 pgmap v1275: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:37.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:37 smithi050 bash[29221]: audit 2024-04-06T03:52:36.808267+0000 mon.c (mon.2) 138 : audit 0 from='client.? 172.21.15.50:0/1100915794' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:52:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:37 smithi050 bash[21467]: audit 2024-04-06T03:52:36.260784+0000 mon.a (mon.0) 1941 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:37 smithi050 bash[21467]: cluster 2024-04-06T03:52:36.720402+0000 mgr.y (mgr.24368) 1159 : cluster 0 pgmap v1275: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:37.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:37 smithi050 bash[21467]: audit 2024-04-06T03:52:36.808267+0000 mon.c (mon.2) 138 : audit 0 from='client.? 172.21.15.50:0/1100915794' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:52:37.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:37 smithi153 bash[23852]: audit 2024-04-06T03:52:36.260784+0000 mon.a (mon.0) 1941 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:37.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:37 smithi153 bash[23852]: cluster 2024-04-06T03:52:36.720402+0000 mgr.y (mgr.24368) 1159 : cluster 0 pgmap v1275: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:37.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:37 smithi153 bash[23852]: audit 2024-04-06T03:52:36.808267+0000 mon.c (mon.2) 138 : audit 0 from='client.? 172.21.15.50:0/1100915794' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-06T03:52:38.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[21467]: cluster 2024-04-06T03:52:36.860439+0000 osd.6 (osd.6) 143 : cluster 0 5.11 scrub starts 2024-04-06T03:52:38.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[21467]: cluster 2024-04-06T03:52:36.863041+0000 osd.6 (osd.6) 144 : cluster 0 5.11 scrub ok 2024-04-06T03:52:38.411 INFO:teuthology.orchestra.run.smithi050.stdout:150323855784 2024-04-06T03:52:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:38 smithi153 bash[23852]: cluster 2024-04-06T03:52:36.860439+0000 osd.6 (osd.6) 143 : cluster 0 5.11 scrub starts 2024-04-06T03:52:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:38 smithi153 bash[23852]: cluster 2024-04-06T03:52:36.863041+0000 osd.6 (osd.6) 144 : cluster 0 5.11 scrub ok 2024-04-06T03:52:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:38 smithi153 bash[23852]: cluster 2024-04-06T03:52:37.450363+0000 osd.3 (osd.3) 779 : cluster 0 6.11 scrub starts 2024-04-06T03:52:38.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:38 smithi153 bash[23852]: cluster 2024-04-06T03:52:37.502783+0000 osd.3 (osd.3) 780 : cluster 0 6.11 scrub ok 2024-04-06T03:52:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[21467]: cluster 2024-04-06T03:52:37.450363+0000 osd.3 (osd.3) 779 : cluster 0 6.11 scrub starts 2024-04-06T03:52:38.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[21467]: cluster 2024-04-06T03:52:37.502783+0000 osd.3 (osd.3) 780 : cluster 0 6.11 scrub ok 2024-04-06T03:52:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[29221]: cluster 2024-04-06T03:52:36.860439+0000 osd.6 (osd.6) 143 : cluster 0 5.11 scrub starts 2024-04-06T03:52:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[29221]: cluster 2024-04-06T03:52:36.863041+0000 osd.6 (osd.6) 144 : cluster 0 5.11 scrub ok 2024-04-06T03:52:38.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[29221]: cluster 2024-04-06T03:52:37.450363+0000 osd.3 (osd.3) 779 : cluster 0 6.11 scrub starts 2024-04-06T03:52:38.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:38 smithi050 bash[29221]: cluster 2024-04-06T03:52:37.502783+0000 osd.3 (osd.3) 780 : cluster 0 6.11 scrub ok 2024-04-06T03:52:39.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:39 smithi050 bash[21467]: audit 2024-04-06T03:52:38.412124+0000 mon.c (mon.2) 139 : audit 0 from='client.? 172.21.15.50:0/836432369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:52:39.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:39 smithi050 bash[21467]: cluster 2024-04-06T03:52:38.721111+0000 mgr.y (mgr.24368) 1160 : cluster 0 pgmap v1276: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-04-06T03:52:39.527 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:52:39 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:52:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:52:39.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:39 smithi050 bash[29221]: audit 2024-04-06T03:52:38.412124+0000 mon.c (mon.2) 139 : audit 0 from='client.? 172.21.15.50:0/836432369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:52:39.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:39 smithi050 bash[29221]: cluster 2024-04-06T03:52:38.721111+0000 mgr.y (mgr.24368) 1160 : cluster 0 pgmap v1276: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-04-06T03:52:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:39 smithi153 bash[23852]: audit 2024-04-06T03:52:38.412124+0000 mon.c (mon.2) 139 : audit 0 from='client.? 172.21.15.50:0/836432369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-06T03:52:39.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:39 smithi153 bash[23852]: cluster 2024-04-06T03:52:38.721111+0000 mgr.y (mgr.24368) 1160 : cluster 0 pgmap v1276: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-04-06T03:52:40.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:40 smithi050 bash[29221]: cluster 2024-04-06T03:52:38.414514+0000 osd.1 (osd.1) 533 : cluster 0 3.19 scrub starts 2024-04-06T03:52:40.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:40 smithi050 bash[29221]: cluster 2024-04-06T03:52:38.418654+0000 osd.1 (osd.1) 534 : cluster 0 3.19 scrub ok 2024-04-06T03:52:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:40 smithi050 bash[21467]: cluster 2024-04-06T03:52:38.414514+0000 osd.1 (osd.1) 533 : cluster 0 3.19 scrub starts 2024-04-06T03:52:40.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:40 smithi050 bash[21467]: cluster 2024-04-06T03:52:38.418654+0000 osd.1 (osd.1) 534 : cluster 0 3.19 scrub ok 2024-04-06T03:52:40.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:40 smithi153 bash[23852]: cluster 2024-04-06T03:52:38.414514+0000 osd.1 (osd.1) 533 : cluster 0 3.19 scrub starts 2024-04-06T03:52:40.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:40 smithi153 bash[23852]: cluster 2024-04-06T03:52:38.418654+0000 osd.1 (osd.1) 534 : cluster 0 3.19 scrub ok 2024-04-06T03:52:40.554 INFO:teuthology.orchestra.run.smithi050.stdout:55834575303 2024-04-06T03:52:41.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:52:40 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:52:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: cluster 2024-04-06T03:52:39.844889+0000 osd.2 (osd.2) 623 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: cluster 2024-04-06T03:52:39.845008+0000 osd.2 (osd.2) 624 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.109781+0000 osd.5 (osd.5) 215 : cluster 0 5.7 scrub starts 2024-04-06T03:52:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.163995+0000 mon.a (mon.0) 1942 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-04-06T03:52:41.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.170059+0000 osd.5 (osd.5) 216 : cluster 0 5.7 scrub ok 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: audit 2024-04-06T03:52:40.556138+0000 mon.a (mon.0) 1943 : audit 0 from='client.? 172.21.15.50:0/1316278135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: audit 2024-04-06T03:52:40.591793+0000 mgr.y (mgr.24368) 1161 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.721625+0000 mgr.y (mgr.24368) 1162 : cluster 0 pgmap v1278: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: cluster 2024-04-06T03:52:39.844889+0000 osd.2 (osd.2) 623 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: cluster 2024-04-06T03:52:39.845008+0000 osd.2 (osd.2) 624 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.109781+0000 osd.5 (osd.5) 215 : cluster 0 5.7 scrub starts 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.163995+0000 mon.a (mon.0) 1942 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.170059+0000 osd.5 (osd.5) 216 : cluster 0 5.7 scrub ok 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: audit 2024-04-06T03:52:40.556138+0000 mon.a (mon.0) 1943 : audit 0 from='client.? 172.21.15.50:0/1316278135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: audit 2024-04-06T03:52:40.591793+0000 mgr.y (mgr.24368) 1161 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:41.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:41 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.721625+0000 mgr.y (mgr.24368) 1162 : cluster 0 pgmap v1278: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: cluster 2024-04-06T03:52:39.844889+0000 osd.2 (osd.2) 623 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: cluster 2024-04-06T03:52:39.845008+0000 osd.2 (osd.2) 624 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.109781+0000 osd.5 (osd.5) 215 : cluster 0 5.7 scrub starts 2024-04-06T03:52:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.163995+0000 mon.a (mon.0) 1942 : cluster 0 osdmap e447: 8 total, 8 up, 8 in 2024-04-06T03:52:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.170059+0000 osd.5 (osd.5) 216 : cluster 0 5.7 scrub ok 2024-04-06T03:52:41.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: audit 2024-04-06T03:52:40.556138+0000 mon.a (mon.0) 1943 : audit 0 from='client.? 172.21.15.50:0/1316278135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-06T03:52:41.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: audit 2024-04-06T03:52:40.591793+0000 mgr.y (mgr.24368) 1161 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:41.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:41 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.721625+0000 mgr.y (mgr.24368) 1162 : cluster 0 pgmap v1278: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:41.771 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738828 got 34359738833 for osd.0 2024-04-06T03:52:41.772 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:42.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.858627+0000 osd.6 (osd.6) 145 : cluster 0 2.14 scrub starts 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.861096+0000 osd.6 (osd.6) 146 : cluster 0 2.14 scrub ok 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.888123+0000 osd.2 (osd.2) 625 : cluster 0 2.1 scrub starts 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:40.890458+0000 osd.2 (osd.2) 626 : cluster 0 2.1 scrub ok 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:41.375948+0000 osd.0 (osd.0) 748 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:41.376105+0000 osd.0 (osd.0) 749 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:41.465059+0000 osd.3 (osd.3) 781 : cluster 0 5.0 scrub starts 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[21467]: cluster 2024-04-06T03:52:41.468131+0000 osd.3 (osd.3) 782 : cluster 0 5.0 scrub ok 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.858627+0000 osd.6 (osd.6) 145 : cluster 0 2.14 scrub starts 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.861096+0000 osd.6 (osd.6) 146 : cluster 0 2.14 scrub ok 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.888123+0000 osd.2 (osd.2) 625 : cluster 0 2.1 scrub starts 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:40.890458+0000 osd.2 (osd.2) 626 : cluster 0 2.1 scrub ok 2024-04-06T03:52:42.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:41.375948+0000 osd.0 (osd.0) 748 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:41.376105+0000 osd.0 (osd.0) 749 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:41.465059+0000 osd.3 (osd.3) 781 : cluster 0 5.0 scrub starts 2024-04-06T03:52:42.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:42 smithi050 bash[29221]: cluster 2024-04-06T03:52:41.468131+0000 osd.3 (osd.3) 782 : cluster 0 5.0 scrub ok 2024-04-06T03:52:42.545 INFO:teuthology.orchestra.run.smithi050.stdout:81604379070 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.858627+0000 osd.6 (osd.6) 145 : cluster 0 2.14 scrub starts 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.861096+0000 osd.6 (osd.6) 146 : cluster 0 2.14 scrub ok 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.888123+0000 osd.2 (osd.2) 625 : cluster 0 2.1 scrub starts 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:40.890458+0000 osd.2 (osd.2) 626 : cluster 0 2.1 scrub ok 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:41.375948+0000 osd.0 (osd.0) 748 : cluster 0 purged_snaps scrub starts 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:41.376105+0000 osd.0 (osd.0) 749 : cluster 0 purged_snaps scrub ok 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:41.465059+0000 osd.3 (osd.3) 781 : cluster 0 5.0 scrub starts 2024-04-06T03:52:42.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:42 smithi153 bash[23852]: cluster 2024-04-06T03:52:41.468131+0000 osd.3 (osd.3) 782 : cluster 0 5.0 scrub ok 2024-04-06T03:52:42.630 INFO:teuthology.orchestra.run.smithi050.stdout:244813136275 2024-04-06T03:52:43.169 INFO:teuthology.orchestra.run.smithi050.stdout:120259084722 2024-04-06T03:52:43.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: cluster 2024-04-06T03:52:42.076200+0000 osd.4 (osd.4) 741 : cluster 0 4.1 scrub starts 2024-04-06T03:52:43.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: cluster 2024-04-06T03:52:42.078661+0000 osd.4 (osd.4) 742 : cluster 0 4.1 scrub ok 2024-04-06T03:52:43.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: cluster 2024-04-06T03:52:42.183556+0000 mon.a (mon.0) 1944 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-04-06T03:52:43.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: audit 2024-04-06T03:52:42.546895+0000 mon.a (mon.0) 1945 : audit 0 from='client.? 172.21.15.50:0/3186761612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: audit 2024-04-06T03:52:42.631721+0000 mon.c (mon.2) 140 : audit 0 from='client.? 172.21.15.50:0/682986778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: cluster 2024-04-06T03:52:42.722270+0000 mgr.y (mgr.24368) 1163 : cluster 0 pgmap v1280: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: audit 2024-04-06T03:52:42.874782+0000 mon.a (mon.0) 1946 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[29221]: audit 2024-04-06T03:52:43.170654+0000 mon.c (mon.2) 141 : audit 0 from='client.? 172.21.15.50:0/3179923716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: cluster 2024-04-06T03:52:42.076200+0000 osd.4 (osd.4) 741 : cluster 0 4.1 scrub starts 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: cluster 2024-04-06T03:52:42.078661+0000 osd.4 (osd.4) 742 : cluster 0 4.1 scrub ok 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: cluster 2024-04-06T03:52:42.183556+0000 mon.a (mon.0) 1944 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: audit 2024-04-06T03:52:42.546895+0000 mon.a (mon.0) 1945 : audit 0 from='client.? 172.21.15.50:0/3186761612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: audit 2024-04-06T03:52:42.631721+0000 mon.c (mon.2) 140 : audit 0 from='client.? 172.21.15.50:0/682986778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: cluster 2024-04-06T03:52:42.722270+0000 mgr.y (mgr.24368) 1163 : cluster 0 pgmap v1280: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:43.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: audit 2024-04-06T03:52:42.874782+0000 mon.a (mon.0) 1946 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:52:43.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:43 smithi050 bash[21467]: audit 2024-04-06T03:52:43.170654+0000 mon.c (mon.2) 141 : audit 0 from='client.? 172.21.15.50:0/3179923716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:52:43.506 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855779 got 150323855784 for osd.4 2024-04-06T03:52:43.506 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: cluster 2024-04-06T03:52:42.076200+0000 osd.4 (osd.4) 741 : cluster 0 4.1 scrub starts 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: cluster 2024-04-06T03:52:42.078661+0000 osd.4 (osd.4) 742 : cluster 0 4.1 scrub ok 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: cluster 2024-04-06T03:52:42.183556+0000 mon.a (mon.0) 1944 : cluster 0 osdmap e448: 8 total, 8 up, 8 in 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: audit 2024-04-06T03:52:42.546895+0000 mon.a (mon.0) 1945 : audit 0 from='client.? 172.21.15.50:0/3186761612' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: audit 2024-04-06T03:52:42.631721+0000 mon.c (mon.2) 140 : audit 0 from='client.? 172.21.15.50:0/682986778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: cluster 2024-04-06T03:52:42.722270+0000 mgr.y (mgr.24368) 1163 : cluster 0 pgmap v1280: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: audit 2024-04-06T03:52:42.874782+0000 mon.a (mon.0) 1946 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:52:43.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:43 smithi153 bash[23852]: audit 2024-04-06T03:52:43.170654+0000 mon.c (mon.2) 141 : audit 0 from='client.? 172.21.15.50:0/3179923716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-06T03:52:44.325 INFO:teuthology.orchestra.run.smithi050.stdout:197568496031 2024-04-06T03:52:44.561 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575298 got 55834575303 for osd.1 2024-04-06T03:52:44.561 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: cluster 2024-04-06T03:52:43.119503+0000 osd.4 (osd.4) 743 : cluster 0 3.9 scrub starts 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: cluster 2024-04-06T03:52:43.125483+0000 osd.4 (osd.4) 744 : cluster 0 3.9 scrub ok 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: audit 2024-04-06T03:52:43.346015+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: audit 2024-04-06T03:52:43.350712+0000 mon.a (mon.0) 1948 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: cluster 2024-04-06T03:52:43.392992+0000 osd.1 (osd.1) 535 : cluster 0 2.6 deep-scrub starts 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: cluster 2024-04-06T03:52:43.395837+0000 osd.1 (osd.1) 536 : cluster 0 2.6 deep-scrub ok 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: audit 2024-04-06T03:52:43.983833+0000 mon.a (mon.0) 1949 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: audit 2024-04-06T03:52:43.989290+0000 mon.a (mon.0) 1950 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[21467]: audit 2024-04-06T03:52:44.326283+0000 mon.a (mon.0) 1951 : audit 0 from='client.? 172.21.15.50:0/3432214409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: cluster 2024-04-06T03:52:43.119503+0000 osd.4 (osd.4) 743 : cluster 0 3.9 scrub starts 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: cluster 2024-04-06T03:52:43.125483+0000 osd.4 (osd.4) 744 : cluster 0 3.9 scrub ok 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: audit 2024-04-06T03:52:43.346015+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: audit 2024-04-06T03:52:43.350712+0000 mon.a (mon.0) 1948 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: cluster 2024-04-06T03:52:43.392992+0000 osd.1 (osd.1) 535 : cluster 0 2.6 deep-scrub starts 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: cluster 2024-04-06T03:52:43.395837+0000 osd.1 (osd.1) 536 : cluster 0 2.6 deep-scrub ok 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: audit 2024-04-06T03:52:43.983833+0000 mon.a (mon.0) 1949 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: audit 2024-04-06T03:52:43.989290+0000 mon.a (mon.0) 1950 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.650 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:44 smithi050 bash[29221]: audit 2024-04-06T03:52:44.326283+0000 mon.a (mon.0) 1951 : audit 0 from='client.? 172.21.15.50:0/3432214409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:52:44.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: cluster 2024-04-06T03:52:43.119503+0000 osd.4 (osd.4) 743 : cluster 0 3.9 scrub starts 2024-04-06T03:52:44.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: cluster 2024-04-06T03:52:43.125483+0000 osd.4 (osd.4) 744 : cluster 0 3.9 scrub ok 2024-04-06T03:52:44.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: audit 2024-04-06T03:52:43.346015+0000 mon.a (mon.0) 1947 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: audit 2024-04-06T03:52:43.350712+0000 mon.a (mon.0) 1948 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: cluster 2024-04-06T03:52:43.392992+0000 osd.1 (osd.1) 535 : cluster 0 2.6 deep-scrub starts 2024-04-06T03:52:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: cluster 2024-04-06T03:52:43.395837+0000 osd.1 (osd.1) 536 : cluster 0 2.6 deep-scrub ok 2024-04-06T03:52:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: audit 2024-04-06T03:52:43.983833+0000 mon.a (mon.0) 1949 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: audit 2024-04-06T03:52:43.989290+0000 mon.a (mon.0) 1950 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:44.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:44 smithi153 bash[23852]: audit 2024-04-06T03:52:44.326283+0000 mon.a (mon.0) 1951 : audit 0 from='client.? 172.21.15.50:0/3432214409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-06T03:52:44.822 INFO:teuthology.orchestra.run.smithi050.stdout:296352743817 2024-04-06T03:52:45.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.003423+0000 osd.0 (osd.0) 750 : cluster 0 3.17 scrub starts 2024-04-06T03:52:45.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.005634+0000 osd.0 (osd.0) 751 : cluster 0 3.17 scrub ok 2024-04-06T03:52:45.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.348087+0000 osd.1 (osd.1) 537 : cluster 0 2.9 scrub starts 2024-04-06T03:52:45.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.003423+0000 osd.0 (osd.0) 750 : cluster 0 3.17 scrub starts 2024-04-06T03:52:45.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.005634+0000 osd.0 (osd.0) 751 : cluster 0 3.17 scrub ok 2024-04-06T03:52:45.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.348087+0000 osd.1 (osd.1) 537 : cluster 0 2.9 scrub starts 2024-04-06T03:52:45.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.349485+0000 osd.1 (osd.1) 538 : cluster 0 2.9 scrub ok 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: audit 2024-04-06T03:52:44.544605+0000 mon.a (mon.0) 1952 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: audit 2024-04-06T03:52:44.550429+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: audit 2024-04-06T03:52:44.551573+0000 mon.a (mon.0) 1954 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: audit 2024-04-06T03:52:44.552309+0000 mon.a (mon.0) 1955 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: audit 2024-04-06T03:52:44.558443+0000 mon.a (mon.0) 1956 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.609394+0000 osd.7 (osd.7) 785 : cluster 0 5.9 scrub starts 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.612561+0000 osd.7 (osd.7) 786 : cluster 0 5.9 scrub ok 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:44.723199+0000 mgr.y (mgr.24368) 1164 : cluster 0 pgmap v1281: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: audit 2024-04-06T03:52:44.824108+0000 mon.a (mon.0) 1957 : audit 0 from='client.? 172.21.15.50:0/3877536433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:45.146309+0000 osd.4 (osd.4) 745 : cluster 0 5.18 scrub starts 2024-04-06T03:52:45.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:45 smithi153 bash[23852]: cluster 2024-04-06T03:52:45.148784+0000 osd.4 (osd.4) 746 : cluster 0 5.18 scrub ok 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.349485+0000 osd.1 (osd.1) 538 : cluster 0 2.9 scrub ok 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: audit 2024-04-06T03:52:44.544605+0000 mon.a (mon.0) 1952 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: audit 2024-04-06T03:52:44.550429+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: audit 2024-04-06T03:52:44.551573+0000 mon.a (mon.0) 1954 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: audit 2024-04-06T03:52:44.552309+0000 mon.a (mon.0) 1955 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: audit 2024-04-06T03:52:44.558443+0000 mon.a (mon.0) 1956 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.609394+0000 osd.7 (osd.7) 785 : cluster 0 5.9 scrub starts 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.612561+0000 osd.7 (osd.7) 786 : cluster 0 5.9 scrub ok 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:44.723199+0000 mgr.y (mgr.24368) 1164 : cluster 0 pgmap v1281: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: audit 2024-04-06T03:52:44.824108+0000 mon.a (mon.0) 1957 : audit 0 from='client.? 172.21.15.50:0/3877536433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:45.146309+0000 osd.4 (osd.4) 745 : cluster 0 5.18 scrub starts 2024-04-06T03:52:45.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[21467]: cluster 2024-04-06T03:52:45.148784+0000 osd.4 (osd.4) 746 : cluster 0 5.18 scrub ok 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.003423+0000 osd.0 (osd.0) 750 : cluster 0 3.17 scrub starts 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.005634+0000 osd.0 (osd.0) 751 : cluster 0 3.17 scrub ok 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.348087+0000 osd.1 (osd.1) 537 : cluster 0 2.9 scrub starts 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.349485+0000 osd.1 (osd.1) 538 : cluster 0 2.9 scrub ok 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: audit 2024-04-06T03:52:44.544605+0000 mon.a (mon.0) 1952 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: audit 2024-04-06T03:52:44.550429+0000 mon.a (mon.0) 1953 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: audit 2024-04-06T03:52:44.551573+0000 mon.a (mon.0) 1954 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: audit 2024-04-06T03:52:44.552309+0000 mon.a (mon.0) 1955 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: audit 2024-04-06T03:52:44.558443+0000 mon.a (mon.0) 1956 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.609394+0000 osd.7 (osd.7) 785 : cluster 0 5.9 scrub starts 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.612561+0000 osd.7 (osd.7) 786 : cluster 0 5.9 scrub ok 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:44.723199+0000 mgr.y (mgr.24368) 1164 : cluster 0 pgmap v1281: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:45.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: audit 2024-04-06T03:52:44.824108+0000 mon.a (mon.0) 1957 : audit 0 from='client.? 172.21.15.50:0/3877536433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-06T03:52:45.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:45.146309+0000 osd.4 (osd.4) 745 : cluster 0 5.18 scrub starts 2024-04-06T03:52:45.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:45 smithi050 bash[29221]: cluster 2024-04-06T03:52:45.148784+0000 osd.4 (osd.4) 746 : cluster 0 5.18 scrub ok 2024-04-06T03:52:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:46 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.110529+0000 osd.4 (osd.4) 747 : cluster 0 2.f scrub starts 2024-04-06T03:52:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:46 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.113862+0000 osd.4 (osd.4) 748 : cluster 0 2.f scrub ok 2024-04-06T03:52:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:46 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.110529+0000 osd.4 (osd.4) 747 : cluster 0 2.f scrub starts 2024-04-06T03:52:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:46 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.113862+0000 osd.4 (osd.4) 748 : cluster 0 2.f scrub ok 2024-04-06T03:52:47.046 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813136269 got 244813136275 for osd.6 2024-04-06T03:52:47.047 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:47.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:46 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.110529+0000 osd.4 (osd.4) 747 : cluster 0 2.f scrub starts 2024-04-06T03:52:47.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:46 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.113862+0000 osd.4 (osd.4) 748 : cluster 0 2.f scrub ok 2024-04-06T03:52:47.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084717 got 120259084722 for osd.3 2024-04-06T03:52:47.365 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:47.366 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604379064 got 81604379070 for osd.2 2024-04-06T03:52:47.366 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:47.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.163621+0000 osd.5 (osd.5) 217 : cluster 0 3.1d scrub starts 2024-04-06T03:52:47.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.168271+0000 osd.5 (osd.5) 218 : cluster 0 3.1d scrub ok 2024-04-06T03:52:47.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.723871+0000 mgr.y (mgr.24368) 1165 : cluster 0 pgmap v1282: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:47.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.970376+0000 osd.0 (osd.0) 752 : cluster 0 6.0 scrub starts 2024-04-06T03:52:47.714 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[21467]: cluster 2024-04-06T03:52:47.065290+0000 osd.0 (osd.0) 753 : cluster 0 6.0 scrub ok 2024-04-06T03:52:47.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.163621+0000 osd.5 (osd.5) 217 : cluster 0 3.1d scrub starts 2024-04-06T03:52:47.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.168271+0000 osd.5 (osd.5) 218 : cluster 0 3.1d scrub ok 2024-04-06T03:52:47.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.723871+0000 mgr.y (mgr.24368) 1165 : cluster 0 pgmap v1282: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:47.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.970376+0000 osd.0 (osd.0) 752 : cluster 0 6.0 scrub starts 2024-04-06T03:52:47.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:47 smithi050 bash[29221]: cluster 2024-04-06T03:52:47.065290+0000 osd.0 (osd.0) 753 : cluster 0 6.0 scrub ok 2024-04-06T03:52:47.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743812 got 296352743817 for osd.7 2024-04-06T03:52:47.717 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:47.718 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496026 got 197568496031 for osd.5 2024-04-06T03:52:47.719 DEBUG:teuthology.parallel:result is None 2024-04-06T03:52:47.719 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-06T03:52:47.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:52:48.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:47 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.163621+0000 osd.5 (osd.5) 217 : cluster 0 3.1d scrub starts 2024-04-06T03:52:48.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:47 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.168271+0000 osd.5 (osd.5) 218 : cluster 0 3.1d scrub ok 2024-04-06T03:52:48.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:47 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.723871+0000 mgr.y (mgr.24368) 1165 : cluster 0 pgmap v1282: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:48.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:47 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.970376+0000 osd.0 (osd.0) 752 : cluster 0 6.0 scrub starts 2024-04-06T03:52:48.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:47 smithi153 bash[23852]: cluster 2024-04-06T03:52:47.065290+0000 osd.0 (osd.0) 753 : cluster 0 6.0 scrub ok 2024-04-06T03:52:48.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.663369+0000 osd.7 (osd.7) 787 : cluster 0 3.d scrub starts 2024-04-06T03:52:48.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[29221]: cluster 2024-04-06T03:52:46.668755+0000 osd.7 (osd.7) 788 : cluster 0 3.d scrub ok 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[29221]: cluster 2024-04-06T03:52:47.305609+0000 osd.1 (osd.1) 539 : cluster 0 5.12 scrub starts 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[29221]: cluster 2024-04-06T03:52:47.308302+0000 osd.1 (osd.1) 540 : cluster 0 5.12 scrub ok 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[29221]: cluster 2024-04-06T03:52:48.036812+0000 osd.4 (osd.4) 749 : cluster 0 5.1 scrub starts 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[29221]: cluster 2024-04-06T03:52:48.039167+0000 osd.4 (osd.4) 750 : cluster 0 5.1 scrub ok 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.663369+0000 osd.7 (osd.7) 787 : cluster 0 3.d scrub starts 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[21467]: cluster 2024-04-06T03:52:46.668755+0000 osd.7 (osd.7) 788 : cluster 0 3.d scrub ok 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[21467]: cluster 2024-04-06T03:52:47.305609+0000 osd.1 (osd.1) 539 : cluster 0 5.12 scrub starts 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[21467]: cluster 2024-04-06T03:52:47.308302+0000 osd.1 (osd.1) 540 : cluster 0 5.12 scrub ok 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[21467]: cluster 2024-04-06T03:52:48.036812+0000 osd.4 (osd.4) 749 : cluster 0 5.1 scrub starts 2024-04-06T03:52:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:48 smithi050 bash[21467]: cluster 2024-04-06T03:52:48.039167+0000 osd.4 (osd.4) 750 : cluster 0 5.1 scrub ok 2024-04-06T03:52:49.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:48 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.663369+0000 osd.7 (osd.7) 787 : cluster 0 3.d scrub starts 2024-04-06T03:52:49.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:48 smithi153 bash[23852]: cluster 2024-04-06T03:52:46.668755+0000 osd.7 (osd.7) 788 : cluster 0 3.d scrub ok 2024-04-06T03:52:49.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:48 smithi153 bash[23852]: cluster 2024-04-06T03:52:47.305609+0000 osd.1 (osd.1) 539 : cluster 0 5.12 scrub starts 2024-04-06T03:52:49.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:48 smithi153 bash[23852]: cluster 2024-04-06T03:52:47.308302+0000 osd.1 (osd.1) 540 : cluster 0 5.12 scrub ok 2024-04-06T03:52:49.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:48 smithi153 bash[23852]: cluster 2024-04-06T03:52:48.036812+0000 osd.4 (osd.4) 749 : cluster 0 5.1 scrub starts 2024-04-06T03:52:49.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:48 smithi153 bash[23852]: cluster 2024-04-06T03:52:48.039167+0000 osd.4 (osd.4) 750 : cluster 0 5.1 scrub ok 2024-04-06T03:52:49.588 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:52:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:52:49.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[21467]: cluster 2024-04-06T03:52:47.660355+0000 osd.7 (osd.7) 789 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:52:49.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[21467]: cluster 2024-04-06T03:52:47.666003+0000 osd.7 (osd.7) 790 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:52:49.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[21467]: cluster 2024-04-06T03:52:48.724603+0000 mgr.y (mgr.24368) 1166 : cluster 0 pgmap v1283: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-06T03:52:49.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[29221]: cluster 2024-04-06T03:52:47.660355+0000 osd.7 (osd.7) 789 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:52:49.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[29221]: cluster 2024-04-06T03:52:47.666003+0000 osd.7 (osd.7) 790 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:52:49.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:49 smithi050 bash[29221]: cluster 2024-04-06T03:52:48.724603+0000 mgr.y (mgr.24368) 1166 : cluster 0 pgmap v1283: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-06T03:52:50.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:49 smithi153 bash[23852]: cluster 2024-04-06T03:52:47.660355+0000 osd.7 (osd.7) 789 : cluster 0 3.11 deep-scrub starts 2024-04-06T03:52:50.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:49 smithi153 bash[23852]: cluster 2024-04-06T03:52:47.666003+0000 osd.7 (osd.7) 790 : cluster 0 3.11 deep-scrub ok 2024-04-06T03:52:50.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:49 smithi153 bash[23852]: cluster 2024-04-06T03:52:48.724603+0000 mgr.y (mgr.24368) 1166 : cluster 0 pgmap v1283: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-06T03:52:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:50 smithi050 bash[21467]: cluster 2024-04-06T03:52:48.662304+0000 osd.7 (osd.7) 791 : cluster 0 3.13 scrub starts 2024-04-06T03:52:50.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:50 smithi050 bash[21467]: cluster 2024-04-06T03:52:48.666296+0000 osd.7 (osd.7) 792 : cluster 0 3.13 scrub ok 2024-04-06T03:52:50.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:50 smithi050 bash[29221]: cluster 2024-04-06T03:52:48.662304+0000 osd.7 (osd.7) 791 : cluster 0 3.13 scrub starts 2024-04-06T03:52:50.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:50 smithi050 bash[29221]: cluster 2024-04-06T03:52:48.666296+0000 osd.7 (osd.7) 792 : cluster 0 3.13 scrub ok 2024-04-06T03:52:51.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:52:50 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:52:51.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:50 smithi153 bash[23852]: cluster 2024-04-06T03:52:48.662304+0000 osd.7 (osd.7) 791 : cluster 0 3.13 scrub starts 2024-04-06T03:52:51.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:50 smithi153 bash[23852]: cluster 2024-04-06T03:52:48.666296+0000 osd.7 (osd.7) 792 : cluster 0 3.13 scrub ok 2024-04-06T03:52:51.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:49.840781+0000 osd.2 (osd.2) 627 : cluster 0 6.f scrub starts 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:49.862003+0000 osd.6 (osd.6) 147 : cluster 0 3.10 deep-scrub starts 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:49.865732+0000 osd.6 (osd.6) 148 : cluster 0 3.10 deep-scrub ok 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:50.052396+0000 osd.2 (osd.2) 628 : cluster 0 6.f scrub ok 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:50.138653+0000 osd.5 (osd.5) 219 : cluster 0 5.15 scrub starts 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:50.140983+0000 osd.5 (osd.5) 220 : cluster 0 5.15 scrub ok 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:50.324968+0000 osd.1 (osd.1) 541 : cluster 0 2.15 scrub starts 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:50.327506+0000 osd.1 (osd.1) 542 : cluster 0 2.15 scrub ok 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: audit 2024-04-06T03:52:50.599765+0000 mgr.y (mgr.24368) 1167 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: cluster 2024-04-06T03:52:50.726491+0000 mgr.y (mgr.24368) 1168 : cluster 0 pgmap v1284: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[21467]: audit 2024-04-06T03:52:51.292808+0000 mon.a (mon.0) 1958 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:49.840781+0000 osd.2 (osd.2) 627 : cluster 0 6.f scrub starts 2024-04-06T03:52:51.991 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:49.862003+0000 osd.6 (osd.6) 147 : cluster 0 3.10 deep-scrub starts 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:49.865732+0000 osd.6 (osd.6) 148 : cluster 0 3.10 deep-scrub ok 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:50.052396+0000 osd.2 (osd.2) 628 : cluster 0 6.f scrub ok 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:50.138653+0000 osd.5 (osd.5) 219 : cluster 0 5.15 scrub starts 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:50.140983+0000 osd.5 (osd.5) 220 : cluster 0 5.15 scrub ok 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:50.324968+0000 osd.1 (osd.1) 541 : cluster 0 2.15 scrub starts 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:50.327506+0000 osd.1 (osd.1) 542 : cluster 0 2.15 scrub ok 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: audit 2024-04-06T03:52:50.599765+0000 mgr.y (mgr.24368) 1167 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: cluster 2024-04-06T03:52:50.726491+0000 mgr.y (mgr.24368) 1168 : cluster 0 pgmap v1284: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:51.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:51 smithi050 bash[29221]: audit 2024-04-06T03:52:51.292808+0000 mon.a (mon.0) 1958 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:49.840781+0000 osd.2 (osd.2) 627 : cluster 0 6.f scrub starts 2024-04-06T03:52:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:49.862003+0000 osd.6 (osd.6) 147 : cluster 0 3.10 deep-scrub starts 2024-04-06T03:52:52.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:49.865732+0000 osd.6 (osd.6) 148 : cluster 0 3.10 deep-scrub ok 2024-04-06T03:52:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:50.052396+0000 osd.2 (osd.2) 628 : cluster 0 6.f scrub ok 2024-04-06T03:52:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:50.138653+0000 osd.5 (osd.5) 219 : cluster 0 5.15 scrub starts 2024-04-06T03:52:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:50.140983+0000 osd.5 (osd.5) 220 : cluster 0 5.15 scrub ok 2024-04-06T03:52:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:50.324968+0000 osd.1 (osd.1) 541 : cluster 0 2.15 scrub starts 2024-04-06T03:52:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:50.327506+0000 osd.1 (osd.1) 542 : cluster 0 2.15 scrub ok 2024-04-06T03:52:52.048 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: audit 2024-04-06T03:52:50.599765+0000 mgr.y (mgr.24368) 1167 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:52:52.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: cluster 2024-04-06T03:52:50.726491+0000 mgr.y (mgr.24368) 1168 : cluster 0 pgmap v1284: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:52:52.049 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:51 smithi153 bash[23852]: audit 2024-04-06T03:52:51.292808+0000 mon.a (mon.0) 1958 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:52:52.515 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:52:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:52 smithi050 bash[29221]: cluster 2024-04-06T03:52:51.178107+0000 osd.5 (osd.5) 221 : cluster 0 2.3 scrub starts 2024-04-06T03:52:52.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:52 smithi050 bash[29221]: cluster 2024-04-06T03:52:51.180603+0000 osd.5 (osd.5) 222 : cluster 0 2.3 scrub ok 2024-04-06T03:52:52.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:52 smithi050 bash[21467]: cluster 2024-04-06T03:52:51.178107+0000 osd.5 (osd.5) 221 : cluster 0 2.3 scrub starts 2024-04-06T03:52:52.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:52 smithi050 bash[21467]: cluster 2024-04-06T03:52:51.180603+0000 osd.5 (osd.5) 222 : cluster 0 2.3 scrub ok 2024-04-06T03:52:53.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:52 smithi153 bash[23852]: cluster 2024-04-06T03:52:51.178107+0000 osd.5 (osd.5) 221 : cluster 0 2.3 scrub starts 2024-04-06T03:52:53.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:52 smithi153 bash[23852]: cluster 2024-04-06T03:52:51.180603+0000 osd.5 (osd.5) 222 : cluster 0 2.3 scrub ok 2024-04-06T03:52:53.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.619404+0000 osd.7 (osd.7) 793 : cluster 0 2.1d scrub starts 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.622500+0000 osd.7 (osd.7) 794 : cluster 0 2.1d scrub ok 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.648007+0000 osd.3 (osd.3) 783 : cluster 0 2.1e deep-scrub starts 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.650542+0000 osd.3 (osd.3) 784 : cluster 0 2.1e deep-scrub ok 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.727333+0000 mgr.y (mgr.24368) 1169 : cluster 0 pgmap v1285: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.933793+0000 osd.2 (osd.2) 629 : cluster 0 2.e deep-scrub starts 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[21467]: cluster 2024-04-06T03:52:52.936106+0000 osd.2 (osd.2) 630 : cluster 0 2.e deep-scrub ok 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.619404+0000 osd.7 (osd.7) 793 : cluster 0 2.1d scrub starts 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.622500+0000 osd.7 (osd.7) 794 : cluster 0 2.1d scrub ok 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.648007+0000 osd.3 (osd.3) 783 : cluster 0 2.1e deep-scrub starts 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.650542+0000 osd.3 (osd.3) 784 : cluster 0 2.1e deep-scrub ok 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.727333+0000 mgr.y (mgr.24368) 1169 : cluster 0 pgmap v1285: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.933793+0000 osd.2 (osd.2) 629 : cluster 0 2.e deep-scrub starts 2024-04-06T03:52:53.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:53 smithi050 bash[29221]: cluster 2024-04-06T03:52:52.936106+0000 osd.2 (osd.2) 630 : cluster 0 2.e deep-scrub ok 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.619404+0000 osd.7 (osd.7) 793 : cluster 0 2.1d scrub starts 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.622500+0000 osd.7 (osd.7) 794 : cluster 0 2.1d scrub ok 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.648007+0000 osd.3 (osd.3) 783 : cluster 0 2.1e deep-scrub starts 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.650542+0000 osd.3 (osd.3) 784 : cluster 0 2.1e deep-scrub ok 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.727333+0000 mgr.y (mgr.24368) 1169 : cluster 0 pgmap v1285: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.933793+0000 osd.2 (osd.2) 629 : cluster 0 2.e deep-scrub starts 2024-04-06T03:52:54.047 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:53 smithi153 bash[23852]: cluster 2024-04-06T03:52:52.936106+0000 osd.2 (osd.2) 630 : cluster 0 2.e deep-scrub ok 2024-04-06T03:52:55.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[21467]: cluster 2024-04-06T03:52:53.887483+0000 osd.0 (osd.0) 754 : cluster 0 2.1f scrub starts 2024-04-06T03:52:55.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[21467]: cluster 2024-04-06T03:52:53.889302+0000 osd.0 (osd.0) 755 : cluster 0 2.1f scrub ok 2024-04-06T03:52:55.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[21467]: cluster 2024-04-06T03:52:54.605277+0000 osd.3 (osd.3) 785 : cluster 0 3.18 deep-scrub starts 2024-04-06T03:52:55.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[21467]: cluster 2024-04-06T03:52:54.610355+0000 osd.3 (osd.3) 786 : cluster 0 3.18 deep-scrub ok 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[21467]: cluster 2024-04-06T03:52:54.615243+0000 osd.7 (osd.7) 795 : cluster 0 6.15 scrub starts 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[21467]: cluster 2024-04-06T03:52:54.728296+0000 mgr.y (mgr.24368) 1170 : cluster 0 pgmap v1286: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[29221]: cluster 2024-04-06T03:52:53.887483+0000 osd.0 (osd.0) 754 : cluster 0 2.1f scrub starts 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[29221]: cluster 2024-04-06T03:52:53.889302+0000 osd.0 (osd.0) 755 : cluster 0 2.1f scrub ok 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[29221]: cluster 2024-04-06T03:52:54.605277+0000 osd.3 (osd.3) 785 : cluster 0 3.18 deep-scrub starts 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[29221]: cluster 2024-04-06T03:52:54.610355+0000 osd.3 (osd.3) 786 : cluster 0 3.18 deep-scrub ok 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[29221]: cluster 2024-04-06T03:52:54.615243+0000 osd.7 (osd.7) 795 : cluster 0 6.15 scrub starts 2024-04-06T03:52:55.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:55 smithi050 bash[29221]: cluster 2024-04-06T03:52:54.728296+0000 mgr.y (mgr.24368) 1170 : cluster 0 pgmap v1286: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:55 smithi153 bash[23852]: cluster 2024-04-06T03:52:53.887483+0000 osd.0 (osd.0) 754 : cluster 0 2.1f scrub starts 2024-04-06T03:52:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:55 smithi153 bash[23852]: cluster 2024-04-06T03:52:53.889302+0000 osd.0 (osd.0) 755 : cluster 0 2.1f scrub ok 2024-04-06T03:52:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:55 smithi153 bash[23852]: cluster 2024-04-06T03:52:54.605277+0000 osd.3 (osd.3) 785 : cluster 0 3.18 deep-scrub starts 2024-04-06T03:52:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:55 smithi153 bash[23852]: cluster 2024-04-06T03:52:54.610355+0000 osd.3 (osd.3) 786 : cluster 0 3.18 deep-scrub ok 2024-04-06T03:52:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:55 smithi153 bash[23852]: cluster 2024-04-06T03:52:54.615243+0000 osd.7 (osd.7) 795 : cluster 0 6.15 scrub starts 2024-04-06T03:52:55.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:55 smithi153 bash[23852]: cluster 2024-04-06T03:52:54.728296+0000 mgr.y (mgr.24368) 1170 : cluster 0 pgmap v1286: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:52:55.996 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:52:55.998 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:52:56.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[21467]: cluster 2024-04-06T03:52:54.708822+0000 osd.7 (osd.7) 796 : cluster 0 6.15 scrub ok 2024-04-06T03:52:56.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[21467]: cluster 2024-04-06T03:52:55.626291+0000 osd.3 (osd.3) 787 : cluster 0 6.6 scrub starts 2024-04-06T03:52:56.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[21467]: cluster 2024-04-06T03:52:55.670208+0000 osd.3 (osd.3) 788 : cluster 0 6.6 scrub ok 2024-04-06T03:52:56.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[21467]: audit 2024-04-06T03:52:55.975741+0000 mgr.y (mgr.24368) 1171 : audit 0 from='client.25817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:52:56.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[29221]: cluster 2024-04-06T03:52:54.708822+0000 osd.7 (osd.7) 796 : cluster 0 6.15 scrub ok 2024-04-06T03:52:56.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[29221]: cluster 2024-04-06T03:52:55.626291+0000 osd.3 (osd.3) 787 : cluster 0 6.6 scrub starts 2024-04-06T03:52:56.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[29221]: cluster 2024-04-06T03:52:55.670208+0000 osd.3 (osd.3) 788 : cluster 0 6.6 scrub ok 2024-04-06T03:52:56.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:56 smithi050 bash[29221]: audit 2024-04-06T03:52:55.975741+0000 mgr.y (mgr.24368) 1171 : audit 0 from='client.25817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:52:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:56 smithi153 bash[23852]: cluster 2024-04-06T03:52:54.708822+0000 osd.7 (osd.7) 796 : cluster 0 6.15 scrub ok 2024-04-06T03:52:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:56 smithi153 bash[23852]: cluster 2024-04-06T03:52:55.626291+0000 osd.3 (osd.3) 787 : cluster 0 6.6 scrub starts 2024-04-06T03:52:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:56 smithi153 bash[23852]: cluster 2024-04-06T03:52:55.670208+0000 osd.3 (osd.3) 788 : cluster 0 6.6 scrub ok 2024-04-06T03:52:56.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:56 smithi153 bash[23852]: audit 2024-04-06T03:52:55.975741+0000 mgr.y (mgr.24368) 1171 : audit 0 from='client.25817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:52:56.801 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1286,"stamp":"2024-04-06T03:52:54.727622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":4391,"num_read_kb":4205,"num_write":1665,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":4736933,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1907,"ondisk_log_size":1907,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4515676,"kb_used_data":8348,"kb_used_omap":12,"kb_used_meta":4507251,"kb_avail":745445540,"statfs":{"total":767960285184,"available":763336232960,"internally_reserved":0,"allocated":8548352,"data_stored":5619970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4615425585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005748"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":867,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181756+0000","last_change":"2024-04-06T03:51:35.538693+0000","last_active":"2024-04-06T03:52:42.181756+0000","last_peered":"2024-04-06T03:52:42.181756+0000","last_clean":"2024-04-06T03:52:42.181756+0000","last_became_active":"2024-04-06T03:42:23.588459+0000","last_became_peered":"2024-04-06T03:42:23.588459+0000","last_unstale":"2024-04-06T03:52:42.181756+0000","last_undegraded":"2024-04-06T03:52:42.181756+0000","last_fullsized":"2024-04-06T03:52:42.181756+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:35.538528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_clean_scrub_stamp":"2024-04-06T03:51:35.538528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:55.284493+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":892,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:45.148942+0000","last_change":"2024-04-06T03:52:45.148998+0000","last_active":"2024-04-06T03:52:45.148942+0000","last_peered":"2024-04-06T03:52:45.148942+0000","last_clean":"2024-04-06T03:52:45.148942+0000","last_became_active":"2024-04-06T03:48:21.308873+0000","last_became_peered":"2024-04-06T03:48:21.308873+0000","last_unstale":"2024-04-06T03:52:45.148942+0000","last_undegraded":"2024-04-06T03:52:45.148942+0000","last_fullsized":"2024-04-06T03:52:45.148942+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:45.148794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:11.395397+0000","last_clean_scrub_stamp":"2024-04-06T03:52:45.148794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:10.335782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":133,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:53.889428+0000","last_change":"2024-04-06T03:52:53.889476+0000","last_active":"2024-04-06T03:52:53.889428+0000","last_peered":"2024-04-06T03:52:53.889428+0000","last_clean":"2024-04-06T03:52:53.889428+0000","last_became_active":"2024-04-06T03:48:32.447661+0000","last_became_peered":"2024-04-06T03:48:32.447661+0000","last_unstale":"2024-04-06T03:52:53.889428+0000","last_undegraded":"2024-04-06T03:52:53.889428+0000","last_fullsized":"2024-04-06T03:52:53.889428+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:53.889310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_clean_scrub_stamp":"2024-04-06T03:52:53.889310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:21.498483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"433'52","reported_seq":984,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.229608+0000","last_change":"2024-04-06T03:51:43.336368+0000","last_active":"2024-04-06T03:52:42.229608+0000","last_peered":"2024-04-06T03:52:42.229608+0000","last_clean":"2024-04-06T03:52:42.229608+0000","last_became_active":"2024-04-06T03:42:23.588857+0000","last_became_peered":"2024-04-06T03:42:23.588857+0000","last_unstale":"2024-04-06T03:52:42.229608+0000","last_undegraded":"2024-04-06T03:52:42.229608+0000","last_fullsized":"2024-04-06T03:52:42.229608+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'52","last_scrub_stamp":"2024-04-06T03:51:43.335943+0000","last_deep_scrub":"433'52","last_deep_scrub_stamp":"2024-04-06T03:51:43.335943+0000","last_clean_scrub_stamp":"2024-04-06T03:51:43.335943+0000","objects_scrubbed":6,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:58.038617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":63,"num_write":46,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671368+0000","last_change":"2024-04-06T03:52:18.525011+0000","last_active":"2024-04-06T03:52:42.671368+0000","last_peered":"2024-04-06T03:52:42.671368+0000","last_clean":"2024-04-06T03:52:42.671368+0000","last_became_active":"2024-04-06T03:48:21.296906+0000","last_became_peered":"2024-04-06T03:48:21.296906+0000","last_unstale":"2024-04-06T03:52:42.671368+0000","last_undegraded":"2024-04-06T03:52:42.671368+0000","last_fullsized":"2024-04-06T03:52:42.671368+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:18.524759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:52:18.524759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:45.251770+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":719,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669609+0000","last_change":"2024-04-06T03:51:43.429918+0000","last_active":"2024-04-06T03:52:42.669609+0000","last_peered":"2024-04-06T03:52:42.669609+0000","last_clean":"2024-04-06T03:52:42.669609+0000","last_became_active":"2024-04-06T03:49:27.616011+0000","last_became_peered":"2024-04-06T03:49:27.616011+0000","last_unstale":"2024-04-06T03:52:42.669609+0000","last_undegraded":"2024-04-06T03:52:42.669609+0000","last_fullsized":"2024-04-06T03:52:42.669609+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:43.429712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:28.642157+0000","last_clean_scrub_stamp":"2024-04-06T03:51:43.429712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:52.108295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":893,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.650717+0000","last_change":"2024-04-06T03:52:52.650774+0000","last_active":"2024-04-06T03:52:52.650717+0000","last_peered":"2024-04-06T03:52:52.650717+0000","last_clean":"2024-04-06T03:52:52.650717+0000","last_became_active":"2024-04-06T03:48:32.467675+0000","last_became_peered":"2024-04-06T03:48:32.467675+0000","last_unstale":"2024-04-06T03:52:52.650717+0000","last_undegraded":"2024-04-06T03:52:52.650717+0000","last_fullsized":"2024-04-06T03:52:52.650717+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:52.650552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:52.650552+0000","last_clean_scrub_stamp":"2024-04-06T03:52:52.650552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:09.968970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"433'47","reported_seq":151,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671828+0000","last_change":"2024-04-06T03:51:58.262615+0000","last_active":"2024-04-06T03:52:42.671828+0000","last_peered":"2024-04-06T03:52:42.671828+0000","last_clean":"2024-04-06T03:52:42.671828+0000","last_became_active":"2024-04-06T03:48:32.464269+0000","last_became_peered":"2024-04-06T03:48:32.464269+0000","last_unstale":"2024-04-06T03:52:42.671828+0000","last_undegraded":"2024-04-06T03:52:42.671828+0000","last_fullsized":"2024-04-06T03:52:42.671828+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'47","last_scrub_stamp":"2024-04-06T03:51:58.262487+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:51:58.262487+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.209317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":64,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":189,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.677086+0000","last_change":"2024-04-06T03:52:21.009573+0000","last_active":"2024-04-06T03:52:42.677086+0000","last_peered":"2024-04-06T03:52:42.677086+0000","last_clean":"2024-04-06T03:52:42.677086+0000","last_became_active":"2024-04-06T03:50:47.188362+0000","last_became_peered":"2024-04-06T03:50:47.188362+0000","last_unstale":"2024-04-06T03:52:42.677086+0000","last_undegraded":"2024-04-06T03:52:42.677086+0000","last_fullsized":"2024-04-06T03:52:42.677086+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:21.009411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:52:21.009411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:42.768587+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183256+0000","last_change":"2024-04-06T03:51:53.594917+0000","last_active":"2024-04-06T03:52:42.183256+0000","last_peered":"2024-04-06T03:52:42.183256+0000","last_clean":"2024-04-06T03:52:42.183256+0000","last_became_active":"2024-04-06T03:48:21.326952+0000","last_became_peered":"2024-04-06T03:48:21.326952+0000","last_unstale":"2024-04-06T03:52:42.183256+0000","last_undegraded":"2024-04-06T03:52:42.183256+0000","last_fullsized":"2024-04-06T03:52:42.183256+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:53.594698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_clean_scrub_stamp":"2024-04-06T03:51:53.594698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:06.426677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":929,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.622674+0000","last_change":"2024-04-06T03:52:52.622750+0000","last_active":"2024-04-06T03:52:52.622674+0000","last_peered":"2024-04-06T03:52:52.622674+0000","last_clean":"2024-04-06T03:52:52.622674+0000","last_became_active":"2024-04-06T03:48:32.436036+0000","last_became_peered":"2024-04-06T03:48:32.436036+0000","last_unstale":"2024-04-06T03:52:52.622674+0000","last_undegraded":"2024-04-06T03:52:52.622674+0000","last_fullsized":"2024-04-06T03:52:52.622674+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:52:52.622511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:52.622511+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:53.418332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"433'58","reported_seq":641,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.677033+0000","last_change":"2024-04-06T03:52:31.983093+0000","last_active":"2024-04-06T03:52:42.677033+0000","last_peered":"2024-04-06T03:52:42.677033+0000","last_clean":"2024-04-06T03:52:42.677033+0000","last_became_active":"2024-04-06T03:50:47.196236+0000","last_became_peered":"2024-04-06T03:50:47.196236+0000","last_unstale":"2024-04-06T03:52:42.677033+0000","last_undegraded":"2024-04-06T03:52:42.677033+0000","last_fullsized":"2024-04-06T03:52:42.677033+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'58","last_scrub_stamp":"2024-04-06T03:52:31.982714+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:52:31.982714+0000","objects_scrubbed":7,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:48.140404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":70,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":132,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669803+0000","last_change":"2024-04-06T03:52:34.107753+0000","last_active":"2024-04-06T03:52:42.669803+0000","last_peered":"2024-04-06T03:52:42.669803+0000","last_clean":"2024-04-06T03:52:42.669803+0000","last_became_active":"2024-04-06T03:48:32.458724+0000","last_became_peered":"2024-04-06T03:48:32.458724+0000","last_unstale":"2024-04-06T03:52:42.669803+0000","last_undegraded":"2024-04-06T03:52:42.669803+0000","last_fullsized":"2024-04-06T03:52:42.669803+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:34.107597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:55.288085+0000","last_clean_scrub_stamp":"2024-04-06T03:52:34.107597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:40.005227+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":184,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670880+0000","last_change":"2024-04-06T03:52:05.923917+0000","last_active":"2024-04-06T03:52:42.670880+0000","last_peered":"2024-04-06T03:52:42.670880+0000","last_clean":"2024-04-06T03:52:42.670880+0000","last_became_active":"2024-04-06T03:50:47.201333+0000","last_became_peered":"2024-04-06T03:50:47.201333+0000","last_unstale":"2024-04-06T03:52:42.670880+0000","last_undegraded":"2024-04-06T03:52:42.670880+0000","last_fullsized":"2024-04-06T03:52:42.670880+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:05.923705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:47:24.939189+0000","last_clean_scrub_stamp":"2024-04-06T03:52:05.923705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:31.238147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":873,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183499+0000","last_change":"2024-04-06T03:52:30.650730+0000","last_active":"2024-04-06T03:52:42.183499+0000","last_peered":"2024-04-06T03:52:42.183499+0000","last_clean":"2024-04-06T03:52:42.183499+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:52:42.183499+0000","last_undegraded":"2024-04-06T03:52:42.183499+0000","last_fullsized":"2024-04-06T03:52:42.183499+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:30.650506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:52:30.650506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:36.321214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"433'42","reported_seq":593,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:46.168435+0000","last_change":"2024-04-06T03:52:46.168521+0000","last_active":"2024-04-06T03:52:46.168435+0000","last_peered":"2024-04-06T03:52:46.168435+0000","last_clean":"2024-04-06T03:52:46.168435+0000","last_became_active":"2024-04-06T03:50:47.210996+0000","last_became_peered":"2024-04-06T03:50:47.210996+0000","last_unstale":"2024-04-06T03:52:46.168435+0000","last_undegraded":"2024-04-06T03:52:46.168435+0000","last_fullsized":"2024-04-06T03:52:46.168435+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'42","last_scrub_stamp":"2024-04-06T03:52:46.168285+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:52:46.168285+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:47.603652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227934+0000","last_change":"2024-04-06T03:52:08.753656+0000","last_active":"2024-04-06T03:52:42.227934+0000","last_peered":"2024-04-06T03:52:42.227934+0000","last_clean":"2024-04-06T03:52:42.227934+0000","last_became_active":"2024-04-06T03:42:23.577258+0000","last_became_peered":"2024-04-06T03:42:23.577258+0000","last_unstale":"2024-04-06T03:52:42.227934+0000","last_undegraded":"2024-04-06T03:52:42.227934+0000","last_fullsized":"2024-04-06T03:52:42.227934+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:08.753549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:23.012827+0000","last_clean_scrub_stamp":"2024-04-06T03:52:08.753549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:25.835619+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671950+0000","last_change":"2024-04-06T03:52:29.268192+0000","last_active":"2024-04-06T03:52:42.671950+0000","last_peered":"2024-04-06T03:52:42.671950+0000","last_clean":"2024-04-06T03:52:42.671950+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:52:42.671950+0000","last_undegraded":"2024-04-06T03:52:42.671950+0000","last_fullsized":"2024-04-06T03:52:42.671950+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:29.267905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:39.155872+0000","last_clean_scrub_stamp":"2024-04-06T03:52:29.267905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:32.835359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":871,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182155+0000","last_change":"2024-04-06T03:51:41.334777+0000","last_active":"2024-04-06T03:52:42.182155+0000","last_peered":"2024-04-06T03:52:42.182155+0000","last_clean":"2024-04-06T03:52:42.182155+0000","last_became_active":"2024-04-06T03:42:23.588672+0000","last_became_peered":"2024-04-06T03:42:23.588672+0000","last_unstale":"2024-04-06T03:52:42.182155+0000","last_undegraded":"2024-04-06T03:52:42.182155+0000","last_fullsized":"2024-04-06T03:52:42.182155+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:41.334631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:41.334631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:59.164585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"433'45","reported_seq":1012,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671827+0000","last_change":"2024-04-06T03:52:19.411643+0000","last_active":"2024-04-06T03:52:42.671827+0000","last_peered":"2024-04-06T03:52:42.671827+0000","last_clean":"2024-04-06T03:52:42.671827+0000","last_became_active":"2024-04-06T03:48:21.316917+0000","last_became_peered":"2024-04-06T03:48:21.316917+0000","last_unstale":"2024-04-06T03:52:42.671827+0000","last_undegraded":"2024-04-06T03:52:42.671827+0000","last_fullsized":"2024-04-06T03:52:42.671827+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'45","last_scrub_stamp":"2024-04-06T03:52:19.411319+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:52:19.411319+0000","objects_scrubbed":6,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:22.874698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":62,"num_read_kb":54,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673625+0000","last_change":"2024-04-06T03:52:30.295661+0000","last_active":"2024-04-06T03:52:42.673625+0000","last_peered":"2024-04-06T03:52:42.673625+0000","last_clean":"2024-04-06T03:52:42.673625+0000","last_became_active":"2024-04-06T03:42:23.594112+0000","last_became_peered":"2024-04-06T03:42:23.594112+0000","last_unstale":"2024-04-06T03:52:42.673625+0000","last_undegraded":"2024-04-06T03:52:42.673625+0000","last_fullsized":"2024-04-06T03:52:42.673625+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:30.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:30.295451+0000","last_clean_scrub_stamp":"2024-04-06T03:52:30.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:58.323225+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670250+0000","last_change":"2024-04-06T03:52:06.534507+0000","last_active":"2024-04-06T03:52:42.670250+0000","last_peered":"2024-04-06T03:52:42.670250+0000","last_clean":"2024-04-06T03:52:42.670250+0000","last_became_active":"2024-04-06T03:48:32.469910+0000","last_became_peered":"2024-04-06T03:48:32.469910+0000","last_unstale":"2024-04-06T03:52:42.670250+0000","last_undegraded":"2024-04-06T03:52:42.670250+0000","last_fullsized":"2024-04-06T03:52:42.670250+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:06.534157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:52:06.534157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:24.217203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":497,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673797+0000","last_change":"2024-04-06T03:51:56.823283+0000","last_active":"2024-04-06T03:52:42.673797+0000","last_peered":"2024-04-06T03:52:42.673797+0000","last_clean":"2024-04-06T03:52:42.673797+0000","last_became_active":"2024-04-06T03:51:06.727485+0000","last_became_peered":"2024-04-06T03:51:06.727485+0000","last_unstale":"2024-04-06T03:52:42.673797+0000","last_undegraded":"2024-04-06T03:52:42.673797+0000","last_fullsized":"2024-04-06T03:52:42.673797+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:56.823070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:56.823070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:13.269898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"433'20","reported_seq":135,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671233+0000","last_change":"2024-04-06T03:51:46.161345+0000","last_active":"2024-04-06T03:52:42.671233+0000","last_peered":"2024-04-06T03:52:42.671233+0000","last_clean":"2024-04-06T03:52:42.671233+0000","last_became_active":"2024-04-06T03:48:32.448235+0000","last_became_peered":"2024-04-06T03:48:32.448235+0000","last_unstale":"2024-04-06T03:52:42.671233+0000","last_undegraded":"2024-04-06T03:52:42.671233+0000","last_fullsized":"2024-04-06T03:52:42.671233+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'20","last_scrub_stamp":"2024-04-06T03:51:46.161101+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:51:46.161101+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:02.178241+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":717,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669756+0000","last_change":"2024-04-06T03:52:14.571480+0000","last_active":"2024-04-06T03:52:42.669756+0000","last_peered":"2024-04-06T03:52:42.669756+0000","last_clean":"2024-04-06T03:52:42.669756+0000","last_became_active":"2024-04-06T03:49:27.620657+0000","last_became_peered":"2024-04-06T03:49:27.620657+0000","last_unstale":"2024-04-06T03:52:42.669756+0000","last_undegraded":"2024-04-06T03:52:42.669756+0000","last_fullsized":"2024-04-06T03:52:42.669756+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:14.571351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:52:14.571351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:35.447873+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":125,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671601+0000","last_change":"2024-04-06T03:52:12.201535+0000","last_active":"2024-04-06T03:52:42.671601+0000","last_peered":"2024-04-06T03:52:42.671601+0000","last_clean":"2024-04-06T03:52:42.671601+0000","last_became_active":"2024-04-06T03:48:32.463740+0000","last_became_peered":"2024-04-06T03:48:32.463740+0000","last_unstale":"2024-04-06T03:52:42.671601+0000","last_undegraded":"2024-04-06T03:52:42.671601+0000","last_fullsized":"2024-04-06T03:52:42.671601+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:12.201328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_clean_scrub_stamp":"2024-04-06T03:52:12.201328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:32.720165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"414'2","reported_seq":910,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181299+0000","last_change":"2024-04-06T03:52:11.507399+0000","last_active":"2024-04-06T03:52:42.181299+0000","last_peered":"2024-04-06T03:52:42.181299+0000","last_clean":"2024-04-06T03:52:42.181299+0000","last_became_active":"2024-04-06T03:48:32.467749+0000","last_became_peered":"2024-04-06T03:48:32.467749+0000","last_unstale":"2024-04-06T03:52:42.181299+0000","last_undegraded":"2024-04-06T03:52:42.181299+0000","last_fullsized":"2024-04-06T03:52:42.181299+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'2","last_scrub_stamp":"2024-04-06T03:52:11.507256+0000","last_deep_scrub":"414'2","last_deep_scrub_stamp":"2024-04-06T03:51:06.320142+0000","last_clean_scrub_stamp":"2024-04-06T03:52:11.507256+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:16.419280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"433'38","reported_seq":988,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180662+0000","last_change":"2024-04-06T03:51:28.329120+0000","last_active":"2024-04-06T03:52:42.180662+0000","last_peered":"2024-04-06T03:52:42.180662+0000","last_clean":"2024-04-06T03:52:42.180662+0000","last_became_active":"2024-04-06T03:48:32.466273+0000","last_became_peered":"2024-04-06T03:48:32.466273+0000","last_unstale":"2024-04-06T03:52:42.180662+0000","last_undegraded":"2024-04-06T03:52:42.180662+0000","last_fullsized":"2024-04-06T03:52:42.180662+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'38","last_scrub_stamp":"2024-04-06T03:51:28.328866+0000","last_deep_scrub":"109'20","last_deep_scrub_stamp":"2024-04-06T03:22:49.786480+0000","last_clean_scrub_stamp":"2024-04-06T03:51:28.328866+0000","objects_scrubbed":5,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:50.962037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":864,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.185491+0000","last_change":"2024-04-06T03:52:29.670766+0000","last_active":"2024-04-06T03:52:42.185491+0000","last_peered":"2024-04-06T03:52:42.185491+0000","last_clean":"2024-04-06T03:52:42.185491+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:52:42.185491+0000","last_undegraded":"2024-04-06T03:52:42.185491+0000","last_fullsized":"2024-04-06T03:52:42.185491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:52:29.670537+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:41:29.127643+0000","last_clean_scrub_stamp":"2024-04-06T03:52:29.670537+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:57.455009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"436'16","reported_seq":793,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697486+0000","last_change":"2024-04-06T03:51:49.838298+0000","last_active":"2024-04-06T03:52:47.697486+0000","last_peered":"2024-04-06T03:52:47.697486+0000","last_clean":"2024-04-06T03:52:47.697486+0000","last_became_active":"2024-04-06T03:51:06.732897+0000","last_became_peered":"2024-04-06T03:51:06.732897+0000","last_unstale":"2024-04-06T03:52:47.697486+0000","last_undegraded":"2024-04-06T03:52:47.697486+0000","last_fullsized":"2024-04-06T03:52:47.697486+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'16","last_scrub_stamp":"2024-04-06T03:51:49.838074+0000","last_deep_scrub":"327'15","last_deep_scrub_stamp":"2024-04-06T03:50:22.454509+0000","last_clean_scrub_stamp":"2024-04-06T03:51:49.838074+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:54.040195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":201,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670524+0000","last_change":"2024-04-06T03:51:52.902749+0000","last_active":"2024-04-06T03:52:42.670524+0000","last_peered":"2024-04-06T03:52:42.670524+0000","last_clean":"2024-04-06T03:52:42.670524+0000","last_became_active":"2024-04-06T03:50:47.206404+0000","last_became_peered":"2024-04-06T03:50:47.206404+0000","last_unstale":"2024-04-06T03:52:42.670524+0000","last_undegraded":"2024-04-06T03:52:42.670524+0000","last_fullsized":"2024-04-06T03:52:42.670524+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:52.902538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:15.156474+0000","last_clean_scrub_stamp":"2024-04-06T03:51:52.902538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:14.649304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"433'86","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669855+0000","last_change":"2024-04-06T03:52:38.418870+0000","last_active":"2024-04-06T03:52:42.669855+0000","last_peered":"2024-04-06T03:52:42.669855+0000","last_clean":"2024-04-06T03:52:42.669855+0000","last_became_active":"2024-04-06T03:48:21.323684+0000","last_became_peered":"2024-04-06T03:48:21.323684+0000","last_unstale":"2024-04-06T03:52:42.669855+0000","last_undegraded":"2024-04-06T03:52:42.669855+0000","last_fullsized":"2024-04-06T03:52:42.669855+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'86","last_scrub_stamp":"2024-04-06T03:52:38.418664+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:52:38.418664+0000","objects_scrubbed":11,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:40.129651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":115,"num_read_kb":102,"num_write":76,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":880,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227457+0000","last_change":"2024-04-06T03:52:22.571818+0000","last_active":"2024-04-06T03:52:42.227457+0000","last_peered":"2024-04-06T03:52:42.227457+0000","last_clean":"2024-04-06T03:52:42.227457+0000","last_became_active":"2024-04-06T03:48:32.466118+0000","last_became_peered":"2024-04-06T03:48:32.466118+0000","last_unstale":"2024-04-06T03:52:42.227457+0000","last_undegraded":"2024-04-06T03:52:42.227457+0000","last_fullsized":"2024-04-06T03:52:42.227457+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:22.571589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:52:22.571589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:51.789866+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183508+0000","last_change":"2024-04-06T03:52:23.509135+0000","last_active":"2024-04-06T03:52:42.183508+0000","last_peered":"2024-04-06T03:52:42.183508+0000","last_clean":"2024-04-06T03:52:42.183508+0000","last_became_active":"2024-04-06T03:42:23.577555+0000","last_became_peered":"2024-04-06T03:42:23.577555+0000","last_unstale":"2024-04-06T03:52:42.183508+0000","last_undegraded":"2024-04-06T03:52:42.183508+0000","last_fullsized":"2024-04-06T03:52:42.183508+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:23.508929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.508929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:51.057458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":523,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678119+0000","last_change":"2024-04-06T03:52:33.854635+0000","last_active":"2024-04-06T03:52:42.678119+0000","last_peered":"2024-04-06T03:52:42.678119+0000","last_clean":"2024-04-06T03:52:42.678119+0000","last_became_active":"2024-04-06T03:51:06.720828+0000","last_became_peered":"2024-04-06T03:51:06.720828+0000","last_unstale":"2024-04-06T03:52:42.678119+0000","last_undegraded":"2024-04-06T03:52:42.678119+0000","last_fullsized":"2024-04-06T03:52:42.678119+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:33.854432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:49.470404+0000","last_clean_scrub_stamp":"2024-04-06T03:52:33.854432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:38.651018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"433'45","reported_seq":662,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675697+0000","last_change":"2024-04-06T03:52:02.843565+0000","last_active":"2024-04-06T03:52:42.675697+0000","last_peered":"2024-04-06T03:52:42.675697+0000","last_clean":"2024-04-06T03:52:42.675697+0000","last_became_active":"2024-04-06T03:50:47.200759+0000","last_became_peered":"2024-04-06T03:50:47.200759+0000","last_unstale":"2024-04-06T03:52:42.675697+0000","last_undegraded":"2024-04-06T03:52:42.675697+0000","last_fullsized":"2024-04-06T03:52:42.675697+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'45","last_scrub_stamp":"2024-04-06T03:52:02.843317+0000","last_deep_scrub":"344'39","last_deep_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.843317+0000","objects_scrubbed":6,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:09.007353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":61,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":870,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183416+0000","last_change":"2024-04-06T03:52:34.699039+0000","last_active":"2024-04-06T03:52:42.183416+0000","last_peered":"2024-04-06T03:52:42.183416+0000","last_clean":"2024-04-06T03:52:42.183416+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:52:42.183416+0000","last_undegraded":"2024-04-06T03:52:42.183416+0000","last_fullsized":"2024-04-06T03:52:42.183416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:34.698755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:01.484422+0000","last_clean_scrub_stamp":"2024-04-06T03:52:34.698755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:44.344379+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":506,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673611+0000","last_change":"2024-04-06T03:52:36.863272+0000","last_active":"2024-04-06T03:52:42.673611+0000","last_peered":"2024-04-06T03:52:42.673611+0000","last_clean":"2024-04-06T03:52:42.673611+0000","last_became_active":"2024-04-06T03:51:06.732712+0000","last_became_peered":"2024-04-06T03:51:06.732712+0000","last_unstale":"2024-04-06T03:52:42.673611+0000","last_undegraded":"2024-04-06T03:52:42.673611+0000","last_fullsized":"2024-04-06T03:52:42.673611+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:36.863051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:36.863051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:39.902161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":526,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675976+0000","last_change":"2024-04-06T03:51:45.027088+0000","last_active":"2024-04-06T03:52:42.675976+0000","last_peered":"2024-04-06T03:52:42.675976+0000","last_clean":"2024-04-06T03:52:42.675976+0000","last_became_active":"2024-04-06T03:50:47.205196+0000","last_became_peered":"2024-04-06T03:50:47.205196+0000","last_unstale":"2024-04-06T03:52:42.675976+0000","last_undegraded":"2024-04-06T03:52:42.675976+0000","last_fullsized":"2024-04-06T03:52:42.675976+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:45.026839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:45.026839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:07.088793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"433'34","reported_seq":151,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:44.005717+0000","last_change":"2024-04-06T03:52:44.005774+0000","last_active":"2024-04-06T03:52:44.005717+0000","last_peered":"2024-04-06T03:52:44.005717+0000","last_clean":"2024-04-06T03:52:44.005717+0000","last_became_active":"2024-04-06T03:48:32.462539+0000","last_became_peered":"2024-04-06T03:48:32.462539+0000","last_unstale":"2024-04-06T03:52:44.005717+0000","last_undegraded":"2024-04-06T03:52:44.005717+0000","last_fullsized":"2024-04-06T03:52:44.005717+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'34","last_scrub_stamp":"2024-04-06T03:52:44.005638+0000","last_deep_scrub":"433'34","last_deep_scrub_stamp":"2024-04-06T03:51:25.163875+0000","last_clean_scrub_stamp":"2024-04-06T03:52:44.005638+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:09.604521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":868,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181304+0000","last_change":"2024-04-06T03:52:37.503026+0000","last_active":"2024-04-06T03:52:42.181304+0000","last_peered":"2024-04-06T03:52:42.181304+0000","last_clean":"2024-04-06T03:52:42.181304+0000","last_became_active":"2024-04-06T03:48:32.464518+0000","last_became_peered":"2024-04-06T03:48:32.464518+0000","last_unstale":"2024-04-06T03:52:42.181304+0000","last_undegraded":"2024-04-06T03:52:42.181304+0000","last_fullsized":"2024-04-06T03:52:42.181304+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:37.502797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:55.881161+0000","last_clean_scrub_stamp":"2024-04-06T03:52:37.502797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:46.923594+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":146,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.308530+0000","last_change":"2024-04-06T03:52:47.308643+0000","last_active":"2024-04-06T03:52:47.308530+0000","last_peered":"2024-04-06T03:52:47.308530+0000","last_clean":"2024-04-06T03:52:47.308530+0000","last_became_active":"2024-04-06T03:48:21.324233+0000","last_became_peered":"2024-04-06T03:48:21.324233+0000","last_unstale":"2024-04-06T03:52:47.308530+0000","last_undegraded":"2024-04-06T03:52:47.308530+0000","last_fullsized":"2024-04-06T03:52:47.308530+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:47.308312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:47.308312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:15.194624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:50.327666+0000","last_change":"2024-04-06T03:52:50.327722+0000","last_active":"2024-04-06T03:52:50.327666+0000","last_peered":"2024-04-06T03:52:50.327666+0000","last_clean":"2024-04-06T03:52:50.327666+0000","last_became_active":"2024-04-06T03:48:32.470368+0000","last_became_peered":"2024-04-06T03:48:32.470368+0000","last_unstale":"2024-04-06T03:52:50.327666+0000","last_undegraded":"2024-04-06T03:52:50.327666+0000","last_fullsized":"2024-04-06T03:52:50.327666+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:50.327516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:52:50.327516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:13.023613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"433'60","reported_seq":1016,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671608+0000","last_change":"2024-04-06T03:52:03.513260+0000","last_active":"2024-04-06T03:52:42.671608+0000","last_peered":"2024-04-06T03:52:42.671608+0000","last_clean":"2024-04-06T03:52:42.671608+0000","last_became_active":"2024-04-06T03:42:23.592079+0000","last_became_peered":"2024-04-06T03:42:23.592079+0000","last_unstale":"2024-04-06T03:52:42.671608+0000","last_undegraded":"2024-04-06T03:52:42.671608+0000","last_fullsized":"2024-04-06T03:52:42.671608+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'60","last_scrub_stamp":"2024-04-06T03:52:03.513006+0000","last_deep_scrub":"430'50","last_deep_scrub_stamp":"2024-04-06T03:50:46.333320+0000","last_clean_scrub_stamp":"2024-04-06T03:52:03.513006+0000","objects_scrubbed":7,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.010964+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":78,"num_read_kb":71,"num_write":52,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":124,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669437+0000","last_change":"2024-04-06T03:51:49.257334+0000","last_active":"2024-04-06T03:52:42.669437+0000","last_peered":"2024-04-06T03:52:42.669437+0000","last_clean":"2024-04-06T03:52:42.669437+0000","last_became_active":"2024-04-06T03:48:32.461335+0000","last_became_peered":"2024-04-06T03:48:32.461335+0000","last_unstale":"2024-04-06T03:52:42.669437+0000","last_undegraded":"2024-04-06T03:52:42.669437+0000","last_fullsized":"2024-04-06T03:52:42.669437+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:49.257168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_clean_scrub_stamp":"2024-04-06T03:51:49.257168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:10.434059+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":884,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182448+0000","last_change":"2024-04-06T03:52:20.437248+0000","last_active":"2024-04-06T03:52:42.182448+0000","last_peered":"2024-04-06T03:52:42.182448+0000","last_clean":"2024-04-06T03:52:42.182448+0000","last_became_active":"2024-04-06T03:48:32.469863+0000","last_became_peered":"2024-04-06T03:48:32.469863+0000","last_unstale":"2024-04-06T03:52:42.182448+0000","last_undegraded":"2024-04-06T03:52:42.182448+0000","last_fullsized":"2024-04-06T03:52:42.182448+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:20.436937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:33.624959+0000","last_clean_scrub_stamp":"2024-04-06T03:52:20.436937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:44.571761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":497,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.674839+0000","last_change":"2024-04-06T03:52:40.861328+0000","last_active":"2024-04-06T03:52:42.674839+0000","last_peered":"2024-04-06T03:52:42.674839+0000","last_clean":"2024-04-06T03:52:42.674839+0000","last_became_active":"2024-04-06T03:51:07.366122+0000","last_became_peered":"2024-04-06T03:51:07.366122+0000","last_unstale":"2024-04-06T03:52:42.674839+0000","last_undegraded":"2024-04-06T03:52:42.674839+0000","last_fullsized":"2024-04-06T03:52:42.674839+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:40.861105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:30.734198+0000","last_clean_scrub_stamp":"2024-04-06T03:52:40.861105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:41.086066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"433'38","reported_seq":930,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184634+0000","last_change":"2024-04-06T03:52:28.638446+0000","last_active":"2024-04-06T03:52:42.184634+0000","last_peered":"2024-04-06T03:52:42.184634+0000","last_clean":"2024-04-06T03:52:42.184634+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:52:42.184634+0000","last_undegraded":"2024-04-06T03:52:42.184634+0000","last_fullsized":"2024-04-06T03:52:42.184634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'38","last_scrub_stamp":"2024-04-06T03:52:28.638198+0000","last_deep_scrub":"432'32","last_deep_scrub_stamp":"2024-04-06T03:51:00.500705+0000","last_clean_scrub_stamp":"2024-04-06T03:52:28.638198+0000","objects_scrubbed":5,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:49.035812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":872,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673516+0000","last_change":"2024-04-06T03:52:33.312479+0000","last_active":"2024-04-06T03:52:42.673516+0000","last_peered":"2024-04-06T03:52:42.673516+0000","last_clean":"2024-04-06T03:52:42.673516+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:52:42.673516+0000","last_undegraded":"2024-04-06T03:52:42.673516+0000","last_fullsized":"2024-04-06T03:52:42.673516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:33.312337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:33.312337+0000","last_clean_scrub_stamp":"2024-04-06T03:52:33.312337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:47.571104+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":1256,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697573+0000","last_change":"2024-04-06T03:51:51.419871+0000","last_active":"2024-04-06T03:52:47.697573+0000","last_peered":"2024-04-06T03:52:47.697573+0000","last_clean":"2024-04-06T03:52:47.697573+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:52:47.697573+0000","last_undegraded":"2024-04-06T03:52:47.697573+0000","last_fullsized":"2024-04-06T03:52:47.697573+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:51:51.419749+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:50:20.079528+0000","last_clean_scrub_stamp":"2024-04-06T03:51:51.419749+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:09.576512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":124,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671333+0000","last_change":"2024-04-06T03:52:00.217593+0000","last_active":"2024-04-06T03:52:42.671333+0000","last_peered":"2024-04-06T03:52:42.671333+0000","last_clean":"2024-04-06T03:52:42.671333+0000","last_became_active":"2024-04-06T03:48:32.454942+0000","last_became_peered":"2024-04-06T03:48:32.454942+0000","last_unstale":"2024-04-06T03:52:42.671333+0000","last_undegraded":"2024-04-06T03:52:42.671333+0000","last_fullsized":"2024-04-06T03:52:42.671333+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:00.217476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:00.217476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.375475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"433'31","reported_seq":143,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669436+0000","last_change":"2024-04-06T03:52:14.131244+0000","last_active":"2024-04-06T03:52:42.669436+0000","last_peered":"2024-04-06T03:52:42.669436+0000","last_clean":"2024-04-06T03:52:42.669436+0000","last_became_active":"2024-04-06T03:48:32.452795+0000","last_became_peered":"2024-04-06T03:48:32.452795+0000","last_unstale":"2024-04-06T03:52:42.669436+0000","last_undegraded":"2024-04-06T03:52:42.669436+0000","last_fullsized":"2024-04-06T03:52:42.669436+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'31","last_scrub_stamp":"2024-04-06T03:52:14.130995+0000","last_deep_scrub":"433'31","last_deep_scrub_stamp":"2024-04-06T03:52:14.130995+0000","last_clean_scrub_stamp":"2024-04-06T03:52:14.130995+0000","objects_scrubbed":4,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:20.533112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":38,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":124,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671516+0000","last_change":"2024-04-06T03:52:01.296078+0000","last_active":"2024-04-06T03:52:42.671516+0000","last_peered":"2024-04-06T03:52:42.671516+0000","last_clean":"2024-04-06T03:52:42.671516+0000","last_became_active":"2024-04-06T03:48:32.452785+0000","last_became_peered":"2024-04-06T03:48:32.452785+0000","last_unstale":"2024-04-06T03:52:42.671516+0000","last_undegraded":"2024-04-06T03:52:42.671516+0000","last_fullsized":"2024-04-06T03:52:42.671516+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:01.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:52:01.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:05.902219+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"431'20","reported_seq":664,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:50.141077+0000","last_change":"2024-04-06T03:52:50.141077+0000","last_active":"2024-04-06T03:52:50.141077+0000","last_peered":"2024-04-06T03:52:50.141077+0000","last_clean":"2024-04-06T03:52:50.141077+0000","last_became_active":"2024-04-06T03:50:47.201652+0000","last_became_peered":"2024-04-06T03:50:47.201652+0000","last_unstale":"2024-04-06T03:52:50.141077+0000","last_undegraded":"2024-04-06T03:52:50.141077+0000","last_fullsized":"2024-04-06T03:52:50.141077+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'20","last_scrub_stamp":"2024-04-06T03:52:50.140987+0000","last_deep_scrub":"431'20","last_deep_scrub_stamp":"2024-04-06T03:51:44.005147+0000","last_clean_scrub_stamp":"2024-04-06T03:52:50.140987+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:14.352697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":203,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675185+0000","last_change":"2024-04-06T03:52:11.954060+0000","last_active":"2024-04-06T03:52:42.675185+0000","last_peered":"2024-04-06T03:52:42.675185+0000","last_clean":"2024-04-06T03:52:42.675185+0000","last_became_active":"2024-04-06T03:50:47.195653+0000","last_became_peered":"2024-04-06T03:50:47.195653+0000","last_unstale":"2024-04-06T03:52:42.675185+0000","last_undegraded":"2024-04-06T03:52:42.675185+0000","last_fullsized":"2024-04-06T03:52:42.675185+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:11.953851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:21.406614+0000","last_clean_scrub_stamp":"2024-04-06T03:52:11.953851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:12.115475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"433'40","reported_seq":930,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:48.666469+0000","last_change":"2024-04-06T03:52:48.666549+0000","last_active":"2024-04-06T03:52:48.666469+0000","last_peered":"2024-04-06T03:52:48.666469+0000","last_clean":"2024-04-06T03:52:48.666469+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:52:48.666469+0000","last_undegraded":"2024-04-06T03:52:48.666469+0000","last_fullsized":"2024-04-06T03:52:48.666469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'40","last_scrub_stamp":"2024-04-06T03:52:48.666309+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:52:48.666309+0000","objects_scrubbed":5,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:11.029047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":49,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":874,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183817+0000","last_change":"2024-04-06T03:51:16.588641+0000","last_active":"2024-04-06T03:52:42.183817+0000","last_peered":"2024-04-06T03:52:42.183817+0000","last_clean":"2024-04-06T03:52:42.183817+0000","last_became_active":"2024-04-06T03:42:23.574848+0000","last_became_peered":"2024-04-06T03:42:23.574848+0000","last_unstale":"2024-04-06T03:52:42.183817+0000","last_undegraded":"2024-04-06T03:52:42.183817+0000","last_fullsized":"2024-04-06T03:52:42.183817+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:16.588378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:16.588378+0000","last_clean_scrub_stamp":"2024-04-06T03:51:16.588378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":664,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675848+0000","last_change":"2024-04-06T03:52:19.911267+0000","last_active":"2024-04-06T03:52:42.675848+0000","last_peered":"2024-04-06T03:52:42.675848+0000","last_clean":"2024-04-06T03:52:42.675848+0000","last_became_active":"2024-04-06T03:50:47.193241+0000","last_became_peered":"2024-04-06T03:50:47.193241+0000","last_unstale":"2024-04-06T03:52:42.675848+0000","last_undegraded":"2024-04-06T03:52:42.675848+0000","last_fullsized":"2024-04-06T03:52:42.675848+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:19.911031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:19.911031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:31.221461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":660,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673889+0000","last_change":"2024-04-06T03:51:41.758468+0000","last_active":"2024-04-06T03:52:42.673889+0000","last_peered":"2024-04-06T03:52:42.673889+0000","last_clean":"2024-04-06T03:52:42.673889+0000","last_became_active":"2024-04-06T03:51:06.725760+0000","last_became_peered":"2024-04-06T03:51:06.725760+0000","last_unstale":"2024-04-06T03:52:42.673889+0000","last_undegraded":"2024-04-06T03:52:42.673889+0000","last_fullsized":"2024-04-06T03:52:42.673889+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:41.758246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:41.758246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:05.842684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"433'18","reported_seq":148,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:49.865912+0000","last_change":"2024-04-06T03:52:49.865982+0000","last_active":"2024-04-06T03:52:49.865912+0000","last_peered":"2024-04-06T03:52:49.865912+0000","last_clean":"2024-04-06T03:52:49.865912+0000","last_became_active":"2024-04-06T03:51:06.720346+0000","last_became_peered":"2024-04-06T03:51:06.720346+0000","last_unstale":"2024-04-06T03:52:49.865912+0000","last_undegraded":"2024-04-06T03:52:49.865912+0000","last_fullsized":"2024-04-06T03:52:49.865912+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'18","last_scrub_stamp":"2024-04-06T03:52:49.865747+0000","last_deep_scrub":"433'18","last_deep_scrub_stamp":"2024-04-06T03:52:49.865747+0000","last_clean_scrub_stamp":"2024-04-06T03:52:49.865747+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:13.620635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":898,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180521+0000","last_change":"2024-04-06T03:52:09.573051+0000","last_active":"2024-04-06T03:52:42.180521+0000","last_peered":"2024-04-06T03:52:42.180521+0000","last_clean":"2024-04-06T03:52:42.180521+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:52:42.180521+0000","last_undegraded":"2024-04-06T03:52:42.180521+0000","last_fullsized":"2024-04-06T03:52:42.180521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:52:09.572891+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:52:09.572891+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:39.466312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":886,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.228205+0000","last_change":"2024-04-06T03:51:55.401163+0000","last_active":"2024-04-06T03:52:42.228205+0000","last_peered":"2024-04-06T03:52:42.228205+0000","last_clean":"2024-04-06T03:52:42.228205+0000","last_became_active":"2024-04-06T03:48:21.332734+0000","last_became_peered":"2024-04-06T03:48:21.332734+0000","last_unstale":"2024-04-06T03:52:42.228205+0000","last_undegraded":"2024-04-06T03:52:42.228205+0000","last_fullsized":"2024-04-06T03:52:42.228205+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:55.400946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:51:55.400946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:15.090910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":932,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179145+0000","last_change":"2024-04-06T03:51:56.558185+0000","last_active":"2024-04-06T03:52:42.179145+0000","last_peered":"2024-04-06T03:52:42.179145+0000","last_clean":"2024-04-06T03:52:42.179145+0000","last_became_active":"2024-04-06T03:48:32.438669+0000","last_became_peered":"2024-04-06T03:48:32.438669+0000","last_unstale":"2024-04-06T03:52:42.179145+0000","last_undegraded":"2024-04-06T03:52:42.179145+0000","last_fullsized":"2024-04-06T03:52:42.179145+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:56.557974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:56.557974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:06.175275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"433'47","reported_seq":991,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.666259+0000","last_change":"2024-04-06T03:52:47.666415+0000","last_active":"2024-04-06T03:52:47.666259+0000","last_peered":"2024-04-06T03:52:47.666259+0000","last_clean":"2024-04-06T03:52:47.666259+0000","last_became_active":"2024-04-06T03:42:23.574886+0000","last_became_peered":"2024-04-06T03:42:23.574886+0000","last_unstale":"2024-04-06T03:52:47.666259+0000","last_undegraded":"2024-04-06T03:52:47.666259+0000","last_fullsized":"2024-04-06T03:52:47.666259+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'47","last_scrub_stamp":"2024-04-06T03:52:47.666017+0000","last_deep_scrub":"433'47","last_deep_scrub_stamp":"2024-04-06T03:52:47.666017+0000","last_clean_scrub_stamp":"2024-04-06T03:52:47.666017+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:11.857289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":65,"num_read_kb":57,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":870,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227036+0000","last_change":"2024-04-06T03:52:00.708590+0000","last_active":"2024-04-06T03:52:42.227036+0000","last_peered":"2024-04-06T03:52:42.227036+0000","last_clean":"2024-04-06T03:52:42.227036+0000","last_became_active":"2024-04-06T03:48:21.333229+0000","last_became_peered":"2024-04-06T03:48:21.333229+0000","last_unstale":"2024-04-06T03:52:42.227036+0000","last_undegraded":"2024-04-06T03:52:42.227036+0000","last_fullsized":"2024-04-06T03:52:42.227036+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:00.708379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:52:00.708379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:16.477498+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":927,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:46.114039+0000","last_change":"2024-04-06T03:52:46.114114+0000","last_active":"2024-04-06T03:52:46.114039+0000","last_peered":"2024-04-06T03:52:46.114039+0000","last_clean":"2024-04-06T03:52:46.114039+0000","last_became_active":"2024-04-06T03:48:32.447783+0000","last_became_peered":"2024-04-06T03:48:32.447783+0000","last_unstale":"2024-04-06T03:52:46.114039+0000","last_undegraded":"2024-04-06T03:52:46.114039+0000","last_fullsized":"2024-04-06T03:52:46.114039+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:52:46.113874+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:52:46.113874+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:05.267012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"433'47","reported_seq":1012,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183977+0000","last_change":"2024-04-06T03:51:50.469297+0000","last_active":"2024-04-06T03:52:42.183977+0000","last_peered":"2024-04-06T03:52:42.183977+0000","last_clean":"2024-04-06T03:52:42.183977+0000","last_became_active":"2024-04-06T03:48:21.325889+0000","last_became_peered":"2024-04-06T03:48:21.325889+0000","last_unstale":"2024-04-06T03:52:42.183977+0000","last_undegraded":"2024-04-06T03:52:42.183977+0000","last_fullsized":"2024-04-06T03:52:42.183977+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'47","last_scrub_stamp":"2024-04-06T03:51:50.468976+0000","last_deep_scrub":"307'39","last_deep_scrub_stamp":"2024-04-06T03:50:36.490189+0000","last_clean_scrub_stamp":"2024-04-06T03:51:50.468976+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:56.503346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":65,"num_read_kb":57,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":917,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179294+0000","last_change":"2024-04-06T03:51:34.596574+0000","last_active":"2024-04-06T03:52:42.179294+0000","last_peered":"2024-04-06T03:52:42.179294+0000","last_clean":"2024-04-06T03:52:42.179294+0000","last_became_active":"2024-04-06T03:48:32.438050+0000","last_became_peered":"2024-04-06T03:48:32.438050+0000","last_unstale":"2024-04-06T03:52:42.179294+0000","last_undegraded":"2024-04-06T03:52:42.179294+0000","last_fullsized":"2024-04-06T03:52:42.179294+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:34.596324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:51:34.596324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:02.246202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":133,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.065401+0000","last_change":"2024-04-06T03:52:47.065441+0000","last_active":"2024-04-06T03:52:47.065401+0000","last_peered":"2024-04-06T03:52:47.065401+0000","last_clean":"2024-04-06T03:52:47.065401+0000","last_became_active":"2024-04-06T03:48:32.463557+0000","last_became_peered":"2024-04-06T03:48:32.463557+0000","last_unstale":"2024-04-06T03:52:47.065401+0000","last_undegraded":"2024-04-06T03:52:47.065401+0000","last_fullsized":"2024-04-06T03:52:47.065401+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:47.065296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:31.197990+0000","last_clean_scrub_stamp":"2024-04-06T03:52:47.065296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:47.270460+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":248,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670037+0000","last_change":"2024-04-06T03:52:02.563263+0000","last_active":"2024-04-06T03:52:42.670037+0000","last_peered":"2024-04-06T03:52:42.670037+0000","last_clean":"2024-04-06T03:52:42.670037+0000","last_became_active":"2024-04-06T03:49:27.609263+0000","last_became_peered":"2024-04-06T03:49:27.609263+0000","last_unstale":"2024-04-06T03:52:42.670037+0000","last_undegraded":"2024-04-06T03:52:42.670037+0000","last_fullsized":"2024-04-06T03:52:42.670037+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:52:02.563108+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.563108+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:31.829646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"433'62","reported_seq":782,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697562+0000","last_change":"2024-04-06T03:52:12.915030+0000","last_active":"2024-04-06T03:52:47.697562+0000","last_peered":"2024-04-06T03:52:47.697562+0000","last_clean":"2024-04-06T03:52:47.697562+0000","last_became_active":"2024-04-06T03:50:47.192671+0000","last_became_peered":"2024-04-06T03:50:47.192671+0000","last_unstale":"2024-04-06T03:52:47.697562+0000","last_undegraded":"2024-04-06T03:52:47.697562+0000","last_fullsized":"2024-04-06T03:52:47.697562+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'62","last_scrub_stamp":"2024-04-06T03:52:12.914736+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:52:12.914736+0000","objects_scrubbed":8,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:39.627666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":77,"num_read_kb":67,"num_write":49,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":151,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670128+0000","last_change":"2024-04-06T03:52:28.407248+0000","last_active":"2024-04-06T03:52:42.670128+0000","last_peered":"2024-04-06T03:52:42.670128+0000","last_clean":"2024-04-06T03:52:42.670128+0000","last_became_active":"2024-04-06T03:48:21.312637+0000","last_became_peered":"2024-04-06T03:48:21.312637+0000","last_unstale":"2024-04-06T03:52:42.670128+0000","last_undegraded":"2024-04-06T03:52:42.670128+0000","last_fullsized":"2024-04-06T03:52:42.670128+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:52:28.407143+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:52:28.407143+0000","last_clean_scrub_stamp":"2024-04-06T03:52:28.407143+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:54.361111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"404'15","reported_seq":182,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697899+0000","last_change":"2024-04-06T03:52:24.006456+0000","last_active":"2024-04-06T03:52:47.697899+0000","last_peered":"2024-04-06T03:52:47.697899+0000","last_clean":"2024-04-06T03:52:47.697899+0000","last_became_active":"2024-04-06T03:48:32.462990+0000","last_became_peered":"2024-04-06T03:48:32.462990+0000","last_unstale":"2024-04-06T03:52:47.697899+0000","last_undegraded":"2024-04-06T03:52:47.697899+0000","last_fullsized":"2024-04-06T03:52:47.697899+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15","last_scrub_stamp":"2024-04-06T03:52:24.006233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:24.006233+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:52.216685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":857,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.229169+0000","last_change":"2024-04-06T03:51:39.387969+0000","last_active":"2024-04-06T03:52:42.229169+0000","last_peered":"2024-04-06T03:52:42.229169+0000","last_clean":"2024-04-06T03:52:42.229169+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:52:42.229169+0000","last_undegraded":"2024-04-06T03:52:42.229169+0000","last_fullsized":"2024-04-06T03:52:42.229169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:39.387751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:51:39.387751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:51.763588+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":530,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675562+0000","last_change":"2024-04-06T03:51:26.020953+0000","last_active":"2024-04-06T03:52:42.675562+0000","last_peered":"2024-04-06T03:52:42.675562+0000","last_clean":"2024-04-06T03:52:42.675562+0000","last_became_active":"2024-04-06T03:50:47.205131+0000","last_became_peered":"2024-04-06T03:50:47.205131+0000","last_unstale":"2024-04-06T03:52:42.675562+0000","last_undegraded":"2024-04-06T03:52:42.675562+0000","last_fullsized":"2024-04-06T03:52:42.675562+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:26.020724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:51.337979+0000","last_clean_scrub_stamp":"2024-04-06T03:51:26.020724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"433'64","reported_seq":1106,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671920+0000","last_change":"2024-04-06T03:52:23.414492+0000","last_active":"2024-04-06T03:52:42.671920+0000","last_peered":"2024-04-06T03:52:42.671920+0000","last_clean":"2024-04-06T03:52:42.671920+0000","last_became_active":"2024-04-06T03:48:32.447729+0000","last_became_peered":"2024-04-06T03:48:32.447729+0000","last_unstale":"2024-04-06T03:52:42.671920+0000","last_undegraded":"2024-04-06T03:52:42.671920+0000","last_fullsized":"2024-04-06T03:52:42.671920+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'64","last_scrub_stamp":"2024-04-06T03:52:23.414258+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.414258+0000","objects_scrubbed":9,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:33.443894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":93,"num_read_kb":78,"num_write":58,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":125,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671582+0000","last_change":"2024-04-06T03:52:03.146201+0000","last_active":"2024-04-06T03:52:42.671582+0000","last_peered":"2024-04-06T03:52:42.671582+0000","last_clean":"2024-04-06T03:52:42.671582+0000","last_became_active":"2024-04-06T03:48:32.458833+0000","last_became_peered":"2024-04-06T03:48:32.458833+0000","last_unstale":"2024-04-06T03:52:42.671582+0000","last_undegraded":"2024-04-06T03:52:42.671582+0000","last_fullsized":"2024-04-06T03:52:42.671582+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:03.146086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:14.960678+0000","last_clean_scrub_stamp":"2024-04-06T03:52:03.146086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.037034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":876,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183892+0000","last_change":"2024-04-06T03:51:20.533177+0000","last_active":"2024-04-06T03:52:42.183892+0000","last_peered":"2024-04-06T03:52:42.183892+0000","last_clean":"2024-04-06T03:52:42.183892+0000","last_became_active":"2024-04-06T03:42:23.569974+0000","last_became_peered":"2024-04-06T03:42:23.569974+0000","last_unstale":"2024-04-06T03:52:42.183892+0000","last_undegraded":"2024-04-06T03:52:42.183892+0000","last_fullsized":"2024-04-06T03:52:42.183892+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:20.533028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:51:20.533028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"433'66","reported_seq":931,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669672+0000","last_change":"2024-04-06T03:51:37.413260+0000","last_active":"2024-04-06T03:52:42.669672+0000","last_peered":"2024-04-06T03:52:42.669672+0000","last_clean":"2024-04-06T03:52:42.669672+0000","last_became_active":"2024-04-06T03:49:27.612734+0000","last_became_peered":"2024-04-06T03:49:27.612734+0000","last_unstale":"2024-04-06T03:52:42.669672+0000","last_undegraded":"2024-04-06T03:52:42.669672+0000","last_fullsized":"2024-04-06T03:52:42.669672+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'66","last_scrub_stamp":"2024-04-06T03:51:37.413007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:51:37.413007+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:01.539492+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":105,"num_read_kb":91,"num_write":68,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":928,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180876+0000","last_change":"2024-04-06T03:52:40.890687+0000","last_active":"2024-04-06T03:52:42.180876+0000","last_peered":"2024-04-06T03:52:42.180876+0000","last_clean":"2024-04-06T03:52:42.180876+0000","last_became_active":"2024-04-06T03:48:32.438362+0000","last_became_peered":"2024-04-06T03:48:32.438362+0000","last_unstale":"2024-04-06T03:52:42.180876+0000","last_undegraded":"2024-04-06T03:52:42.180876+0000","last_fullsized":"2024-04-06T03:52:42.180876+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:40.890470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:52:40.890470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:06.120298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":909,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180207+0000","last_change":"2024-04-06T03:52:04.562655+0000","last_active":"2024-04-06T03:52:42.180207+0000","last_peered":"2024-04-06T03:52:42.180207+0000","last_clean":"2024-04-06T03:52:42.180207+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:52:42.180207+0000","last_undegraded":"2024-04-06T03:52:42.180207+0000","last_fullsized":"2024-04-06T03:52:42.180207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:04.562403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:04.562403+0000","last_clean_scrub_stamp":"2024-04-06T03:52:04.562403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.076317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678161+0000","last_change":"2024-04-06T03:51:41.225716+0000","last_active":"2024-04-06T03:52:42.678161+0000","last_peered":"2024-04-06T03:52:42.678161+0000","last_clean":"2024-04-06T03:52:42.678161+0000","last_became_active":"2024-04-06T03:50:47.199384+0000","last_became_peered":"2024-04-06T03:50:47.199384+0000","last_unstale":"2024-04-06T03:52:42.678161+0000","last_undegraded":"2024-04-06T03:52:42.678161+0000","last_fullsized":"2024-04-06T03:52:42.678161+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:41.225572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:41.225572+0000","last_clean_scrub_stamp":"2024-04-06T03:51:41.225572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:03.844311+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"433'52","reported_seq":992,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181611+0000","last_change":"2024-04-06T03:52:26.395076+0000","last_active":"2024-04-06T03:52:42.181611+0000","last_peered":"2024-04-06T03:52:42.181611+0000","last_clean":"2024-04-06T03:52:42.181611+0000","last_became_active":"2024-04-06T03:42:23.588643+0000","last_became_peered":"2024-04-06T03:42:23.588643+0000","last_unstale":"2024-04-06T03:52:42.181611+0000","last_undegraded":"2024-04-06T03:52:42.181611+0000","last_fullsized":"2024-04-06T03:52:42.181611+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'52","last_scrub_stamp":"2024-04-06T03:52:26.394825+0000","last_deep_scrub":"307'42","last_deep_scrub_stamp":"2024-04-06T03:41:26.470096+0000","last_clean_scrub_stamp":"2024-04-06T03:52:26.394825+0000","objects_scrubbed":6,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:43.769865+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":63,"num_write":46,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":210,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:51.180929+0000","last_change":"2024-04-06T03:52:51.180929+0000","last_active":"2024-04-06T03:52:51.180929+0000","last_peered":"2024-04-06T03:52:51.180929+0000","last_clean":"2024-04-06T03:52:51.180929+0000","last_became_active":"2024-04-06T03:50:47.200613+0000","last_became_peered":"2024-04-06T03:50:47.200613+0000","last_unstale":"2024-04-06T03:52:51.180929+0000","last_undegraded":"2024-04-06T03:52:51.180929+0000","last_fullsized":"2024-04-06T03:52:51.180929+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:51.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:51.180613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:02.014870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":1211,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182828+0000","last_change":"2024-04-06T03:52:17.582249+0000","last_active":"2024-04-06T03:52:42.182828+0000","last_peered":"2024-04-06T03:52:42.182828+0000","last_clean":"2024-04-06T03:52:42.182828+0000","last_became_active":"2024-04-06T03:48:35.061820+0000","last_became_peered":"2024-04-06T03:48:35.061820+0000","last_unstale":"2024-04-06T03:52:42.182828+0000","last_undegraded":"2024-04-06T03:52:42.182828+0000","last_fullsized":"2024-04-06T03:52:42.182828+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:52:17.581994+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:52:17.581994+0000","last_clean_scrub_stamp":"2024-04-06T03:52:17.581994+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.126174+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4722944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":868,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184887+0000","last_change":"2024-04-06T03:51:42.545477+0000","last_active":"2024-04-06T03:52:42.184887+0000","last_peered":"2024-04-06T03:52:42.184887+0000","last_clean":"2024-04-06T03:52:42.184887+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:52:42.184887+0000","last_undegraded":"2024-04-06T03:52:42.184887+0000","last_fullsized":"2024-04-06T03:52:42.184887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:42.545401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:51:42.545401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:45.823051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670336+0000","last_change":"2024-04-06T03:52:18.473669+0000","last_active":"2024-04-06T03:52:42.670336+0000","last_peered":"2024-04-06T03:52:42.670336+0000","last_clean":"2024-04-06T03:52:42.670336+0000","last_became_active":"2024-04-06T03:48:21.322411+0000","last_became_peered":"2024-04-06T03:48:21.322411+0000","last_unstale":"2024-04-06T03:52:42.670336+0000","last_undegraded":"2024-04-06T03:52:42.670336+0000","last_fullsized":"2024-04-06T03:52:42.670336+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:18.473388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:36.552917+0000","last_clean_scrub_stamp":"2024-04-06T03:52:18.473388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:37.057796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"441'71","reported_seq":172,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670036+0000","last_change":"2024-04-06T03:51:47.133865+0000","last_active":"2024-04-06T03:52:42.670036+0000","last_peered":"2024-04-06T03:52:42.670036+0000","last_clean":"2024-04-06T03:52:42.670036+0000","last_became_active":"2024-04-06T03:48:32.452730+0000","last_became_peered":"2024-04-06T03:48:32.452730+0000","last_unstale":"2024-04-06T03:52:42.670036+0000","last_undegraded":"2024-04-06T03:52:42.670036+0000","last_fullsized":"2024-04-06T03:52:42.670036+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'71","last_scrub_stamp":"2024-04-06T03:51:47.133583+0000","last_deep_scrub":"441'71","last_deep_scrub_stamp":"2024-04-06T03:51:47.133583+0000","last_clean_scrub_stamp":"2024-04-06T03:51:47.133583+0000","objects_scrubbed":8,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:49.888993+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":92,"num_read_kb":84,"num_write":62,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":900,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182595+0000","last_change":"2024-04-06T03:52:05.612748+0000","last_active":"2024-04-06T03:52:42.182595+0000","last_peered":"2024-04-06T03:52:42.182595+0000","last_clean":"2024-04-06T03:52:42.182595+0000","last_became_active":"2024-04-06T03:48:32.436634+0000","last_became_peered":"2024-04-06T03:48:32.436634+0000","last_unstale":"2024-04-06T03:52:42.182595+0000","last_undegraded":"2024-04-06T03:52:42.182595+0000","last_fullsized":"2024-04-06T03:52:42.182595+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:05.612536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:05.612536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.003869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.677907+0000","last_change":"2024-04-06T03:52:40.170322+0000","last_active":"2024-04-06T03:52:42.677907+0000","last_peered":"2024-04-06T03:52:42.677907+0000","last_clean":"2024-04-06T03:52:42.677907+0000","last_became_active":"2024-04-06T03:50:47.198358+0000","last_became_peered":"2024-04-06T03:50:47.198358+0000","last_unstale":"2024-04-06T03:52:42.677907+0000","last_undegraded":"2024-04-06T03:52:42.677907+0000","last_fullsized":"2024-04-06T03:52:42.677907+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:40.170070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_clean_scrub_stamp":"2024-04-06T03:52:40.170070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:05.441402+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":139,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669433+0000","last_change":"2024-04-06T03:51:46.527140+0000","last_active":"2024-04-06T03:52:42.669433+0000","last_peered":"2024-04-06T03:52:42.669433+0000","last_clean":"2024-04-06T03:52:42.669433+0000","last_became_active":"2024-04-06T03:48:21.315429+0000","last_became_peered":"2024-04-06T03:48:21.315429+0000","last_unstale":"2024-04-06T03:52:42.669433+0000","last_undegraded":"2024-04-06T03:52:42.669433+0000","last_fullsized":"2024-04-06T03:52:42.669433+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:46.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:46.526935+0000","last_clean_scrub_stamp":"2024-04-06T03:51:46.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:53.642546+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":894,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184123+0000","last_change":"2024-04-06T03:52:10.679166+0000","last_active":"2024-04-06T03:52:42.184123+0000","last_peered":"2024-04-06T03:52:42.184123+0000","last_clean":"2024-04-06T03:52:42.184123+0000","last_became_active":"2024-04-06T03:48:32.437232+0000","last_became_peered":"2024-04-06T03:48:32.437232+0000","last_unstale":"2024-04-06T03:52:42.184123+0000","last_undegraded":"2024-04-06T03:52:42.184123+0000","last_fullsized":"2024-04-06T03:52:42.184123+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:10.678935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:52:10.678935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.226580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"433'90","reported_seq":1366,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.698294+0000","last_change":"2024-04-06T03:52:27.385824+0000","last_active":"2024-04-06T03:52:47.698294+0000","last_peered":"2024-04-06T03:52:47.698294+0000","last_clean":"2024-04-06T03:52:47.698294+0000","last_became_active":"2024-04-06T03:49:27.611624+0000","last_became_peered":"2024-04-06T03:49:27.611624+0000","last_unstale":"2024-04-06T03:52:47.698294+0000","last_undegraded":"2024-04-06T03:52:47.698294+0000","last_fullsized":"2024-04-06T03:52:47.698294+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'90","last_scrub_stamp":"2024-04-06T03:52:27.385680+0000","last_deep_scrub":"414'78","last_deep_scrub_stamp":"2024-04-06T03:49:47.447699+0000","last_clean_scrub_stamp":"2024-04-06T03:52:27.385680+0000","objects_scrubbed":11,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:42.788391+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":114,"num_read_kb":98,"num_write":68,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":507,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.674013+0000","last_change":"2024-04-06T03:52:03.827033+0000","last_active":"2024-04-06T03:52:42.674013+0000","last_peered":"2024-04-06T03:52:42.674013+0000","last_clean":"2024-04-06T03:52:42.674013+0000","last_became_active":"2024-04-06T03:51:06.720527+0000","last_became_peered":"2024-04-06T03:51:06.720527+0000","last_unstale":"2024-04-06T03:52:42.674013+0000","last_undegraded":"2024-04-06T03:52:42.674013+0000","last_fullsized":"2024-04-06T03:52:42.674013+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:03.826789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:03.826789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:19.295188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":866,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671683+0000","last_change":"2024-04-06T03:51:57.328265+0000","last_active":"2024-04-06T03:52:42.671683+0000","last_peered":"2024-04-06T03:52:42.671683+0000","last_clean":"2024-04-06T03:52:42.671683+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:52:42.671683+0000","last_undegraded":"2024-04-06T03:52:42.671683+0000","last_fullsized":"2024-04-06T03:52:42.671683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:57.327982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:51:57.327982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:16.629680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"448'2","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:43.395921+0000","last_change":"2024-04-06T03:52:43.395921+0000","last_active":"2024-04-06T03:52:43.395921+0000","last_peered":"2024-04-06T03:52:43.395921+0000","last_clean":"2024-04-06T03:52:43.395921+0000","last_became_active":"2024-04-06T03:48:21.323713+0000","last_became_peered":"2024-04-06T03:48:21.323713+0000","last_unstale":"2024-04-06T03:52:43.395921+0000","last_undegraded":"2024-04-06T03:52:43.395921+0000","last_fullsized":"2024-04-06T03:52:43.395921+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'2","last_scrub_stamp":"2024-04-06T03:52:43.395842+0000","last_deep_scrub":"448'2","last_deep_scrub_stamp":"2024-04-06T03:52:43.395842+0000","last_clean_scrub_stamp":"2024-04-06T03:52:43.395842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:05.808931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"433'31","reported_seq":962,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227620+0000","last_change":"2024-04-06T03:51:56.452210+0000","last_active":"2024-04-06T03:52:42.227620+0000","last_peered":"2024-04-06T03:52:42.227620+0000","last_clean":"2024-04-06T03:52:42.227620+0000","last_became_active":"2024-04-06T03:48:32.468746+0000","last_became_peered":"2024-04-06T03:48:32.468746+0000","last_unstale":"2024-04-06T03:52:42.227620+0000","last_undegraded":"2024-04-06T03:52:42.227620+0000","last_fullsized":"2024-04-06T03:52:42.227620+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'31","last_scrub_stamp":"2024-04-06T03:51:56.451944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:51:56.451944+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:23.762368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":45,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"428'6","reported_seq":2791,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.034632+0000","last_change":"2024-04-06T03:51:59.468913+0000","last_active":"2024-04-06T03:52:52.034632+0000","last_peered":"2024-04-06T03:52:52.034632+0000","last_clean":"2024-04-06T03:52:52.034632+0000","last_became_active":"2024-04-06T03:48:32.466606+0000","last_became_peered":"2024-04-06T03:48:32.466606+0000","last_unstale":"2024-04-06T03:52:52.034632+0000","last_undegraded":"2024-04-06T03:52:52.034632+0000","last_fullsized":"2024-04-06T03:52:52.034632+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'6","last_scrub_stamp":"2024-04-06T03:51:59.468813+0000","last_deep_scrub":"428'6","last_deep_scrub_stamp":"2024-04-06T03:50:42.386898+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.468813+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:09.991308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1881,"num_read_kb":1876,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:48.039335+0000","last_change":"2024-04-06T03:52:48.039391+0000","last_active":"2024-04-06T03:52:48.039335+0000","last_peered":"2024-04-06T03:52:48.039335+0000","last_clean":"2024-04-06T03:52:48.039335+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:52:48.039335+0000","last_undegraded":"2024-04-06T03:52:48.039335+0000","last_fullsized":"2024-04-06T03:52:48.039335+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:48.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:39.430710+0000","last_clean_scrub_stamp":"2024-04-06T03:52:48.039180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:55.018004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":877,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183799+0000","last_change":"2024-04-06T03:52:19.739171+0000","last_active":"2024-04-06T03:52:42.183799+0000","last_peered":"2024-04-06T03:52:42.183799+0000","last_clean":"2024-04-06T03:52:42.183799+0000","last_became_active":"2024-04-06T03:42:23.562715+0000","last_became_peered":"2024-04-06T03:42:23.562715+0000","last_unstale":"2024-04-06T03:52:42.183799+0000","last_undegraded":"2024-04-06T03:52:42.183799+0000","last_fullsized":"2024-04-06T03:52:42.183799+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:19.739026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:52:19.739026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:25.673156+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":498,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678247+0000","last_change":"2024-04-06T03:51:48.824750+0000","last_active":"2024-04-06T03:52:42.678247+0000","last_peered":"2024-04-06T03:52:42.678247+0000","last_clean":"2024-04-06T03:52:42.678247+0000","last_became_active":"2024-04-06T03:51:06.733099+0000","last_became_peered":"2024-04-06T03:51:06.733099+0000","last_unstale":"2024-04-06T03:52:42.678247+0000","last_undegraded":"2024-04-06T03:52:42.678247+0000","last_fullsized":"2024-04-06T03:52:42.678247+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:48.824466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:51:48.824466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:14.788630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"433'54","reported_seq":157,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671534+0000","last_change":"2024-04-06T03:52:07.147768+0000","last_active":"2024-04-06T03:52:42.671534+0000","last_peered":"2024-04-06T03:52:42.671534+0000","last_clean":"2024-04-06T03:52:42.671534+0000","last_became_active":"2024-04-06T03:48:32.459468+0000","last_became_peered":"2024-04-06T03:48:32.459468+0000","last_unstale":"2024-04-06T03:52:42.671534+0000","last_undegraded":"2024-04-06T03:52:42.671534+0000","last_fullsized":"2024-04-06T03:52:42.671534+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'54","last_scrub_stamp":"2024-04-06T03:52:07.147604+0000","last_deep_scrub":"433'54","last_deep_scrub_stamp":"2024-04-06T03:52:07.147604+0000","last_clean_scrub_stamp":"2024-04-06T03:52:07.147604+0000","objects_scrubbed":6,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:11.468923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":66,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":891,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.672299+0000","last_change":"2024-04-06T03:52:42.078891+0000","last_active":"2024-04-06T03:52:42.672299+0000","last_peered":"2024-04-06T03:52:42.672299+0000","last_clean":"2024-04-06T03:52:42.672299+0000","last_became_active":"2024-04-06T03:42:23.576890+0000","last_became_peered":"2024-04-06T03:42:23.576890+0000","last_unstale":"2024-04-06T03:52:42.672299+0000","last_undegraded":"2024-04-06T03:52:42.672299+0000","last_fullsized":"2024-04-06T03:52:42.672299+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:42.078672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:19.191417+0000","last_clean_scrub_stamp":"2024-04-06T03:52:42.078672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:09.137096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"397'15","reported_seq":1298,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697316+0000","last_change":"2024-04-06T03:52:41.468507+0000","last_active":"2024-04-06T03:52:47.697316+0000","last_peered":"2024-04-06T03:52:47.697316+0000","last_clean":"2024-04-06T03:52:47.697316+0000","last_became_active":"2024-04-06T03:48:21.346584+0000","last_became_peered":"2024-04-06T03:48:21.346584+0000","last_unstale":"2024-04-06T03:52:47.697316+0000","last_undegraded":"2024-04-06T03:52:47.697316+0000","last_fullsized":"2024-04-06T03:52:47.697316+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'15","last_scrub_stamp":"2024-04-06T03:52:41.468142+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:41:18.436223+0000","last_clean_scrub_stamp":"2024-04-06T03:52:41.468142+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:58.130229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":869,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181972+0000","last_change":"2024-04-06T03:51:50.437671+0000","last_active":"2024-04-06T03:52:42.181972+0000","last_peered":"2024-04-06T03:52:42.181972+0000","last_clean":"2024-04-06T03:52:42.181972+0000","last_became_active":"2024-04-06T03:42:23.590773+0000","last_became_peered":"2024-04-06T03:42:23.590773+0000","last_unstale":"2024-04-06T03:52:42.181972+0000","last_undegraded":"2024-04-06T03:52:42.181972+0000","last_fullsized":"2024-04-06T03:52:42.181972+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:50.437449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:51:50.437449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:19.095643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":506,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678031+0000","last_change":"2024-04-06T03:51:59.849971+0000","last_active":"2024-04-06T03:52:42.678031+0000","last_peered":"2024-04-06T03:52:42.678031+0000","last_clean":"2024-04-06T03:52:42.678031+0000","last_became_active":"2024-04-06T03:50:47.210897+0000","last_became_peered":"2024-04-06T03:50:47.210897+0000","last_unstale":"2024-04-06T03:52:42.678031+0000","last_undegraded":"2024-04-06T03:52:42.678031+0000","last_fullsized":"2024-04-06T03:52:42.678031+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:59.849738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:27.569064+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.849738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:12.972343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":891,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182901+0000","last_change":"2024-04-06T03:51:34.508607+0000","last_active":"2024-04-06T03:52:42.182901+0000","last_peered":"2024-04-06T03:52:42.182901+0000","last_clean":"2024-04-06T03:52:42.182901+0000","last_became_active":"2024-04-06T03:48:32.435449+0000","last_became_peered":"2024-04-06T03:48:32.435449+0000","last_unstale":"2024-04-06T03:52:42.182901+0000","last_undegraded":"2024-04-06T03:52:42.182901+0000","last_fullsized":"2024-04-06T03:52:42.182901+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:34.508293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:34.508293+0000","last_clean_scrub_stamp":"2024-04-06T03:51:34.508293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:44.125080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"433'49","reported_seq":954,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:43.125667+0000","last_change":"2024-04-06T03:52:43.125762+0000","last_active":"2024-04-06T03:52:43.125667+0000","last_peered":"2024-04-06T03:52:43.125667+0000","last_clean":"2024-04-06T03:52:43.125667+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:52:43.125667+0000","last_undegraded":"2024-04-06T03:52:43.125667+0000","last_fullsized":"2024-04-06T03:52:43.125667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'49","last_scrub_stamp":"2024-04-06T03:52:43.125497+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:52:43.125497+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:07.774408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":514,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.676402+0000","last_change":"2024-04-06T03:52:02.035919+0000","last_active":"2024-04-06T03:52:42.676402+0000","last_peered":"2024-04-06T03:52:42.676402+0000","last_clean":"2024-04-06T03:52:42.676402+0000","last_became_active":"2024-04-06T03:50:47.200323+0000","last_became_peered":"2024-04-06T03:50:47.200323+0000","last_unstale":"2024-04-06T03:52:42.676402+0000","last_undegraded":"2024-04-06T03:52:42.676402+0000","last_fullsized":"2024-04-06T03:52:42.676402+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:02.035691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:37.734841+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.035691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:10.406611+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"404'14","reported_seq":1294,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697859+0000","last_change":"2024-04-06T03:52:08.487093+0000","last_active":"2024-04-06T03:52:47.697859+0000","last_peered":"2024-04-06T03:52:47.697859+0000","last_clean":"2024-04-06T03:52:47.697859+0000","last_became_active":"2024-04-06T03:48:32.447676+0000","last_became_peered":"2024-04-06T03:48:32.447676+0000","last_unstale":"2024-04-06T03:52:47.697859+0000","last_undegraded":"2024-04-06T03:52:47.697859+0000","last_fullsized":"2024-04-06T03:52:47.697859+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-04-06T03:52:08.486745+0000","last_deep_scrub":"404'14","last_deep_scrub_stamp":"2024-04-06T03:52:08.486745+0000","last_clean_scrub_stamp":"2024-04-06T03:52:08.486745+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:38.080900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:44.349553+0000","last_change":"2024-04-06T03:52:44.349573+0000","last_active":"2024-04-06T03:52:44.349553+0000","last_peered":"2024-04-06T03:52:44.349553+0000","last_clean":"2024-04-06T03:52:44.349553+0000","last_became_active":"2024-04-06T03:48:21.342086+0000","last_became_peered":"2024-04-06T03:48:21.342086+0000","last_unstale":"2024-04-06T03:52:44.349553+0000","last_undegraded":"2024-04-06T03:52:44.349553+0000","last_fullsized":"2024-04-06T03:52:44.349553+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:44.349489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:36.411646+0000","last_clean_scrub_stamp":"2024-04-06T03:52:44.349489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:03.010061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"433'72","reported_seq":1029,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180821+0000","last_change":"2024-04-06T03:51:57.441813+0000","last_active":"2024-04-06T03:52:42.180821+0000","last_peered":"2024-04-06T03:52:42.180821+0000","last_clean":"2024-04-06T03:52:42.180821+0000","last_became_active":"2024-04-06T03:48:21.345525+0000","last_became_peered":"2024-04-06T03:48:21.345525+0000","last_unstale":"2024-04-06T03:52:42.180821+0000","last_undegraded":"2024-04-06T03:52:42.180821+0000","last_fullsized":"2024-04-06T03:52:42.180821+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'72","last_scrub_stamp":"2024-04-06T03:51:57.441544+0000","last_deep_scrub":"428'60","last_deep_scrub_stamp":"2024-04-06T03:50:38.375224+0000","last_clean_scrub_stamp":"2024-04-06T03:51:57.441544+0000","objects_scrubbed":9,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:04.682835+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":98,"num_read_kb":87,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":877,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673568+0000","last_change":"2024-04-06T03:51:52.371847+0000","last_active":"2024-04-06T03:52:42.673568+0000","last_peered":"2024-04-06T03:52:42.673568+0000","last_clean":"2024-04-06T03:52:42.673568+0000","last_became_active":"2024-04-06T03:48:21.308499+0000","last_became_peered":"2024-04-06T03:48:21.308499+0000","last_unstale":"2024-04-06T03:52:42.673568+0000","last_undegraded":"2024-04-06T03:52:42.673568+0000","last_fullsized":"2024-04-06T03:52:42.673568+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:52.371606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:37.509500+0000","last_clean_scrub_stamp":"2024-04-06T03:51:52.371606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:14.280369+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":1305,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697574+0000","last_change":"2024-04-06T03:52:23.737147+0000","last_active":"2024-04-06T03:52:47.697574+0000","last_peered":"2024-04-06T03:52:47.697574+0000","last_clean":"2024-04-06T03:52:47.697574+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:52:47.697574+0000","last_undegraded":"2024-04-06T03:52:47.697574+0000","last_fullsized":"2024-04-06T03:52:47.697574+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:52:23.736924+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.736924+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:34.562602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":726,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669771+0000","last_change":"2024-04-06T03:51:59.569896+0000","last_active":"2024-04-06T03:52:42.669771+0000","last_peered":"2024-04-06T03:52:42.669771+0000","last_clean":"2024-04-06T03:52:42.669771+0000","last_became_active":"2024-04-06T03:49:27.623051+0000","last_became_peered":"2024-04-06T03:49:27.623051+0000","last_unstale":"2024-04-06T03:52:42.669771+0000","last_undegraded":"2024-04-06T03:52:42.669771+0000","last_fullsized":"2024-04-06T03:52:42.669771+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:59.569693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:59.569693+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.569693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.001786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"433'33","reported_seq":974,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182265+0000","last_change":"2024-04-06T03:51:45.362474+0000","last_active":"2024-04-06T03:52:42.182265+0000","last_peered":"2024-04-06T03:52:42.182265+0000","last_clean":"2024-04-06T03:52:42.182265+0000","last_became_active":"2024-04-06T03:48:32.468304+0000","last_became_peered":"2024-04-06T03:48:32.468304+0000","last_unstale":"2024-04-06T03:52:42.182265+0000","last_undegraded":"2024-04-06T03:52:42.182265+0000","last_fullsized":"2024-04-06T03:52:42.182265+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'33","last_scrub_stamp":"2024-04-06T03:51:45.362127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:51:45.362127+0000","objects_scrubbed":5,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:00.873159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":905,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:50.052577+0000","last_change":"2024-04-06T03:52:50.052631+0000","last_active":"2024-04-06T03:52:50.052577+0000","last_peered":"2024-04-06T03:52:50.052577+0000","last_clean":"2024-04-06T03:52:50.052577+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:52:50.052577+0000","last_undegraded":"2024-04-06T03:52:50.052577+0000","last_fullsized":"2024-04-06T03:52:50.052577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:50.052419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:48:47.654196+0000","last_clean_scrub_stamp":"2024-04-06T03:52:50.052419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:58.671364+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":698,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670325+0000","last_change":"2024-04-06T03:52:34.443299+0000","last_active":"2024-04-06T03:52:42.670325+0000","last_peered":"2024-04-06T03:52:42.670325+0000","last_clean":"2024-04-06T03:52:42.670325+0000","last_became_active":"2024-04-06T03:49:27.614179+0000","last_became_peered":"2024-04-06T03:49:27.614179+0000","last_unstale":"2024-04-06T03:52:42.670325+0000","last_undegraded":"2024-04-06T03:52:42.670325+0000","last_fullsized":"2024-04-06T03:52:42.670325+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:34.442971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:59.438940+0000","last_clean_scrub_stamp":"2024-04-06T03:52:34.442971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:38.721866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183492+0000","last_change":"2024-04-06T03:51:31.499337+0000","last_active":"2024-04-06T03:52:42.183492+0000","last_peered":"2024-04-06T03:52:42.183492+0000","last_clean":"2024-04-06T03:52:42.183492+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:52:42.183492+0000","last_undegraded":"2024-04-06T03:52:42.183492+0000","last_fullsized":"2024-04-06T03:52:42.183492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:31.499126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:51:31.499126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:59.598040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"433'73","reported_seq":995,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.672393+0000","last_change":"2024-04-06T03:52:07.817533+0000","last_active":"2024-04-06T03:52:42.672393+0000","last_peered":"2024-04-06T03:52:42.672393+0000","last_clean":"2024-04-06T03:52:42.672393+0000","last_became_active":"2024-04-06T03:51:06.723822+0000","last_became_peered":"2024-04-06T03:51:06.723822+0000","last_unstale":"2024-04-06T03:52:42.672393+0000","last_undegraded":"2024-04-06T03:52:42.672393+0000","last_fullsized":"2024-04-06T03:52:42.672393+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'73","last_scrub_stamp":"2024-04-06T03:52:07.817268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:52:07.817268+0000","objects_scrubbed":10,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:30.962715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":111,"num_read_kb":99,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":855,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182498+0000","last_change":"2024-04-06T03:51:36.450457+0000","last_active":"2024-04-06T03:52:42.182498+0000","last_peered":"2024-04-06T03:52:42.182498+0000","last_clean":"2024-04-06T03:52:42.182498+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:52:42.182498+0000","last_undegraded":"2024-04-06T03:52:42.182498+0000","last_fullsized":"2024-04-06T03:52:42.182498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:36.450157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:20.575364+0000","last_clean_scrub_stamp":"2024-04-06T03:51:36.450157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:53.561542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":913,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179561+0000","last_change":"2024-04-06T03:51:54.618796+0000","last_active":"2024-04-06T03:52:42.179561+0000","last_peered":"2024-04-06T03:52:42.179561+0000","last_clean":"2024-04-06T03:52:42.179561+0000","last_became_active":"2024-04-06T03:48:32.439955+0000","last_became_peered":"2024-04-06T03:48:32.439955+0000","last_unstale":"2024-04-06T03:52:42.179561+0000","last_undegraded":"2024-04-06T03:52:42.179561+0000","last_fullsized":"2024-04-06T03:52:42.179561+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:54.618581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:33.770813+0000","last_clean_scrub_stamp":"2024-04-06T03:51:54.618581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:18.523727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":919,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179950+0000","last_change":"2024-04-06T03:51:27.580126+0000","last_active":"2024-04-06T03:52:42.179950+0000","last_peered":"2024-04-06T03:52:42.179950+0000","last_clean":"2024-04-06T03:52:42.179950+0000","last_became_active":"2024-04-06T03:48:32.439252+0000","last_became_peered":"2024-04-06T03:48:32.439252+0000","last_unstale":"2024-04-06T03:52:42.179950+0000","last_undegraded":"2024-04-06T03:52:42.179950+0000","last_fullsized":"2024-04-06T03:52:42.179950+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:27.579882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:12.446876+0000","last_clean_scrub_stamp":"2024-04-06T03:51:27.579882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:51.449223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"433'67","reported_seq":1018,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:46.668947+0000","last_change":"2024-04-06T03:52:46.669041+0000","last_active":"2024-04-06T03:52:46.668947+0000","last_peered":"2024-04-06T03:52:46.668947+0000","last_clean":"2024-04-06T03:52:46.668947+0000","last_became_active":"2024-04-06T03:42:23.570628+0000","last_became_peered":"2024-04-06T03:42:23.570628+0000","last_unstale":"2024-04-06T03:52:46.668947+0000","last_undegraded":"2024-04-06T03:52:46.668947+0000","last_fullsized":"2024-04-06T03:52:46.668947+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'67","last_scrub_stamp":"2024-04-06T03:52:46.668769+0000","last_deep_scrub":"307'53","last_deep_scrub_stamp":"2024-04-06T03:49:52.351719+0000","last_clean_scrub_stamp":"2024-04-06T03:52:46.668769+0000","objects_scrubbed":8,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:01.443665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":92,"num_read_kb":82,"num_write":60,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":125,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670250+0000","last_change":"2024-04-06T03:52:02.315577+0000","last_active":"2024-04-06T03:52:42.670250+0000","last_peered":"2024-04-06T03:52:42.670250+0000","last_clean":"2024-04-06T03:52:42.670250+0000","last_became_active":"2024-04-06T03:48:32.463995+0000","last_became_peered":"2024-04-06T03:48:32.463995+0000","last_unstale":"2024-04-06T03:52:42.670250+0000","last_undegraded":"2024-04-06T03:52:42.670250+0000","last_fullsized":"2024-04-06T03:52:42.670250+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:02.315422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:02.315422+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.315422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:06.859785+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":901,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179689+0000","last_change":"2024-04-06T03:52:10.622320+0000","last_active":"2024-04-06T03:52:42.179689+0000","last_peered":"2024-04-06T03:52:42.179689+0000","last_clean":"2024-04-06T03:52:42.179689+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:52:42.179689+0000","last_undegraded":"2024-04-06T03:52:42.179689+0000","last_fullsized":"2024-04-06T03:52:42.179689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:10.622097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:10.622097+0000","last_clean_scrub_stamp":"2024-04-06T03:52:10.622097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:25.546094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670293+0000","last_change":"2024-04-06T03:52:23.441902+0000","last_active":"2024-04-06T03:52:42.670293+0000","last_peered":"2024-04-06T03:52:42.670293+0000","last_clean":"2024-04-06T03:52:42.670293+0000","last_became_active":"2024-04-06T03:48:21.325094+0000","last_became_peered":"2024-04-06T03:48:21.325094+0000","last_unstale":"2024-04-06T03:52:42.670293+0000","last_undegraded":"2024-04-06T03:52:42.670293+0000","last_fullsized":"2024-04-06T03:52:42.670293+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:23.441690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.441690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:49.516959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"433'45","reported_seq":640,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.676750+0000","last_change":"2024-04-06T03:52:28.933090+0000","last_active":"2024-04-06T03:52:42.676750+0000","last_peered":"2024-04-06T03:52:42.676750+0000","last_clean":"2024-04-06T03:52:42.676750+0000","last_became_active":"2024-04-06T03:50:47.205748+0000","last_became_peered":"2024-04-06T03:50:47.205748+0000","last_unstale":"2024-04-06T03:52:42.676750+0000","last_undegraded":"2024-04-06T03:52:42.676750+0000","last_fullsized":"2024-04-06T03:52:42.676750+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'45","last_scrub_stamp":"2024-04-06T03:52:28.932835+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:52:28.932835+0000","objects_scrubbed":5,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:52.609018+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":40,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":520,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.674883+0000","last_change":"2024-04-06T03:52:18.908569+0000","last_active":"2024-04-06T03:52:42.674883+0000","last_peered":"2024-04-06T03:52:42.674883+0000","last_clean":"2024-04-06T03:52:42.674883+0000","last_became_active":"2024-04-06T03:50:47.206374+0000","last_became_peered":"2024-04-06T03:50:47.206374+0000","last_unstale":"2024-04-06T03:52:42.674883+0000","last_undegraded":"2024-04-06T03:52:42.674883+0000","last_fullsized":"2024-04-06T03:52:42.674883+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:18.908322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:52:18.908322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:21.944455+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":917,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.936286+0000","last_change":"2024-04-06T03:52:52.936365+0000","last_active":"2024-04-06T03:52:52.936286+0000","last_peered":"2024-04-06T03:52:52.936286+0000","last_clean":"2024-04-06T03:52:52.936286+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:52:52.936286+0000","last_undegraded":"2024-04-06T03:52:52.936286+0000","last_fullsized":"2024-04-06T03:52:52.936286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:52.936118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:52.936118+0000","last_clean_scrub_stamp":"2024-04-06T03:52:52.936118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:00.340491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"433'58","reported_seq":1023,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184449+0000","last_change":"2024-04-06T03:51:59.656696+0000","last_active":"2024-04-06T03:52:42.184449+0000","last_peered":"2024-04-06T03:52:42.184449+0000","last_clean":"2024-04-06T03:52:42.184449+0000","last_became_active":"2024-04-06T03:48:32.446386+0000","last_became_peered":"2024-04-06T03:48:32.446386+0000","last_unstale":"2024-04-06T03:52:42.184449+0000","last_undegraded":"2024-04-06T03:52:42.184449+0000","last_fullsized":"2024-04-06T03:52:42.184449+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'58","last_scrub_stamp":"2024-04-06T03:51:59.656460+0000","last_deep_scrub":"307'46","last_deep_scrub_stamp":"2024-04-06T03:49:07.245123+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.656460+0000","objects_scrubbed":7,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:22.661084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":71,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"327'14","reported_seq":1293,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.697450+0000","last_change":"2024-04-06T03:52:44.612822+0000","last_active":"2024-04-06T03:52:47.697450+0000","last_peered":"2024-04-06T03:52:47.697450+0000","last_clean":"2024-04-06T03:52:47.697450+0000","last_became_active":"2024-04-06T03:42:23.576196+0000","last_became_peered":"2024-04-06T03:42:23.576196+0000","last_unstale":"2024-04-06T03:52:47.697450+0000","last_undegraded":"2024-04-06T03:52:47.697450+0000","last_fullsized":"2024-04-06T03:52:47.697450+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'14","last_scrub_stamp":"2024-04-06T03:52:44.612575+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:52:44.612575+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:00.322812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4722944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":2261,"num_read_kb":1986,"num_write":1467,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":8472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1635,"ondisk_log_size":1635,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1881,"num_read_kb":1876,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743820,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546284,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93198868,"statfs":{"total":95995035648,"available":95435640832,"internally_reserved":0,"allocated":1478656,"data_stored":1073390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":557906378},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136278,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582328,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162824,"statfs":{"total":95995035648,"available":95398731776,"internally_reserved":0,"allocated":1425408,"data_stored":1071312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575307,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460996,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93284156,"statfs":{"total":95995035648,"available":95522975744,"internally_reserved":0,"allocated":847872,"data_stored":480798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738837,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646220,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098932,"statfs":{"total":95995035648,"available":95333306368,"internally_reserved":0,"allocated":1445888,"data_stored":1072365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379073,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537336,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207816,"statfs":{"total":95995035648,"available":95444803584,"internally_reserved":0,"allocated":835584,"data_stored":480542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084725,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605360,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139792,"statfs":{"total":95995035648,"available":95375147008,"internally_reserved":0,"allocated":827392,"data_stored":480456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855788,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533212,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211940,"statfs":{"total":95995035648,"available":95449026560,"internally_reserved":0,"allocated":872448,"data_stored":481027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568496034,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603940,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93141212,"statfs":{"total":95995035648,"available":95376601088,"internally_reserved":0,"allocated":815104,"data_stored":480080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:52:56.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-06T03:52:57.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:55.251767+0000 osd.1 (osd.1) 543 : cluster 0 5.19 scrub starts 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:55.253803+0000 osd.1 (osd.1) 544 : cluster 0 5.19 scrub ok 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:55.922203+0000 osd.2 (osd.2) 631 : cluster 0 2.c scrub starts 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:55.924502+0000 osd.2 (osd.2) 632 : cluster 0 2.c scrub ok 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.234428+0000 osd.5 (osd.5) 223 : cluster 0 2.2 scrub starts 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.236860+0000 osd.5 (osd.5) 224 : cluster 0 2.2 scrub ok 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.639086+0000 osd.3 (osd.3) 789 : cluster 0 6.1b scrub starts 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.728866+0000 mgr.y (mgr.24368) 1172 : cluster 0 pgmap v1287: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.845024+0000 osd.3 (osd.3) 790 : cluster 0 6.1b scrub ok 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:55.251767+0000 osd.1 (osd.1) 543 : cluster 0 5.19 scrub starts 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:55.253803+0000 osd.1 (osd.1) 544 : cluster 0 5.19 scrub ok 2024-04-06T03:52:57.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:55.922203+0000 osd.2 (osd.2) 631 : cluster 0 2.c scrub starts 2024-04-06T03:52:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:55.924502+0000 osd.2 (osd.2) 632 : cluster 0 2.c scrub ok 2024-04-06T03:52:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.234428+0000 osd.5 (osd.5) 223 : cluster 0 2.2 scrub starts 2024-04-06T03:52:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.236860+0000 osd.5 (osd.5) 224 : cluster 0 2.2 scrub ok 2024-04-06T03:52:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.639086+0000 osd.3 (osd.3) 789 : cluster 0 6.1b scrub starts 2024-04-06T03:52:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.728866+0000 mgr.y (mgr.24368) 1172 : cluster 0 pgmap v1287: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:57.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:57 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.845024+0000 osd.3 (osd.3) 790 : cluster 0 6.1b scrub ok 2024-04-06T03:52:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:55.251767+0000 osd.1 (osd.1) 543 : cluster 0 5.19 scrub starts 2024-04-06T03:52:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:55.253803+0000 osd.1 (osd.1) 544 : cluster 0 5.19 scrub ok 2024-04-06T03:52:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:55.922203+0000 osd.2 (osd.2) 631 : cluster 0 2.c scrub starts 2024-04-06T03:52:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:55.924502+0000 osd.2 (osd.2) 632 : cluster 0 2.c scrub ok 2024-04-06T03:52:57.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.234428+0000 osd.5 (osd.5) 223 : cluster 0 2.2 scrub starts 2024-04-06T03:52:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.236860+0000 osd.5 (osd.5) 224 : cluster 0 2.2 scrub ok 2024-04-06T03:52:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.639086+0000 osd.3 (osd.3) 789 : cluster 0 6.1b scrub starts 2024-04-06T03:52:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.728866+0000 mgr.y (mgr.24368) 1172 : cluster 0 pgmap v1287: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:57.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:57 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.845024+0000 osd.3 (osd.3) 790 : cluster 0 6.1b scrub ok 2024-04-06T03:52:58.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.659762+0000 osd.7 (osd.7) 797 : cluster 0 6.5 scrub starts 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.756637+0000 osd.7 (osd.7) 798 : cluster 0 6.5 scrub ok 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.795456+0000 osd.6 (osd.6) 149 : cluster 0 5.1f scrub starts 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.798158+0000 osd.6 (osd.6) 150 : cluster 0 5.1f scrub ok 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.884006+0000 osd.0 (osd.0) 756 : cluster 0 3.1 scrub starts 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[21467]: cluster 2024-04-06T03:52:56.888797+0000 osd.0 (osd.0) 757 : cluster 0 3.1 scrub ok 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.659762+0000 osd.7 (osd.7) 797 : cluster 0 6.5 scrub starts 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.756637+0000 osd.7 (osd.7) 798 : cluster 0 6.5 scrub ok 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.795456+0000 osd.6 (osd.6) 149 : cluster 0 5.1f scrub starts 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.798158+0000 osd.6 (osd.6) 150 : cluster 0 5.1f scrub ok 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.884006+0000 osd.0 (osd.0) 756 : cluster 0 3.1 scrub starts 2024-04-06T03:52:58.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:58 smithi050 bash[29221]: cluster 2024-04-06T03:52:56.888797+0000 osd.0 (osd.0) 757 : cluster 0 3.1 scrub ok 2024-04-06T03:52:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:58 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.659762+0000 osd.7 (osd.7) 797 : cluster 0 6.5 scrub starts 2024-04-06T03:52:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:58 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.756637+0000 osd.7 (osd.7) 798 : cluster 0 6.5 scrub ok 2024-04-06T03:52:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:58 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.795456+0000 osd.6 (osd.6) 149 : cluster 0 5.1f scrub starts 2024-04-06T03:52:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:58 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.798158+0000 osd.6 (osd.6) 150 : cluster 0 5.1f scrub ok 2024-04-06T03:52:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:58 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.884006+0000 osd.0 (osd.0) 756 : cluster 0 3.1 scrub starts 2024-04-06T03:52:58.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:58 smithi153 bash[23852]: cluster 2024-04-06T03:52:56.888797+0000 osd.0 (osd.0) 757 : cluster 0 3.1 scrub ok 2024-04-06T03:52:59.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:52:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:52:59.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[29221]: cluster 2024-04-06T03:52:58.615462+0000 osd.7 (osd.7) 799 : cluster 0 2.8 scrub starts 2024-04-06T03:52:59.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[29221]: cluster 2024-04-06T03:52:58.618324+0000 osd.7 (osd.7) 800 : cluster 0 2.8 scrub ok 2024-04-06T03:52:59.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[29221]: cluster 2024-04-06T03:52:58.729520+0000 mgr.y (mgr.24368) 1173 : cluster 0 pgmap v1288: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:59.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[21467]: cluster 2024-04-06T03:52:58.615462+0000 osd.7 (osd.7) 799 : cluster 0 2.8 scrub starts 2024-04-06T03:52:59.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[21467]: cluster 2024-04-06T03:52:58.618324+0000 osd.7 (osd.7) 800 : cluster 0 2.8 scrub ok 2024-04-06T03:52:59.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:52:59 smithi050 bash[21467]: cluster 2024-04-06T03:52:58.729520+0000 mgr.y (mgr.24368) 1173 : cluster 0 pgmap v1288: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:52:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:59 smithi153 bash[23852]: cluster 2024-04-06T03:52:58.615462+0000 osd.7 (osd.7) 799 : cluster 0 2.8 scrub starts 2024-04-06T03:52:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:59 smithi153 bash[23852]: cluster 2024-04-06T03:52:58.618324+0000 osd.7 (osd.7) 800 : cluster 0 2.8 scrub ok 2024-04-06T03:52:59.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:52:59 smithi153 bash[23852]: cluster 2024-04-06T03:52:58.729520+0000 mgr.y (mgr.24368) 1173 : cluster 0 pgmap v1288: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:00 smithi153 bash[23852]: cluster 2024-04-06T03:52:59.238703+0000 osd.1 (osd.1) 545 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:53:00.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:00 smithi153 bash[23852]: cluster 2024-04-06T03:52:59.323423+0000 osd.1 (osd.1) 546 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:53:00.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:00 smithi050 bash[21467]: cluster 2024-04-06T03:52:59.238703+0000 osd.1 (osd.1) 545 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:53:00.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:00 smithi050 bash[21467]: cluster 2024-04-06T03:52:59.323423+0000 osd.1 (osd.1) 546 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:53:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:00 smithi050 bash[29221]: cluster 2024-04-06T03:52:59.238703+0000 osd.1 (osd.1) 545 : cluster 0 6.1 deep-scrub starts 2024-04-06T03:53:00.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:00 smithi050 bash[29221]: cluster 2024-04-06T03:52:59.323423+0000 osd.1 (osd.1) 546 : cluster 0 6.1 deep-scrub ok 2024-04-06T03:53:01.046 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:53:00 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:53:01.547 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:53:01 smithi153 bash[51781]: logger=cleanup t=2024-04-06T03:53:01.216893434Z level=info msg="Completed cleanup jobs" duration=56.564544ms 2024-04-06T03:53:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:01 smithi153 bash[23852]: audit 2024-04-06T03:53:00.607868+0000 mgr.y (mgr.24368) 1174 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:53:01.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:01 smithi153 bash[23852]: cluster 2024-04-06T03:53:00.731033+0000 mgr.y (mgr.24368) 1175 : cluster 0 pgmap v1289: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:53:01.595 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:53:01.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:01 smithi050 bash[29221]: audit 2024-04-06T03:53:00.607868+0000 mgr.y (mgr.24368) 1174 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:53:01.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:01 smithi050 bash[29221]: cluster 2024-04-06T03:53:00.731033+0000 mgr.y (mgr.24368) 1175 : cluster 0 pgmap v1289: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:53:01.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:01 smithi050 bash[21467]: audit 2024-04-06T03:53:00.607868+0000 mgr.y (mgr.24368) 1174 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:53:01.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:01 smithi050 bash[21467]: cluster 2024-04-06T03:53:00.731033+0000 mgr.y (mgr.24368) 1175 : cluster 0 pgmap v1289: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:53:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:02 smithi153 bash[23852]: cluster 2024-04-06T03:53:00.629146+0000 osd.7 (osd.7) 801 : cluster 0 5.4 scrub starts 2024-04-06T03:53:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:02 smithi153 bash[23852]: cluster 2024-04-06T03:53:00.631979+0000 osd.7 (osd.7) 802 : cluster 0 5.4 scrub ok 2024-04-06T03:53:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:02 smithi153 bash[23852]: cluster 2024-04-06T03:53:01.621339+0000 osd.7 (osd.7) 803 : cluster 0 6.8 scrub starts 2024-04-06T03:53:02.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:02 smithi153 bash[23852]: cluster 2024-04-06T03:53:01.623371+0000 osd.7 (osd.7) 804 : cluster 0 6.8 scrub ok 2024-04-06T03:53:02.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[21467]: cluster 2024-04-06T03:53:00.629146+0000 osd.7 (osd.7) 801 : cluster 0 5.4 scrub starts 2024-04-06T03:53:02.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[21467]: cluster 2024-04-06T03:53:00.631979+0000 osd.7 (osd.7) 802 : cluster 0 5.4 scrub ok 2024-04-06T03:53:02.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[21467]: cluster 2024-04-06T03:53:01.621339+0000 osd.7 (osd.7) 803 : cluster 0 6.8 scrub starts 2024-04-06T03:53:02.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[21467]: cluster 2024-04-06T03:53:01.623371+0000 osd.7 (osd.7) 804 : cluster 0 6.8 scrub ok 2024-04-06T03:53:02.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[29221]: cluster 2024-04-06T03:53:00.629146+0000 osd.7 (osd.7) 801 : cluster 0 5.4 scrub starts 2024-04-06T03:53:02.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[29221]: cluster 2024-04-06T03:53:00.631979+0000 osd.7 (osd.7) 802 : cluster 0 5.4 scrub ok 2024-04-06T03:53:02.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[29221]: cluster 2024-04-06T03:53:01.621339+0000 osd.7 (osd.7) 803 : cluster 0 6.8 scrub starts 2024-04-06T03:53:02.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:02 smithi050 bash[29221]: cluster 2024-04-06T03:53:01.623371+0000 osd.7 (osd.7) 804 : cluster 0 6.8 scrub ok 2024-04-06T03:53:03.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.250995+0000 osd.1 (osd.1) 547 : cluster 0 3.0 scrub starts 2024-04-06T03:53:03.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.257054+0000 osd.1 (osd.1) 548 : cluster 0 3.0 scrub ok 2024-04-06T03:53:03.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.624532+0000 osd.7 (osd.7) 805 : cluster 0 3.e deep-scrub starts 2024-04-06T03:53:03.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.731483+0000 mgr.y (mgr.24368) 1176 : cluster 0 pgmap v1290: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.250995+0000 osd.1 (osd.1) 547 : cluster 0 3.0 scrub starts 2024-04-06T03:53:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.257054+0000 osd.1 (osd.1) 548 : cluster 0 3.0 scrub ok 2024-04-06T03:53:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.624532+0000 osd.7 (osd.7) 805 : cluster 0 3.e deep-scrub starts 2024-04-06T03:53:03.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:03 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.731483+0000 mgr.y (mgr.24368) 1176 : cluster 0 pgmap v1290: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:03 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.250995+0000 osd.1 (osd.1) 547 : cluster 0 3.0 scrub starts 2024-04-06T03:53:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:03 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.257054+0000 osd.1 (osd.1) 548 : cluster 0 3.0 scrub ok 2024-04-06T03:53:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:03 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.624532+0000 osd.7 (osd.7) 805 : cluster 0 3.e deep-scrub starts 2024-04-06T03:53:03.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:03 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.731483+0000 mgr.y (mgr.24368) 1176 : cluster 0 pgmap v1290: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:04 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.629545+0000 osd.7 (osd.7) 806 : cluster 0 3.e deep-scrub ok 2024-04-06T03:53:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:04 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.862982+0000 osd.2 (osd.2) 633 : cluster 0 5.8 scrub starts 2024-04-06T03:53:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:04 smithi153 bash[23852]: cluster 2024-04-06T03:53:02.864302+0000 osd.2 (osd.2) 634 : cluster 0 5.8 scrub ok 2024-04-06T03:53:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:04 smithi153 bash[23852]: cluster 2024-04-06T03:53:03.642558+0000 osd.3 (osd.3) 791 : cluster 0 3.1e scrub starts 2024-04-06T03:53:04.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:04 smithi153 bash[23852]: cluster 2024-04-06T03:53:03.645295+0000 osd.3 (osd.3) 792 : cluster 0 3.1e scrub ok 2024-04-06T03:53:04.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.629545+0000 osd.7 (osd.7) 806 : cluster 0 3.e deep-scrub ok 2024-04-06T03:53:04.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.862982+0000 osd.2 (osd.2) 633 : cluster 0 5.8 scrub starts 2024-04-06T03:53:04.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[29221]: cluster 2024-04-06T03:53:02.864302+0000 osd.2 (osd.2) 634 : cluster 0 5.8 scrub ok 2024-04-06T03:53:04.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[29221]: cluster 2024-04-06T03:53:03.642558+0000 osd.3 (osd.3) 791 : cluster 0 3.1e scrub starts 2024-04-06T03:53:04.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[29221]: cluster 2024-04-06T03:53:03.645295+0000 osd.3 (osd.3) 792 : cluster 0 3.1e scrub ok 2024-04-06T03:53:04.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.629545+0000 osd.7 (osd.7) 806 : cluster 0 3.e deep-scrub ok 2024-04-06T03:53:04.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.862982+0000 osd.2 (osd.2) 633 : cluster 0 5.8 scrub starts 2024-04-06T03:53:04.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[21467]: cluster 2024-04-06T03:53:02.864302+0000 osd.2 (osd.2) 634 : cluster 0 5.8 scrub ok 2024-04-06T03:53:04.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[21467]: cluster 2024-04-06T03:53:03.642558+0000 osd.3 (osd.3) 791 : cluster 0 3.1e scrub starts 2024-04-06T03:53:04.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:04 smithi050 bash[21467]: cluster 2024-04-06T03:53:03.645295+0000 osd.3 (osd.3) 792 : cluster 0 3.1e scrub ok 2024-04-06T03:53:04.980 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-06T03:53:04.982 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-06T03:53:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:05 smithi153 bash[23852]: cluster 2024-04-06T03:53:03.913755+0000 osd.0 (osd.0) 758 : cluster 0 3.1b scrub starts 2024-04-06T03:53:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:05 smithi153 bash[23852]: cluster 2024-04-06T03:53:03.917773+0000 osd.0 (osd.0) 759 : cluster 0 3.1b scrub ok 2024-04-06T03:53:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:05 smithi153 bash[23852]: cluster 2024-04-06T03:53:04.293417+0000 osd.5 (osd.5) 225 : cluster 0 6.7 scrub starts 2024-04-06T03:53:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:05 smithi153 bash[23852]: cluster 2024-04-06T03:53:04.467063+0000 osd.5 (osd.5) 226 : cluster 0 6.7 scrub ok 2024-04-06T03:53:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:05 smithi153 bash[23852]: cluster 2024-04-06T03:53:04.732183+0000 mgr.y (mgr.24368) 1177 : cluster 0 pgmap v1291: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:53:05.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:05 smithi153 bash[23852]: audit 2024-04-06T03:53:04.944862+0000 mgr.y (mgr.24368) 1178 : audit 0 from='client.25823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:53:05.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[21467]: cluster 2024-04-06T03:53:03.913755+0000 osd.0 (osd.0) 758 : cluster 0 3.1b scrub starts 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[21467]: cluster 2024-04-06T03:53:03.917773+0000 osd.0 (osd.0) 759 : cluster 0 3.1b scrub ok 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[21467]: cluster 2024-04-06T03:53:04.293417+0000 osd.5 (osd.5) 225 : cluster 0 6.7 scrub starts 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[21467]: cluster 2024-04-06T03:53:04.467063+0000 osd.5 (osd.5) 226 : cluster 0 6.7 scrub ok 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[21467]: cluster 2024-04-06T03:53:04.732183+0000 mgr.y (mgr.24368) 1177 : cluster 0 pgmap v1291: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[21467]: audit 2024-04-06T03:53:04.944862+0000 mgr.y (mgr.24368) 1178 : audit 0 from='client.25823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[29221]: cluster 2024-04-06T03:53:03.913755+0000 osd.0 (osd.0) 758 : cluster 0 3.1b scrub starts 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[29221]: cluster 2024-04-06T03:53:03.917773+0000 osd.0 (osd.0) 759 : cluster 0 3.1b scrub ok 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[29221]: cluster 2024-04-06T03:53:04.293417+0000 osd.5 (osd.5) 225 : cluster 0 6.7 scrub starts 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[29221]: cluster 2024-04-06T03:53:04.467063+0000 osd.5 (osd.5) 226 : cluster 0 6.7 scrub ok 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[29221]: cluster 2024-04-06T03:53:04.732183+0000 mgr.y (mgr.24368) 1177 : cluster 0 pgmap v1291: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-06T03:53:05.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:05 smithi050 bash[29221]: audit 2024-04-06T03:53:04.944862+0000 mgr.y (mgr.24368) 1178 : audit 0 from='client.25823 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-06T03:53:05.837 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1291,"stamp":"2024-04-06T03:53:04.731653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":4401,"num_read_kb":4215,"num_write":1665,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":4736933,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1907,"ondisk_log_size":1907,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4515676,"kb_used_data":8348,"kb_used_omap":12,"kb_used_meta":4507251,"kb_avail":745445540,"statfs":{"total":767960285184,"available":763336232960,"internally_reserved":0,"allocated":8548352,"data_stored":5619970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4615425585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004801"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:56.845124+0000","last_change":"2024-04-06T03:52:56.845154+0000","last_active":"2024-04-06T03:52:56.845124+0000","last_peered":"2024-04-06T03:52:56.845124+0000","last_clean":"2024-04-06T03:52:56.845124+0000","last_became_active":"2024-04-06T03:42:23.588459+0000","last_became_peered":"2024-04-06T03:42:23.588459+0000","last_unstale":"2024-04-06T03:52:56.845124+0000","last_undegraded":"2024-04-06T03:52:56.845124+0000","last_fullsized":"2024-04-06T03:52:56.845124+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:56.845030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:33.261299+0000","last_clean_scrub_stamp":"2024-04-06T03:52:56.845030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:22.131439+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":892,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:45.148942+0000","last_change":"2024-04-06T03:52:45.148998+0000","last_active":"2024-04-06T03:52:45.148942+0000","last_peered":"2024-04-06T03:52:45.148942+0000","last_clean":"2024-04-06T03:52:45.148942+0000","last_became_active":"2024-04-06T03:48:21.308873+0000","last_became_peered":"2024-04-06T03:48:21.308873+0000","last_unstale":"2024-04-06T03:52:45.148942+0000","last_undegraded":"2024-04-06T03:52:45.148942+0000","last_fullsized":"2024-04-06T03:52:45.148942+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:45.148794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:11.395397+0000","last_clean_scrub_stamp":"2024-04-06T03:52:45.148794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:10.335782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":133,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:53.889428+0000","last_change":"2024-04-06T03:52:53.889476+0000","last_active":"2024-04-06T03:52:53.889428+0000","last_peered":"2024-04-06T03:52:53.889428+0000","last_clean":"2024-04-06T03:52:53.889428+0000","last_became_active":"2024-04-06T03:48:32.447661+0000","last_became_peered":"2024-04-06T03:48:32.447661+0000","last_unstale":"2024-04-06T03:52:53.889428+0000","last_undegraded":"2024-04-06T03:52:53.889428+0000","last_fullsized":"2024-04-06T03:52:53.889428+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:53.889310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:21.990975+0000","last_clean_scrub_stamp":"2024-04-06T03:52:53.889310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:21.498483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"433'52","reported_seq":993,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:03.645418+0000","last_change":"2024-04-06T03:53:03.645418+0000","last_active":"2024-04-06T03:53:03.645418+0000","last_peered":"2024-04-06T03:53:03.645418+0000","last_clean":"2024-04-06T03:53:03.645418+0000","last_became_active":"2024-04-06T03:42:23.588857+0000","last_became_peered":"2024-04-06T03:42:23.588857+0000","last_unstale":"2024-04-06T03:53:03.645418+0000","last_undegraded":"2024-04-06T03:53:03.645418+0000","last_fullsized":"2024-04-06T03:53:03.645418+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'52","last_scrub_stamp":"2024-04-06T03:53:03.645300+0000","last_deep_scrub":"433'52","last_deep_scrub_stamp":"2024-04-06T03:51:43.335943+0000","last_clean_scrub_stamp":"2024-04-06T03:53:03.645300+0000","objects_scrubbed":6,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:32.838769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":63,"num_write":46,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671368+0000","last_change":"2024-04-06T03:52:18.525011+0000","last_active":"2024-04-06T03:52:42.671368+0000","last_peered":"2024-04-06T03:52:42.671368+0000","last_clean":"2024-04-06T03:52:42.671368+0000","last_became_active":"2024-04-06T03:48:21.296906+0000","last_became_peered":"2024-04-06T03:48:21.296906+0000","last_unstale":"2024-04-06T03:52:42.671368+0000","last_undegraded":"2024-04-06T03:52:42.671368+0000","last_fullsized":"2024-04-06T03:52:42.671368+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:18.524759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:52:18.524759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:45.251770+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":727,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:55.253894+0000","last_change":"2024-04-06T03:52:55.253924+0000","last_active":"2024-04-06T03:52:55.253894+0000","last_peered":"2024-04-06T03:52:55.253894+0000","last_clean":"2024-04-06T03:52:55.253894+0000","last_became_active":"2024-04-06T03:49:27.616011+0000","last_became_peered":"2024-04-06T03:49:27.616011+0000","last_unstale":"2024-04-06T03:52:55.253894+0000","last_undegraded":"2024-04-06T03:52:55.253894+0000","last_fullsized":"2024-04-06T03:52:55.253894+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:55.253808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:28.642157+0000","last_clean_scrub_stamp":"2024-04-06T03:52:55.253808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:16.028263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":894,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.650774+0000","last_change":"2024-04-06T03:52:52.650774+0000","last_active":"2024-04-06T03:52:52.650774+0000","last_peered":"2024-04-06T03:52:52.650774+0000","last_clean":"2024-04-06T03:52:52.650774+0000","last_became_active":"2024-04-06T03:48:32.467675+0000","last_became_peered":"2024-04-06T03:48:32.467675+0000","last_unstale":"2024-04-06T03:52:52.650774+0000","last_undegraded":"2024-04-06T03:52:52.650774+0000","last_fullsized":"2024-04-06T03:52:52.650774+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:52.650552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:52.650552+0000","last_clean_scrub_stamp":"2024-04-06T03:52:52.650552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:09.968970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"433'47","reported_seq":151,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671828+0000","last_change":"2024-04-06T03:51:58.262615+0000","last_active":"2024-04-06T03:52:42.671828+0000","last_peered":"2024-04-06T03:52:42.671828+0000","last_clean":"2024-04-06T03:52:42.671828+0000","last_became_active":"2024-04-06T03:48:32.464269+0000","last_became_peered":"2024-04-06T03:48:32.464269+0000","last_unstale":"2024-04-06T03:52:42.671828+0000","last_undegraded":"2024-04-06T03:52:42.671828+0000","last_fullsized":"2024-04-06T03:52:42.671828+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'47","last_scrub_stamp":"2024-04-06T03:51:58.262487+0000","last_deep_scrub":"199'31","last_deep_scrub_stamp":"2024-04-06T03:39:01.848222+0000","last_clean_scrub_stamp":"2024-04-06T03:51:58.262487+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.209317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":64,"num_read_kb":56,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":189,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.677086+0000","last_change":"2024-04-06T03:52:21.009573+0000","last_active":"2024-04-06T03:52:42.677086+0000","last_peered":"2024-04-06T03:52:42.677086+0000","last_clean":"2024-04-06T03:52:42.677086+0000","last_became_active":"2024-04-06T03:50:47.188362+0000","last_became_peered":"2024-04-06T03:50:47.188362+0000","last_unstale":"2024-04-06T03:52:42.677086+0000","last_undegraded":"2024-04-06T03:52:42.677086+0000","last_fullsized":"2024-04-06T03:52:42.677086+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:21.009411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:11.124952+0000","last_clean_scrub_stamp":"2024-04-06T03:52:21.009411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:42.768587+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183256+0000","last_change":"2024-04-06T03:51:53.594917+0000","last_active":"2024-04-06T03:52:42.183256+0000","last_peered":"2024-04-06T03:52:42.183256+0000","last_clean":"2024-04-06T03:52:42.183256+0000","last_became_active":"2024-04-06T03:48:21.326952+0000","last_became_peered":"2024-04-06T03:48:21.326952+0000","last_unstale":"2024-04-06T03:52:42.183256+0000","last_undegraded":"2024-04-06T03:52:42.183256+0000","last_fullsized":"2024-04-06T03:52:42.183256+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:53.594698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:25.274778+0000","last_clean_scrub_stamp":"2024-04-06T03:51:53.594698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:06.426677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"78'1","reported_seq":929,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.622674+0000","last_change":"2024-04-06T03:52:52.622750+0000","last_active":"2024-04-06T03:52:52.622674+0000","last_peered":"2024-04-06T03:52:52.622674+0000","last_clean":"2024-04-06T03:52:52.622674+0000","last_became_active":"2024-04-06T03:48:32.436036+0000","last_became_peered":"2024-04-06T03:48:32.436036+0000","last_unstale":"2024-04-06T03:52:52.622674+0000","last_undegraded":"2024-04-06T03:52:52.622674+0000","last_fullsized":"2024-04-06T03:52:52.622674+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'1","last_scrub_stamp":"2024-04-06T03:52:52.622511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:52.622511+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:53.418332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"433'58","reported_seq":641,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.677033+0000","last_change":"2024-04-06T03:52:31.983093+0000","last_active":"2024-04-06T03:52:42.677033+0000","last_peered":"2024-04-06T03:52:42.677033+0000","last_clean":"2024-04-06T03:52:42.677033+0000","last_became_active":"2024-04-06T03:50:47.196236+0000","last_became_peered":"2024-04-06T03:50:47.196236+0000","last_unstale":"2024-04-06T03:52:42.677033+0000","last_undegraded":"2024-04-06T03:52:42.677033+0000","last_fullsized":"2024-04-06T03:52:42.677033+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'58","last_scrub_stamp":"2024-04-06T03:52:31.982714+0000","last_deep_scrub":"112'22","last_deep_scrub_stamp":"2024-04-06T03:26:40.864830+0000","last_clean_scrub_stamp":"2024-04-06T03:52:31.982714+0000","objects_scrubbed":7,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:48.140404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":70,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":132,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669803+0000","last_change":"2024-04-06T03:52:34.107753+0000","last_active":"2024-04-06T03:52:42.669803+0000","last_peered":"2024-04-06T03:52:42.669803+0000","last_clean":"2024-04-06T03:52:42.669803+0000","last_became_active":"2024-04-06T03:48:32.458724+0000","last_became_peered":"2024-04-06T03:48:32.458724+0000","last_unstale":"2024-04-06T03:52:42.669803+0000","last_undegraded":"2024-04-06T03:52:42.669803+0000","last_fullsized":"2024-04-06T03:52:42.669803+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:34.107597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:55.288085+0000","last_clean_scrub_stamp":"2024-04-06T03:52:34.107597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:40.005227+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":184,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670880+0000","last_change":"2024-04-06T03:52:05.923917+0000","last_active":"2024-04-06T03:52:42.670880+0000","last_peered":"2024-04-06T03:52:42.670880+0000","last_clean":"2024-04-06T03:52:42.670880+0000","last_became_active":"2024-04-06T03:50:47.201333+0000","last_became_peered":"2024-04-06T03:50:47.201333+0000","last_unstale":"2024-04-06T03:52:42.670880+0000","last_undegraded":"2024-04-06T03:52:42.670880+0000","last_fullsized":"2024-04-06T03:52:42.670880+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:05.923705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:47:24.939189+0000","last_clean_scrub_stamp":"2024-04-06T03:52:05.923705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:31.238147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":873,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183499+0000","last_change":"2024-04-06T03:52:30.650730+0000","last_active":"2024-04-06T03:52:42.183499+0000","last_peered":"2024-04-06T03:52:42.183499+0000","last_clean":"2024-04-06T03:52:42.183499+0000","last_became_active":"2024-04-06T03:20:51.145853+0000","last_became_peered":"2024-04-06T03:20:51.145853+0000","last_unstale":"2024-04-06T03:52:42.183499+0000","last_undegraded":"2024-04-06T03:52:42.183499+0000","last_fullsized":"2024-04-06T03:52:42.183499+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:30.650506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:28.609910+0000","last_clean_scrub_stamp":"2024-04-06T03:52:30.650506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:36.321214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"433'42","reported_seq":593,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:46.168435+0000","last_change":"2024-04-06T03:52:46.168521+0000","last_active":"2024-04-06T03:52:46.168435+0000","last_peered":"2024-04-06T03:52:46.168435+0000","last_clean":"2024-04-06T03:52:46.168435+0000","last_became_active":"2024-04-06T03:50:47.210996+0000","last_became_peered":"2024-04-06T03:50:47.210996+0000","last_unstale":"2024-04-06T03:52:46.168435+0000","last_undegraded":"2024-04-06T03:52:46.168435+0000","last_fullsized":"2024-04-06T03:52:46.168435+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'42","last_scrub_stamp":"2024-04-06T03:52:46.168285+0000","last_deep_scrub":"105'24","last_deep_scrub_stamp":"2024-04-06T03:22:21.783165+0000","last_clean_scrub_stamp":"2024-04-06T03:52:46.168285+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:47.603652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227934+0000","last_change":"2024-04-06T03:52:08.753656+0000","last_active":"2024-04-06T03:52:42.227934+0000","last_peered":"2024-04-06T03:52:42.227934+0000","last_clean":"2024-04-06T03:52:42.227934+0000","last_became_active":"2024-04-06T03:42:23.577258+0000","last_became_peered":"2024-04-06T03:42:23.577258+0000","last_unstale":"2024-04-06T03:52:42.227934+0000","last_undegraded":"2024-04-06T03:52:42.227934+0000","last_fullsized":"2024-04-06T03:52:42.227934+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:08.753549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:23.012827+0000","last_clean_scrub_stamp":"2024-04-06T03:52:08.753549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:25.835619+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671950+0000","last_change":"2024-04-06T03:52:29.268192+0000","last_active":"2024-04-06T03:52:42.671950+0000","last_peered":"2024-04-06T03:52:42.671950+0000","last_clean":"2024-04-06T03:52:42.671950+0000","last_became_active":"2024-04-06T03:20:55.761632+0000","last_became_peered":"2024-04-06T03:20:55.761632+0000","last_unstale":"2024-04-06T03:52:42.671950+0000","last_undegraded":"2024-04-06T03:52:42.671950+0000","last_fullsized":"2024-04-06T03:52:42.671950+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:29.267905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:39.155872+0000","last_clean_scrub_stamp":"2024-04-06T03:52:29.267905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:32.835359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":871,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182155+0000","last_change":"2024-04-06T03:51:41.334777+0000","last_active":"2024-04-06T03:52:42.182155+0000","last_peered":"2024-04-06T03:52:42.182155+0000","last_clean":"2024-04-06T03:52:42.182155+0000","last_became_active":"2024-04-06T03:42:23.588672+0000","last_became_peered":"2024-04-06T03:42:23.588672+0000","last_unstale":"2024-04-06T03:52:42.182155+0000","last_undegraded":"2024-04-06T03:52:42.182155+0000","last_fullsized":"2024-04-06T03:52:42.182155+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:41.334631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:41.334631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:59.164585+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"433'45","reported_seq":1012,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671827+0000","last_change":"2024-04-06T03:52:19.411643+0000","last_active":"2024-04-06T03:52:42.671827+0000","last_peered":"2024-04-06T03:52:42.671827+0000","last_clean":"2024-04-06T03:52:42.671827+0000","last_became_active":"2024-04-06T03:48:21.316917+0000","last_became_peered":"2024-04-06T03:48:21.316917+0000","last_unstale":"2024-04-06T03:52:42.671827+0000","last_undegraded":"2024-04-06T03:52:42.671827+0000","last_fullsized":"2024-04-06T03:52:42.671827+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'45","last_scrub_stamp":"2024-04-06T03:52:19.411319+0000","last_deep_scrub":"199'33","last_deep_scrub_stamp":"2024-04-06T03:38:57.058408+0000","last_clean_scrub_stamp":"2024-04-06T03:52:19.411319+0000","objects_scrubbed":6,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:22.874698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":62,"num_read_kb":54,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673625+0000","last_change":"2024-04-06T03:52:30.295661+0000","last_active":"2024-04-06T03:52:42.673625+0000","last_peered":"2024-04-06T03:52:42.673625+0000","last_clean":"2024-04-06T03:52:42.673625+0000","last_became_active":"2024-04-06T03:42:23.594112+0000","last_became_peered":"2024-04-06T03:42:23.594112+0000","last_unstale":"2024-04-06T03:52:42.673625+0000","last_undegraded":"2024-04-06T03:52:42.673625+0000","last_fullsized":"2024-04-06T03:52:42.673625+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:30.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:30.295451+0000","last_clean_scrub_stamp":"2024-04-06T03:52:30.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:58.323225+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670250+0000","last_change":"2024-04-06T03:52:06.534507+0000","last_active":"2024-04-06T03:52:42.670250+0000","last_peered":"2024-04-06T03:52:42.670250+0000","last_clean":"2024-04-06T03:52:42.670250+0000","last_became_active":"2024-04-06T03:48:32.469910+0000","last_became_peered":"2024-04-06T03:48:32.469910+0000","last_unstale":"2024-04-06T03:52:42.670250+0000","last_undegraded":"2024-04-06T03:52:42.670250+0000","last_fullsized":"2024-04-06T03:52:42.670250+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:06.534157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:50.181452+0000","last_clean_scrub_stamp":"2024-04-06T03:52:06.534157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:24.217203+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":497,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673797+0000","last_change":"2024-04-06T03:51:56.823283+0000","last_active":"2024-04-06T03:52:42.673797+0000","last_peered":"2024-04-06T03:52:42.673797+0000","last_clean":"2024-04-06T03:52:42.673797+0000","last_became_active":"2024-04-06T03:51:06.727485+0000","last_became_peered":"2024-04-06T03:51:06.727485+0000","last_unstale":"2024-04-06T03:52:42.673797+0000","last_undegraded":"2024-04-06T03:52:42.673797+0000","last_fullsized":"2024-04-06T03:52:42.673797+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:56.823070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:56.823070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:13.269898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"433'20","reported_seq":143,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:03.917957+0000","last_change":"2024-04-06T03:53:03.918047+0000","last_active":"2024-04-06T03:53:03.917957+0000","last_peered":"2024-04-06T03:53:03.917957+0000","last_clean":"2024-04-06T03:53:03.917957+0000","last_became_active":"2024-04-06T03:48:32.448235+0000","last_became_peered":"2024-04-06T03:48:32.448235+0000","last_unstale":"2024-04-06T03:53:03.917957+0000","last_undegraded":"2024-04-06T03:53:03.917957+0000","last_fullsized":"2024-04-06T03:53:03.917957+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'20","last_scrub_stamp":"2024-04-06T03:53:03.917787+0000","last_deep_scrub":"224'16","last_deep_scrub_stamp":"2024-04-06T03:33:51.165059+0000","last_clean_scrub_stamp":"2024-04-06T03:53:03.917787+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:21.704799+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":717,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669756+0000","last_change":"2024-04-06T03:52:14.571480+0000","last_active":"2024-04-06T03:52:42.669756+0000","last_peered":"2024-04-06T03:52:42.669756+0000","last_clean":"2024-04-06T03:52:42.669756+0000","last_became_active":"2024-04-06T03:49:27.620657+0000","last_became_peered":"2024-04-06T03:49:27.620657+0000","last_unstale":"2024-04-06T03:52:42.669756+0000","last_undegraded":"2024-04-06T03:52:42.669756+0000","last_fullsized":"2024-04-06T03:52:42.669756+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:14.571351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:52:14.571351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:35.447873+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":125,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671601+0000","last_change":"2024-04-06T03:52:12.201535+0000","last_active":"2024-04-06T03:52:42.671601+0000","last_peered":"2024-04-06T03:52:42.671601+0000","last_clean":"2024-04-06T03:52:42.671601+0000","last_became_active":"2024-04-06T03:48:32.463740+0000","last_became_peered":"2024-04-06T03:48:32.463740+0000","last_unstale":"2024-04-06T03:52:42.671601+0000","last_undegraded":"2024-04-06T03:52:42.671601+0000","last_fullsized":"2024-04-06T03:52:42.671601+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:12.201328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:26.134284+0000","last_clean_scrub_stamp":"2024-04-06T03:52:12.201328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:32.720165+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"414'2","reported_seq":910,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181299+0000","last_change":"2024-04-06T03:52:11.507399+0000","last_active":"2024-04-06T03:52:42.181299+0000","last_peered":"2024-04-06T03:52:42.181299+0000","last_clean":"2024-04-06T03:52:42.181299+0000","last_became_active":"2024-04-06T03:48:32.467749+0000","last_became_peered":"2024-04-06T03:48:32.467749+0000","last_unstale":"2024-04-06T03:52:42.181299+0000","last_undegraded":"2024-04-06T03:52:42.181299+0000","last_fullsized":"2024-04-06T03:52:42.181299+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'2","last_scrub_stamp":"2024-04-06T03:52:11.507256+0000","last_deep_scrub":"414'2","last_deep_scrub_stamp":"2024-04-06T03:51:06.320142+0000","last_clean_scrub_stamp":"2024-04-06T03:52:11.507256+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:16.419280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"433'38","reported_seq":996,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:54.610544+0000","last_change":"2024-04-06T03:52:54.610633+0000","last_active":"2024-04-06T03:52:54.610544+0000","last_peered":"2024-04-06T03:52:54.610544+0000","last_clean":"2024-04-06T03:52:54.610544+0000","last_became_active":"2024-04-06T03:48:32.466273+0000","last_became_peered":"2024-04-06T03:48:32.466273+0000","last_unstale":"2024-04-06T03:52:54.610544+0000","last_undegraded":"2024-04-06T03:52:54.610544+0000","last_fullsized":"2024-04-06T03:52:54.610544+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'38","last_scrub_stamp":"2024-04-06T03:52:54.610373+0000","last_deep_scrub":"433'38","last_deep_scrub_stamp":"2024-04-06T03:52:54.610373+0000","last_clean_scrub_stamp":"2024-04-06T03:52:54.610373+0000","objects_scrubbed":5,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:04.645889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"207'2","reported_seq":864,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.185491+0000","last_change":"2024-04-06T03:52:29.670766+0000","last_active":"2024-04-06T03:52:42.185491+0000","last_peered":"2024-04-06T03:52:42.185491+0000","last_clean":"2024-04-06T03:52:42.185491+0000","last_became_active":"2024-04-06T03:20:57.844201+0000","last_became_peered":"2024-04-06T03:20:57.844201+0000","last_unstale":"2024-04-06T03:52:42.185491+0000","last_undegraded":"2024-04-06T03:52:42.185491+0000","last_fullsized":"2024-04-06T03:52:42.185491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'2","last_scrub_stamp":"2024-04-06T03:52:29.670537+0000","last_deep_scrub":"207'2","last_deep_scrub_stamp":"2024-04-06T03:41:29.127643+0000","last_clean_scrub_stamp":"2024-04-06T03:52:29.670537+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:57.455009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"436'16","reported_seq":803,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705459+0000","last_change":"2024-04-06T03:52:56.798393+0000","last_active":"2024-04-06T03:52:57.705459+0000","last_peered":"2024-04-06T03:52:57.705459+0000","last_clean":"2024-04-06T03:52:57.705459+0000","last_became_active":"2024-04-06T03:51:06.732897+0000","last_became_peered":"2024-04-06T03:51:06.732897+0000","last_unstale":"2024-04-06T03:52:57.705459+0000","last_undegraded":"2024-04-06T03:52:57.705459+0000","last_fullsized":"2024-04-06T03:52:57.705459+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'16","last_scrub_stamp":"2024-04-06T03:52:56.798169+0000","last_deep_scrub":"327'15","last_deep_scrub_stamp":"2024-04-06T03:50:22.454509+0000","last_clean_scrub_stamp":"2024-04-06T03:52:56.798169+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:01.839018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":201,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670524+0000","last_change":"2024-04-06T03:51:52.902749+0000","last_active":"2024-04-06T03:52:42.670524+0000","last_peered":"2024-04-06T03:52:42.670524+0000","last_clean":"2024-04-06T03:52:42.670524+0000","last_became_active":"2024-04-06T03:50:47.206404+0000","last_became_peered":"2024-04-06T03:50:47.206404+0000","last_unstale":"2024-04-06T03:52:42.670524+0000","last_undegraded":"2024-04-06T03:52:42.670524+0000","last_fullsized":"2024-04-06T03:52:42.670524+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:52.902538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:15.156474+0000","last_clean_scrub_stamp":"2024-04-06T03:51:52.902538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:14.649304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"433'86","reported_seq":188,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669855+0000","last_change":"2024-04-06T03:52:38.418870+0000","last_active":"2024-04-06T03:52:42.669855+0000","last_peered":"2024-04-06T03:52:42.669855+0000","last_clean":"2024-04-06T03:52:42.669855+0000","last_became_active":"2024-04-06T03:48:21.323684+0000","last_became_peered":"2024-04-06T03:48:21.323684+0000","last_unstale":"2024-04-06T03:52:42.669855+0000","last_undegraded":"2024-04-06T03:52:42.669855+0000","last_fullsized":"2024-04-06T03:52:42.669855+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'86","last_scrub_stamp":"2024-04-06T03:52:38.418664+0000","last_deep_scrub":"111'50","last_deep_scrub_stamp":"2024-04-06T03:28:28.306994+0000","last_clean_scrub_stamp":"2024-04-06T03:52:38.418664+0000","objects_scrubbed":11,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:40.129651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":115,"num_read_kb":102,"num_write":76,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":880,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227457+0000","last_change":"2024-04-06T03:52:22.571818+0000","last_active":"2024-04-06T03:52:42.227457+0000","last_peered":"2024-04-06T03:52:42.227457+0000","last_clean":"2024-04-06T03:52:42.227457+0000","last_became_active":"2024-04-06T03:48:32.466118+0000","last_became_peered":"2024-04-06T03:48:32.466118+0000","last_unstale":"2024-04-06T03:52:42.227457+0000","last_undegraded":"2024-04-06T03:52:42.227457+0000","last_fullsized":"2024-04-06T03:52:42.227457+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:22.571589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:47.788083+0000","last_clean_scrub_stamp":"2024-04-06T03:52:22.571589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:51.789866+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183508+0000","last_change":"2024-04-06T03:52:23.509135+0000","last_active":"2024-04-06T03:52:42.183508+0000","last_peered":"2024-04-06T03:52:42.183508+0000","last_clean":"2024-04-06T03:52:42.183508+0000","last_became_active":"2024-04-06T03:42:23.577555+0000","last_became_peered":"2024-04-06T03:42:23.577555+0000","last_unstale":"2024-04-06T03:52:42.183508+0000","last_undegraded":"2024-04-06T03:52:42.183508+0000","last_fullsized":"2024-04-06T03:52:42.183508+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:23.508929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:35.558625+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.508929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:51.057458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":523,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678119+0000","last_change":"2024-04-06T03:52:33.854635+0000","last_active":"2024-04-06T03:52:42.678119+0000","last_peered":"2024-04-06T03:52:42.678119+0000","last_clean":"2024-04-06T03:52:42.678119+0000","last_became_active":"2024-04-06T03:51:06.720828+0000","last_became_peered":"2024-04-06T03:51:06.720828+0000","last_unstale":"2024-04-06T03:52:42.678119+0000","last_undegraded":"2024-04-06T03:52:42.678119+0000","last_fullsized":"2024-04-06T03:52:42.678119+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:33.854432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:49.470404+0000","last_clean_scrub_stamp":"2024-04-06T03:52:33.854432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:38.651018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"433'45","reported_seq":662,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675697+0000","last_change":"2024-04-06T03:52:02.843565+0000","last_active":"2024-04-06T03:52:42.675697+0000","last_peered":"2024-04-06T03:52:42.675697+0000","last_clean":"2024-04-06T03:52:42.675697+0000","last_became_active":"2024-04-06T03:50:47.200759+0000","last_became_peered":"2024-04-06T03:50:47.200759+0000","last_unstale":"2024-04-06T03:52:42.675697+0000","last_undegraded":"2024-04-06T03:52:42.675697+0000","last_fullsized":"2024-04-06T03:52:42.675697+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'45","last_scrub_stamp":"2024-04-06T03:52:02.843317+0000","last_deep_scrub":"344'39","last_deep_scrub_stamp":"2024-04-06T03:43:22.294366+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.843317+0000","objects_scrubbed":6,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:09.007353+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":61,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":870,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183416+0000","last_change":"2024-04-06T03:52:34.699039+0000","last_active":"2024-04-06T03:52:42.183416+0000","last_peered":"2024-04-06T03:52:42.183416+0000","last_clean":"2024-04-06T03:52:42.183416+0000","last_became_active":"2024-04-06T03:20:57.840691+0000","last_became_peered":"2024-04-06T03:20:57.840691+0000","last_unstale":"2024-04-06T03:52:42.183416+0000","last_undegraded":"2024-04-06T03:52:42.183416+0000","last_fullsized":"2024-04-06T03:52:42.183416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:34.698755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:01.484422+0000","last_clean_scrub_stamp":"2024-04-06T03:52:34.698755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:44.344379+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":506,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673611+0000","last_change":"2024-04-06T03:52:36.863272+0000","last_active":"2024-04-06T03:52:42.673611+0000","last_peered":"2024-04-06T03:52:42.673611+0000","last_clean":"2024-04-06T03:52:42.673611+0000","last_became_active":"2024-04-06T03:51:06.732712+0000","last_became_peered":"2024-04-06T03:51:06.732712+0000","last_unstale":"2024-04-06T03:52:42.673611+0000","last_undegraded":"2024-04-06T03:52:42.673611+0000","last_fullsized":"2024-04-06T03:52:42.673611+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:36.863051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:36.863051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:39.902161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":526,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675976+0000","last_change":"2024-04-06T03:51:45.027088+0000","last_active":"2024-04-06T03:52:42.675976+0000","last_peered":"2024-04-06T03:52:42.675976+0000","last_clean":"2024-04-06T03:52:42.675976+0000","last_became_active":"2024-04-06T03:50:47.205196+0000","last_became_peered":"2024-04-06T03:50:47.205196+0000","last_unstale":"2024-04-06T03:52:42.675976+0000","last_undegraded":"2024-04-06T03:52:42.675976+0000","last_fullsized":"2024-04-06T03:52:42.675976+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:45.026839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:45.026839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:07.088793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"433'34","reported_seq":151,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:44.005717+0000","last_change":"2024-04-06T03:52:44.005774+0000","last_active":"2024-04-06T03:52:44.005717+0000","last_peered":"2024-04-06T03:52:44.005717+0000","last_clean":"2024-04-06T03:52:44.005717+0000","last_became_active":"2024-04-06T03:48:32.462539+0000","last_became_peered":"2024-04-06T03:48:32.462539+0000","last_unstale":"2024-04-06T03:52:44.005717+0000","last_undegraded":"2024-04-06T03:52:44.005717+0000","last_fullsized":"2024-04-06T03:52:44.005717+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'34","last_scrub_stamp":"2024-04-06T03:52:44.005638+0000","last_deep_scrub":"433'34","last_deep_scrub_stamp":"2024-04-06T03:51:25.163875+0000","last_clean_scrub_stamp":"2024-04-06T03:52:44.005638+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:09.604521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":868,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181304+0000","last_change":"2024-04-06T03:52:37.503026+0000","last_active":"2024-04-06T03:52:42.181304+0000","last_peered":"2024-04-06T03:52:42.181304+0000","last_clean":"2024-04-06T03:52:42.181304+0000","last_became_active":"2024-04-06T03:48:32.464518+0000","last_became_peered":"2024-04-06T03:48:32.464518+0000","last_unstale":"2024-04-06T03:52:42.181304+0000","last_undegraded":"2024-04-06T03:52:42.181304+0000","last_fullsized":"2024-04-06T03:52:42.181304+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:37.502797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:55.881161+0000","last_clean_scrub_stamp":"2024-04-06T03:52:37.502797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:46.923594+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.308643+0000","last_change":"2024-04-06T03:52:47.308643+0000","last_active":"2024-04-06T03:52:47.308643+0000","last_peered":"2024-04-06T03:52:47.308643+0000","last_clean":"2024-04-06T03:52:47.308643+0000","last_became_active":"2024-04-06T03:48:21.324233+0000","last_became_peered":"2024-04-06T03:48:21.324233+0000","last_unstale":"2024-04-06T03:52:47.308643+0000","last_undegraded":"2024-04-06T03:52:47.308643+0000","last_fullsized":"2024-04-06T03:52:47.308643+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:47.308312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:47.308312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:15.194624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:50.327666+0000","last_change":"2024-04-06T03:52:50.327722+0000","last_active":"2024-04-06T03:52:50.327666+0000","last_peered":"2024-04-06T03:52:50.327666+0000","last_clean":"2024-04-06T03:52:50.327666+0000","last_became_active":"2024-04-06T03:48:32.470368+0000","last_became_peered":"2024-04-06T03:48:32.470368+0000","last_unstale":"2024-04-06T03:52:50.327666+0000","last_undegraded":"2024-04-06T03:52:50.327666+0000","last_fullsized":"2024-04-06T03:52:50.327666+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:50.327516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:30.204990+0000","last_clean_scrub_stamp":"2024-04-06T03:52:50.327516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:13.023613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"433'60","reported_seq":1016,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671608+0000","last_change":"2024-04-06T03:52:03.513260+0000","last_active":"2024-04-06T03:52:42.671608+0000","last_peered":"2024-04-06T03:52:42.671608+0000","last_clean":"2024-04-06T03:52:42.671608+0000","last_became_active":"2024-04-06T03:42:23.592079+0000","last_became_peered":"2024-04-06T03:42:23.592079+0000","last_unstale":"2024-04-06T03:52:42.671608+0000","last_undegraded":"2024-04-06T03:52:42.671608+0000","last_fullsized":"2024-04-06T03:52:42.671608+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'60","last_scrub_stamp":"2024-04-06T03:52:03.513006+0000","last_deep_scrub":"430'50","last_deep_scrub_stamp":"2024-04-06T03:50:46.333320+0000","last_clean_scrub_stamp":"2024-04-06T03:52:03.513006+0000","objects_scrubbed":7,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.010964+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":78,"num_read_kb":71,"num_write":52,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":124,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669437+0000","last_change":"2024-04-06T03:51:49.257334+0000","last_active":"2024-04-06T03:52:42.669437+0000","last_peered":"2024-04-06T03:52:42.669437+0000","last_clean":"2024-04-06T03:52:42.669437+0000","last_became_active":"2024-04-06T03:48:32.461335+0000","last_became_peered":"2024-04-06T03:48:32.461335+0000","last_unstale":"2024-04-06T03:52:42.669437+0000","last_undegraded":"2024-04-06T03:52:42.669437+0000","last_fullsized":"2024-04-06T03:52:42.669437+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:49.257168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:45.956977+0000","last_clean_scrub_stamp":"2024-04-06T03:51:49.257168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:10.434059+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":884,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182448+0000","last_change":"2024-04-06T03:52:20.437248+0000","last_active":"2024-04-06T03:52:42.182448+0000","last_peered":"2024-04-06T03:52:42.182448+0000","last_clean":"2024-04-06T03:52:42.182448+0000","last_became_active":"2024-04-06T03:48:32.469863+0000","last_became_peered":"2024-04-06T03:48:32.469863+0000","last_unstale":"2024-04-06T03:52:42.182448+0000","last_undegraded":"2024-04-06T03:52:42.182448+0000","last_fullsized":"2024-04-06T03:52:42.182448+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:20.436937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:33.624959+0000","last_clean_scrub_stamp":"2024-04-06T03:52:20.436937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:44.571761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":497,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.674839+0000","last_change":"2024-04-06T03:52:40.861328+0000","last_active":"2024-04-06T03:52:42.674839+0000","last_peered":"2024-04-06T03:52:42.674839+0000","last_clean":"2024-04-06T03:52:42.674839+0000","last_became_active":"2024-04-06T03:51:07.366122+0000","last_became_peered":"2024-04-06T03:51:07.366122+0000","last_unstale":"2024-04-06T03:52:42.674839+0000","last_undegraded":"2024-04-06T03:52:42.674839+0000","last_fullsized":"2024-04-06T03:52:42.674839+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:40.861105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:30.734198+0000","last_clean_scrub_stamp":"2024-04-06T03:52:40.861105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:41.086066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"433'38","reported_seq":930,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184634+0000","last_change":"2024-04-06T03:52:28.638446+0000","last_active":"2024-04-06T03:52:42.184634+0000","last_peered":"2024-04-06T03:52:42.184634+0000","last_clean":"2024-04-06T03:52:42.184634+0000","last_became_active":"2024-04-06T03:20:53.350984+0000","last_became_peered":"2024-04-06T03:20:53.350984+0000","last_unstale":"2024-04-06T03:52:42.184634+0000","last_undegraded":"2024-04-06T03:52:42.184634+0000","last_fullsized":"2024-04-06T03:52:42.184634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'38","last_scrub_stamp":"2024-04-06T03:52:28.638198+0000","last_deep_scrub":"432'32","last_deep_scrub_stamp":"2024-04-06T03:51:00.500705+0000","last_clean_scrub_stamp":"2024-04-06T03:52:28.638198+0000","objects_scrubbed":5,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:49.035812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":872,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673516+0000","last_change":"2024-04-06T03:52:33.312479+0000","last_active":"2024-04-06T03:52:42.673516+0000","last_peered":"2024-04-06T03:52:42.673516+0000","last_clean":"2024-04-06T03:52:42.673516+0000","last_became_active":"2024-04-06T03:20:57.812683+0000","last_became_peered":"2024-04-06T03:20:57.812683+0000","last_unstale":"2024-04-06T03:52:42.673516+0000","last_undegraded":"2024-04-06T03:52:42.673516+0000","last_fullsized":"2024-04-06T03:52:42.673516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:33.312337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:33.312337+0000","last_clean_scrub_stamp":"2024-04-06T03:52:33.312337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:47.571104+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"109'12","reported_seq":1258,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705705+0000","last_change":"2024-04-06T03:51:51.419871+0000","last_active":"2024-04-06T03:52:57.705705+0000","last_peered":"2024-04-06T03:52:57.705705+0000","last_clean":"2024-04-06T03:52:57.705705+0000","last_became_active":"2024-04-06T03:20:55.757193+0000","last_became_peered":"2024-04-06T03:20:55.757193+0000","last_unstale":"2024-04-06T03:52:57.705705+0000","last_undegraded":"2024-04-06T03:52:57.705705+0000","last_fullsized":"2024-04-06T03:52:57.705705+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-06T03:51:51.419749+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:50:20.079528+0000","last_clean_scrub_stamp":"2024-04-06T03:51:51.419749+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:09.576512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":124,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671333+0000","last_change":"2024-04-06T03:52:00.217593+0000","last_active":"2024-04-06T03:52:42.671333+0000","last_peered":"2024-04-06T03:52:42.671333+0000","last_clean":"2024-04-06T03:52:42.671333+0000","last_became_active":"2024-04-06T03:48:32.454942+0000","last_became_peered":"2024-04-06T03:48:32.454942+0000","last_unstale":"2024-04-06T03:52:42.671333+0000","last_undegraded":"2024-04-06T03:52:42.671333+0000","last_fullsized":"2024-04-06T03:52:42.671333+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:00.217476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:00.217476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.375475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"433'31","reported_seq":143,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669436+0000","last_change":"2024-04-06T03:52:14.131244+0000","last_active":"2024-04-06T03:52:42.669436+0000","last_peered":"2024-04-06T03:52:42.669436+0000","last_clean":"2024-04-06T03:52:42.669436+0000","last_became_active":"2024-04-06T03:48:32.452795+0000","last_became_peered":"2024-04-06T03:48:32.452795+0000","last_unstale":"2024-04-06T03:52:42.669436+0000","last_undegraded":"2024-04-06T03:52:42.669436+0000","last_fullsized":"2024-04-06T03:52:42.669436+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'31","last_scrub_stamp":"2024-04-06T03:52:14.130995+0000","last_deep_scrub":"433'31","last_deep_scrub_stamp":"2024-04-06T03:52:14.130995+0000","last_clean_scrub_stamp":"2024-04-06T03:52:14.130995+0000","objects_scrubbed":4,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:20.533112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":38,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":124,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671516+0000","last_change":"2024-04-06T03:52:01.296078+0000","last_active":"2024-04-06T03:52:42.671516+0000","last_peered":"2024-04-06T03:52:42.671516+0000","last_clean":"2024-04-06T03:52:42.671516+0000","last_became_active":"2024-04-06T03:48:32.452785+0000","last_became_peered":"2024-04-06T03:48:32.452785+0000","last_unstale":"2024-04-06T03:52:42.671516+0000","last_undegraded":"2024-04-06T03:52:42.671516+0000","last_fullsized":"2024-04-06T03:52:42.671516+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:01.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:28.067970+0000","last_clean_scrub_stamp":"2024-04-06T03:52:01.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:05.902219+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"431'20","reported_seq":666,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705467+0000","last_change":"2024-04-06T03:52:50.141077+0000","last_active":"2024-04-06T03:52:57.705467+0000","last_peered":"2024-04-06T03:52:57.705467+0000","last_clean":"2024-04-06T03:52:57.705467+0000","last_became_active":"2024-04-06T03:50:47.201652+0000","last_became_peered":"2024-04-06T03:50:47.201652+0000","last_unstale":"2024-04-06T03:52:57.705467+0000","last_undegraded":"2024-04-06T03:52:57.705467+0000","last_fullsized":"2024-04-06T03:52:57.705467+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'20","last_scrub_stamp":"2024-04-06T03:52:50.140987+0000","last_deep_scrub":"431'20","last_deep_scrub_stamp":"2024-04-06T03:51:44.005147+0000","last_clean_scrub_stamp":"2024-04-06T03:52:50.140987+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:14.352697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":203,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675185+0000","last_change":"2024-04-06T03:52:11.954060+0000","last_active":"2024-04-06T03:52:42.675185+0000","last_peered":"2024-04-06T03:52:42.675185+0000","last_clean":"2024-04-06T03:52:42.675185+0000","last_became_active":"2024-04-06T03:50:47.195653+0000","last_became_peered":"2024-04-06T03:50:47.195653+0000","last_unstale":"2024-04-06T03:52:42.675185+0000","last_undegraded":"2024-04-06T03:52:42.675185+0000","last_fullsized":"2024-04-06T03:52:42.675185+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:11.953851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:21.406614+0000","last_clean_scrub_stamp":"2024-04-06T03:52:11.953851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:12.115475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"433'40","reported_seq":930,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:48.666469+0000","last_change":"2024-04-06T03:52:48.666549+0000","last_active":"2024-04-06T03:52:48.666469+0000","last_peered":"2024-04-06T03:52:48.666469+0000","last_clean":"2024-04-06T03:52:48.666469+0000","last_became_active":"2024-04-06T03:20:53.350062+0000","last_became_peered":"2024-04-06T03:20:53.350062+0000","last_unstale":"2024-04-06T03:52:48.666469+0000","last_undegraded":"2024-04-06T03:52:48.666469+0000","last_fullsized":"2024-04-06T03:52:48.666469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'40","last_scrub_stamp":"2024-04-06T03:52:48.666309+0000","last_deep_scrub":"199'24","last_deep_scrub_stamp":"2024-04-06T03:37:30.552697+0000","last_clean_scrub_stamp":"2024-04-06T03:52:48.666309+0000","objects_scrubbed":5,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:11.029047+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":49,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":882,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:54.709014+0000","last_change":"2024-04-06T03:52:54.709074+0000","last_active":"2024-04-06T03:52:54.709014+0000","last_peered":"2024-04-06T03:52:54.709014+0000","last_clean":"2024-04-06T03:52:54.709014+0000","last_became_active":"2024-04-06T03:42:23.574848+0000","last_became_peered":"2024-04-06T03:42:23.574848+0000","last_unstale":"2024-04-06T03:52:54.709014+0000","last_undegraded":"2024-04-06T03:52:54.709014+0000","last_fullsized":"2024-04-06T03:52:54.709014+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:54.708834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:16.588378+0000","last_clean_scrub_stamp":"2024-04-06T03:52:54.708834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:10.382314+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":664,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675848+0000","last_change":"2024-04-06T03:52:19.911267+0000","last_active":"2024-04-06T03:52:42.675848+0000","last_peered":"2024-04-06T03:52:42.675848+0000","last_clean":"2024-04-06T03:52:42.675848+0000","last_became_active":"2024-04-06T03:50:47.193241+0000","last_became_peered":"2024-04-06T03:50:47.193241+0000","last_unstale":"2024-04-06T03:52:42.675848+0000","last_undegraded":"2024-04-06T03:52:42.675848+0000","last_fullsized":"2024-04-06T03:52:42.675848+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:19.911031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:19.911031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:31.221461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":660,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673889+0000","last_change":"2024-04-06T03:51:41.758468+0000","last_active":"2024-04-06T03:52:42.673889+0000","last_peered":"2024-04-06T03:52:42.673889+0000","last_clean":"2024-04-06T03:52:42.673889+0000","last_became_active":"2024-04-06T03:51:06.725760+0000","last_became_peered":"2024-04-06T03:51:06.725760+0000","last_unstale":"2024-04-06T03:52:42.673889+0000","last_undegraded":"2024-04-06T03:52:42.673889+0000","last_fullsized":"2024-04-06T03:52:42.673889+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:41.758246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:41.758246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:05.842684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"433'18","reported_seq":148,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:49.865912+0000","last_change":"2024-04-06T03:52:49.865982+0000","last_active":"2024-04-06T03:52:49.865912+0000","last_peered":"2024-04-06T03:52:49.865912+0000","last_clean":"2024-04-06T03:52:49.865912+0000","last_became_active":"2024-04-06T03:51:06.720346+0000","last_became_peered":"2024-04-06T03:51:06.720346+0000","last_unstale":"2024-04-06T03:52:49.865912+0000","last_undegraded":"2024-04-06T03:52:49.865912+0000","last_fullsized":"2024-04-06T03:52:49.865912+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'18","last_scrub_stamp":"2024-04-06T03:52:49.865747+0000","last_deep_scrub":"433'18","last_deep_scrub_stamp":"2024-04-06T03:52:49.865747+0000","last_clean_scrub_stamp":"2024-04-06T03:52:49.865747+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:13.620635+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"111'2","reported_seq":898,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180521+0000","last_change":"2024-04-06T03:52:09.573051+0000","last_active":"2024-04-06T03:52:42.180521+0000","last_peered":"2024-04-06T03:52:42.180521+0000","last_clean":"2024-04-06T03:52:42.180521+0000","last_became_active":"2024-04-06T03:20:57.841141+0000","last_became_peered":"2024-04-06T03:20:57.841141+0000","last_unstale":"2024-04-06T03:52:42.180521+0000","last_undegraded":"2024-04-06T03:52:42.180521+0000","last_fullsized":"2024-04-06T03:52:42.180521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'2","last_scrub_stamp":"2024-04-06T03:52:09.572891+0000","last_deep_scrub":"111'2","last_deep_scrub_stamp":"2024-04-06T03:23:10.631051+0000","last_clean_scrub_stamp":"2024-04-06T03:52:09.572891+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:39.466312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":886,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.228205+0000","last_change":"2024-04-06T03:51:55.401163+0000","last_active":"2024-04-06T03:52:42.228205+0000","last_peered":"2024-04-06T03:52:42.228205+0000","last_clean":"2024-04-06T03:52:42.228205+0000","last_became_active":"2024-04-06T03:48:21.332734+0000","last_became_peered":"2024-04-06T03:48:21.332734+0000","last_unstale":"2024-04-06T03:52:42.228205+0000","last_undegraded":"2024-04-06T03:52:42.228205+0000","last_fullsized":"2024-04-06T03:52:42.228205+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:55.400946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:51:55.400946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:15.090910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":932,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179145+0000","last_change":"2024-04-06T03:51:56.558185+0000","last_active":"2024-04-06T03:52:42.179145+0000","last_peered":"2024-04-06T03:52:42.179145+0000","last_clean":"2024-04-06T03:52:42.179145+0000","last_became_active":"2024-04-06T03:48:32.438669+0000","last_became_peered":"2024-04-06T03:48:32.438669+0000","last_unstale":"2024-04-06T03:52:42.179145+0000","last_undegraded":"2024-04-06T03:52:42.179145+0000","last_fullsized":"2024-04-06T03:52:42.179145+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:56.557974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:51:56.557974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:06.175275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"433'47","reported_seq":991,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.666259+0000","last_change":"2024-04-06T03:52:47.666415+0000","last_active":"2024-04-06T03:52:47.666259+0000","last_peered":"2024-04-06T03:52:47.666259+0000","last_clean":"2024-04-06T03:52:47.666259+0000","last_became_active":"2024-04-06T03:42:23.574886+0000","last_became_peered":"2024-04-06T03:42:23.574886+0000","last_unstale":"2024-04-06T03:52:47.666259+0000","last_undegraded":"2024-04-06T03:52:47.666259+0000","last_fullsized":"2024-04-06T03:52:47.666259+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'47","last_scrub_stamp":"2024-04-06T03:52:47.666017+0000","last_deep_scrub":"433'47","last_deep_scrub_stamp":"2024-04-06T03:52:47.666017+0000","last_clean_scrub_stamp":"2024-04-06T03:52:47.666017+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:11.857289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":65,"num_read_kb":57,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":870,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227036+0000","last_change":"2024-04-06T03:52:00.708590+0000","last_active":"2024-04-06T03:52:42.227036+0000","last_peered":"2024-04-06T03:52:42.227036+0000","last_clean":"2024-04-06T03:52:42.227036+0000","last_became_active":"2024-04-06T03:48:21.333229+0000","last_became_peered":"2024-04-06T03:48:21.333229+0000","last_unstale":"2024-04-06T03:52:42.227036+0000","last_undegraded":"2024-04-06T03:52:42.227036+0000","last_fullsized":"2024-04-06T03:52:42.227036+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:00.708379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:56.774056+0000","last_clean_scrub_stamp":"2024-04-06T03:52:00.708379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:16.477498+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"188'4","reported_seq":927,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:46.114039+0000","last_change":"2024-04-06T03:52:46.114114+0000","last_active":"2024-04-06T03:52:46.114039+0000","last_peered":"2024-04-06T03:52:46.114039+0000","last_clean":"2024-04-06T03:52:46.114039+0000","last_became_active":"2024-04-06T03:48:32.447783+0000","last_became_peered":"2024-04-06T03:48:32.447783+0000","last_unstale":"2024-04-06T03:52:46.114039+0000","last_undegraded":"2024-04-06T03:52:46.114039+0000","last_fullsized":"2024-04-06T03:52:46.114039+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'4","last_scrub_stamp":"2024-04-06T03:52:46.113874+0000","last_deep_scrub":"188'4","last_deep_scrub_stamp":"2024-04-06T03:30:01.533601+0000","last_clean_scrub_stamp":"2024-04-06T03:52:46.113874+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:05.267012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"433'47","reported_seq":1020,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:02.629779+0000","last_change":"2024-04-06T03:53:02.629935+0000","last_active":"2024-04-06T03:53:02.629779+0000","last_peered":"2024-04-06T03:53:02.629779+0000","last_clean":"2024-04-06T03:53:02.629779+0000","last_became_active":"2024-04-06T03:48:21.325889+0000","last_became_peered":"2024-04-06T03:48:21.325889+0000","last_unstale":"2024-04-06T03:53:02.629779+0000","last_undegraded":"2024-04-06T03:53:02.629779+0000","last_fullsized":"2024-04-06T03:53:02.629779+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'47","last_scrub_stamp":"2024-04-06T03:53:02.629560+0000","last_deep_scrub":"433'47","last_deep_scrub_stamp":"2024-04-06T03:53:02.629560+0000","last_clean_scrub_stamp":"2024-04-06T03:53:02.629560+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:15.540025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":65,"num_read_kb":57,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":925,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:02.864388+0000","last_change":"2024-04-06T03:53:02.864432+0000","last_active":"2024-04-06T03:53:02.864388+0000","last_peered":"2024-04-06T03:53:02.864388+0000","last_clean":"2024-04-06T03:53:02.864388+0000","last_became_active":"2024-04-06T03:48:32.438050+0000","last_became_peered":"2024-04-06T03:48:32.438050+0000","last_unstale":"2024-04-06T03:53:02.864388+0000","last_undegraded":"2024-04-06T03:53:02.864388+0000","last_fullsized":"2024-04-06T03:53:02.864388+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:53:02.864307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:53:02.864307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:32.676015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":133,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:47.065401+0000","last_change":"2024-04-06T03:52:47.065441+0000","last_active":"2024-04-06T03:52:47.065401+0000","last_peered":"2024-04-06T03:52:47.065401+0000","last_clean":"2024-04-06T03:52:47.065401+0000","last_became_active":"2024-04-06T03:48:32.463557+0000","last_became_peered":"2024-04-06T03:48:32.463557+0000","last_unstale":"2024-04-06T03:52:47.065401+0000","last_undegraded":"2024-04-06T03:52:47.065401+0000","last_fullsized":"2024-04-06T03:52:47.065401+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:47.065296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:31.197990+0000","last_clean_scrub_stamp":"2024-04-06T03:52:47.065296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:47.270460+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"121'2","reported_seq":248,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670037+0000","last_change":"2024-04-06T03:52:02.563263+0000","last_active":"2024-04-06T03:52:42.670037+0000","last_peered":"2024-04-06T03:52:42.670037+0000","last_clean":"2024-04-06T03:52:42.670037+0000","last_became_active":"2024-04-06T03:49:27.609263+0000","last_became_peered":"2024-04-06T03:49:27.609263+0000","last_unstale":"2024-04-06T03:52:42.670037+0000","last_undegraded":"2024-04-06T03:52:42.670037+0000","last_fullsized":"2024-04-06T03:52:42.670037+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'2","last_scrub_stamp":"2024-04-06T03:52:02.563108+0000","last_deep_scrub":"121'2","last_deep_scrub_stamp":"2024-04-06T03:30:14.578519+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.563108+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:31.829646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"433'62","reported_seq":785,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:58.958944+0000","last_change":"2024-04-06T03:52:12.915030+0000","last_active":"2024-04-06T03:52:58.958944+0000","last_peered":"2024-04-06T03:52:58.958944+0000","last_clean":"2024-04-06T03:52:58.958944+0000","last_became_active":"2024-04-06T03:50:47.192671+0000","last_became_peered":"2024-04-06T03:50:47.192671+0000","last_unstale":"2024-04-06T03:52:58.958944+0000","last_undegraded":"2024-04-06T03:52:58.958944+0000","last_fullsized":"2024-04-06T03:52:58.958944+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'62","last_scrub_stamp":"2024-04-06T03:52:12.914736+0000","last_deep_scrub":"199'37","last_deep_scrub_stamp":"2024-04-06T03:37:26.091713+0000","last_clean_scrub_stamp":"2024-04-06T03:52:12.914736+0000","objects_scrubbed":8,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:39.627666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":77,"num_read_kb":67,"num_write":49,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"105'3","reported_seq":151,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670128+0000","last_change":"2024-04-06T03:52:28.407248+0000","last_active":"2024-04-06T03:52:42.670128+0000","last_peered":"2024-04-06T03:52:42.670128+0000","last_clean":"2024-04-06T03:52:42.670128+0000","last_became_active":"2024-04-06T03:48:21.312637+0000","last_became_peered":"2024-04-06T03:48:21.312637+0000","last_unstale":"2024-04-06T03:52:42.670128+0000","last_undegraded":"2024-04-06T03:52:42.670128+0000","last_fullsized":"2024-04-06T03:52:42.670128+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'3","last_scrub_stamp":"2024-04-06T03:52:28.407143+0000","last_deep_scrub":"105'3","last_deep_scrub_stamp":"2024-04-06T03:52:28.407143+0000","last_clean_scrub_stamp":"2024-04-06T03:52:28.407143+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:54.361111+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024079850000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"404'15","reported_seq":184,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705480+0000","last_change":"2024-04-06T03:52:24.006456+0000","last_active":"2024-04-06T03:52:57.705480+0000","last_peered":"2024-04-06T03:52:57.705480+0000","last_clean":"2024-04-06T03:52:57.705480+0000","last_became_active":"2024-04-06T03:48:32.462990+0000","last_became_peered":"2024-04-06T03:48:32.462990+0000","last_unstale":"2024-04-06T03:52:57.705480+0000","last_undegraded":"2024-04-06T03:52:57.705480+0000","last_fullsized":"2024-04-06T03:52:57.705480+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'15","last_scrub_stamp":"2024-04-06T03:52:24.006233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:24.006233+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:52.216685+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":865,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:55.670319+0000","last_change":"2024-04-06T03:52:55.670346+0000","last_active":"2024-04-06T03:52:55.670319+0000","last_peered":"2024-04-06T03:52:55.670319+0000","last_clean":"2024-04-06T03:52:55.670319+0000","last_became_active":"2024-04-06T03:20:57.829529+0000","last_became_peered":"2024-04-06T03:20:57.829529+0000","last_unstale":"2024-04-06T03:52:55.670319+0000","last_undegraded":"2024-04-06T03:52:55.670319+0000","last_fullsized":"2024-04-06T03:52:55.670319+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:55.670213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:24.595337+0000","last_clean_scrub_stamp":"2024-04-06T03:52:55.670213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:12.880116+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":538,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:56.237060+0000","last_change":"2024-04-06T03:52:56.237156+0000","last_active":"2024-04-06T03:52:56.237060+0000","last_peered":"2024-04-06T03:52:56.237060+0000","last_clean":"2024-04-06T03:52:56.237060+0000","last_became_active":"2024-04-06T03:50:47.205131+0000","last_became_peered":"2024-04-06T03:50:47.205131+0000","last_unstale":"2024-04-06T03:52:56.237060+0000","last_undegraded":"2024-04-06T03:52:56.237060+0000","last_fullsized":"2024-04-06T03:52:56.237060+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:56.236870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:51.337979+0000","last_clean_scrub_stamp":"2024-04-06T03:52:56.236870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:03.784057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"433'64","reported_seq":1106,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671920+0000","last_change":"2024-04-06T03:52:23.414492+0000","last_active":"2024-04-06T03:52:42.671920+0000","last_peered":"2024-04-06T03:52:42.671920+0000","last_clean":"2024-04-06T03:52:42.671920+0000","last_became_active":"2024-04-06T03:48:32.447729+0000","last_became_peered":"2024-04-06T03:48:32.447729+0000","last_unstale":"2024-04-06T03:52:42.671920+0000","last_undegraded":"2024-04-06T03:52:42.671920+0000","last_fullsized":"2024-04-06T03:52:42.671920+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'64","last_scrub_stamp":"2024-04-06T03:52:23.414258+0000","last_deep_scrub":"106'34","last_deep_scrub_stamp":"2024-04-06T03:30:17.494006+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.414258+0000","objects_scrubbed":9,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:33.443894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":93,"num_read_kb":78,"num_write":58,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":125,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671582+0000","last_change":"2024-04-06T03:52:03.146201+0000","last_active":"2024-04-06T03:52:42.671582+0000","last_peered":"2024-04-06T03:52:42.671582+0000","last_clean":"2024-04-06T03:52:42.671582+0000","last_became_active":"2024-04-06T03:48:32.458833+0000","last_became_peered":"2024-04-06T03:48:32.458833+0000","last_unstale":"2024-04-06T03:52:42.671582+0000","last_undegraded":"2024-04-06T03:52:42.671582+0000","last_fullsized":"2024-04-06T03:52:42.671582+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:03.146086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:14.960678+0000","last_clean_scrub_stamp":"2024-04-06T03:52:03.146086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.037034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":884,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:56.756869+0000","last_change":"2024-04-06T03:52:56.756955+0000","last_active":"2024-04-06T03:52:56.756869+0000","last_peered":"2024-04-06T03:52:56.756869+0000","last_clean":"2024-04-06T03:52:56.756869+0000","last_became_active":"2024-04-06T03:42:23.569974+0000","last_became_peered":"2024-04-06T03:42:23.569974+0000","last_unstale":"2024-04-06T03:52:56.756869+0000","last_undegraded":"2024-04-06T03:52:56.756869+0000","last_fullsized":"2024-04-06T03:52:56.756869+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:56.756654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:30.391413+0000","last_clean_scrub_stamp":"2024-04-06T03:52:56.756654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:01.238579+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"433'66","reported_seq":939,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:02.257362+0000","last_change":"2024-04-06T03:53:02.257513+0000","last_active":"2024-04-06T03:53:02.257362+0000","last_peered":"2024-04-06T03:53:02.257362+0000","last_clean":"2024-04-06T03:53:02.257362+0000","last_became_active":"2024-04-06T03:49:27.612734+0000","last_became_peered":"2024-04-06T03:49:27.612734+0000","last_unstale":"2024-04-06T03:53:02.257362+0000","last_undegraded":"2024-04-06T03:53:02.257362+0000","last_fullsized":"2024-04-06T03:53:02.257362+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'66","last_scrub_stamp":"2024-04-06T03:53:02.257067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:53:02.257067+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:14.622457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":105,"num_read_kb":91,"num_write":68,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":928,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180876+0000","last_change":"2024-04-06T03:52:40.890687+0000","last_active":"2024-04-06T03:52:42.180876+0000","last_peered":"2024-04-06T03:52:42.180876+0000","last_clean":"2024-04-06T03:52:42.180876+0000","last_became_active":"2024-04-06T03:48:32.438362+0000","last_became_peered":"2024-04-06T03:48:32.438362+0000","last_unstale":"2024-04-06T03:52:42.180876+0000","last_undegraded":"2024-04-06T03:52:42.180876+0000","last_fullsized":"2024-04-06T03:52:42.180876+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:40.890470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:30:08.253651+0000","last_clean_scrub_stamp":"2024-04-06T03:52:40.890470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:06.120298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":909,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180207+0000","last_change":"2024-04-06T03:52:04.562655+0000","last_active":"2024-04-06T03:52:42.180207+0000","last_peered":"2024-04-06T03:52:42.180207+0000","last_clean":"2024-04-06T03:52:42.180207+0000","last_became_active":"2024-04-06T03:20:55.759595+0000","last_became_peered":"2024-04-06T03:20:55.759595+0000","last_unstale":"2024-04-06T03:52:42.180207+0000","last_undegraded":"2024-04-06T03:52:42.180207+0000","last_fullsized":"2024-04-06T03:52:42.180207+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:04.562403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:04.562403+0000","last_clean_scrub_stamp":"2024-04-06T03:52:04.562403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.076317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":190,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678220+0000","last_change":"2024-04-06T03:51:41.225716+0000","last_active":"2024-04-06T03:52:42.678220+0000","last_peered":"2024-04-06T03:52:42.678220+0000","last_clean":"2024-04-06T03:52:42.678220+0000","last_became_active":"2024-04-06T03:50:47.199384+0000","last_became_peered":"2024-04-06T03:50:47.199384+0000","last_unstale":"2024-04-06T03:52:42.678220+0000","last_undegraded":"2024-04-06T03:52:42.678220+0000","last_fullsized":"2024-04-06T03:52:42.678220+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:41.225572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:41.225572+0000","last_clean_scrub_stamp":"2024-04-06T03:51:41.225572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:03.844311+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"433'52","reported_seq":992,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181611+0000","last_change":"2024-04-06T03:52:26.395076+0000","last_active":"2024-04-06T03:52:42.181611+0000","last_peered":"2024-04-06T03:52:42.181611+0000","last_clean":"2024-04-06T03:52:42.181611+0000","last_became_active":"2024-04-06T03:42:23.588643+0000","last_became_peered":"2024-04-06T03:42:23.588643+0000","last_unstale":"2024-04-06T03:52:42.181611+0000","last_undegraded":"2024-04-06T03:52:42.181611+0000","last_fullsized":"2024-04-06T03:52:42.181611+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'52","last_scrub_stamp":"2024-04-06T03:52:26.394825+0000","last_deep_scrub":"307'42","last_deep_scrub_stamp":"2024-04-06T03:41:26.470096+0000","last_clean_scrub_stamp":"2024-04-06T03:52:26.394825+0000","objects_scrubbed":6,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:43.769865+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":63,"num_write":46,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":210,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:51.180929+0000","last_change":"2024-04-06T03:52:51.180929+0000","last_active":"2024-04-06T03:52:51.180929+0000","last_peered":"2024-04-06T03:52:51.180929+0000","last_clean":"2024-04-06T03:52:51.180929+0000","last_became_active":"2024-04-06T03:50:47.200613+0000","last_became_peered":"2024-04-06T03:50:47.200613+0000","last_unstale":"2024-04-06T03:52:51.180929+0000","last_undegraded":"2024-04-06T03:52:51.180929+0000","last_fullsized":"2024-04-06T03:52:51.180929+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:51.180613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:51.180613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:02.014870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"107'130","reported_seq":1211,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182828+0000","last_change":"2024-04-06T03:52:17.582249+0000","last_active":"2024-04-06T03:52:42.182828+0000","last_peered":"2024-04-06T03:52:42.182828+0000","last_clean":"2024-04-06T03:52:42.182828+0000","last_became_active":"2024-04-06T03:48:35.061820+0000","last_became_peered":"2024-04-06T03:48:35.061820+0000","last_unstale":"2024-04-06T03:52:42.182828+0000","last_undegraded":"2024-04-06T03:52:42.182828+0000","last_fullsized":"2024-04-06T03:52:42.182828+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'130","last_scrub_stamp":"2024-04-06T03:52:17.581994+0000","last_deep_scrub":"107'130","last_deep_scrub_stamp":"2024-04-06T03:52:17.581994+0000","last_clean_scrub_stamp":"2024-04-06T03:52:17.581994+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.126174+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4722944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":876,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:00.632195+0000","last_change":"2024-04-06T03:53:00.632293+0000","last_active":"2024-04-06T03:53:00.632195+0000","last_peered":"2024-04-06T03:53:00.632195+0000","last_clean":"2024-04-06T03:53:00.632195+0000","last_became_active":"2024-04-06T03:20:55.756587+0000","last_became_peered":"2024-04-06T03:20:55.756587+0000","last_unstale":"2024-04-06T03:53:00.632195+0000","last_undegraded":"2024-04-06T03:53:00.632195+0000","last_fullsized":"2024-04-06T03:53:00.632195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:53:00.631989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:14.817089+0000","last_clean_scrub_stamp":"2024-04-06T03:53:00.631989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:18.041010+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670336+0000","last_change":"2024-04-06T03:52:18.473669+0000","last_active":"2024-04-06T03:52:42.670336+0000","last_peered":"2024-04-06T03:52:42.670336+0000","last_clean":"2024-04-06T03:52:42.670336+0000","last_became_active":"2024-04-06T03:48:21.322411+0000","last_became_peered":"2024-04-06T03:48:21.322411+0000","last_unstale":"2024-04-06T03:52:42.670336+0000","last_undegraded":"2024-04-06T03:52:42.670336+0000","last_fullsized":"2024-04-06T03:52:42.670336+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:18.473388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:49:36.552917+0000","last_clean_scrub_stamp":"2024-04-06T03:52:18.473388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:37.057796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"441'71","reported_seq":180,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:56.888983+0000","last_change":"2024-04-06T03:52:56.889085+0000","last_active":"2024-04-06T03:52:56.888983+0000","last_peered":"2024-04-06T03:52:56.888983+0000","last_clean":"2024-04-06T03:52:56.888983+0000","last_became_active":"2024-04-06T03:48:32.452730+0000","last_became_peered":"2024-04-06T03:48:32.452730+0000","last_unstale":"2024-04-06T03:52:56.888983+0000","last_undegraded":"2024-04-06T03:52:56.888983+0000","last_fullsized":"2024-04-06T03:52:56.888983+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'71","last_scrub_stamp":"2024-04-06T03:52:56.888810+0000","last_deep_scrub":"441'71","last_deep_scrub_stamp":"2024-04-06T03:51:47.133583+0000","last_clean_scrub_stamp":"2024-04-06T03:52:56.888810+0000","objects_scrubbed":8,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:16.961657+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":92,"num_read_kb":84,"num_write":62,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":901,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182681+0000","last_change":"2024-04-06T03:52:05.612748+0000","last_active":"2024-04-06T03:52:42.182681+0000","last_peered":"2024-04-06T03:52:42.182681+0000","last_clean":"2024-04-06T03:52:42.182681+0000","last_became_active":"2024-04-06T03:48:32.436634+0000","last_became_peered":"2024-04-06T03:48:32.436634+0000","last_unstale":"2024-04-06T03:52:42.182681+0000","last_undegraded":"2024-04-06T03:52:42.182681+0000","last_fullsized":"2024-04-06T03:52:42.182681+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:05.612536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:50.108708+0000","last_clean_scrub_stamp":"2024-04-06T03:52:05.612536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.003869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":202,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.677963+0000","last_change":"2024-04-06T03:52:40.170322+0000","last_active":"2024-04-06T03:52:42.677963+0000","last_peered":"2024-04-06T03:52:42.677963+0000","last_clean":"2024-04-06T03:52:42.677963+0000","last_became_active":"2024-04-06T03:50:47.198358+0000","last_became_peered":"2024-04-06T03:50:47.198358+0000","last_unstale":"2024-04-06T03:52:42.677963+0000","last_undegraded":"2024-04-06T03:52:42.677963+0000","last_fullsized":"2024-04-06T03:52:42.677963+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:40.170070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:45:32.117767+0000","last_clean_scrub_stamp":"2024-04-06T03:52:40.170070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:05.441402+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:59.323586+0000","last_change":"2024-04-06T03:52:59.323652+0000","last_active":"2024-04-06T03:52:59.323586+0000","last_peered":"2024-04-06T03:52:59.323586+0000","last_clean":"2024-04-06T03:52:59.323586+0000","last_became_active":"2024-04-06T03:48:21.315429+0000","last_became_peered":"2024-04-06T03:48:21.315429+0000","last_unstale":"2024-04-06T03:52:59.323586+0000","last_undegraded":"2024-04-06T03:52:59.323586+0000","last_fullsized":"2024-04-06T03:52:59.323586+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:59.323430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:59.323430+0000","last_clean_scrub_stamp":"2024-04-06T03:52:59.323430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:26.286110+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":894,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184123+0000","last_change":"2024-04-06T03:52:10.679166+0000","last_active":"2024-04-06T03:52:42.184123+0000","last_peered":"2024-04-06T03:52:42.184123+0000","last_clean":"2024-04-06T03:52:42.184123+0000","last_became_active":"2024-04-06T03:48:32.437232+0000","last_became_peered":"2024-04-06T03:48:32.437232+0000","last_unstale":"2024-04-06T03:52:42.184123+0000","last_undegraded":"2024-04-06T03:52:42.184123+0000","last_fullsized":"2024-04-06T03:52:42.184123+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:10.678935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:28:34.057999+0000","last_clean_scrub_stamp":"2024-04-06T03:52:10.678935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:17.226580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"433'90","reported_seq":1368,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705685+0000","last_change":"2024-04-06T03:52:27.385824+0000","last_active":"2024-04-06T03:52:57.705685+0000","last_peered":"2024-04-06T03:52:57.705685+0000","last_clean":"2024-04-06T03:52:57.705685+0000","last_became_active":"2024-04-06T03:49:27.611624+0000","last_became_peered":"2024-04-06T03:49:27.611624+0000","last_unstale":"2024-04-06T03:52:57.705685+0000","last_undegraded":"2024-04-06T03:52:57.705685+0000","last_fullsized":"2024-04-06T03:52:57.705685+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'90","last_scrub_stamp":"2024-04-06T03:52:27.385680+0000","last_deep_scrub":"414'78","last_deep_scrub_stamp":"2024-04-06T03:49:47.447699+0000","last_clean_scrub_stamp":"2024-04-06T03:52:27.385680+0000","objects_scrubbed":11,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:42.788391+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":114,"num_read_kb":98,"num_write":68,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":507,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.674013+0000","last_change":"2024-04-06T03:52:03.827033+0000","last_active":"2024-04-06T03:52:42.674013+0000","last_peered":"2024-04-06T03:52:42.674013+0000","last_clean":"2024-04-06T03:52:42.674013+0000","last_became_active":"2024-04-06T03:51:06.720527+0000","last_became_peered":"2024-04-06T03:51:06.720527+0000","last_unstale":"2024-04-06T03:52:42.674013+0000","last_undegraded":"2024-04-06T03:52:42.674013+0000","last_fullsized":"2024-04-06T03:52:42.674013+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:03.826789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:54.714222+0000","last_clean_scrub_stamp":"2024-04-06T03:52:03.826789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:19.295188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":866,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671683+0000","last_change":"2024-04-06T03:51:57.328265+0000","last_active":"2024-04-06T03:52:42.671683+0000","last_peered":"2024-04-06T03:52:42.671683+0000","last_clean":"2024-04-06T03:52:42.671683+0000","last_became_active":"2024-04-06T03:20:57.815071+0000","last_became_peered":"2024-04-06T03:20:57.815071+0000","last_unstale":"2024-04-06T03:52:42.671683+0000","last_undegraded":"2024-04-06T03:52:42.671683+0000","last_fullsized":"2024-04-06T03:52:42.671683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:57.327982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:24:54.826483+0000","last_clean_scrub_stamp":"2024-04-06T03:51:57.327982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:16.629680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"448'2","reported_seq":152,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:43.395921+0000","last_change":"2024-04-06T03:52:43.395921+0000","last_active":"2024-04-06T03:52:43.395921+0000","last_peered":"2024-04-06T03:52:43.395921+0000","last_clean":"2024-04-06T03:52:43.395921+0000","last_became_active":"2024-04-06T03:48:21.323713+0000","last_became_peered":"2024-04-06T03:48:21.323713+0000","last_unstale":"2024-04-06T03:52:43.395921+0000","last_undegraded":"2024-04-06T03:52:43.395921+0000","last_fullsized":"2024-04-06T03:52:43.395921+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'2","last_scrub_stamp":"2024-04-06T03:52:43.395842+0000","last_deep_scrub":"448'2","last_deep_scrub_stamp":"2024-04-06T03:52:43.395842+0000","last_clean_scrub_stamp":"2024-04-06T03:52:43.395842+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:05.808931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"433'31","reported_seq":962,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.227620+0000","last_change":"2024-04-06T03:51:56.452210+0000","last_active":"2024-04-06T03:52:42.227620+0000","last_peered":"2024-04-06T03:52:42.227620+0000","last_clean":"2024-04-06T03:52:42.227620+0000","last_became_active":"2024-04-06T03:48:32.468746+0000","last_became_peered":"2024-04-06T03:48:32.468746+0000","last_unstale":"2024-04-06T03:52:42.227620+0000","last_undegraded":"2024-04-06T03:52:42.227620+0000","last_fullsized":"2024-04-06T03:52:42.227620+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'31","last_scrub_stamp":"2024-04-06T03:51:56.451944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:51:56.451944+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:23.762368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":45,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"428'6","reported_seq":2802,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:03.059862+0000","last_change":"2024-04-06T03:51:59.468913+0000","last_active":"2024-04-06T03:53:03.059862+0000","last_peered":"2024-04-06T03:53:03.059862+0000","last_clean":"2024-04-06T03:53:03.059862+0000","last_became_active":"2024-04-06T03:48:32.466606+0000","last_became_peered":"2024-04-06T03:48:32.466606+0000","last_unstale":"2024-04-06T03:53:03.059862+0000","last_undegraded":"2024-04-06T03:53:03.059862+0000","last_fullsized":"2024-04-06T03:53:03.059862+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"428'6","last_scrub_stamp":"2024-04-06T03:51:59.468813+0000","last_deep_scrub":"428'6","last_deep_scrub_stamp":"2024-04-06T03:50:42.386898+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.468813+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:09.991308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011044449999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1891,"num_read_kb":1886,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":878,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:48.039335+0000","last_change":"2024-04-06T03:52:48.039391+0000","last_active":"2024-04-06T03:52:48.039335+0000","last_peered":"2024-04-06T03:52:48.039335+0000","last_clean":"2024-04-06T03:52:48.039335+0000","last_became_active":"2024-04-06T03:20:55.771633+0000","last_became_peered":"2024-04-06T03:20:55.771633+0000","last_unstale":"2024-04-06T03:52:48.039335+0000","last_undegraded":"2024-04-06T03:52:48.039335+0000","last_fullsized":"2024-04-06T03:52:48.039335+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:48.039180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:39.430710+0000","last_clean_scrub_stamp":"2024-04-06T03:52:48.039180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:55.018004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":877,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183799+0000","last_change":"2024-04-06T03:52:19.739171+0000","last_active":"2024-04-06T03:52:42.183799+0000","last_peered":"2024-04-06T03:52:42.183799+0000","last_clean":"2024-04-06T03:52:42.183799+0000","last_became_active":"2024-04-06T03:42:23.562715+0000","last_became_peered":"2024-04-06T03:42:23.562715+0000","last_unstale":"2024-04-06T03:52:42.183799+0000","last_undegraded":"2024-04-06T03:52:42.183799+0000","last_fullsized":"2024-04-06T03:52:42.183799+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:19.739026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:37:31.563716+0000","last_clean_scrub_stamp":"2024-04-06T03:52:19.739026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:25.673156+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":498,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678247+0000","last_change":"2024-04-06T03:51:48.824750+0000","last_active":"2024-04-06T03:52:42.678247+0000","last_peered":"2024-04-06T03:52:42.678247+0000","last_clean":"2024-04-06T03:52:42.678247+0000","last_became_active":"2024-04-06T03:51:06.733099+0000","last_became_peered":"2024-04-06T03:51:06.733099+0000","last_unstale":"2024-04-06T03:52:42.678247+0000","last_undegraded":"2024-04-06T03:52:42.678247+0000","last_fullsized":"2024-04-06T03:52:42.678247+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:48.824466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:17.864183+0000","last_clean_scrub_stamp":"2024-04-06T03:51:48.824466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:14.788630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"433'54","reported_seq":157,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.671534+0000","last_change":"2024-04-06T03:52:07.147768+0000","last_active":"2024-04-06T03:52:42.671534+0000","last_peered":"2024-04-06T03:52:42.671534+0000","last_clean":"2024-04-06T03:52:42.671534+0000","last_became_active":"2024-04-06T03:48:32.459468+0000","last_became_peered":"2024-04-06T03:48:32.459468+0000","last_unstale":"2024-04-06T03:52:42.671534+0000","last_undegraded":"2024-04-06T03:52:42.671534+0000","last_fullsized":"2024-04-06T03:52:42.671534+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'54","last_scrub_stamp":"2024-04-06T03:52:07.147604+0000","last_deep_scrub":"433'54","last_deep_scrub_stamp":"2024-04-06T03:52:07.147604+0000","last_clean_scrub_stamp":"2024-04-06T03:52:07.147604+0000","objects_scrubbed":6,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:11.468923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":66,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":891,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.672299+0000","last_change":"2024-04-06T03:52:42.078891+0000","last_active":"2024-04-06T03:52:42.672299+0000","last_peered":"2024-04-06T03:52:42.672299+0000","last_clean":"2024-04-06T03:52:42.672299+0000","last_became_active":"2024-04-06T03:42:23.576890+0000","last_became_peered":"2024-04-06T03:42:23.576890+0000","last_unstale":"2024-04-06T03:52:42.672299+0000","last_undegraded":"2024-04-06T03:52:42.672299+0000","last_fullsized":"2024-04-06T03:52:42.672299+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:42.078672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:43:19.191417+0000","last_clean_scrub_stamp":"2024-04-06T03:52:42.078672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:09.137096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012182339999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"397'15","reported_seq":1301,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:02.709133+0000","last_change":"2024-04-06T03:52:41.468507+0000","last_active":"2024-04-06T03:53:02.709133+0000","last_peered":"2024-04-06T03:53:02.709133+0000","last_clean":"2024-04-06T03:53:02.709133+0000","last_became_active":"2024-04-06T03:48:21.346584+0000","last_became_peered":"2024-04-06T03:48:21.346584+0000","last_unstale":"2024-04-06T03:53:02.709133+0000","last_undegraded":"2024-04-06T03:53:02.709133+0000","last_fullsized":"2024-04-06T03:53:02.709133+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'15","last_scrub_stamp":"2024-04-06T03:52:41.468142+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-06T03:41:18.436223+0000","last_clean_scrub_stamp":"2024-04-06T03:52:41.468142+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:58.130229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":869,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.181972+0000","last_change":"2024-04-06T03:51:50.437671+0000","last_active":"2024-04-06T03:52:42.181972+0000","last_peered":"2024-04-06T03:52:42.181972+0000","last_clean":"2024-04-06T03:52:42.181972+0000","last_became_active":"2024-04-06T03:42:23.590773+0000","last_became_peered":"2024-04-06T03:42:23.590773+0000","last_unstale":"2024-04-06T03:52:42.181972+0000","last_undegraded":"2024-04-06T03:52:42.181972+0000","last_fullsized":"2024-04-06T03:52:42.181972+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:50.437449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:23:38.882269+0000","last_clean_scrub_stamp":"2024-04-06T03:51:50.437449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:19.095643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":506,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.678031+0000","last_change":"2024-04-06T03:51:59.849971+0000","last_active":"2024-04-06T03:52:42.678031+0000","last_peered":"2024-04-06T03:52:42.678031+0000","last_clean":"2024-04-06T03:52:42.678031+0000","last_became_active":"2024-04-06T03:50:47.210897+0000","last_became_peered":"2024-04-06T03:50:47.210897+0000","last_unstale":"2024-04-06T03:52:42.678031+0000","last_undegraded":"2024-04-06T03:52:42.678031+0000","last_fullsized":"2024-04-06T03:52:42.678031+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:59.849738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:27.569064+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.849738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:12.972343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":899,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:58.618495+0000","last_change":"2024-04-06T03:52:58.618565+0000","last_active":"2024-04-06T03:52:58.618495+0000","last_peered":"2024-04-06T03:52:58.618495+0000","last_clean":"2024-04-06T03:52:58.618495+0000","last_became_active":"2024-04-06T03:48:32.435449+0000","last_became_peered":"2024-04-06T03:48:32.435449+0000","last_unstale":"2024-04-06T03:52:58.618495+0000","last_undegraded":"2024-04-06T03:52:58.618495+0000","last_fullsized":"2024-04-06T03:52:58.618495+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:58.618339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:34.508293+0000","last_clean_scrub_stamp":"2024-04-06T03:52:58.618339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:26.212470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"433'49","reported_seq":954,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:43.125667+0000","last_change":"2024-04-06T03:52:43.125762+0000","last_active":"2024-04-06T03:52:43.125667+0000","last_peered":"2024-04-06T03:52:43.125667+0000","last_clean":"2024-04-06T03:52:43.125667+0000","last_became_active":"2024-04-06T03:20:53.338248+0000","last_became_peered":"2024-04-06T03:20:53.338248+0000","last_unstale":"2024-04-06T03:52:43.125667+0000","last_undegraded":"2024-04-06T03:52:43.125667+0000","last_fullsized":"2024-04-06T03:52:43.125667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'49","last_scrub_stamp":"2024-04-06T03:52:43.125497+0000","last_deep_scrub":"122'31","last_deep_scrub_stamp":"2024-04-06T03:25:06.862413+0000","last_clean_scrub_stamp":"2024-04-06T03:52:43.125497+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:07.774408+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":515,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.676523+0000","last_change":"2024-04-06T03:52:02.035919+0000","last_active":"2024-04-06T03:52:42.676523+0000","last_peered":"2024-04-06T03:52:42.676523+0000","last_clean":"2024-04-06T03:52:42.676523+0000","last_became_active":"2024-04-06T03:50:47.200323+0000","last_became_peered":"2024-04-06T03:50:47.200323+0000","last_unstale":"2024-04-06T03:52:42.676523+0000","last_undegraded":"2024-04-06T03:52:42.676523+0000","last_fullsized":"2024-04-06T03:52:42.676523+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:02.035691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:37.734841+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.035691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:10.406611+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"404'14","reported_seq":1296,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705487+0000","last_change":"2024-04-06T03:52:08.487093+0000","last_active":"2024-04-06T03:52:57.705487+0000","last_peered":"2024-04-06T03:52:57.705487+0000","last_clean":"2024-04-06T03:52:57.705487+0000","last_became_active":"2024-04-06T03:48:32.447676+0000","last_became_peered":"2024-04-06T03:48:32.447676+0000","last_unstale":"2024-04-06T03:52:57.705487+0000","last_undegraded":"2024-04-06T03:52:57.705487+0000","last_fullsized":"2024-04-06T03:52:57.705487+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'14","last_scrub_stamp":"2024-04-06T03:52:08.486745+0000","last_deep_scrub":"404'14","last_deep_scrub_stamp":"2024-04-06T03:52:08.486745+0000","last_clean_scrub_stamp":"2024-04-06T03:52:08.486745+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:38.080900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:44.349553+0000","last_change":"2024-04-06T03:52:44.349573+0000","last_active":"2024-04-06T03:52:44.349553+0000","last_peered":"2024-04-06T03:52:44.349553+0000","last_clean":"2024-04-06T03:52:44.349553+0000","last_became_active":"2024-04-06T03:48:21.342086+0000","last_became_peered":"2024-04-06T03:48:21.342086+0000","last_unstale":"2024-04-06T03:52:44.349553+0000","last_undegraded":"2024-04-06T03:52:44.349553+0000","last_fullsized":"2024-04-06T03:52:44.349553+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:44.349489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:36.411646+0000","last_clean_scrub_stamp":"2024-04-06T03:52:44.349489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:03.010061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"433'72","reported_seq":1029,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.180821+0000","last_change":"2024-04-06T03:51:57.441813+0000","last_active":"2024-04-06T03:52:42.180821+0000","last_peered":"2024-04-06T03:52:42.180821+0000","last_clean":"2024-04-06T03:52:42.180821+0000","last_became_active":"2024-04-06T03:48:21.345525+0000","last_became_peered":"2024-04-06T03:48:21.345525+0000","last_unstale":"2024-04-06T03:52:42.180821+0000","last_undegraded":"2024-04-06T03:52:42.180821+0000","last_fullsized":"2024-04-06T03:52:42.180821+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'72","last_scrub_stamp":"2024-04-06T03:51:57.441544+0000","last_deep_scrub":"428'60","last_deep_scrub_stamp":"2024-04-06T03:50:38.375224+0000","last_clean_scrub_stamp":"2024-04-06T03:51:57.441544+0000","objects_scrubbed":9,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:04.682835+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":98,"num_read_kb":87,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":877,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.673568+0000","last_change":"2024-04-06T03:51:52.371847+0000","last_active":"2024-04-06T03:52:42.673568+0000","last_peered":"2024-04-06T03:52:42.673568+0000","last_clean":"2024-04-06T03:52:42.673568+0000","last_became_active":"2024-04-06T03:48:21.308499+0000","last_became_peered":"2024-04-06T03:48:21.308499+0000","last_unstale":"2024-04-06T03:52:42.673568+0000","last_undegraded":"2024-04-06T03:52:42.673568+0000","last_fullsized":"2024-04-06T03:52:42.673568+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:52.371606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:37.509500+0000","last_clean_scrub_stamp":"2024-04-06T03:51:52.371606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:14.280369+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"111'12","reported_seq":1307,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705524+0000","last_change":"2024-04-06T03:52:23.737147+0000","last_active":"2024-04-06T03:52:57.705524+0000","last_peered":"2024-04-06T03:52:57.705524+0000","last_clean":"2024-04-06T03:52:57.705524+0000","last_became_active":"2024-04-06T03:20:55.742915+0000","last_became_peered":"2024-04-06T03:20:55.742915+0000","last_unstale":"2024-04-06T03:52:57.705524+0000","last_undegraded":"2024-04-06T03:52:57.705524+0000","last_fullsized":"2024-04-06T03:52:57.705524+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'12","last_scrub_stamp":"2024-04-06T03:52:23.736924+0000","last_deep_scrub":"111'12","last_deep_scrub_stamp":"2024-04-06T03:23:29.528005+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.736924+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:34.562602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":726,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.669771+0000","last_change":"2024-04-06T03:51:59.569896+0000","last_active":"2024-04-06T03:52:42.669771+0000","last_peered":"2024-04-06T03:52:42.669771+0000","last_clean":"2024-04-06T03:52:42.669771+0000","last_became_active":"2024-04-06T03:49:27.623051+0000","last_became_peered":"2024-04-06T03:49:27.623051+0000","last_unstale":"2024-04-06T03:52:42.669771+0000","last_undegraded":"2024-04-06T03:52:42.669771+0000","last_fullsized":"2024-04-06T03:52:42.669771+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:59.569693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:51:59.569693+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.569693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:27.001786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"433'33","reported_seq":974,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.182265+0000","last_change":"2024-04-06T03:51:45.362474+0000","last_active":"2024-04-06T03:52:42.182265+0000","last_peered":"2024-04-06T03:52:42.182265+0000","last_clean":"2024-04-06T03:52:42.182265+0000","last_became_active":"2024-04-06T03:48:32.468304+0000","last_became_peered":"2024-04-06T03:48:32.468304+0000","last_unstale":"2024-04-06T03:52:42.182265+0000","last_undegraded":"2024-04-06T03:52:42.182265+0000","last_fullsized":"2024-04-06T03:52:42.182265+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'33","last_scrub_stamp":"2024-04-06T03:51:45.362127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:51:45.362127+0000","objects_scrubbed":5,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:00.873159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":905,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:50.052577+0000","last_change":"2024-04-06T03:52:50.052631+0000","last_active":"2024-04-06T03:52:50.052577+0000","last_peered":"2024-04-06T03:52:50.052577+0000","last_clean":"2024-04-06T03:52:50.052577+0000","last_became_active":"2024-04-06T03:20:57.841572+0000","last_became_peered":"2024-04-06T03:20:57.841572+0000","last_unstale":"2024-04-06T03:52:50.052577+0000","last_undegraded":"2024-04-06T03:52:50.052577+0000","last_fullsized":"2024-04-06T03:52:50.052577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:50.052419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:48:47.654196+0000","last_clean_scrub_stamp":"2024-04-06T03:52:50.052419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:58.671364+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":698,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670325+0000","last_change":"2024-04-06T03:52:34.443299+0000","last_active":"2024-04-06T03:52:42.670325+0000","last_peered":"2024-04-06T03:52:42.670325+0000","last_clean":"2024-04-06T03:52:42.670325+0000","last_became_active":"2024-04-06T03:49:27.614179+0000","last_became_peered":"2024-04-06T03:49:27.614179+0000","last_unstale":"2024-04-06T03:52:42.670325+0000","last_undegraded":"2024-04-06T03:52:42.670325+0000","last_fullsized":"2024-04-06T03:52:42.670325+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:34.442971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:59.438940+0000","last_clean_scrub_stamp":"2024-04-06T03:52:34.442971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:38.721866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":875,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.183492+0000","last_change":"2024-04-06T03:51:31.499337+0000","last_active":"2024-04-06T03:52:42.183492+0000","last_peered":"2024-04-06T03:52:42.183492+0000","last_clean":"2024-04-06T03:52:42.183492+0000","last_became_active":"2024-04-06T03:20:51.143211+0000","last_became_peered":"2024-04-06T03:20:51.143211+0000","last_unstale":"2024-04-06T03:52:42.183492+0000","last_undegraded":"2024-04-06T03:52:42.183492+0000","last_fullsized":"2024-04-06T03:52:42.183492+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:31.499126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:27.906531+0000","last_clean_scrub_stamp":"2024-04-06T03:51:31.499126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:52:59.598040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"433'73","reported_seq":995,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.672393+0000","last_change":"2024-04-06T03:52:07.817533+0000","last_active":"2024-04-06T03:52:42.672393+0000","last_peered":"2024-04-06T03:52:42.672393+0000","last_clean":"2024-04-06T03:52:42.672393+0000","last_became_active":"2024-04-06T03:51:06.723822+0000","last_became_peered":"2024-04-06T03:51:06.723822+0000","last_unstale":"2024-04-06T03:52:42.672393+0000","last_undegraded":"2024-04-06T03:52:42.672393+0000","last_fullsized":"2024-04-06T03:52:42.672393+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'73","last_scrub_stamp":"2024-04-06T03:52:07.817268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:20:52.127727+0000","last_clean_scrub_stamp":"2024-04-06T03:52:07.817268+0000","objects_scrubbed":10,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:30.962715+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":111,"num_read_kb":99,"num_write":74,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":863,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:53:01.623668+0000","last_change":"2024-04-06T03:53:01.623821+0000","last_active":"2024-04-06T03:53:01.623668+0000","last_peered":"2024-04-06T03:53:01.623668+0000","last_clean":"2024-04-06T03:53:01.623668+0000","last_became_active":"2024-04-06T03:20:57.829293+0000","last_became_peered":"2024-04-06T03:20:57.829293+0000","last_unstale":"2024-04-06T03:53:01.623668+0000","last_undegraded":"2024-04-06T03:53:01.623668+0000","last_fullsized":"2024-04-06T03:53:01.623668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:53:01.623397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:20.575364+0000","last_clean_scrub_stamp":"2024-04-06T03:53:01.623397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:02.737456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":913,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179561+0000","last_change":"2024-04-06T03:51:54.618796+0000","last_active":"2024-04-06T03:52:42.179561+0000","last_peered":"2024-04-06T03:52:42.179561+0000","last_clean":"2024-04-06T03:52:42.179561+0000","last_became_active":"2024-04-06T03:48:32.439955+0000","last_became_peered":"2024-04-06T03:48:32.439955+0000","last_unstale":"2024-04-06T03:52:42.179561+0000","last_undegraded":"2024-04-06T03:52:42.179561+0000","last_fullsized":"2024-04-06T03:52:42.179561+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:51:54.618581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:41:33.770813+0000","last_clean_scrub_stamp":"2024-04-06T03:51:54.618581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:18.523727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":927,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:55.924657+0000","last_change":"2024-04-06T03:52:55.924710+0000","last_active":"2024-04-06T03:52:55.924657+0000","last_peered":"2024-04-06T03:52:55.924657+0000","last_clean":"2024-04-06T03:52:55.924657+0000","last_became_active":"2024-04-06T03:48:32.439252+0000","last_became_peered":"2024-04-06T03:48:32.439252+0000","last_unstale":"2024-04-06T03:52:55.924657+0000","last_undegraded":"2024-04-06T03:52:55.924657+0000","last_fullsized":"2024-04-06T03:52:55.924657+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:55.924511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:50:12.446876+0000","last_clean_scrub_stamp":"2024-04-06T03:52:55.924511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:25.842798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"433'67","reported_seq":1018,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:46.668947+0000","last_change":"2024-04-06T03:52:46.669041+0000","last_active":"2024-04-06T03:52:46.668947+0000","last_peered":"2024-04-06T03:52:46.668947+0000","last_clean":"2024-04-06T03:52:46.668947+0000","last_became_active":"2024-04-06T03:42:23.570628+0000","last_became_peered":"2024-04-06T03:42:23.570628+0000","last_unstale":"2024-04-06T03:52:46.668947+0000","last_undegraded":"2024-04-06T03:52:46.668947+0000","last_fullsized":"2024-04-06T03:52:46.668947+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'67","last_scrub_stamp":"2024-04-06T03:52:46.668769+0000","last_deep_scrub":"307'53","last_deep_scrub_stamp":"2024-04-06T03:49:52.351719+0000","last_clean_scrub_stamp":"2024-04-06T03:52:46.668769+0000","objects_scrubbed":8,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:01.443665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":92,"num_read_kb":82,"num_write":60,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":125,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670250+0000","last_change":"2024-04-06T03:52:02.315577+0000","last_active":"2024-04-06T03:52:42.670250+0000","last_peered":"2024-04-06T03:52:42.670250+0000","last_clean":"2024-04-06T03:52:42.670250+0000","last_became_active":"2024-04-06T03:48:32.463995+0000","last_became_peered":"2024-04-06T03:48:32.463995+0000","last_unstale":"2024-04-06T03:52:42.670250+0000","last_undegraded":"2024-04-06T03:52:42.670250+0000","last_fullsized":"2024-04-06T03:52:42.670250+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:02.315422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:02.315422+0000","last_clean_scrub_stamp":"2024-04-06T03:52:02.315422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:06.859785+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":901,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.179689+0000","last_change":"2024-04-06T03:52:10.622320+0000","last_active":"2024-04-06T03:52:42.179689+0000","last_peered":"2024-04-06T03:52:42.179689+0000","last_clean":"2024-04-06T03:52:42.179689+0000","last_became_active":"2024-04-06T03:20:55.748210+0000","last_became_peered":"2024-04-06T03:20:55.748210+0000","last_unstale":"2024-04-06T03:52:42.179689+0000","last_undegraded":"2024-04-06T03:52:42.179689+0000","last_fullsized":"2024-04-06T03:52:42.179689+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:10.622097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:10.622097+0000","last_clean_scrub_stamp":"2024-04-06T03:52:10.622097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:25.546094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":147,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.670293+0000","last_change":"2024-04-06T03:52:23.441902+0000","last_active":"2024-04-06T03:52:42.670293+0000","last_peered":"2024-04-06T03:52:42.670293+0000","last_clean":"2024-04-06T03:52:42.670293+0000","last_became_active":"2024-04-06T03:48:21.325094+0000","last_became_peered":"2024-04-06T03:48:21.325094+0000","last_unstale":"2024-04-06T03:52:42.670293+0000","last_undegraded":"2024-04-06T03:52:42.670293+0000","last_fullsized":"2024-04-06T03:52:42.670293+0000","mapping_epoch":396,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":397,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:23.441690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:39:13.053965+0000","last_clean_scrub_stamp":"2024-04-06T03:52:23.441690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:49.516959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"433'45","reported_seq":641,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.676878+0000","last_change":"2024-04-06T03:52:28.933090+0000","last_active":"2024-04-06T03:52:42.676878+0000","last_peered":"2024-04-06T03:52:42.676878+0000","last_clean":"2024-04-06T03:52:42.676878+0000","last_became_active":"2024-04-06T03:50:47.205748+0000","last_became_peered":"2024-04-06T03:50:47.205748+0000","last_unstale":"2024-04-06T03:52:42.676878+0000","last_undegraded":"2024-04-06T03:52:42.676878+0000","last_fullsized":"2024-04-06T03:52:42.676878+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'45","last_scrub_stamp":"2024-04-06T03:52:28.932835+0000","last_deep_scrub":"199'25","last_deep_scrub_stamp":"2024-04-06T03:39:25.598852+0000","last_clean_scrub_stamp":"2024-04-06T03:52:28.932835+0000","objects_scrubbed":5,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:52.609018+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":59,"num_read_kb":54,"num_write":40,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":521,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.675036+0000","last_change":"2024-04-06T03:52:18.908569+0000","last_active":"2024-04-06T03:52:42.675036+0000","last_peered":"2024-04-06T03:52:42.675036+0000","last_clean":"2024-04-06T03:52:42.675036+0000","last_became_active":"2024-04-06T03:50:47.206374+0000","last_became_peered":"2024-04-06T03:50:47.206374+0000","last_unstale":"2024-04-06T03:52:42.675036+0000","last_undegraded":"2024-04-06T03:52:42.675036+0000","last_fullsized":"2024-04-06T03:52:42.675036+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:18.908322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:22:46.720611+0000","last_clean_scrub_stamp":"2024-04-06T03:52:18.908322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:21.944455+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":917,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:52.936286+0000","last_change":"2024-04-06T03:52:52.936365+0000","last_active":"2024-04-06T03:52:52.936286+0000","last_peered":"2024-04-06T03:52:52.936286+0000","last_clean":"2024-04-06T03:52:52.936286+0000","last_became_active":"2024-04-06T03:20:51.146367+0000","last_became_peered":"2024-04-06T03:20:51.146367+0000","last_unstale":"2024-04-06T03:52:52.936286+0000","last_undegraded":"2024-04-06T03:52:52.936286+0000","last_fullsized":"2024-04-06T03:52:52.936286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-06T03:52:52.936118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-06T03:52:52.936118+0000","last_clean_scrub_stamp":"2024-04-06T03:52:52.936118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:00.340491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"433'58","reported_seq":1023,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:42.184449+0000","last_change":"2024-04-06T03:51:59.656696+0000","last_active":"2024-04-06T03:52:42.184449+0000","last_peered":"2024-04-06T03:52:42.184449+0000","last_clean":"2024-04-06T03:52:42.184449+0000","last_became_active":"2024-04-06T03:48:32.446386+0000","last_became_peered":"2024-04-06T03:48:32.446386+0000","last_unstale":"2024-04-06T03:52:42.184449+0000","last_undegraded":"2024-04-06T03:52:42.184449+0000","last_fullsized":"2024-04-06T03:52:42.184449+0000","mapping_epoch":403,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'58","last_scrub_stamp":"2024-04-06T03:51:59.656460+0000","last_deep_scrub":"307'46","last_deep_scrub_stamp":"2024-04-06T03:49:07.245123+0000","last_clean_scrub_stamp":"2024-04-06T03:51:59.656460+0000","objects_scrubbed":7,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:53:22.661084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":71,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"327'14","reported_seq":1295,"reported_epoch":448,"state":"active+clean","last_fresh":"2024-04-06T03:52:57.705571+0000","last_change":"2024-04-06T03:52:44.612822+0000","last_active":"2024-04-06T03:52:57.705571+0000","last_peered":"2024-04-06T03:52:57.705571+0000","last_clean":"2024-04-06T03:52:57.705571+0000","last_became_active":"2024-04-06T03:42:23.576196+0000","last_became_peered":"2024-04-06T03:42:23.576196+0000","last_unstale":"2024-04-06T03:52:57.705571+0000","last_undegraded":"2024-04-06T03:52:57.705571+0000","last_fullsized":"2024-04-06T03:52:57.705571+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'14","last_scrub_stamp":"2024-04-06T03:52:44.612575+0000","last_deep_scrub":"209'12","last_deep_scrub_stamp":"2024-04-06T03:32:06.805186+0000","last_clean_scrub_stamp":"2024-04-06T03:52:44.612575+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T03:54:00.322812+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4722944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":2261,"num_read_kb":1986,"num_write":1467,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":8472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1635,"ondisk_log_size":1635,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1891,"num_read_kb":1886,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743822,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546284,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93198868,"statfs":{"total":95995035648,"available":95435640832,"internally_reserved":0,"allocated":1478656,"data_stored":1073390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":557906378},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136280,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582328,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162824,"statfs":{"total":95995035648,"available":95398731776,"internally_reserved":0,"allocated":1425408,"data_stored":1071312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575309,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460996,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93284156,"statfs":{"total":95995035648,"available":95522975744,"internally_reserved":0,"allocated":847872,"data_stored":480798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":471202246},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738839,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646220,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098932,"statfs":{"total":95995035648,"available":95333306368,"internally_reserved":0,"allocated":1445888,"data_stored":1072365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604379075,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537336,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207816,"statfs":{"total":95995035648,"available":95444803584,"internally_reserved":0,"allocated":835584,"data_stored":480542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084727,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605360,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93139792,"statfs":{"total":95995035648,"available":95375147008,"internally_reserved":0,"allocated":827392,"data_stored":480456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855790,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533212,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211940,"statfs":{"total":95995035648,"available":95449026560,"internally_reserved":0,"allocated":872448,"data_stored":481027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":46,"seq":197568496036,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603940,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":603134,"kb_avail":93141212,"statfs":{"total":95995035648,"available":95376601088,"internally_reserved":0,"allocated":815104,"data_stored":480080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":617609669},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-06T03:53:05.842 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-06T03:53:05.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9f192350ba5fe6d0a365b251a8c2af61f56a5fa6 shell --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-06T03:53:06.546 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:06 smithi153 bash[23852]: cluster 2024-04-06T03:53:05.644154+0000 osd.3 (osd.3) 793 : cluster 0 2.1b scrub starts 2024-04-06T03:53:06.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:06 smithi153 bash[23852]: cluster 2024-04-06T03:53:05.646818+0000 osd.3 (osd.3) 794 : cluster 0 2.1b scrub ok 2024-04-06T03:53:06.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:06 smithi050 bash[29221]: cluster 2024-04-06T03:53:05.644154+0000 osd.3 (osd.3) 793 : cluster 0 2.1b scrub starts 2024-04-06T03:53:06.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:06 smithi050 bash[29221]: cluster 2024-04-06T03:53:05.646818+0000 osd.3 (osd.3) 794 : cluster 0 2.1b scrub ok 2024-04-06T03:53:06.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:06 smithi050 bash[21467]: cluster 2024-04-06T03:53:05.644154+0000 osd.3 (osd.3) 793 : cluster 0 2.1b scrub starts 2024-04-06T03:53:06.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:06 smithi050 bash[21467]: cluster 2024-04-06T03:53:05.646818+0000 osd.3 (osd.3) 794 : cluster 0 2.1b scrub ok 2024-04-06T03:53:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:07 smithi153 bash[23852]: audit 2024-04-06T03:53:06.325257+0000 mon.a (mon.0) 1959 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:53:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:07 smithi153 bash[23852]: cluster 2024-04-06T03:53:06.612018+0000 osd.3 (osd.3) 795 : cluster 0 3.b scrub starts 2024-04-06T03:53:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:07 smithi153 bash[23852]: cluster 2024-04-06T03:53:06.616473+0000 osd.3 (osd.3) 796 : cluster 0 3.b scrub ok 2024-04-06T03:53:07.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:07 smithi153 bash[23852]: cluster 2024-04-06T03:53:06.732818+0000 mgr.y (mgr.24368) 1179 : cluster 0 pgmap v1292: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:07.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[21467]: audit 2024-04-06T03:53:06.325257+0000 mon.a (mon.0) 1959 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:53:07.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[21467]: cluster 2024-04-06T03:53:06.612018+0000 osd.3 (osd.3) 795 : cluster 0 3.b scrub starts 2024-04-06T03:53:07.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[21467]: cluster 2024-04-06T03:53:06.616473+0000 osd.3 (osd.3) 796 : cluster 0 3.b scrub ok 2024-04-06T03:53:07.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[21467]: cluster 2024-04-06T03:53:06.732818+0000 mgr.y (mgr.24368) 1179 : cluster 0 pgmap v1292: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:07.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[29221]: audit 2024-04-06T03:53:06.325257+0000 mon.a (mon.0) 1959 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-06T03:53:07.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[29221]: cluster 2024-04-06T03:53:06.612018+0000 osd.3 (osd.3) 795 : cluster 0 3.b scrub starts 2024-04-06T03:53:07.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[29221]: cluster 2024-04-06T03:53:06.616473+0000 osd.3 (osd.3) 796 : cluster 0 3.b scrub ok 2024-04-06T03:53:07.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:07 smithi050 bash[29221]: cluster 2024-04-06T03:53:06.732818+0000 mgr.y (mgr.24368) 1179 : cluster 0 pgmap v1292: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:06.902062+0000 osd.2 (osd.2) 635 : cluster 0 2.10 scrub starts 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:06.904728+0000 osd.2 (osd.2) 636 : cluster 0 2.10 scrub ok 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.171013+0000 osd.4 (osd.4) 751 : cluster 0 purged_snaps scrub starts 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.171155+0000 osd.4 (osd.4) 752 : cluster 0 purged_snaps scrub ok 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.297014+0000 mon.a (mon.0) 1960 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.589988+0000 osd.3 (osd.3) 797 : cluster 0 3.8 deep-scrub starts 2024-04-06T03:53:08.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:08 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.595851+0000 osd.3 (osd.3) 798 : cluster 0 3.8 deep-scrub ok 2024-04-06T03:53:08.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:06.902062+0000 osd.2 (osd.2) 635 : cluster 0 2.10 scrub starts 2024-04-06T03:53:08.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:06.904728+0000 osd.2 (osd.2) 636 : cluster 0 2.10 scrub ok 2024-04-06T03:53:08.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.171013+0000 osd.4 (osd.4) 751 : cluster 0 purged_snaps scrub starts 2024-04-06T03:53:08.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.171155+0000 osd.4 (osd.4) 752 : cluster 0 purged_snaps scrub ok 2024-04-06T03:53:08.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.297014+0000 mon.a (mon.0) 1960 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-04-06T03:53:08.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.589988+0000 osd.3 (osd.3) 797 : cluster 0 3.8 deep-scrub starts 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.595851+0000 osd.3 (osd.3) 798 : cluster 0 3.8 deep-scrub ok 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:06.902062+0000 osd.2 (osd.2) 635 : cluster 0 2.10 scrub starts 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:06.904728+0000 osd.2 (osd.2) 636 : cluster 0 2.10 scrub ok 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.171013+0000 osd.4 (osd.4) 751 : cluster 0 purged_snaps scrub starts 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.171155+0000 osd.4 (osd.4) 752 : cluster 0 purged_snaps scrub ok 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.297014+0000 mon.a (mon.0) 1960 : cluster 0 osdmap e449: 8 total, 8 up, 8 in 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.589988+0000 osd.3 (osd.3) 797 : cluster 0 3.8 deep-scrub starts 2024-04-06T03:53:08.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:08 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.595851+0000 osd.3 (osd.3) 798 : cluster 0 3.8 deep-scrub ok 2024-04-06T03:53:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:09 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.714895+0000 osd.7 (osd.7) 807 : cluster 0 2.b scrub starts 2024-04-06T03:53:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:09 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.717422+0000 osd.7 (osd.7) 808 : cluster 0 2.b scrub ok 2024-04-06T03:53:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:09 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.838811+0000 osd.6 (osd.6) 151 : cluster 0 2.11 scrub starts 2024-04-06T03:53:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:09 smithi153 bash[23852]: cluster 2024-04-06T03:53:07.841194+0000 osd.6 (osd.6) 152 : cluster 0 2.11 scrub ok 2024-04-06T03:53:09.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:09 smithi153 bash[23852]: cluster 2024-04-06T03:53:08.733478+0000 mgr.y (mgr.24368) 1180 : cluster 0 pgmap v1294: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:09.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.714895+0000 osd.7 (osd.7) 807 : cluster 0 2.b scrub starts 2024-04-06T03:53:09.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.717422+0000 osd.7 (osd.7) 808 : cluster 0 2.b scrub ok 2024-04-06T03:53:09.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.838811+0000 osd.6 (osd.6) 151 : cluster 0 2.11 scrub starts 2024-04-06T03:53:09.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[21467]: cluster 2024-04-06T03:53:07.841194+0000 osd.6 (osd.6) 152 : cluster 0 2.11 scrub ok 2024-04-06T03:53:09.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[21467]: cluster 2024-04-06T03:53:08.733478+0000 mgr.y (mgr.24368) 1180 : cluster 0 pgmap v1294: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:09.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:53:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:53:09.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.714895+0000 osd.7 (osd.7) 807 : cluster 0 2.b scrub starts 2024-04-06T03:53:09.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.717422+0000 osd.7 (osd.7) 808 : cluster 0 2.b scrub ok 2024-04-06T03:53:09.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.838811+0000 osd.6 (osd.6) 151 : cluster 0 2.11 scrub starts 2024-04-06T03:53:09.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[29221]: cluster 2024-04-06T03:53:07.841194+0000 osd.6 (osd.6) 152 : cluster 0 2.11 scrub ok 2024-04-06T03:53:09.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:09 smithi050 bash[29221]: cluster 2024-04-06T03:53:08.733478+0000 mgr.y (mgr.24368) 1180 : cluster 0 pgmap v1294: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:10.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:10 smithi153 bash[23852]: cluster 2024-04-06T03:53:08.875240+0000 osd.0 (osd.0) 760 : cluster 0 6.16 scrub starts 2024-04-06T03:53:10.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:10 smithi153 bash[23852]: cluster 2024-04-06T03:53:08.969783+0000 osd.0 (osd.0) 761 : cluster 0 6.16 scrub ok 2024-04-06T03:53:10.633 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/mon.a/config 2024-04-06T03:53:10.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:10 smithi050 bash[21467]: cluster 2024-04-06T03:53:08.875240+0000 osd.0 (osd.0) 760 : cluster 0 6.16 scrub starts 2024-04-06T03:53:10.658 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:10 smithi050 bash[21467]: cluster 2024-04-06T03:53:08.969783+0000 osd.0 (osd.0) 761 : cluster 0 6.16 scrub ok 2024-04-06T03:53:10.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:10 smithi050 bash[29221]: cluster 2024-04-06T03:53:08.875240+0000 osd.0 (osd.0) 760 : cluster 0 6.16 scrub starts 2024-04-06T03:53:10.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:10 smithi050 bash[29221]: cluster 2024-04-06T03:53:08.969783+0000 osd.0 (osd.0) 761 : cluster 0 6.16 scrub ok 2024-04-06T03:53:11.046 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:53:10 smithi153 bash[48791]: debug there is no tcmu-runner data available 2024-04-06T03:53:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[21467]: cluster 2024-04-06T03:53:09.920619+0000 osd.0 (osd.0) 762 : cluster 0 6.9 scrub starts 2024-04-06T03:53:11.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[21467]: cluster 2024-04-06T03:53:10.083084+0000 osd.0 (osd.0) 763 : cluster 0 6.9 scrub ok 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[21467]: cluster 2024-04-06T03:53:10.273161+0000 osd.5 (osd.5) 227 : cluster 0 2.16 scrub starts 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[21467]: cluster 2024-04-06T03:53:10.275609+0000 osd.5 (osd.5) 228 : cluster 0 2.16 scrub ok 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[21467]: audit 2024-04-06T03:53:10.619252+0000 mgr.y (mgr.24368) 1181 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[21467]: cluster 2024-04-06T03:53:10.734997+0000 mgr.y (mgr.24368) 1182 : cluster 0 pgmap v1295: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[29221]: cluster 2024-04-06T03:53:09.920619+0000 osd.0 (osd.0) 762 : cluster 0 6.9 scrub starts 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[29221]: cluster 2024-04-06T03:53:10.083084+0000 osd.0 (osd.0) 763 : cluster 0 6.9 scrub ok 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[29221]: cluster 2024-04-06T03:53:10.273161+0000 osd.5 (osd.5) 227 : cluster 0 2.16 scrub starts 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[29221]: cluster 2024-04-06T03:53:10.275609+0000 osd.5 (osd.5) 228 : cluster 0 2.16 scrub ok 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[29221]: audit 2024-04-06T03:53:10.619252+0000 mgr.y (mgr.24368) 1181 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:53:11.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:11 smithi050 bash[29221]: cluster 2024-04-06T03:53:10.734997+0000 mgr.y (mgr.24368) 1182 : cluster 0 pgmap v1295: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:11 smithi153 bash[23852]: cluster 2024-04-06T03:53:09.920619+0000 osd.0 (osd.0) 762 : cluster 0 6.9 scrub starts 2024-04-06T03:53:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:11 smithi153 bash[23852]: cluster 2024-04-06T03:53:10.083084+0000 osd.0 (osd.0) 763 : cluster 0 6.9 scrub ok 2024-04-06T03:53:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:11 smithi153 bash[23852]: cluster 2024-04-06T03:53:10.273161+0000 osd.5 (osd.5) 227 : cluster 0 2.16 scrub starts 2024-04-06T03:53:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:11 smithi153 bash[23852]: cluster 2024-04-06T03:53:10.275609+0000 osd.5 (osd.5) 228 : cluster 0 2.16 scrub ok 2024-04-06T03:53:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:11 smithi153 bash[23852]: audit 2024-04-06T03:53:10.619252+0000 mgr.y (mgr.24368) 1181 : audit 0 from='client.24415 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-06T03:53:11.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:11 smithi153 bash[23852]: cluster 2024-04-06T03:53:10.734997+0000 mgr.y (mgr.24368) 1182 : cluster 0 pgmap v1295: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:12.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[29221]: cluster 2024-04-06T03:53:10.923358+0000 osd.0 (osd.0) 764 : cluster 0 6.10 scrub starts 2024-04-06T03:53:12.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.014321+0000 osd.0 (osd.0) 765 : cluster 0 6.10 scrub ok 2024-04-06T03:53:12.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.298948+0000 osd.5 (osd.5) 229 : cluster 0 3.16 scrub starts 2024-04-06T03:53:12.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.304046+0000 osd.5 (osd.5) 230 : cluster 0 3.16 scrub ok 2024-04-06T03:53:12.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.523643+0000 osd.3 (osd.3) 799 : cluster 0 5.14 scrub starts 2024-04-06T03:53:12.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.526707+0000 osd.3 (osd.3) 800 : cluster 0 5.14 scrub ok 2024-04-06T03:53:12.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[21467]: cluster 2024-04-06T03:53:10.923358+0000 osd.0 (osd.0) 764 : cluster 0 6.10 scrub starts 2024-04-06T03:53:12.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.014321+0000 osd.0 (osd.0) 765 : cluster 0 6.10 scrub ok 2024-04-06T03:53:12.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.298948+0000 osd.5 (osd.5) 229 : cluster 0 3.16 scrub starts 2024-04-06T03:53:12.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.304046+0000 osd.5 (osd.5) 230 : cluster 0 3.16 scrub ok 2024-04-06T03:53:12.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.523643+0000 osd.3 (osd.3) 799 : cluster 0 5.14 scrub starts 2024-04-06T03:53:12.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:12 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.526707+0000 osd.3 (osd.3) 800 : cluster 0 5.14 scrub ok 2024-04-06T03:53:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:12 smithi153 bash[23852]: cluster 2024-04-06T03:53:10.923358+0000 osd.0 (osd.0) 764 : cluster 0 6.10 scrub starts 2024-04-06T03:53:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:12 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.014321+0000 osd.0 (osd.0) 765 : cluster 0 6.10 scrub ok 2024-04-06T03:53:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:12 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.298948+0000 osd.5 (osd.5) 229 : cluster 0 3.16 scrub starts 2024-04-06T03:53:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:12 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.304046+0000 osd.5 (osd.5) 230 : cluster 0 3.16 scrub ok 2024-04-06T03:53:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:12 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.523643+0000 osd.3 (osd.3) 799 : cluster 0 5.14 scrub starts 2024-04-06T03:53:12.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:12 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.526707+0000 osd.3 (osd.3) 800 : cluster 0 5.14 scrub ok 2024-04-06T03:53:13.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:13 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.652246+0000 osd.7 (osd.7) 809 : cluster 0 5.1a scrub starts 2024-04-06T03:53:13.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:13 smithi050 bash[29221]: cluster 2024-04-06T03:53:11.654553+0000 osd.7 (osd.7) 810 : cluster 0 5.1a scrub ok 2024-04-06T03:53:13.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:13 smithi050 bash[29221]: cluster 2024-04-06T03:53:12.735707+0000 mgr.y (mgr.24368) 1183 : cluster 0 pgmap v1296: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:13.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:13 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.652246+0000 osd.7 (osd.7) 809 : cluster 0 5.1a scrub starts 2024-04-06T03:53:13.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:13 smithi050 bash[21467]: cluster 2024-04-06T03:53:11.654553+0000 osd.7 (osd.7) 810 : cluster 0 5.1a scrub ok 2024-04-06T03:53:13.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:13 smithi050 bash[21467]: cluster 2024-04-06T03:53:12.735707+0000 mgr.y (mgr.24368) 1183 : cluster 0 pgmap v1296: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:13 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.652246+0000 osd.7 (osd.7) 809 : cluster 0 5.1a scrub starts 2024-04-06T03:53:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:13 smithi153 bash[23852]: cluster 2024-04-06T03:53:11.654553+0000 osd.7 (osd.7) 810 : cluster 0 5.1a scrub ok 2024-04-06T03:53:13.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:13 smithi153 bash[23852]: cluster 2024-04-06T03:53:12.735707+0000 mgr.y (mgr.24368) 1183 : cluster 0 pgmap v1296: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:14.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21467]: cluster 2024-04-06T03:53:13.305244+0000 osd.5 (osd.5) 231 : cluster 0 6.d scrub starts 2024-04-06T03:53:14.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21467]: cluster 2024-04-06T03:53:13.478212+0000 osd.3 (osd.3) 801 : cluster 0 4.0 scrub starts 2024-04-06T03:53:14.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21467]: cluster 2024-04-06T03:53:13.480314+0000 osd.5 (osd.5) 232 : cluster 0 6.d scrub ok 2024-04-06T03:53:14.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21467]: cluster 2024-04-06T03:53:13.481191+0000 osd.3 (osd.3) 802 : cluster 0 4.0 scrub ok 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[29221]: cluster 2024-04-06T03:53:13.305244+0000 osd.5 (osd.5) 231 : cluster 0 6.d scrub starts 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[29221]: cluster 2024-04-06T03:53:13.478212+0000 osd.3 (osd.3) 801 : cluster 0 4.0 scrub starts 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[29221]: cluster 2024-04-06T03:53:13.480314+0000 osd.5 (osd.5) 232 : cluster 0 6.d scrub ok 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[29221]: cluster 2024-04-06T03:53:13.481191+0000 osd.3 (osd.3) 802 : cluster 0 4.0 scrub ok 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21725]: [06/Apr/2024:03:53:14] ENGINE Bus STOPPING 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21725]: [06/Apr/2024:03:53:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21725]: [06/Apr/2024:03:53:14] ENGINE Bus STOPPED 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21725]: [06/Apr/2024:03:53:14] ENGINE Bus STARTING 2024-04-06T03:53:14.741 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21725]: [06/Apr/2024:03:53:14] ENGINE Serving on http://:::9283 2024-04-06T03:53:14.742 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:14 smithi050 bash[21725]: [06/Apr/2024:03:53:14] ENGINE Bus STARTED 2024-04-06T03:53:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:14 smithi153 bash[23852]: cluster 2024-04-06T03:53:13.305244+0000 osd.5 (osd.5) 231 : cluster 0 6.d scrub starts 2024-04-06T03:53:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:14 smithi153 bash[23852]: cluster 2024-04-06T03:53:13.478212+0000 osd.3 (osd.3) 801 : cluster 0 4.0 scrub starts 2024-04-06T03:53:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:14 smithi153 bash[23852]: cluster 2024-04-06T03:53:13.480314+0000 osd.5 (osd.5) 232 : cluster 0 6.d scrub ok 2024-04-06T03:53:14.797 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:14 smithi153 bash[23852]: cluster 2024-04-06T03:53:13.481191+0000 osd.3 (osd.3) 802 : cluster 0 4.0 scrub ok 2024-04-06T03:53:15.211 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-06T03:53:15.220 INFO:tasks.cephadm:Teardown begin 2024-04-06T03:53:15.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:53:15.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:53:15.270 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-06T03:53:15.270 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:53:15.310 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-06T03:53:15.327 INFO:tasks.cephadm:Stopping all daemons... 2024-04-06T03:53:15.327 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-06T03:53:15.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a 2024-04-06T03:53:15.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: cluster 2024-04-06T03:53:13.964389+0000 osd.0 (osd.0) 766 : cluster 0 3.6 scrub starts 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: cluster 2024-04-06T03:53:13.967966+0000 osd.0 (osd.0) 767 : cluster 0 3.6 scrub ok 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.346325+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.50:0/870117792' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.346928+0000 mon.a (mon.0) 1961 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.351070+0000 mon.a (mon.0) 1962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.375999+0000 mon.a (mon.0) 1963 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.379479+0000 mon.a (mon.0) 1964 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.380181+0000 mon.a (mon.0) 1965 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: audit 2024-04-06T03:53:14.385443+0000 mon.a (mon.0) 1966 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:53:15.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[29221]: cluster 2024-04-06T03:53:14.736771+0000 mgr.y (mgr.24368) 1184 : cluster 0 pgmap v1297: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: cluster 2024-04-06T03:53:13.964389+0000 osd.0 (osd.0) 766 : cluster 0 3.6 scrub starts 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: cluster 2024-04-06T03:53:13.967966+0000 osd.0 (osd.0) 767 : cluster 0 3.6 scrub ok 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.346325+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.50:0/870117792' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.346928+0000 mon.a (mon.0) 1961 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.351070+0000 mon.a (mon.0) 1962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.375999+0000 mon.a (mon.0) 1963 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.379479+0000 mon.a (mon.0) 1964 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.380181+0000 mon.a (mon.0) 1965 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: audit 2024-04-06T03:53:14.385443+0000 mon.a (mon.0) 1966 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: cluster 2024-04-06T03:53:14.736771+0000 mgr.y (mgr.24368) 1184 : cluster 0 pgmap v1297: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:15.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 systemd[1]: Stopping Ceph mon.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: cluster 2024-04-06T03:53:13.964389+0000 osd.0 (osd.0) 766 : cluster 0 3.6 scrub starts 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: cluster 2024-04-06T03:53:13.967966+0000 osd.0 (osd.0) 767 : cluster 0 3.6 scrub ok 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.346325+0000 mon.c (mon.2) 142 : audit 1 from='client.? 172.21.15.50:0/870117792' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.346928+0000 mon.a (mon.0) 1961 : audit 1 from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.351070+0000 mon.a (mon.0) 1962 : audit 1 from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.375999+0000 mon.a (mon.0) 1963 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-06T03:53:15.547 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.379479+0000 mon.a (mon.0) 1964 : audit 0 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-06T03:53:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.380181+0000 mon.a (mon.0) 1965 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-06T03:53:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: audit 2024-04-06T03:53:14.385443+0000 mon.a (mon.0) 1966 : audit 1 from='mgr.24368 172.21.15.50:0/2066281285' entity='mgr.y' 2024-04-06T03:53:15.548 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:15 smithi153 bash[23852]: cluster 2024-04-06T03:53:14.736771+0000 mgr.y (mgr.24368) 1184 : cluster 0 pgmap v1297: 132 pgs: 132 active+clean; 582 KiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-06T03:53:15.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: debug 2024-04-06T03:53:15.548+0000 7f7030e2c700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-06T03:53:15.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21467]: debug 2024-04-06T03:53:15.548+0000 7f7030e2c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-06T03:53:15.578 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21725]: [06/Apr/2024:03:53:15] ENGINE Bus STOPPING 2024-04-06T03:53:15.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21725]: [06/Apr/2024:03:53:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-06T03:53:15.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21725]: [06/Apr/2024:03:53:15] ENGINE Bus STOPPED 2024-04-06T03:53:15.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21725]: [06/Apr/2024:03:53:15] ENGINE Bus STARTING 2024-04-06T03:53:15.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21725]: [06/Apr/2024:03:53:15] ENGINE Serving on http://:::9283 2024-04-06T03:53:15.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:15 smithi050 bash[21725]: [06/Apr/2024:03:53:15] ENGINE Bus STARTED 2024-04-06T03:53:16.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.a.service' 2024-04-06T03:53:16.684 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:16.685 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-06T03:53:16.685 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-06T03:53:16.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.c 2024-04-06T03:53:16.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:16 smithi050 systemd[1]: Stopping Ceph mon.c for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:16.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:16 smithi050 bash[29221]: debug 2024-04-06T03:53:16.864+0000 7fcb842e1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-06T03:53:16.990 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 06 03:53:16 smithi050 bash[29221]: debug 2024-04-06T03:53:16.864+0000 7fcb842e1700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-06T03:53:18.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.c.service' 2024-04-06T03:53:18.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:18.107 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-06T03:53:18.107 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-06T03:53:18.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.b 2024-04-06T03:53:18.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:18 smithi153 systemd[1]: Stopping Ceph mon.b for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:18.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:18 smithi153 bash[23852]: debug 2024-04-06T03:53:18.299+0000 7f6c542f9700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-06T03:53:18.514 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:18 smithi153 bash[23852]: debug 2024-04-06T03:53:18.299+0000 7f6c542f9700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-06T03:53:19.417 INFO:journalctl@ceph.mon.b.smithi153.stdout:Apr 06 03:53:19 smithi153 bash[114010]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-mon-b 2024-04-06T03:53:19.421 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mon.b.service' 2024-04-06T03:53:19.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:19.469 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-06T03:53:19.469 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-06T03:53:19.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y 2024-04-06T03:53:19.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:19 smithi050 bash[21725]: ::ffff:172.21.15.153 - - [06/Apr/2024:03:53:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-06T03:53:19.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:19 smithi050 systemd[1]: Stopping Ceph mgr.y for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:19.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:19 smithi050 bash[21725]: debug 2024-04-06T03:53:19.612+0000 7f6504b8c700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:53:19.740 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:19 smithi050 bash[21725]: debug 2024-04-06T03:53:19.612+0000 7f6504b8c700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-06T03:53:20.664 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 06 03:53:20 smithi050 bash[167819]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-mgr-y 2024-04-06T03:53:20.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.y.service' 2024-04-06T03:53:20.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:20.698 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-06T03:53:20.698 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-06T03:53:20.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.x 2024-04-06T03:53:21.047 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Apr 06 03:53:20 smithi153 systemd[1]: Stopping Ceph mgr.x for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:21.739 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@mgr.x.service' 2024-04-06T03:53:21.764 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:21.764 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-06T03:53:21.764 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-06T03:53:21.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.0 2024-04-06T03:53:22.240 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:53:21 smithi050 systemd[1]: Stopping Ceph osd.0 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:22.241 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:53:21 smithi050 bash[32260]: debug 2024-04-06T03:53:21.900+0000 7fe64eca8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:53:22.241 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:53:21 smithi050 bash[32260]: debug 2024-04-06T03:53:21.900+0000 7fe64eca8700 -1 osd.0 449 *** Got signal Terminated *** 2024-04-06T03:53:22.241 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:53:21 smithi050 bash[32260]: debug 2024-04-06T03:53:21.900+0000 7fe64eca8700 -1 osd.0 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:53:28.532 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:53:28 smithi050 bash[167919]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T03:53:28.990 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 06 03:53:28 smithi050 bash[167984]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-0 2024-04-06T03:53:31.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-06T03:53:31.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:31.168 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-06T03:53:31.168 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-06T03:53:31.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.1 2024-04-06T03:53:31.490 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:53:31 smithi050 systemd[1]: Stopping Ceph osd.1 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:31.490 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:53:31 smithi050 bash[37631]: debug 2024-04-06T03:53:31.336+0000 7fc5a3c83700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:53:31.491 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:53:31 smithi050 bash[37631]: debug 2024-04-06T03:53:31.336+0000 7fc5a3c83700 -1 osd.1 449 *** Got signal Terminated *** 2024-04-06T03:53:31.491 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:53:31 smithi050 bash[37631]: debug 2024-04-06T03:53:31.336+0000 7fc5a3c83700 -1 osd.1 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:53:37.862 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:53:37 smithi050 bash[168119]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T03:53:37.862 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 06 03:53:37 smithi050 bash[168185]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-1 2024-04-06T03:53:39.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-06T03:53:39.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:39.775 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-06T03:53:39.775 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-06T03:53:39.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.2 2024-04-06T03:53:40.240 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:53:39 smithi050 systemd[1]: Stopping Ceph osd.2 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:40.240 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:53:39 smithi050 bash[43017]: debug 2024-04-06T03:53:39.955+0000 7fa79cb12700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:53:40.240 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:53:39 smithi050 bash[43017]: debug 2024-04-06T03:53:39.955+0000 7fa79cb12700 -1 osd.2 449 *** Got signal Terminated *** 2024-04-06T03:53:40.240 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:53:39 smithi050 bash[43017]: debug 2024-04-06T03:53:39.955+0000 7fa79cb12700 -1 osd.2 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:53:46.740 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:53:46 smithi050 bash[168317]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T03:53:46.740 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 06 03:53:46 smithi050 bash[168381]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-2 2024-04-06T03:53:48.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-06T03:53:48.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:48.457 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-06T03:53:48.458 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-06T03:53:48.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.3 2024-04-06T03:53:48.740 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:48 smithi050 systemd[1]: Stopping Ceph osd.3 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:48.740 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:48 smithi050 bash[48910]: debug 2024-04-06T03:53:48.663+0000 7f904e72b700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:53:48.740 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:48 smithi050 bash[48910]: debug 2024-04-06T03:53:48.663+0000 7f904e72b700 -1 osd.3 449 *** Got signal Terminated *** 2024-04-06T03:53:48.740 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:48 smithi050 bash[48910]: debug 2024-04-06T03:53:48.663+0000 7f904e72b700 -1 osd.3 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:53:48.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:48 smithi153 bash[44619]: debug 2024-04-06T03:53:48.431+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:49.431 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:49 smithi153 bash[27379]: debug 2024-04-06T03:53:49.051+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:49.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:49 smithi153 bash[44619]: debug 2024-04-06T03:53:49.427+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:50.396 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:50 smithi153 bash[27379]: debug 2024-04-06T03:53:50.091+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:50.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:50 smithi153 bash[44619]: debug 2024-04-06T03:53:50.391+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:50.797 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:50 smithi153 bash[33079]: debug 2024-04-06T03:53:50.431+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:50.990 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:50 smithi050 bash[48910]: debug 2024-04-06T03:53:50.563+0000 7f904ad1f700 -1 osd.3 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.336785+0000 front 2024-04-06T03:53:26.336522+0000 (oldest deadline 2024-04-06T03:53:50.436637+0000) 2024-04-06T03:53:51.375 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:51 smithi153 bash[27379]: debug 2024-04-06T03:53:51.115+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:51.375 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:51 smithi153 bash[38802]: debug 2024-04-06T03:53:51.091+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:51.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:51 smithi153 bash[44619]: debug 2024-04-06T03:53:51.371+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:51.797 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:51 smithi153 bash[33079]: debug 2024-04-06T03:53:51.411+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:51.990 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:51 smithi050 bash[48910]: debug 2024-04-06T03:53:51.523+0000 7f904ad1f700 -1 osd.3 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.336785+0000 front 2024-04-06T03:53:26.336522+0000 (oldest deadline 2024-04-06T03:53:50.436637+0000) 2024-04-06T03:53:52.333 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:52 smithi153 bash[27379]: debug 2024-04-06T03:53:52.091+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:52.333 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:52 smithi153 bash[38802]: debug 2024-04-06T03:53:52.075+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:52.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:52 smithi153 bash[33079]: debug 2024-04-06T03:53:52.443+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:52.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:52 smithi153 bash[44619]: debug 2024-04-06T03:53:52.327+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:52.990 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:52 smithi050 bash[48910]: debug 2024-04-06T03:53:52.575+0000 7f904ad1f700 -1 osd.3 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.336785+0000 front 2024-04-06T03:53:26.336522+0000 (oldest deadline 2024-04-06T03:53:50.436637+0000) 2024-04-06T03:53:53.446 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:53 smithi153 bash[27379]: debug 2024-04-06T03:53:53.127+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:53.446 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:53 smithi153 bash[38802]: debug 2024-04-06T03:53:53.107+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:53.446 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:53 smithi153 bash[44619]: debug 2024-04-06T03:53:53.287+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:53.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:53 smithi153 bash[33079]: debug 2024-04-06T03:53:53.443+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:53.990 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:53 smithi050 bash[48910]: debug 2024-04-06T03:53:53.531+0000 7f904ad1f700 -1 osd.3 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.336785+0000 front 2024-04-06T03:53:26.336522+0000 (oldest deadline 2024-04-06T03:53:50.436637+0000) 2024-04-06T03:53:54.435 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:54 smithi153 bash[27379]: debug 2024-04-06T03:53:54.163+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:54.436 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:54 smithi153 bash[38802]: debug 2024-04-06T03:53:54.155+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:54.436 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:54 smithi153 bash[44619]: debug 2024-04-06T03:53:54.243+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:54.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:54 smithi153 bash[33079]: debug 2024-04-06T03:53:54.431+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:55.278 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:54 smithi050 bash[168514]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T03:53:55.278 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 06 03:53:55 smithi050 bash[168578]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-3 2024-04-06T03:53:55.445 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:55 smithi153 bash[27379]: debug 2024-04-06T03:53:55.135+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:55.446 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:55 smithi153 bash[38802]: debug 2024-04-06T03:53:55.163+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:55.446 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:55 smithi153 bash[44619]: debug 2024-04-06T03:53:55.199+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:55.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:55 smithi153 bash[33079]: debug 2024-04-06T03:53:55.443+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:56.412 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:56 smithi153 bash[27379]: debug 2024-04-06T03:53:56.187+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:56.412 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:56 smithi153 bash[38802]: debug 2024-04-06T03:53:56.155+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:56.412 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:56 smithi153 bash[44619]: debug 2024-04-06T03:53:56.207+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:56.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:56 smithi153 bash[33079]: debug 2024-04-06T03:53:56.411+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:57.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-06T03:53:57.241 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:53:57.241 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-06T03:53:57.241 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-06T03:53:57.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.4 2024-04-06T03:53:57.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[38802]: debug 2024-04-06T03:53:57.163+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:57.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[44619]: debug 2024-04-06T03:53:57.231+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:57.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[33079]: debug 2024-04-06T03:53:57.375+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:57.547 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[27379]: debug 2024-04-06T03:53:57.191+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:57.547 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:57 smithi153 systemd[1]: Stopping Ceph osd.4 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:53:57.547 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[27379]: debug 2024-04-06T03:53:57.379+0000 7f5681a26700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:53:57.547 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[27379]: debug 2024-04-06T03:53:57.379+0000 7f5681a26700 -1 osd.4 449 *** Got signal Terminated *** 2024-04-06T03:53:57.548 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:57 smithi153 bash[27379]: debug 2024-04-06T03:53:57.379+0000 7f5681a26700 -1 osd.4 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:53:58.546 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[27379]: debug 2024-04-06T03:53:58.214+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:58.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[38802]: debug 2024-04-06T03:53:58.166+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:58.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[44619]: debug 2024-04-06T03:53:58.202+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:58.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[33079]: debug 2024-04-06T03:53:58.378+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:59.159 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[52536]: ts=2024-04-06T03:53:58.868Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-06T03:53:59.159 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[52536]: ts=2024-04-06T03:53:58.870Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-06T03:53:59.159 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[52536]: ts=2024-04-06T03:53:58.881Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-06T03:53:59.159 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[52536]: ts=2024-04-06T03:53:58.881Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-06T03:53:59.160 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:53:58 smithi153 bash[52536]: ts=2024-04-06T03:53:58.881Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-06T03:53:59.546 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:53:59 smithi153 bash[27379]: debug 2024-04-06T03:53:59.210+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:53:59.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:53:59 smithi153 bash[44619]: debug 2024-04-06T03:53:59.206+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:53:59.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:59 smithi153 bash[33079]: debug 2024-04-06T03:53:59.378+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:53:59.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:53:59 smithi153 bash[33079]: debug 2024-04-06T03:53:59.378+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:53:59.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:59 smithi153 bash[38802]: debug 2024-04-06T03:53:59.154+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:53:59.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:53:59 smithi153 bash[38802]: debug 2024-04-06T03:53:59.154+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:00.546 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:00 smithi153 bash[27379]: debug 2024-04-06T03:54:00.190+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:00.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:00 smithi153 bash[44619]: debug 2024-04-06T03:54:00.210+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:00.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:00 smithi153 bash[33079]: debug 2024-04-06T03:54:00.334+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:00.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:00 smithi153 bash[33079]: debug 2024-04-06T03:54:00.334+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:00.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:00 smithi153 bash[38802]: debug 2024-04-06T03:54:00.178+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:00.548 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:00 smithi153 bash[38802]: debug 2024-04-06T03:54:00.178+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:01.546 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[27379]: debug 2024-04-06T03:54:01.222+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:01.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[38802]: debug 2024-04-06T03:54:01.146+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:01.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[38802]: debug 2024-04-06T03:54:01.146+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:01.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[44619]: debug 2024-04-06T03:54:01.186+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:01.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[44619]: debug 2024-04-06T03:54:01.186+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:01.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[33079]: debug 2024-04-06T03:54:01.358+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:01.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:01 smithi153 bash[33079]: debug 2024-04-06T03:54:01.358+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:02.546 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug 2024-04-06T03:54:02.262+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:02.547 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug 2024-04-06T03:54:02.262+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:36.201541+0000 front 2024-04-06T03:53:36.201910+0000 (oldest deadline 2024-04-06T03:54:01.501496+0000) 2024-04-06T03:54:02.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[38802]: debug 2024-04-06T03:54:02.114+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:02.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[38802]: debug 2024-04-06T03:54:02.114+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:02.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[44619]: debug 2024-04-06T03:54:02.214+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:02.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[44619]: debug 2024-04-06T03:54:02.214+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:02.548 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[33079]: debug 2024-04-06T03:54:02.346+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:02.548 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[33079]: debug 2024-04-06T03:54:02.346+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:03.047 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: *** Caught signal (Segmentation fault) ** 2024-04-06T03:54:03.047 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: in thread 7f566cff8700 thread_name:safe_timer 2024-04-06T03:54:03.047 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev) 2024-04-06T03:54:03.047 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5688e7cd20] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55c8acfcadca] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55c8ad058968] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55c8acc3e733] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55c8acec9841] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55c8acbba414] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55c8acbbc47a] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55c8acbbe3e3] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55c8acb1f79a] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55c8ad17a735] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55c8ad17bf61] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5688e721ca] 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 13: clone() 2024-04-06T03:54:03.048 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug 2024-04-06T03:54:02.738+0000 7f566cff8700 -1 *** Caught signal (Segmentation fault) ** 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: in thread 7f566cff8700 thread_name:safe_timer 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev) 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5688e7cd20] 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55c8acfcadca] 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55c8ad058968] 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55c8acc3e733] 2024-04-06T03:54:03.049 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55c8acec9841] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55c8acbba414] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55c8acbbc47a] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55c8acbbe3e3] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55c8acb1f79a] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55c8ad17a735] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55c8ad17bf61] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5688e721ca] 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 13: clone() 2024-04-06T03:54:03.050 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -3017> 2024-04-06T03:53:49.051+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -2730> 2024-04-06T03:53:50.091+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -2629> 2024-04-06T03:53:51.115+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -2145> 2024-04-06T03:53:52.091+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1871> 2024-04-06T03:53:53.127+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1677> 2024-04-06T03:53:54.163+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1485> 2024-04-06T03:53:55.135+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1428> 2024-04-06T03:53:56.187+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1339> 2024-04-06T03:53:57.191+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1170> 2024-04-06T03:53:57.379+0000 7f5681a26700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1169> 2024-04-06T03:53:57.379+0000 7f5681a26700 -1 osd.4 449 *** Got signal Terminated *** 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -1167> 2024-04-06T03:53:57.379+0000 7f5681a26700 -1 osd.4 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -990> 2024-04-06T03:53:58.214+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.051 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -860> 2024-04-06T03:53:59.210+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -432> 2024-04-06T03:54:00.190+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -245> 2024-04-06T03:54:01.222+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -141> 2024-04-06T03:54:02.262+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.297947+0000 front 2024-04-06T03:53:24.297443+0000 (oldest deadline 2024-04-06T03:53:48.397266+0000) 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug -140> 2024-04-06T03:54:02.262+0000 7f567e01a700 -1 osd.4 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:36.201541+0000 front 2024-04-06T03:53:36.201910+0000 (oldest deadline 2024-04-06T03:54:01.501496+0000) 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: debug 0> 2024-04-06T03:54:02.738+0000 7f566cff8700 -1 *** Caught signal (Segmentation fault) ** 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: in thread 7f566cff8700 thread_name:safe_timer 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: ceph version 19.0.0-2027-g9f192350 (9f192350ba5fe6d0a365b251a8c2af61f56a5fa6) squid (dev) 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5688e7cd20] 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55c8acfcadca] 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55c8ad058968] 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55c8acc3e733] 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55c8acec9841] 2024-04-06T03:54:03.052 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55c8acbba414] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55c8acbbc47a] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55c8acbbe3e3] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55c8acb1f79a] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x55c8ad17a735] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55c8ad17bf61] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5688e721ca] 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: 13: clone() 2024-04-06T03:54:03.053 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:02 smithi153 bash[27379]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-06T03:54:03.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[38802]: debug 2024-04-06T03:54:03.130+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:03.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[38802]: debug 2024-04-06T03:54:03.130+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:03.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[44619]: debug 2024-04-06T03:54:03.246+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:03.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[44619]: debug 2024-04-06T03:54:03.246+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:03.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[33079]: debug 2024-04-06T03:54:03.342+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:03.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[33079]: debug 2024-04-06T03:54:03.342+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:03.876 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[114213]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T03:54:03.876 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:03 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.4.service: Main process exited, code=exited, status=139/n/a 2024-04-06T03:54:03.877 INFO:journalctl@ceph.osd.4.smithi153.stdout:Apr 06 03:54:03 smithi153 bash[114277]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-4 2024-04-06T03:54:04.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:04 smithi153 bash[38802]: debug 2024-04-06T03:54:04.098+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:04.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:04 smithi153 bash[38802]: debug 2024-04-06T03:54:04.098+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:04.546 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:04 smithi153 bash[33079]: debug 2024-04-06T03:54:04.366+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:04.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:04 smithi153 bash[33079]: debug 2024-04-06T03:54:04.366+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:04.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:04 smithi153 bash[44619]: debug 2024-04-06T03:54:04.250+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:04.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:04 smithi153 bash[44619]: debug 2024-04-06T03:54:04.250+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:05.410 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:05 smithi153 bash[44619]: debug 2024-04-06T03:54:05.210+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:05.410 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:05 smithi153 bash[44619]: debug 2024-04-06T03:54:05.210+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:05.410 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:05 smithi153 bash[38802]: debug 2024-04-06T03:54:05.138+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:05.410 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:05 smithi153 bash[38802]: debug 2024-04-06T03:54:05.138+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:05.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:05 smithi153 bash[33079]: debug 2024-04-06T03:54:05.406+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:05.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:05 smithi153 bash[33079]: debug 2024-04-06T03:54:05.406+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:05.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-06T03:54:06.023 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:54:06.024 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-06T03:54:06.024 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-06T03:54:06.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.5 2024-04-06T03:54:06.296 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[38802]: debug 2024-04-06T03:54:06.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:06.297 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[38802]: debug 2024-04-06T03:54:06.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:06.297 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[44619]: debug 2024-04-06T03:54:06.234+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:06.297 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[44619]: debug 2024-04-06T03:54:06.234+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:06.297 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:06 smithi153 systemd[1]: Stopping Ceph osd.5 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:06.297 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[33079]: debug 2024-04-06T03:54:06.222+0000 7ffa62217700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:54:06.297 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[33079]: debug 2024-04-06T03:54:06.222+0000 7ffa62217700 -1 osd.5 449 *** Got signal Terminated *** 2024-04-06T03:54:06.298 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[33079]: debug 2024-04-06T03:54:06.222+0000 7ffa62217700 -1 osd.5 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:54:06.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[33079]: debug 2024-04-06T03:54:06.358+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:06.796 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:06 smithi153 bash[33079]: debug 2024-04-06T03:54:06.358+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:07.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[38802]: debug 2024-04-06T03:54:07.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:07.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[38802]: debug 2024-04-06T03:54:07.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:07.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[38802]: debug 2024-04-06T03:54:07.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:07.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[33079]: debug 2024-04-06T03:54:07.362+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:07.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[33079]: debug 2024-04-06T03:54:07.362+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:07.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[44619]: debug 2024-04-06T03:54:07.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:07.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[44619]: debug 2024-04-06T03:54:07.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:07.548 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:07 smithi153 bash[44619]: debug 2024-04-06T03:54:07.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:08.546 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[33079]: debug 2024-04-06T03:54:08.354+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[33079]: debug 2024-04-06T03:54:08.354+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[38802]: debug 2024-04-06T03:54:08.142+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[38802]: debug 2024-04-06T03:54:08.142+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[38802]: debug 2024-04-06T03:54:08.142+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[44619]: debug 2024-04-06T03:54:08.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[44619]: debug 2024-04-06T03:54:08.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:08.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:08 smithi153 bash[44619]: debug 2024-04-06T03:54:08.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:09.546 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[33079]: debug 2024-04-06T03:54:09.326+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[33079]: debug 2024-04-06T03:54:09.326+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[44619]: debug 2024-04-06T03:54:09.246+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[44619]: debug 2024-04-06T03:54:09.246+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[44619]: debug 2024-04-06T03:54:09.246+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[38802]: debug 2024-04-06T03:54:09.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[38802]: debug 2024-04-06T03:54:09.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:09.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:09 smithi153 bash[38802]: debug 2024-04-06T03:54:09.150+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:10.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[38802]: debug 2024-04-06T03:54:10.198+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:10.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[38802]: debug 2024-04-06T03:54:10.198+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:10.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[38802]: debug 2024-04-06T03:54:10.198+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:10.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[33079]: debug 2024-04-06T03:54:10.302+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.874247+0000 front 2024-04-06T03:53:24.873919+0000 (oldest deadline 2024-04-06T03:53:50.173937+0000) 2024-04-06T03:54:10.547 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[33079]: debug 2024-04-06T03:54:10.302+0000 7ffa5e80b700 -1 osd.5 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.976672+0000 front 2024-04-06T03:53:35.976420+0000 (oldest deadline 2024-04-06T03:53:58.876295+0000) 2024-04-06T03:54:10.548 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[44619]: debug 2024-04-06T03:54:10.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:10.548 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[44619]: debug 2024-04-06T03:54:10.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:10.548 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:10 smithi153 bash[44619]: debug 2024-04-06T03:54:10.278+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:11.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:11 smithi153 bash[38802]: debug 2024-04-06T03:54:11.198+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:11.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:11 smithi153 bash[38802]: debug 2024-04-06T03:54:11.198+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:11.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:11 smithi153 bash[38802]: debug 2024-04-06T03:54:11.198+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:11.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:11 smithi153 bash[44619]: debug 2024-04-06T03:54:11.262+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:11.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:11 smithi153 bash[44619]: debug 2024-04-06T03:54:11.262+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:11.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:11 smithi153 bash[44619]: debug 2024-04-06T03:54:11.262+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:12.489 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[38802]: debug 2024-04-06T03:54:12.230+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:12.489 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[38802]: debug 2024-04-06T03:54:12.230+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:12.489 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[38802]: debug 2024-04-06T03:54:12.230+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:12.489 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[44619]: debug 2024-04-06T03:54:12.222+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:12.489 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[44619]: debug 2024-04-06T03:54:12.222+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:12.489 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[44619]: debug 2024-04-06T03:54:12.222+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:12.752 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[114409]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T03:54:12.752 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 06 03:54:12 smithi153 bash[114474]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-5 2024-04-06T03:54:13.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:13 smithi153 bash[38802]: debug 2024-04-06T03:54:13.262+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:13.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:13 smithi153 bash[38802]: debug 2024-04-06T03:54:13.262+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:13.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:13 smithi153 bash[38802]: debug 2024-04-06T03:54:13.262+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:13.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:13 smithi153 bash[44619]: debug 2024-04-06T03:54:13.190+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:13.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:13 smithi153 bash[44619]: debug 2024-04-06T03:54:13.190+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:13.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:13 smithi153 bash[44619]: debug 2024-04-06T03:54:13.190+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:14.288 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[44619]: debug 2024-04-06T03:54:14.238+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:14.289 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[44619]: debug 2024-04-06T03:54:14.238+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:14.289 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[44619]: debug 2024-04-06T03:54:14.238+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:14.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[38802]: debug 2024-04-06T03:54:14.286+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:14.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[38802]: debug 2024-04-06T03:54:14.286+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:14.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[38802]: debug 2024-04-06T03:54:14.286+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:14.694 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-06T03:54:14.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:54:14.722 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-06T03:54:14.722 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-06T03:54:14.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.6 2024-04-06T03:54:15.046 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 systemd[1]: Stopping Ceph osd.6 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:15.046 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[38802]: debug 2024-04-06T03:54:14.914+0000 7f03e8a80700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:54:15.047 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[38802]: debug 2024-04-06T03:54:14.914+0000 7f03e8a80700 -1 osd.6 449 *** Got signal Terminated *** 2024-04-06T03:54:15.047 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:14 smithi153 bash[38802]: debug 2024-04-06T03:54:14.914+0000 7f03e8a80700 -1 osd.6 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:54:15.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:15 smithi153 bash[38802]: debug 2024-04-06T03:54:15.258+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:15.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:15 smithi153 bash[38802]: debug 2024-04-06T03:54:15.258+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:15.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:15 smithi153 bash[38802]: debug 2024-04-06T03:54:15.258+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:15.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:15 smithi153 bash[44619]: debug 2024-04-06T03:54:15.218+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:15.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:15 smithi153 bash[44619]: debug 2024-04-06T03:54:15.218+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:15.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:15 smithi153 bash[44619]: debug 2024-04-06T03:54:15.218+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:16.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:16 smithi153 bash[38802]: debug 2024-04-06T03:54:16.294+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:16.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:16 smithi153 bash[38802]: debug 2024-04-06T03:54:16.294+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:16.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:16 smithi153 bash[38802]: debug 2024-04-06T03:54:16.294+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:16.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:16 smithi153 bash[44619]: debug 2024-04-06T03:54:16.254+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:16.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:16 smithi153 bash[44619]: debug 2024-04-06T03:54:16.254+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:16.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:16 smithi153 bash[44619]: debug 2024-04-06T03:54:16.254+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:17.546 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:17 smithi153 bash[38802]: debug 2024-04-06T03:54:17.286+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:17.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:17 smithi153 bash[38802]: debug 2024-04-06T03:54:17.286+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:17.547 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:17 smithi153 bash[38802]: debug 2024-04-06T03:54:17.286+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:17.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:17 smithi153 bash[44619]: debug 2024-04-06T03:54:17.302+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:17.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:17 smithi153 bash[44619]: debug 2024-04-06T03:54:17.302+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:17.547 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:17 smithi153 bash[44619]: debug 2024-04-06T03:54:17.302+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:18.796 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[38802]: debug 2024-04-06T03:54:18.330+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:18.797 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[38802]: debug 2024-04-06T03:54:18.330+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:18.797 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[38802]: debug 2024-04-06T03:54:18.330+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:18.797 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[38802]: debug 2024-04-06T03:54:18.330+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.564419+0000 front 2024-04-06T03:53:53.564463+0000 (oldest deadline 2024-04-06T03:54:17.664377+0000) 2024-04-06T03:54:18.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[44619]: debug 2024-04-06T03:54:18.326+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:18.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[44619]: debug 2024-04-06T03:54:18.326+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:18.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:18 smithi153 bash[44619]: debug 2024-04-06T03:54:18.326+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:19.796 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[38802]: debug 2024-04-06T03:54:19.378+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:26.158705+0000 front 2024-04-06T03:53:26.159233+0000 (oldest deadline 2024-04-06T03:53:50.759510+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[38802]: debug 2024-04-06T03:54:19.378+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:33.061636+0000 front 2024-04-06T03:53:33.061268+0000 (oldest deadline 2024-04-06T03:53:58.961035+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[38802]: debug 2024-04-06T03:54:19.378+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:41.262674+0000 front 2024-04-06T03:53:41.262711+0000 (oldest deadline 2024-04-06T03:54:06.562349+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[38802]: debug 2024-04-06T03:54:19.378+0000 7f03e106c700 -1 osd.6 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.564419+0000 front 2024-04-06T03:53:53.564463+0000 (oldest deadline 2024-04-06T03:54:17.664377+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[44619]: debug 2024-04-06T03:54:19.338+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[44619]: debug 2024-04-06T03:54:19.338+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[44619]: debug 2024-04-06T03:54:19.338+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:19.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:19 smithi153 bash[44619]: debug 2024-04-06T03:54:19.338+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:20.584 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:20 smithi153 bash[44619]: debug 2024-04-06T03:54:20.314+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:20.584 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:20 smithi153 bash[44619]: debug 2024-04-06T03:54:20.314+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:20.584 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:20 smithi153 bash[44619]: debug 2024-04-06T03:54:20.314+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:20.584 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:20 smithi153 bash[44619]: debug 2024-04-06T03:54:20.314+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:21.537 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:21 smithi153 bash[114608]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-6 2024-04-06T03:54:21.537 INFO:journalctl@ceph.osd.6.smithi153.stdout:Apr 06 03:54:21 smithi153 bash[114670]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-6 2024-04-06T03:54:21.537 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:21 smithi153 bash[44619]: debug 2024-04-06T03:54:21.330+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:21.537 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:21 smithi153 bash[44619]: debug 2024-04-06T03:54:21.330+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:21.538 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:21 smithi153 bash[44619]: debug 2024-04-06T03:54:21.334+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:21.538 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:21 smithi153 bash[44619]: debug 2024-04-06T03:54:21.334+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:22.769 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:22 smithi153 bash[44619]: debug 2024-04-06T03:54:22.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:22.769 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:22 smithi153 bash[44619]: debug 2024-04-06T03:54:22.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:22.769 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:22 smithi153 bash[44619]: debug 2024-04-06T03:54:22.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:22.770 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:22 smithi153 bash[44619]: debug 2024-04-06T03:54:22.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:23.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.6.service' 2024-04-06T03:54:23.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:54:23.531 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-06T03:54:23.531 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-06T03:54:23.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.7 2024-04-06T03:54:23.715 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.390+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:23.715 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.390+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:23.716 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.390+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:23.716 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.390+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:23.716 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 systemd[1]: Stopping Ceph osd.7 for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:23.716 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.714+0000 7fc071ccd700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:54:23.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.714+0000 7fc071ccd700 -1 osd.7 449 *** Got signal Terminated *** 2024-04-06T03:54:23.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:23 smithi153 bash[44619]: debug 2024-04-06T03:54:23.714+0000 7fc071ccd700 -1 osd.7 449 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-06T03:54:24.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:24 smithi153 bash[44619]: debug 2024-04-06T03:54:24.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:24.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:24 smithi153 bash[44619]: debug 2024-04-06T03:54:24.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:24.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:24 smithi153 bash[44619]: debug 2024-04-06T03:54:24.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:24.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:24 smithi153 bash[44619]: debug 2024-04-06T03:54:24.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:25.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:25 smithi153 bash[44619]: debug 2024-04-06T03:54:25.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:25.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:25 smithi153 bash[44619]: debug 2024-04-06T03:54:25.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:25.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:25 smithi153 bash[44619]: debug 2024-04-06T03:54:25.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:25.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:25 smithi153 bash[44619]: debug 2024-04-06T03:54:25.382+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:26.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:26 smithi153 bash[44619]: debug 2024-04-06T03:54:26.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:26.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:26 smithi153 bash[44619]: debug 2024-04-06T03:54:26.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:26.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:26 smithi153 bash[44619]: debug 2024-04-06T03:54:26.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:26.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:26 smithi153 bash[44619]: debug 2024-04-06T03:54:26.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:26.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:26 smithi153 bash[44619]: debug 2024-04-06T03:54:26.414+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.153:6804 osd.4 since back 2024-04-06T03:53:59.689134+0000 front 2024-04-06T03:53:59.689091+0000 (oldest deadline 2024-04-06T03:54:25.588705+0000) 2024-04-06T03:54:27.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:27 smithi153 bash[44619]: debug 2024-04-06T03:54:27.446+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:27.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:27 smithi153 bash[44619]: debug 2024-04-06T03:54:27.446+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:27.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:27 smithi153 bash[44619]: debug 2024-04-06T03:54:27.446+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:27.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:27 smithi153 bash[44619]: debug 2024-04-06T03:54:27.446+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:27.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:27 smithi153 bash[44619]: debug 2024-04-06T03:54:27.446+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.153:6804 osd.4 since back 2024-04-06T03:53:59.689134+0000 front 2024-04-06T03:53:59.689091+0000 (oldest deadline 2024-04-06T03:54:25.588705+0000) 2024-04-06T03:54:28.796 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:28 smithi153 bash[44619]: debug 2024-04-06T03:54:28.474+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-06T03:53:24.781656+0000 front 2024-04-06T03:53:24.781472+0000 (oldest deadline 2024-04-06T03:53:47.681297+0000) 2024-04-06T03:54:28.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:28 smithi153 bash[44619]: debug 2024-04-06T03:54:28.474+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-06T03:53:35.184103+0000 front 2024-04-06T03:53:35.184151+0000 (oldest deadline 2024-04-06T03:54:00.484070+0000) 2024-04-06T03:54:28.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:28 smithi153 bash[44619]: debug 2024-04-06T03:54:28.474+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-06T03:53:42.185632+0000 front 2024-04-06T03:53:42.185435+0000 (oldest deadline 2024-04-06T03:54:06.885417+0000) 2024-04-06T03:54:28.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:28 smithi153 bash[44619]: debug 2024-04-06T03:54:28.474+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-06T03:53:53.287845+0000 front 2024-04-06T03:53:53.287807+0000 (oldest deadline 2024-04-06T03:54:19.187643+0000) 2024-04-06T03:54:28.797 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:28 smithi153 bash[44619]: debug 2024-04-06T03:54:28.474+0000 7fc06a2b9700 -1 osd.7 449 heartbeat_check: no reply from 172.21.15.153:6804 osd.4 since back 2024-04-06T03:53:59.689134+0000 front 2024-04-06T03:53:59.689091+0000 (oldest deadline 2024-04-06T03:54:25.588705+0000) 2024-04-06T03:54:30.297 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:30 smithi153 bash[114803]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-7 2024-04-06T03:54:30.298 INFO:journalctl@ceph.osd.7.smithi153.stdout:Apr 06 03:54:30 smithi153 bash[114866]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-osd-7 2024-04-06T03:54:32.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@osd.7.service' 2024-04-06T03:54:32.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:54:32.143 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-06T03:54:32.143 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-04-06T03:54:32.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@rgw.foo.a 2024-04-06T03:54:32.490 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:54:32 smithi050 systemd[1]: Stopping Ceph rgw.foo.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:32.490 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:54:32 smithi050 bash[53160]: debug 2024-04-06T03:54:32.282+0000 7f165146d700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-06T03:54:32.490 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:54:32 smithi050 bash[53160]: debug 2024-04-06T03:54:32.282+0000 7f165fcebac0 -1 shutting down 2024-04-06T03:54:42.990 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 06 03:54:42 smithi050 bash[168712]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-rgw-foo-a 2024-04-06T03:54:43.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@rgw.foo.a.service' 2024-04-06T03:54:43.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:54:43.079 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-04-06T03:54:43.079 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-06T03:54:43.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@prometheus.a 2024-04-06T03:54:43.472 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 systemd[1]: Stopping Ceph prometheus.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:43.472 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.228Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-06T03:54:43.473 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Apr 06 03:54:43 smithi153 bash[52536]: ts=2024-04-06T03:54:43.229Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-06T03:54:43.995 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@prometheus.a.service' 2024-04-06T03:54:44.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-06T03:54:44.018 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-06T03:54:44.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T03:54:44.191 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:54:49.040 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.040 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:48 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.294 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.294 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.550 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.550 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.808 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:49.808 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.058 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.059 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:49 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.433 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.433 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.685 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.685 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.952 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.952 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:50.952 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:50 smithi050 systemd[1]: Stopping Ceph node-exporter.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:51.792 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:51 smithi050 bash[169118]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-node-exporter-a 2024-04-06T03:54:51.792 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:51 smithi050 bash[169180]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-node-exporter-a 2024-04-06T03:54:51.792 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:51 smithi050 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-06T03:54:51.792 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:51 smithi050 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-06T03:54:51.792 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:51 smithi050 systemd[1]: Stopped Ceph node-exporter.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:54:52.132 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 06 03:54:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:52.133 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:51 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:52.133 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:52 smithi050 systemd[1]: Stopping Ceph alertmanager.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:54:52.383 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:52 smithi050 bash[59802]: ts=2024-04-06T03:54:52.134Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-06T03:54:52.988 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:52 smithi050 bash[169251]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T03:54:52.988 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:52 smithi050 bash[169310]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-alertmanager-a 2024-04-06T03:54:53.239 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:52 smithi050 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@alertmanager.a.service: Deactivated successfully. 2024-04-06T03:54:53.240 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:52 smithi050 systemd[1]: Stopped Ceph alertmanager.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:54:53.240 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 06 03:54:53 smithi050 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:53.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-06T03:54:53.953 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T03:54:58.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:54:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:58.797 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:54:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:58.797 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:54:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:54:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.104 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:54:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:54:58 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.373 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.373 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.373 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.644 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.644 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.645 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.918 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.918 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:54:59.918 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:54:59 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:00.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:00 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:00.183 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:00 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:00.183 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:00 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:00.546 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:00 smithi153 systemd[1]: Stopping Ceph iscsi.iscsi.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:55:00.546 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:00 smithi153 bash[48791]: debug Shutdown received 2024-04-06T03:55:11.022 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:10 smithi153 bash[115332]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-iscsi-iscsi-a 2024-04-06T03:55:11.288 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:11.289 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:11.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:11 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-04-06T03:55:11.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:11 smithi153 bash[115402]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-iscsi-iscsi-a 2024-04-06T03:55:11.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:11 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-04-06T03:55:11.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:11 smithi153 systemd[1]: Stopped Ceph iscsi.iscsi.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:55:11.289 INFO:journalctl@ceph.iscsi.iscsi.a.smithi153.stdout:Apr 06 03:55:11 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:12.498 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:12.499 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:12.499 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:12.499 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:12 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:12.753 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:12 smithi153 systemd[1]: Stopping Ceph node-exporter.b for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:55:13.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:13 smithi153 bash[115603]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-node-exporter-b 2024-04-06T03:55:13.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:13 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-06T03:55:13.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:13 smithi153 bash[115663]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-node-exporter-b 2024-04-06T03:55:13.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:13 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-06T03:55:13.546 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:13 smithi153 systemd[1]: Stopped Ceph node-exporter.b for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:55:13.796 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Apr 06 03:55:13 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:13.796 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:13 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:13.796 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:13 smithi153 systemd[1]: Stopping Ceph grafana.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4... 2024-04-06T03:55:13.796 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:13 smithi153 bash[51781]: logger=server t=2024-04-06T03:55:13.788974423Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-06T03:55:13.796 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:13 smithi153 bash[51781]: logger=ticker t=2024-04-06T03:55:13.789049216Z level=info msg=stopped last_tick=2024-04-06T03:55:10Z 2024-04-06T03:55:13.796 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:13 smithi153 bash[51781]: logger=tracing t=2024-04-06T03:55:13.789078526Z level=info msg="Closing tracing" 2024-04-06T03:55:14.546 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:14 smithi153 bash[115730]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-grafana-a 2024-04-06T03:55:14.546 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:14 smithi153 bash[115791]: Error response from daemon: No such container: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4-grafana-a 2024-04-06T03:55:14.546 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:14 smithi153 systemd[1]: ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@grafana.a.service: Deactivated successfully. 2024-04-06T03:55:14.546 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:14 smithi153 systemd[1]: Stopped Ceph grafana.a for 0313e520-f3c3-11ee-b649-cb9ed24678a4. 2024-04-06T03:55:14.917 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Apr 06 03:55:14 smithi153 systemd[1]: /etc/systemd/system/ceph-0313e520-f3c3-11ee-b649-cb9ed24678a4@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-06T03:55:15.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:55:15.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-06T03:55:15.219 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-06T03:55:15.220 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/remote/smithi050/crash 2024-04-06T03:55:15.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T03:55:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:tar: /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T03:55:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:tar: Error is not recoverable: exiting now 2024-04-06T03:55:15.276 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/remote/smithi153/crash 2024-04-06T03:55:15.277 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/crash -- . 2024-04-06T03:55:15.288 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-06T03:55:15.288 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-04-06T03:55:15.289 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-06T03:55:15.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-06T03:55:15.412 INFO:teuthology.orchestra.run.smithi050.stdout:2024-04-06T03:12:53.821259+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-06T03:55:15.413 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-06T03:55:15.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[SEC\]' /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-06T03:55:15.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]' /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-06T03:55:15.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[WRN\]' /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-06T03:55:15.548 INFO:teuthology.orchestra.run.smithi050.stdout:2024-04-06T03:12:53.821259+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-06T03:55:15.549 INFO:tasks.cephadm:Compressing logs... 2024-04-06T03:55:15.549 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T03:55:15.595 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-06T03:55:15.606 INFO:teuthology.orchestra.run.smithi050.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-06T03:55:15.609 INFO:teuthology.orchestra.run.smithi153.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-06T03:55:15.625 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T03:55:15.627 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-06T03:55:15.634 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.y.log 2024-04-06T03:55:15.635 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.7.log 2024-04-06T03:55:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-06T03:55:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-06T03:55:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.a.log 2024-04-06T03:55:15.636 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T03:55:15.637 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.3.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.c.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.a.log 2024-04-06T03:55:15.641 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log: /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.a.log: 58.0% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-client.rgw.foo.a.log.gz 2024-04-06T03:55:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.6.log 2024-04-06T03:55:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.7.log: /var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-06T03:55:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.5.log 2024-04-06T03:55:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/tcmu-runner.log 2024-04-06T03:55:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.x.log 2024-04-06T03:55:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log 2024-04-06T03:55:15.643 INFO:teuthology.orchestra.run.smithi153.stderr: 68.2% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/tcmu-runner.log.gz 2024-04-06T03:55:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose --/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.x.log: /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-06T03:55:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-06T03:55:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.b.log 2024-04-06T03:55:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.4.log 2024-04-06T03:55:15.648 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.b.log: /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.4.log: 92.9% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.x.log.gz 2024-04-06T03:55:15.653 INFO:teuthology.orchestra.run.smithi050.stderr: 91.6% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T03:55:15.693 INFO:teuthology.orchestra.run.smithi050.stderr: 88.5% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T03:55:15.746 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T03:55:15.763 INFO:teuthology.orchestra.run.smithi153.stderr: 80.3% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-06T03:55:15.767 INFO:teuthology.orchestra.run.smithi153.stderr: 90.3% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T03:55:15.803 INFO:teuthology.orchestra.run.smithi153.stderr: 85.1% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-06T03:55:15.814 INFO:teuthology.orchestra.run.smithi153.stderr: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-06T03:55:15.817 INFO:teuthology.orchestra.run.smithi050.stderr: 94.1% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-06T03:55:15.837 INFO:teuthology.orchestra.run.smithi153.stderr: 91.3% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T03:55:16.206 INFO:teuthology.orchestra.run.smithi050.stderr: 91.3% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-06T03:55:16.455 INFO:teuthology.orchestra.run.smithi153.stderr: 91.6% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.b.log.gz 2024-04-06T03:55:18.136 INFO:teuthology.orchestra.run.smithi050.stderr: 89.9% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mgr.y.log.gz 2024-04-06T03:55:18.277 INFO:teuthology.orchestra.run.smithi050.stderr: 91.7% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.c.log.gz 2024-04-06T03:55:40.734 INFO:teuthology.orchestra.run.smithi050.stderr: 90.5% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-mon.a.log.gz 2024-04-06T03:55:51.659 INFO:teuthology.orchestra.run.smithi153.stderr: 94.8% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-06T03:55:51.843 INFO:teuthology.orchestra.run.smithi153.stderr: 94.7% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-06T03:56:26.316 INFO:teuthology.orchestra.run.smithi153.stderr: 93.7% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-06T03:56:27.318 INFO:teuthology.orchestra.run.smithi153.stderr: 93.9% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-06T03:56:27.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-06T03:56:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:real 1m11.720s 2024-04-06T03:56:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:user 3m4.328s 2024-04-06T03:56:27.320 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m7.359s 2024-04-06T04:01:06.968 INFO:teuthology.orchestra.run.smithi050.stderr: 93.8% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-06T04:01:50.900 INFO:teuthology.orchestra.run.smithi050.stderr: 94.1% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-06T04:03:26.526 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-06T04:03:34.845 INFO:teuthology.orchestra.run.smithi050.stderr: 93.7% -- replaced with /var/log/ceph/0313e520-f3c3-11ee-b649-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-06T04:03:34.847 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-04-06T04:03:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:real 8m19.249s 2024-04-06T04:03:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:user 9m40.247s 2024-04-06T04:03:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m25.386s 2024-04-06T04:03:34.848 INFO:tasks.cephadm:Archiving logs... 2024-04-06T04:03:34.848 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/remote/smithi050/log 2024-04-06T04:03:34.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T04:04:10.231 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/remote/smithi153/log 2024-04-06T04:04:10.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-06T04:04:32.112 INFO:tasks.cephadm:Removing cluster... 2024-04-06T04:04:32.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 --force 2024-04-06T04:04:32.313 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T04:04:34.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0313e520-f3c3-11ee-b649-cb9ed24678a4 --force 2024-04-06T04:04:34.409 INFO:teuthology.orchestra.run.smithi153.stdout:Deleting cluster with fsid: 0313e520-f3c3-11ee-b649-cb9ed24678a4 2024-04-06T04:04:36.029 INFO:tasks.cephadm:Removing cephadm ... 2024-04-06T04:04:36.029 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T04:04:36.057 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-06T04:04:36.063 INFO:tasks.cephadm:Teardown complete 2024-04-06T04:04:36.063 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-06T04:04:36.072 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-06T04:04:36.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T04:04:36.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-06T04:04:36.197 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-06T04:04:36.207 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-06T04:04:36.207 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T04:04:36.209 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-06T04:04:36.250 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T04:04:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2024-04-06T04:04:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:*hv01.front.sepi 44.4.53.2 2 u 64 64 377 0.102 -1.911 0.193 2024-04-06T04:04:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:+hv02.front.sepi 63.231.80.2 3 u 57 64 377 0.070 -1.872 0.121 2024-04-06T04:04:36.250 INFO:teuthology.orchestra.run.smithi153.stdout:+hv03.front.sepi 74.6.168.72 3 u 63 64 377 0.082 -2.467 0.136 2024-04-06T04:04:36.250 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-06T04:04:36.259 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2024-04-06T04:04:36.259 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2024-04-06T04:04:36.259 INFO:teuthology.orchestra.run.smithi050.stdout:*hv01.front.sepi 44.4.53.2 2 u 118 128 377 0.091 -1.496 0.333 2024-04-06T04:04:36.259 INFO:teuthology.orchestra.run.smithi050.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 128 377 0.098 -1.853 0.159 2024-04-06T04:04:36.259 INFO:teuthology.orchestra.run.smithi050.stdout:+hv03.front.sepi 74.6.168.72 3 u 130 128 377 0.087 -2.608 0.168 2024-04-06T04:04:36.259 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-06T04:04:36.260 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-06T04:04:36.272 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-06T04:04:36.273 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-06T04:04:36.288 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-06T04:04:36.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-06T04:04:36.316 INFO:teuthology.task.internal:Duration was 3905.713835 seconds 2024-04-06T04:04:36.317 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-06T04:04:36.331 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-06T04:04:36.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T04:04:36.334 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-06T04:04:36.452 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-06T04:04:36.452 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-04-06T04:04:36.452 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T04:04:36.468 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-04-06T04:04:36.468 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-06T04:04:36.479 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-06T04:04:36.479 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T04:04:36.511 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-06T04:04:36.686 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-06T04:04:36.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T04:04:36.689 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-06T04:04:37.140 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-06T04:04:37.151 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-06T04:04:37.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T04:04:37.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-06T04:04:37.245 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-06T04:04:37.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T04:04:37.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-06T04:04:37.274 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-04-06T04:04:37.290 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-04-06T04:04:37.308 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T04:04:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T04:04:37.354 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-06T04:04:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-06T04:04:37.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-06T04:04:37.368 INFO:teuthology.task.internal:Transferring archived files... 2024-04-06T04:04:37.369 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/remote/smithi050 2024-04-06T04:04:37.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T04:04:37.436 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906/remote/smithi153 2024-04-06T04:04:37.437 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-06T04:04:37.467 INFO:teuthology.task.internal:Removing archive directory... 2024-04-06T04:04:37.467 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T04:04:37.470 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-06T04:04:37.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-06T04:04:37.520 INFO:teuthology.task.internal:Not uploading archives. 2024-04-06T04:04:37.521 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-06T04:04:37.533 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-06T04:04:37.533 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T04:04:37.536 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-06T04:04:37.540 INFO:teuthology.orchestra.run.smithi050.stdout: 658635 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 04:04 /home/ubuntu/cephtest 2024-04-06T04:04:37.557 INFO:teuthology.orchestra.run.smithi153.stdout: 658646 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 6 04:04 /home/ubuntu/cephtest 2024-04-06T04:04:37.563 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-06T04:04:37.575 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-06T04:04:37.651 INFO:teuthology.nuke:Checking targets against current locks 2024-04-06T04:04:37.677 DEBUG:teuthology.nuke:shortname: smithi050 2024-04-06T04:04:37.677 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T04:04:37.700 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:50:04.853886', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHS3NGNDwyc+9N/zANprQcxqgUJmXFIpW4mIZqLY+jc+WOAMTCLVpRk4pKDfPLyZwX4lfia2o4km4YptNsAHsm0='} 2024-04-06T04:04:37.703 DEBUG:teuthology.nuke:shortname: smithi153 2024-04-06T04:04:37.703 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-06T04:04:37.722 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-05_21:08:03-orch-squid-distro-default-smithi/7641906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-06 02:50:04.855288', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmbjn0la71LFvZQWmzMqaxBgPwdL/4dFlwQaaVNmnqDOz98eHa9HNzEVBS9MV5Hl9pqHF17g7WyUfrDOdPXADI='} 2024-04-06T04:04:37.755 INFO:teuthology.orchestra.console.smithi050:Power off 2024-04-06T04:04:37.755 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T04:04:37.771 INFO:teuthology.orchestra.console.smithi153:Power off 2024-04-06T04:04:37.771 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-06T04:04:37.781 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-04-06T04:04:37.781 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:37.797 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-04-06T04:04:37.797 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:37.809 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-06T04:04:37.829 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-06T04:04:41.811 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:41.830 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:41.921 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-06T04:04:41.938 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-04-06T04:04:45.927 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:45.939 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:46.037 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-06T04:04:46.049 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-04-06T04:04:46.149 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-04-06T04:04:50.042 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-06T04:04:50.151 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-04-06T04:04:50.252 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-04-06T04:04:50.356 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3905.7138345241547 failure_reason: '"2024-04-06T03:12:53.821259+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-04-06T04:04:50.356 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-06T04:04:50.443 INFO:teuthology.run:FAIL